2024-02-10T07:14:33.938 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T07:14:33.938 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555212 --name yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi --verbose --description orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555212/orig.config.yaml 2024-02-10T07:14:33.983 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T07:14:34.072 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555212 branch: main description: orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} email: yweinste@redhat.com first_in_suite: false job_id: '7555212' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 1610 sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 targets: smithi008.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDT5Z0nOVs84klw14kkTk/VwcgXuWxydE/hPpmGCsA3HeEftC0FePofIpD9URIhSthaYNQ1YNqtlHhVMxfvVml4= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQupySyFXI1xahb593YgRdd8Bp/ewEbjvYKu5rp2civZem77CWEcEaJeIr9jnQ94tTz2l6w7sK4peMzZNkIZco= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_19:44:16 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-10T07:14:34.072 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa; will attempt to use it 2024-02-10T07:14:34.073 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks 2024-02-10T07:14:34.073 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T07:14:34.074 INFO:teuthology.task.internal:Checking packages... 2024-02-10T07:14:34.098 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90' 2024-02-10T07:14:34.098 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T07:14:34.098 INFO:teuthology.packaging:ref: None 2024-02-10T07:14:34.098 INFO:teuthology.packaging:tag: None 2024-02-10T07:14:34.098 INFO:teuthology.packaging:branch: main 2024-02-10T07:14:34.098 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:14:34.098 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-10T07:14:34.490 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1361.gf8e4e331 2024-02-10T07:14:34.492 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T07:14:34.500 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T07:14:34.500 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T07:14:34.507 INFO:teuthology.task.internal:Saving configuration 2024-02-10T07:14:34.519 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T07:14:34.524 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T07:14:34.549 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555212', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 07:08:43.556003', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDT5Z0nOVs84klw14kkTk/VwcgXuWxydE/hPpmGCsA3HeEftC0FePofIpD9URIhSthaYNQ1YNqtlHhVMxfvVml4='} 2024-02-10T07:14:34.570 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555212', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 07:08:43.557917', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQupySyFXI1xahb593YgRdd8Bp/ewEbjvYKu5rp2civZem77CWEcEaJeIr9jnQ94tTz2l6w7sK4peMzZNkIZco='} 2024-02-10T07:14:34.571 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T07:14:34.577 INFO:teuthology.task.internal:roles: ubuntu@smithi008.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-02-10T07:14:34.577 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-02-10T07:14:34.577 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T07:14:34.645 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feaa82084c0>, signals=[15]) 2024-02-10T07:14:34.646 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T07:14:34.653 INFO:teuthology.task.internal:Opening connections... 2024-02-10T07:14:34.653 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T07:14:34.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:14:34.764 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2024-02-10T07:14:34.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:14:34.898 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T07:14:34.905 DEBUG:teuthology.orchestra.run.smithi008:> uname -m 2024-02-10T07:14:34.923 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64 2024-02-10T07:14:34.923 DEBUG:teuthology.orchestra.run.smithi008:> cat /etc/os-release 2024-02-10T07:14:34.981 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="CentOS Stream" 2024-02-10T07:14:34.981 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="9" 2024-02-10T07:14:34.981 INFO:teuthology.orchestra.run.smithi008.stdout:ID="centos" 2024-02-10T07:14:34.981 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE="rhel fedora" 2024-02-10T07:14:34.981 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="9" 2024-02-10T07:14:34.981 INFO:teuthology.orchestra.run.smithi008.stdout:PLATFORM_ID="platform:el9" 2024-02-10T07:14:34.981 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T07:14:34.981 INFO:teuthology.orchestra.run.smithi008.stdout:ANSI_COLOR="0;31" 2024-02-10T07:14:34.981 INFO:teuthology.orchestra.run.smithi008.stdout:LOGO="fedora-logo-icon" 2024-02-10T07:14:34.981 INFO:teuthology.orchestra.run.smithi008.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T07:14:34.981 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://centos.org/" 2024-02-10T07:14:34.981 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T07:14:34.982 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T07:14:34.982 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T07:14:34.982 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2024-02-10T07:14:35.007 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2024-02-10T07:14:35.024 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2024-02-10T07:14:35.025 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2024-02-10T07:14:35.079 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2024-02-10T07:14:35.079 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2024-02-10T07:14:35.079 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2024-02-10T07:14:35.079 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2024-02-10T07:14:35.079 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2024-02-10T07:14:35.079 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2024-02-10T07:14:35.080 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T07:14:35.080 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2024-02-10T07:14:35.080 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2024-02-10T07:14:35.080 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T07:14:35.080 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2024-02-10T07:14:35.080 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T07:14:35.080 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T07:14:35.080 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T07:14:35.080 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2024-02-10T07:14:35.098 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T07:14:35.107 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T07:14:35.113 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T07:14:35.113 DEBUG:teuthology.orchestra.run.smithi008:> test '!' -e /home/ubuntu/cephtest 2024-02-10T07:14:35.116 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2024-02-10T07:14:35.134 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T07:14:35.141 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T07:14:35.141 DEBUG:teuthology.orchestra.run.smithi008:> test -z $(ls -A /var/lib/ceph) 2024-02-10T07:14:35.174 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2024-02-10T07:14:35.197 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T07:14:35.298 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T07:14:35.313 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T07:14:35.313 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T07:14:35.314 DEBUG:teuthology.orchestra.run.smithi008:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T07:14:35.314 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T07:14:35.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:14:35.331 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-02-10T07:14:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:14:35.332 DEBUG:teuthology.orchestra.run.smithi008:> uname -r 2024-02-10T07:14:35.388 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-361.el9.x86_64 2024-02-10T07:14:35.389 INFO:teuthology.task.kernel:Running kernel on smithi170: 5.14.0-361.el9.x86_64 2024-02-10T07:14:35.389 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum install -y kernel 2024-02-10T07:14:35.390 INFO:teuthology.orchestra.run.smithi008.stdout:5.14.0-361.el9.x86_64 2024-02-10T07:14:35.390 INFO:teuthology.task.kernel:Running kernel on smithi008: 5.14.0-361.el9.x86_64 2024-02-10T07:14:35.390 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum install -y kernel 2024-02-10T07:14:37.675 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 8.9 MB/s | 6.7 MB 00:00 2024-02-10T07:14:37.712 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS-9 - AppStream 9.4 MB/s | 6.7 MB 00:00 2024-02-10T07:14:39.120 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-02-10T07:14:39.309 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-02-10T07:14:39.953 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-02-10T07:14:42.554 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-02-10T07:14:42.990 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.0 MB 00:03 2024-02-10T07:14:46.838 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 5.9 MB 00:00 2024-02-10T07:14:48.630 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 15 kB 00:00 2024-02-10T07:14:50.182 INFO:teuthology.orchestra.run.smithi008.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2024-02-10T07:14:55.249 INFO:teuthology.orchestra.run.smithi008.stdout:lab-extras 22 kB/s | 1.7 kB 00:00 2024-02-10T07:14:57.631 INFO:teuthology.orchestra.run.smithi008.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T07:14:57.631 INFO:teuthology.orchestra.run.smithi008.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T07:14:57.728 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:14:57.735 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:14:57.736 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T07:14:57.736 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:14:57.736 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:14:57.736 INFO:teuthology.orchestra.run.smithi008.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T07:14:57.736 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T07:14:57.736 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T07:14:57.736 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T07:14:57.736 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T07:14:57.736 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:14:57.736 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:14:57.736 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:14:57.736 INFO:teuthology.orchestra.run.smithi008.stdout:Install 4 Packages 2024-02-10T07:14:57.736 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:14:57.737 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 99 M 2024-02-10T07:14:57.737 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 122 M 2024-02-10T07:14:57.737 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:14:58.492 INFO:teuthology.orchestra.run.smithi008.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 8.0 MB/s | 6.0 MB 00:00 2024-02-10T07:14:59.600 INFO:teuthology.orchestra.run.smithi008.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 11 MB/s | 21 MB 00:01 2024-02-10T07:15:00.200 INFO:teuthology.orchestra.run.smithi008.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 20 MB/s | 33 MB 00:01 2024-02-10T07:15:00.801 INFO:teuthology.orchestra.run.smithi008.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 13 MB/s | 39 MB 00:03 2024-02-10T07:15:00.804 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:15:00.804 INFO:teuthology.orchestra.run.smithi008.stdout:Total 32 MB/s | 99 MB 00:03 2024-02-10T07:15:00.804 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:15:01.233 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:15:01.233 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:15:02.275 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:15:02.275 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:15:04.791 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:15:05.471 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T07:15:05.602 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T07:15:08.508 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T07:15:08.699 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T07:15:13.802 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T07:15:13.913 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:15.012 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 620 kB/s | 19 MB 00:30 2024-02-10T07:15:19.036 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:19.230 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 8.7 MB/s | 5.9 MB 00:00 2024-02-10T07:15:20.755 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-02-10T07:15:21.982 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 19 MB/s | 20 MB 00:01 2024-02-10T07:15:26.957 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-02-10T07:15:29.238 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T07:15:29.239 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T07:15:29.328 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:15:29.336 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:15:29.336 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-02-10T07:15:29.336 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:15:29.336 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:15:29.336 INFO:teuthology.orchestra.run.smithi170.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T07:15:29.336 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-02-10T07:15:29.337 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T07:15:29.337 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T07:15:29.337 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T07:15:29.337 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:15:29.337 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:15:29.337 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:15:29.337 INFO:teuthology.orchestra.run.smithi170.stdout:Install 4 Packages 2024-02-10T07:15:29.337 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:15:29.338 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 99 M 2024-02-10T07:15:29.338 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 122 M 2024-02-10T07:15:29.338 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:15:29.683 INFO:teuthology.orchestra.run.smithi170.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 17 MB/s | 6.0 MB 00:00 2024-02-10T07:15:30.150 INFO:teuthology.orchestra.run.smithi170.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 26 MB/s | 21 MB 00:00 2024-02-10T07:15:30.808 INFO:teuthology.orchestra.run.smithi170.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 30 MB/s | 33 MB 00:01 2024-02-10T07:15:31.225 INFO:teuthology.orchestra.run.smithi170.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-02-10T07:15:31.228 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:15:31.228 INFO:teuthology.orchestra.run.smithi170.stdout:Total 53 MB/s | 99 MB 00:01 2024-02-10T07:15:31.229 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:15:31.593 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:15:31.594 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:15:32.639 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:15:32.639 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:15:35.043 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:15:35.695 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T07:15:35.834 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T07:15:38.588 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T07:15:38.770 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T07:15:43.493 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T07:15:43.722 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:48.464 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:49.986 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:49.986 INFO:teuthology.orchestra.run.smithi008.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T07:15:49.986 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:15:50.062 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:50.664 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:50.664 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T07:15:50.664 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T07:15:50.665 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T07:15:51.129 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:15:51.129 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:15:51.129 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:15:51.129 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T07:15:51.129 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T07:15:51.129 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T07:15:51.130 INFO:teuthology.orchestra.run.smithi008.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T07:15:51.130 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:15:51.130 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:15:51.668 DEBUG:teuthology.orchestra.run.smithi008:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T07:15:51.800 INFO:teuthology.orchestra.run.smithi008.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T07:15:51.800 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T07:15:51.800 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T07:15:51.800 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T07:15:51.800 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T07:15:51.800 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-10T07:15:51.800 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi008.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T07:15:51.801 DEBUG:teuthology.orchestra.run.smithi008:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T07:15:51.913 INFO:teuthology.orchestra.run.smithi008.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T07:15:51.913 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T07:15:51.913 DEBUG:teuthology.orchestra.run.smithi008:> sudo rpm -qi grub2-tools 2024-02-10T07:15:51.959 INFO:teuthology.orchestra.run.smithi008.stdout:Name : grub2-tools 2024-02-10T07:15:51.959 INFO:teuthology.orchestra.run.smithi008.stdout:Epoch : 1 2024-02-10T07:15:51.959 INFO:teuthology.orchestra.run.smithi008.stdout:Version : 2.06 2024-02-10T07:15:51.959 INFO:teuthology.orchestra.run.smithi008.stdout:Release : 68.el9 2024-02-10T07:15:51.959 INFO:teuthology.orchestra.run.smithi008.stdout:Architecture: x86_64 2024-02-10T07:15:51.959 INFO:teuthology.orchestra.run.smithi008.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T07:15:51.959 INFO:teuthology.orchestra.run.smithi008.stdout:Group : Unspecified 2024-02-10T07:15:51.959 INFO:teuthology.orchestra.run.smithi008.stdout:Size : 8264254 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout:License : GPLv3+ 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout:Packager : builder@centos.org 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout:Vendor : CentOS 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout:Summary : Support tools for GRUB. 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout:Description : 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T07:15:51.960 INFO:teuthology.orchestra.run.smithi008.stdout:hardware devices. 2024-02-10T07:15:51.961 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:15:51.961 INFO:teuthology.orchestra.run.smithi008.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T07:15:51.961 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T07:15:51.962 INFO:teuthology.task.kernel:Updating grub on smithi008 to boot 5.14.0-419.el9.x86_64 2024-02-10T07:15:51.962 DEBUG:teuthology.orchestra.run.smithi008:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T07:15:52.509 INFO:teuthology.orchestra.run.smithi008.stderr:Generating grub configuration file ... 2024-02-10T07:15:53.873 INFO:teuthology.orchestra.run.smithi008.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T07:15:53.910 INFO:teuthology.orchestra.run.smithi008.stderr:done 2024-02-10T07:15:53.913 DEBUG:teuthology.orchestra.run.smithi008:> mktemp 2024-02-10T07:15:53.928 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.d6h3CcRx8h 2024-02-10T07:15:53.928 DEBUG:teuthology.orchestra.run.smithi008:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.d6h3CcRx8h 2024-02-10T07:15:54.000 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.d6h3CcRx8h 2024-02-10T07:15:54.152 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.d6h3CcRx8h is 6KB 2024-02-10T07:15:54.201 DEBUG:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.d6h3CcRx8h 2024-02-10T07:15:54.216 DEBUG:teuthology.orchestra.run.smithi008:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T07:15:54.284 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T07:15:54.285 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T07:15:54.285 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T07:15:54.285 INFO:teuthology.orchestra.run.smithi008.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T07:15:54.286 DEBUG:teuthology.orchestra.run.smithi008:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T07:15:54.433 DEBUG:teuthology.orchestra.run.smithi008:> sudo shutdown -r now 2024-02-10T07:16:17.768 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:16:17.768 INFO:teuthology.orchestra.run.smithi170.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T07:16:17.768 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:16:17.838 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:16:18.292 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:16:18.292 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T07:16:18.292 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T07:16:18.292 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T07:16:18.772 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T07:16:18.772 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:16:18.772 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:16:18.772 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T07:16:18.772 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T07:16:18.772 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T07:16:18.772 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T07:16:18.772 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:16:18.772 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:16:19.273 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T07:16:19.399 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T07:16:19.399 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T07:16:19.400 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T07:16:19.400 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T07:16:19.400 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T07:16:19.400 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-10T07:16:19.400 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi170.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T07:16:19.400 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T07:16:19.507 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T07:16:19.507 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T07:16:19.507 DEBUG:teuthology.orchestra.run.smithi170:> sudo rpm -qi grub2-tools 2024-02-10T07:16:19.552 INFO:teuthology.orchestra.run.smithi170.stdout:Name : grub2-tools 2024-02-10T07:16:19.552 INFO:teuthology.orchestra.run.smithi170.stdout:Epoch : 1 2024-02-10T07:16:19.552 INFO:teuthology.orchestra.run.smithi170.stdout:Version : 2.06 2024-02-10T07:16:19.552 INFO:teuthology.orchestra.run.smithi170.stdout:Release : 68.el9 2024-02-10T07:16:19.552 INFO:teuthology.orchestra.run.smithi170.stdout:Architecture: x86_64 2024-02-10T07:16:19.552 INFO:teuthology.orchestra.run.smithi170.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T07:16:19.552 INFO:teuthology.orchestra.run.smithi170.stdout:Group : Unspecified 2024-02-10T07:16:19.552 INFO:teuthology.orchestra.run.smithi170.stdout:Size : 8264254 2024-02-10T07:16:19.552 INFO:teuthology.orchestra.run.smithi170.stdout:License : GPLv3+ 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout:Packager : builder@centos.org 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout:Vendor : CentOS 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout:Summary : Support tools for GRUB. 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout:Description : 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout:hardware devices. 2024-02-10T07:16:19.553 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:16:19.554 INFO:teuthology.orchestra.run.smithi170.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T07:16:19.554 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T07:16:19.554 INFO:teuthology.task.kernel:Updating grub on smithi170 to boot 5.14.0-419.el9.x86_64 2024-02-10T07:16:19.555 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T07:16:20.123 INFO:teuthology.orchestra.run.smithi170.stderr:Generating grub configuration file ... 2024-02-10T07:16:21.447 INFO:teuthology.orchestra.run.smithi170.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T07:16:21.484 INFO:teuthology.orchestra.run.smithi170.stderr:done 2024-02-10T07:16:21.489 DEBUG:teuthology.orchestra.run.smithi170:> mktemp 2024-02-10T07:16:21.509 INFO:teuthology.orchestra.run.smithi170.stdout:/tmp/tmp.o1gbtPKhTv 2024-02-10T07:16:21.509 DEBUG:teuthology.orchestra.run.smithi170:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.o1gbtPKhTv 2024-02-10T07:16:21.577 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0666 /tmp/tmp.o1gbtPKhTv 2024-02-10T07:16:21.714 DEBUG:teuthology.orchestra.remote:smithi170:/tmp/tmp.o1gbtPKhTv is 6KB 2024-02-10T07:16:21.762 DEBUG:teuthology.orchestra.run.smithi170:> rm -fr /tmp/tmp.o1gbtPKhTv 2024-02-10T07:16:21.776 DEBUG:teuthology.orchestra.run.smithi170:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T07:16:21.841 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T07:16:21.841 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T07:16:21.841 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T07:16:21.841 INFO:teuthology.orchestra.run.smithi170.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T07:16:21.842 DEBUG:teuthology.orchestra.run.smithi170:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T07:16:21.987 DEBUG:teuthology.orchestra.run.smithi170:> sudo shutdown -r now 2024-02-10T07:16:24.437 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-10T07:16:24.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T07:16:24.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:42.940 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-02-10T07:16:51.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T07:16:51.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:51.990 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-10T07:16:51.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-02-10T07:16:51.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:16:55.004 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.8 2024-02-10T07:17:07.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T07:17:07.018 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:17:10.332 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2024-02-10T07:17:19.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-02-10T07:17:19.340 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:17:22.396 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.170 2024-02-10T07:17:34.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-02-10T07:17:34.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:17:39.201 DEBUG:teuthology.orchestra.run.smithi008:> true 2024-02-10T07:17:39.761 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T07:17:39.761 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-419.el9.x86_64"... 2024-02-10T07:17:39.761 DEBUG:teuthology.orchestra.run.smithi008:> uname -r 2024-02-10T07:17:39.775 INFO:teuthology.orchestra.run.smithi008.stdout:5.14.0-419.el9.x86_64 2024-02-10T07:17:39.775 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T07:17:39.775 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T07:17:39.775 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T07:17:40.776 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-10T07:17:40.776 DEBUG:teuthology.orchestra.run.smithi008:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T07:17:40.837 INFO:teuthology.orchestra.run.smithi008.stdout:ttyS1 2024-02-10T07:17:40.864 DEBUG:teuthology.parallel:result is None 2024-02-10T07:18:06.258 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-02-10T07:18:06.688 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-02-10T07:18:06.688 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-419.el9.x86_64"... 2024-02-10T07:18:06.688 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-02-10T07:18:06.702 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-419.el9.x86_64 2024-02-10T07:18:06.702 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T07:18:06.702 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T07:18:06.702 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T07:18:07.703 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-10T07:18:07.704 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T07:18:07.800 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2024-02-10T07:18:07.847 DEBUG:teuthology.parallel:result is None 2024-02-10T07:18:07.847 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T07:18:07.858 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T07:18:07.859 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T07:18:07.861 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T07:18:07.886 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T07:18:07.893 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T07:18:07.903 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T07:18:07.904 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T07:18:07.919 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T07:18:07.976 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T07:18:07.984 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T07:18:07.985 DEBUG:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T07:18:07.987 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T07:18:08.024 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T07:18:08.038 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T07:18:08.235 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T07:18:08.255 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T07:18:08.258 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T07:18:08.265 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T07:18:08.265 DEBUG:teuthology.orchestra.run.smithi008:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T07:18:08.268 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T07:18:08.312 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T07:18:08.323 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T07:18:08.323 DEBUG:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T07:18:08.340 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T07:18:08.363 DEBUG:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T07:18:08.440 DEBUG:teuthology.orchestra.run.smithi008:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T07:18:08.489 DEBUG:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T07:18:08.575 DEBUG:teuthology.orchestra.run.smithi008:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T07:18:08.618 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:18:08.618 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T07:18:08.687 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T07:18:08.728 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T07:18:08.777 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T07:18:08.855 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T07:18:08.897 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-02-10T07:18:08.897 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T07:18:08.966 DEBUG:teuthology.orchestra.run.smithi008:> sudo service rsyslog restart 2024-02-10T07:18:08.975 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2024-02-10T07:18:09.024 INFO:teuthology.orchestra.run.smithi008.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T07:18:09.246 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T07:18:09.369 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T07:18:09.381 INFO:teuthology.task.internal:Starting timer... 2024-02-10T07:18:09.381 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T07:18:09.400 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T07:18:09.414 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-10T07:18:09.414 DEBUG:teuthology.orchestra.run.smithi008:> sudo service auditd rotate 2024-02-10T07:18:09.469 INFO:teuthology.orchestra.run.smithi008.stdout:Rotating logs: 2024-02-10T07:18:09.471 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2024-02-10T07:18:09.551 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2024-02-10T07:18:09.552 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T07:18:09.553 DEBUG:teuthology.orchestra.run.smithi008:> /usr/sbin/getenforce 2024-02-10T07:18:09.570 INFO:teuthology.orchestra.run.smithi008.stdout:Permissive 2024-02-10T07:18:09.571 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2024-02-10T07:18:09.592 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2024-02-10T07:18:09.593 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi008.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi170.front.sepia.ceph.com': 'permissive'} 2024-02-10T07:18:09.593 DEBUG:teuthology.orchestra.run.smithi008:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T07:18:09.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:18:09.640 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T07:18:09.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:18:09.667 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T07:18:09.667 DEBUG:teuthology.orchestra.run.smithi008:> sudo /usr/sbin/setenforce permissive 2024-02-10T07:18:09.715 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2024-02-10T07:18:09.745 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T07:18:09.763 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T07:18:09.781 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T07:18:09.783 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi008.front.sepia.ceph.com,smithi170.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T07:22:23.373 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi008.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com')] 2024-02-10T07:22:23.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T07:22:23.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi008.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:22:23.468 DEBUG:teuthology.orchestra.run.smithi008:> true 2024-02-10T07:22:23.554 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi008.front.sepia.ceph.com' 2024-02-10T07:22:23.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-02-10T07:22:23.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T07:22:23.645 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-02-10T07:22:23.726 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-02-10T07:22:23.727 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T07:22:23.738 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T07:22:23.739 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T07:22:23.739 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T07:22:23.741 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T07:22:23.742 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T07:22:23.771 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T07:22:23.788 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T07:22:23.797 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T07:22:23.814 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T07:22:23.817 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: ntpd: command not found 2024-02-10T07:22:23.831 INFO:teuthology.orchestra.run.smithi008.stdout:506 Cannot talk to daemon 2024-02-10T07:22:23.843 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2024-02-10T07:22:23.847 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T07:22:23.858 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2024-02-10T07:22:23.863 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T07:22:23.873 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T07:22:23.889 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T07:22:23.917 INFO:teuthology.orchestra.run.smithi008.stderr:bash: line 1: ntpq: command not found 2024-02-10T07:22:23.920 INFO:teuthology.orchestra.run.smithi008.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T07:22:23.920 INFO:teuthology.orchestra.run.smithi008.stdout:=============================================================================== 2024-02-10T07:22:23.920 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:22:23.920 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:22:23.920 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:22:23.920 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:22:23.941 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-02-10T07:22:23.944 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T07:22:23.944 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-02-10T07:22:23.944 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:22:23.945 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:22:23.945 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:22:23.945 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T07:22:23.946 INFO:teuthology.run_tasks:Running task install... 2024-02-10T07:22:23.954 DEBUG:teuthology.task.install:project ceph 2024-02-10T07:22:23.954 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}} 2024-02-10T07:22:23.954 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-10T07:22:23.954 INFO:teuthology.task.install:Using flavor: default 2024-02-10T07:22:23.960 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-10T07:22:23.960 INFO:teuthology.task.install:extra packages: [] 2024-02-10T07:22:23.960 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-10T07:22:23.961 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:22:23.963 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-10T07:22:23.963 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:22:24.132 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/ 2024-02-10T07:22:24.132 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1361.g290212a1 2024-02-10T07:22:24.141 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/ 2024-02-10T07:22:24.142 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1361.g290212a1 2024-02-10T07:22:24.261 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T07:22:24.261 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:22:24.261 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T07:22:24.269 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T07:22:24.269 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-02-10T07:22:24.269 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T07:22:24.289 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-10T07:22:24.289 DEBUG:teuthology.orchestra.run.smithi008:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T07:22:24.298 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-10T07:22:24.298 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T07:22:24.358 DEBUG:teuthology.orchestra.run.smithi008:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T07:22:24.365 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T07:22:24.438 DEBUG:teuthology.orchestra.run.smithi008:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T07:22:24.446 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T07:22:24.505 INFO:teuthology.orchestra.run.smithi008.stdout:check_obsoletes = 1 2024-02-10T07:22:24.506 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum clean all 2024-02-10T07:22:24.512 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2024-02-10T07:22:24.514 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2024-02-10T07:22:24.751 INFO:teuthology.orchestra.run.smithi008.stdout:74 files removed 2024-02-10T07:22:24.765 INFO:teuthology.orchestra.run.smithi170.stdout:74 files removed 2024-02-10T07:22:24.778 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-radosgw 2024-02-10T07:22:24.793 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw 2024-02-10T07:22:25.448 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 243 kB/s | 82 kB 00:00 2024-02-10T07:22:25.467 INFO:teuthology.orchestra.run.smithi008.stdout:ceph packages for x86_64 225 kB/s | 82 kB 00:00 2024-02-10T07:22:25.798 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 33 kB/s | 10 kB 00:00 2024-02-10T07:22:25.842 INFO:teuthology.orchestra.run.smithi008.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-02-10T07:22:26.174 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-02-10T07:22:26.235 INFO:teuthology.orchestra.run.smithi008.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-02-10T07:22:26.679 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-02-10T07:22:26.785 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-02-10T07:22:28.246 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-02-10T07:22:28.286 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-02-10T07:22:29.493 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.0 MB 00:00 2024-02-10T07:22:30.595 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.0 MB 00:02 2024-02-10T07:22:31.738 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-02-10T07:22:35.636 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 15 MB/s | 5.9 MB 00:00 2024-02-10T07:22:37.531 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-02-10T07:22:47.353 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-02-10T07:22:52.821 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 5.9 MB 00:01 2024-02-10T07:22:54.846 INFO:teuthology.orchestra.run.smithi008.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 15 kB 00:00 2024-02-10T07:22:57.583 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 20 MB 00:19 2024-02-10T07:23:02.530 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 9.9 kB/s | 1.7 kB 00:00 2024-02-10T07:23:05.175 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:23:05.187 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:23:05.187 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-02-10T07:23:05.187 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:23:05.187 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:23:05.187 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 9.7 M 2024-02-10T07:23:05.188 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2024-02-10T07:23:05.188 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.4 M 2024-02-10T07:23:05.188 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-10T07:23:05.188 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-02-10T07:23:05.188 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options 2024-02-10T07:23:05.188 INFO:teuthology.orchestra.run.smithi170.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-10T07:23:05.188 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.3 M 2024-02-10T07:23:05.188 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 20 M 2024-02-10T07:23:05.188 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:19.0.0-1361.g290212a1.el9 ceph 25 k 2024-02-10T07:23:05.188 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-10T07:23:05.188 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-10T07:23:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-10T07:23:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 743 k 2024-02-10T07:23:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-10T07:23:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-10T07:23:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-10T07:23:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 499 k 2024-02-10T07:23:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-10T07:23:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.1 M 2024-02-10T07:23:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-10T07:23:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-10T07:23:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-10T07:23:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse 2024-02-10T07:23:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: x86_64 2:19.0.0-1361.g290212a1.el9 ceph 45 k 2024-02-10T07:23:05.189 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 131 k 2024-02-10T07:23:05.190 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:19.0.0-1361.g290212a1.el9 ceph 161 k 2024-02-10T07:23:05.190 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-10T07:23:05.190 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:19.0.0-1361.g290212a1.el9 ceph 319 k 2024-02-10T07:23:05.190 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 298 k 2024-02-10T07:23:05.190 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 99 k 2024-02-10T07:23:05.190 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-02-10T07:23:05.190 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-02-10T07:23:05.190 INFO:teuthology.orchestra.run.smithi170.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-10T07:23:05.190 INFO:teuthology.orchestra.run.smithi170.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-10T07:23:05.190 INFO:teuthology.orchestra.run.smithi170.stdout:Installing weak dependencies: 2024-02-10T07:23:05.190 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-02-10T07:23:05.190 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:05.190 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:23:05.190 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:23:05.191 INFO:teuthology.orchestra.run.smithi170.stdout:Install 30 Packages 2024-02-10T07:23:05.191 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 2 Packages 2024-02-10T07:23:05.191 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:05.191 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 58 M 2024-02-10T07:23:05.191 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:23:06.056 INFO:teuthology.orchestra.run.smithi170.stdout:(1/32): ceph-base-19.0.0-1361.g290212a1.el9.x86 6.5 MB/s | 5.3 MB 00:00 2024-02-10T07:23:06.106 INFO:teuthology.orchestra.run.smithi170.stdout:(2/32): ceph-selinux-19.0.0-1361.g290212a1.el9. 505 kB/s | 25 kB 00:00 2024-02-10T07:23:06.240 INFO:teuthology.orchestra.run.smithi170.stdout:(3/32): libcephfs2-19.0.0-1361.g290212a1.el9.x8 5.4 MB/s | 743 kB 00:00 2024-02-10T07:23:06.323 INFO:teuthology.orchestra.run.smithi170.stdout:(4/32): libradosstriper1-19.0.0-1361.g290212a1. 5.9 MB/s | 499 kB 00:00 2024-02-10T07:23:06.716 INFO:teuthology.orchestra.run.smithi170.stdout:(5/32): ceph-radosgw-19.0.0-1361.g290212a1.el9. 6.6 MB/s | 9.7 MB 00:01 2024-02-10T07:23:06.790 INFO:teuthology.orchestra.run.smithi170.stdout:(6/32): python3-ceph-argparse-19.0.0-1361.g2902 603 kB/s | 45 kB 00:00 2024-02-10T07:23:06.882 INFO:teuthology.orchestra.run.smithi170.stdout:(7/32): python3-ceph-common-19.0.0-1361.g290212 1.4 MB/s | 131 kB 00:00 2024-02-10T07:23:06.974 INFO:teuthology.orchestra.run.smithi170.stdout:(8/32): librgw2-19.0.0-1361.g290212a1.el9.x86_6 7.8 MB/s | 5.1 MB 00:00 2024-02-10T07:23:06.999 INFO:teuthology.orchestra.run.smithi170.stdout:(9/32): python3-cephfs-19.0.0-1361.g290212a1.el 1.4 MB/s | 161 kB 00:00 2024-02-10T07:23:07.025 INFO:teuthology.orchestra.run.smithi170.stdout:(10/32): python3-rados-19.0.0-1361.g290212a1.el 6.2 MB/s | 319 kB 00:00 2024-02-10T07:23:07.050 INFO:teuthology.orchestra.run.smithi170.stdout:(11/32): python3-rbd-19.0.0-1361.g290212a1.el9. 5.7 MB/s | 298 kB 00:00 2024-02-10T07:23:07.076 INFO:teuthology.orchestra.run.smithi170.stdout:(12/32): python3-rgw-19.0.0-1361.g290212a1.el9. 1.9 MB/s | 99 kB 00:00 2024-02-10T07:23:07.259 INFO:teuthology.orchestra.run.smithi170.stdout:(13/32): ceph-common-19.0.0-1361.g290212a1.el9. 9.8 MB/s | 20 MB 00:02 2024-02-10T07:23:07.393 INFO:teuthology.orchestra.run.smithi170.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 309 kB/s | 106 kB 00:00 2024-02-10T07:23:07.418 INFO:teuthology.orchestra.run.smithi170.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 465 kB/s | 159 kB 00:00 2024-02-10T07:23:07.435 INFO:teuthology.orchestra.run.smithi170.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 349 kB/s | 44 kB 00:00 2024-02-10T07:23:07.544 INFO:teuthology.orchestra.run.smithi170.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 661 kB 00:00 2024-02-10T07:23:07.561 INFO:teuthology.orchestra.run.smithi170.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 330 kB/s | 41 kB 00:00 2024-02-10T07:23:07.586 INFO:teuthology.orchestra.run.smithi170.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 282 kB 00:00 2024-02-10T07:23:07.619 INFO:teuthology.orchestra.run.smithi170.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 2.8 MB/s | 215 kB 00:00 2024-02-10T07:23:07.644 INFO:teuthology.orchestra.run.smithi170.stdout:(21/32): qatlib-service-23.11.0-1.el9.x86_64.rp 406 kB/s | 34 kB 00:00 2024-02-10T07:23:07.670 INFO:teuthology.orchestra.run.smithi170.stdout:(22/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 519 kB/s | 43 kB 00:00 2024-02-10T07:23:08.612 INFO:teuthology.orchestra.run.smithi170.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 28 kB/s | 26 kB 00:00 2024-02-10T07:23:09.262 INFO:teuthology.orchestra.run.smithi170.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 191 kB/s | 309 kB 00:01 2024-02-10T07:23:09.287 INFO:teuthology.orchestra.run.smithi170.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 72 kB/s | 49 kB 00:00 2024-02-10T07:23:09.512 INFO:teuthology.orchestra.run.smithi170.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 269 kB/s | 67 kB 00:00 2024-02-10T07:23:09.896 INFO:teuthology.orchestra.run.smithi170.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 838 kB 00:00 2024-02-10T07:23:09.921 INFO:teuthology.orchestra.run.smithi170.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 469 kB/s | 192 kB 00:00 2024-02-10T07:23:10.097 INFO:teuthology.orchestra.run.smithi170.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 4.4 MB 00:02 2024-02-10T07:23:10.288 INFO:teuthology.orchestra.run.smithi170.stdout:(30/32): librados2-19.0.0-1361.g290212a1.el9.x8 9.2 MB/s | 3.4 MB 00:00 2024-02-10T07:23:10.355 INFO:teuthology.orchestra.run.smithi170.stdout:(31/32): thrift-0.15.0-2.el9.x86_64.rpm 3.5 MB/s | 1.6 MB 00:00 2024-02-10T07:23:10.448 INFO:teuthology.orchestra.run.smithi170.stdout:(32/32): librbd1-19.0.0-1361.g290212a1.el9.x86_ 8.9 MB/s | 3.1 MB 00:00 2024-02-10T07:23:10.454 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:23:10.454 INFO:teuthology.orchestra.run.smithi170.stdout:Total 11 MB/s | 58 MB 00:05 2024-02-10T07:23:10.455 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:23:10.489 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:23:10.489 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:23:10.555 INFO:teuthology.orchestra.run.smithi008.stdout:Extra Packages for Enterprise Linux 1.3 MB/s | 20 MB 00:15 2024-02-10T07:23:10.962 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:23:10.962 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:23:12.408 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:23:12.495 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-02-10T07:23:12.527 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-02-10T07:23:12.552 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-02-10T07:23:12.592 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-02-10T07:23:12.609 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-02-10T07:23:12.817 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-02-10T07:23:12.872 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-02-10T07:23:13.199 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-02-10T07:23:13.254 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T07:23:13.354 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T07:23:13.459 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 8/34 2024-02-10T07:23:13.489 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 9/34 2024-02-10T07:23:13.536 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 9/34 2024-02-10T07:23:13.564 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-02-10T07:23:13.608 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9. 11/34 2024-02-10T07:23:13.690 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 12/34 2024-02-10T07:23:13.717 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 13/34 2024-02-10T07:23:13.779 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 13/34 2024-02-10T07:23:13.807 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-02-10T07:23:13.917 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-02-10T07:23:13.950 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-02-10T07:23:13.990 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-02-10T07:23:14.023 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-02-10T07:23:14.069 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-02-10T07:23:14.109 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-02-10T07:23:14.436 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-02-10T07:23:14.464 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 22/34 2024-02-10T07:23:14.557 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 22/34 2024-02-10T07:23:14.585 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 23/34 2024-02-10T07:23:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-02-10T07:23:15.091 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-02-10T07:23:15.117 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 26/34 2024-02-10T07:23:15.188 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 26/34 2024-02-10T07:23:15.256 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 27/34 2024-02-10T07:23:15.286 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x8 28/34 2024-02-10T07:23:15.730 INFO:teuthology.orchestra.run.smithi008.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-02-10T07:23:17.481 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T07:23:17.524 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T07:23:18.075 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T07:23:18.111 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 30/34 2024-02-10T07:23:18.238 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 30/34 2024-02-10T07:23:18.238 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-10T07:23:18.238 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-10T07:23:18.238 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:18.279 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T07:23:18.577 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:23:18.589 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:23:18.589 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T07:23:18.589 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:23:18.589 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:23:18.589 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-radosgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 9.7 M 2024-02-10T07:23:18.589 INFO:teuthology.orchestra.run.smithi008.stdout:Upgrading: 2024-02-10T07:23:18.589 INFO:teuthology.orchestra.run.smithi008.stdout: librados2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.4 M 2024-02-10T07:23:18.589 INFO:teuthology.orchestra.run.smithi008.stdout: librbd1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-10T07:23:18.589 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T07:23:18.589 INFO:teuthology.orchestra.run.smithi008.stdout: boost-program-options 2024-02-10T07:23:18.589 INFO:teuthology.orchestra.run.smithi008.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-10T07:23:18.589 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-base x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.3 M 2024-02-10T07:23:18.590 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 20 M 2024-02-10T07:23:18.590 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-selinux x86_64 2:19.0.0-1361.g290212a1.el9 ceph 25 k 2024-02-10T07:23:18.590 INFO:teuthology.orchestra.run.smithi008.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-10T07:23:18.590 INFO:teuthology.orchestra.run.smithi008.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-10T07:23:18.590 INFO:teuthology.orchestra.run.smithi008.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-10T07:23:18.590 INFO:teuthology.orchestra.run.smithi008.stdout: libcephfs2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 743 k 2024-02-10T07:23:18.590 INFO:teuthology.orchestra.run.smithi008.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-10T07:23:18.590 INFO:teuthology.orchestra.run.smithi008.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-10T07:23:18.590 INFO:teuthology.orchestra.run.smithi008.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-10T07:23:18.590 INFO:teuthology.orchestra.run.smithi008.stdout: libradosstriper1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 499 k 2024-02-10T07:23:18.590 INFO:teuthology.orchestra.run.smithi008.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-10T07:23:18.590 INFO:teuthology.orchestra.run.smithi008.stdout: librgw2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.1 M 2024-02-10T07:23:18.590 INFO:teuthology.orchestra.run.smithi008.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-10T07:23:18.590 INFO:teuthology.orchestra.run.smithi008.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-10T07:23:18.591 INFO:teuthology.orchestra.run.smithi008.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-10T07:23:18.591 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-argparse 2024-02-10T07:23:18.591 INFO:teuthology.orchestra.run.smithi008.stdout: x86_64 2:19.0.0-1361.g290212a1.el9 ceph 45 k 2024-02-10T07:23:18.591 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 131 k 2024-02-10T07:23:18.591 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cephfs x86_64 2:19.0.0-1361.g290212a1.el9 ceph 161 k 2024-02-10T07:23:18.591 INFO:teuthology.orchestra.run.smithi008.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-10T07:23:18.591 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rados x86_64 2:19.0.0-1361.g290212a1.el9 ceph 319 k 2024-02-10T07:23:18.591 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 298 k 2024-02-10T07:23:18.591 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 99 k 2024-02-10T07:23:18.591 INFO:teuthology.orchestra.run.smithi008.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-02-10T07:23:18.591 INFO:teuthology.orchestra.run.smithi008.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-02-10T07:23:18.591 INFO:teuthology.orchestra.run.smithi008.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-10T07:23:18.592 INFO:teuthology.orchestra.run.smithi008.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-10T07:23:18.592 INFO:teuthology.orchestra.run.smithi008.stdout:Installing weak dependencies: 2024-02-10T07:23:18.592 INFO:teuthology.orchestra.run.smithi008.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-02-10T07:23:18.592 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:23:18.592 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:23:18.592 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:23:18.592 INFO:teuthology.orchestra.run.smithi008.stdout:Install 30 Packages 2024-02-10T07:23:18.592 INFO:teuthology.orchestra.run.smithi008.stdout:Upgrade 2 Packages 2024-02-10T07:23:18.592 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:23:18.593 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 58 M 2024-02-10T07:23:18.593 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:23:19.423 INFO:teuthology.orchestra.run.smithi008.stdout:(1/32): ceph-base-19.0.0-1361.g290212a1.el9.x86 7.0 MB/s | 5.3 MB 00:00 2024-02-10T07:23:19.489 INFO:teuthology.orchestra.run.smithi008.stdout:(2/32): ceph-selinux-19.0.0-1361.g290212a1.el9. 377 kB/s | 25 kB 00:00 2024-02-10T07:23:19.632 INFO:teuthology.orchestra.run.smithi008.stdout:(3/32): libcephfs2-19.0.0-1361.g290212a1.el9.x8 5.1 MB/s | 743 kB 00:00 2024-02-10T07:23:19.740 INFO:teuthology.orchestra.run.smithi008.stdout:(4/32): ceph-radosgw-19.0.0-1361.g290212a1.el9. 9.0 MB/s | 9.7 MB 00:01 2024-02-10T07:23:19.766 INFO:teuthology.orchestra.run.smithi008.stdout:(5/32): libradosstriper1-19.0.0-1361.g290212a1. 3.6 MB/s | 499 kB 00:00 2024-02-10T07:23:19.816 INFO:teuthology.orchestra.run.smithi008.stdout:(6/32): python3-ceph-argparse-19.0.0-1361.g2902 902 kB/s | 45 kB 00:00 2024-02-10T07:23:19.883 INFO:teuthology.orchestra.run.smithi008.stdout:(7/32): python3-ceph-common-19.0.0-1361.g290212 1.9 MB/s | 131 kB 00:00 2024-02-10T07:23:19.942 INFO:teuthology.orchestra.run.smithi008.stdout:(8/32): python3-cephfs-19.0.0-1361.g290212a1.el 2.7 MB/s | 161 kB 00:00 2024-02-10T07:23:20.017 INFO:teuthology.orchestra.run.smithi008.stdout:(9/32): python3-rados-19.0.0-1361.g290212a1.el9 4.2 MB/s | 319 kB 00:00 2024-02-10T07:23:20.134 INFO:teuthology.orchestra.run.smithi008.stdout:(10/32): librgw2-19.0.0-1361.g290212a1.el9.x86_ 13 MB/s | 5.1 MB 00:00 2024-02-10T07:23:20.159 INFO:teuthology.orchestra.run.smithi008.stdout:(11/32): python3-rbd-19.0.0-1361.g290212a1.el9. 2.1 MB/s | 298 kB 00:00 2024-02-10T07:23:20.184 INFO:teuthology.orchestra.run.smithi008.stdout:(12/32): python3-rgw-19.0.0-1361.g290212a1.el9. 1.9 MB/s | 99 kB 00:00 2024-02-10T07:23:20.310 INFO:teuthology.orchestra.run.smithi008.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-02-10T07:23:20.335 INFO:teuthology.orchestra.run.smithi008.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 603 kB/s | 106 kB 00:00 2024-02-10T07:23:20.352 INFO:teuthology.orchestra.run.smithi008.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-02-10T07:23:20.410 INFO:teuthology.orchestra.run.smithi008.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-02-10T07:23:20.435 INFO:teuthology.orchestra.run.smithi008.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-02-10T07:23:20.452 INFO:teuthology.orchestra.run.smithi008.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 995 kB/s | 41 kB 00:00 2024-02-10T07:23:20.478 INFO:teuthology.orchestra.run.smithi008.stdout:(19/32): qatlib-23.11.0-1.el9.x86_64.rpm 5.0 MB/s | 215 kB 00:00 2024-02-10T07:23:20.495 INFO:teuthology.orchestra.run.smithi008.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 810 kB/s | 34 kB 00:00 2024-02-10T07:23:20.512 INFO:teuthology.orchestra.run.smithi008.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-02-10T07:23:21.120 INFO:teuthology.orchestra.run.smithi008.stdout:(22/32): ceph-common-19.0.0-1361.g290212a1.el9. 8.1 MB/s | 20 MB 00:02 2024-02-10T07:23:21.146 INFO:teuthology.orchestra.run.smithi008.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 475 kB/s | 309 kB 00:00 2024-02-10T07:23:21.238 INFO:teuthology.orchestra.run.smithi008.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-02-10T07:23:21.329 INFO:teuthology.orchestra.run.smithi008.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 735 kB/s | 67 kB 00:00 2024-02-10T07:23:21.388 INFO:teuthology.orchestra.run.smithi008.stdout:(26/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-02-10T07:23:21.538 INFO:teuthology.orchestra.run.smithi008.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 4.4 MB 00:01 2024-02-10T07:23:21.580 INFO:teuthology.orchestra.run.smithi008.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-02-10T07:23:21.656 INFO:teuthology.orchestra.run.smithi008.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 717 kB/s | 192 kB 00:00 2024-02-10T07:23:21.697 INFO:teuthology.orchestra.run.smithi008.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-02-10T07:23:22.039 INFO:teuthology.orchestra.run.smithi008.stdout:(31/32): librados2-19.0.0-1361.g290212a1.el9.x8 7.4 MB/s | 3.4 MB 00:00 2024-02-10T07:23:22.098 INFO:teuthology.orchestra.run.smithi008.stdout:(32/32): librbd1-19.0.0-1361.g290212a1.el9.x86_ 7.1 MB/s | 3.1 MB 00:00 2024-02-10T07:23:22.103 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:23:22.103 INFO:teuthology.orchestra.run.smithi008.stdout:Total 16 MB/s | 58 MB 00:03 2024-02-10T07:23:22.103 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:23:22.145 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:23:22.145 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:23:22.642 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:23:22.642 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:23:24.159 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:23:24.258 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-02-10T07:23:24.294 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-02-10T07:23:24.329 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-02-10T07:23:24.351 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-02-10T07:23:24.370 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-02-10T07:23:24.581 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-02-10T07:23:24.657 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-02-10T07:23:24.991 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-02-10T07:23:25.030 INFO:teuthology.orchestra.run.smithi008.stdout: Upgrading : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T07:23:25.122 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T07:23:25.212 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 8/34 2024-02-10T07:23:25.249 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 9/34 2024-02-10T07:23:25.312 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 9/34 2024-02-10T07:23:25.340 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-02-10T07:23:25.385 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9. 11/34 2024-02-10T07:23:25.454 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 12/34 2024-02-10T07:23:25.476 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 13/34 2024-02-10T07:23:25.547 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 13/34 2024-02-10T07:23:25.575 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-02-10T07:23:25.687 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-02-10T07:23:25.726 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-02-10T07:23:25.767 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-02-10T07:23:25.799 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-02-10T07:23:25.828 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-02-10T07:23:25.861 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-02-10T07:23:26.190 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-02-10T07:23:26.232 INFO:teuthology.orchestra.run.smithi008.stdout: Upgrading : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 22/34 2024-02-10T07:23:26.315 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 22/34 2024-02-10T07:23:26.345 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 23/34 2024-02-10T07:23:26.387 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-02-10T07:23:26.862 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-02-10T07:23:26.893 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 26/34 2024-02-10T07:23:26.956 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 26/34 2024-02-10T07:23:27.027 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 27/34 2024-02-10T07:23:27.062 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x8 28/34 2024-02-10T07:23:27.714 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T07:23:27.714 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2024-02-10T07:23:27.714 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2024-02-10T07:23:27.714 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2024-02-10T07:23:27.714 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2024-02-10T07:23:27.714 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2024-02-10T07:23:27.714 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /root 2024-02-10T07:23:27.714 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2024-02-10T07:23:27.714 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:28.634 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 32/34 2024-02-10T07:23:28.659 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 32/34 2024-02-10T07:23:28.659 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:23:28.659 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-10T07:23:28.659 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T07:23:28.659 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T07:23:28.659 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:28.661 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-02-10T07:23:28.753 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-02-10T07:23:28.753 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T07:23:29.361 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T07:23:29.400 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T07:23:29.951 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T07:23:29.952 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 1/34 2024-02-10T07:23:29.952 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2/34 2024-02-10T07:23:29.952 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 3/34 2024-02-10T07:23:29.952 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 4/34 2024-02-10T07:23:29.952 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 5/34 2024-02-10T07:23:29.952 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 6/34 2024-02-10T07:23:29.952 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T07:23:29.952 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9. 8/34 2024-02-10T07:23:29.952 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x8 9/34 2024-02-10T07:23:29.952 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 10/34 2024-02-10T07:23:29.952 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 11/34 2024-02-10T07:23:29.952 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 12/34 2024-02-10T07:23:29.953 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 13/34 2024-02-10T07:23:29.953 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-02-10T07:23:29.953 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-02-10T07:23:29.953 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-02-10T07:23:29.953 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-02-10T07:23:29.953 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-02-10T07:23:29.953 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-02-10T07:23:29.953 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-02-10T07:23:29.953 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-02-10T07:23:29.953 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-02-10T07:23:29.953 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-02-10T07:23:29.953 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-02-10T07:23:29.953 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-02-10T07:23:29.953 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-02-10T07:23:29.954 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-02-10T07:23:29.954 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-02-10T07:23:29.954 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-02-10T07:23:29.954 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-02-10T07:23:29.954 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T07:23:29.954 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-02-10T07:23:29.954 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 33/34 2024-02-10T07:23:29.972 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 29/34 2024-02-10T07:23:30.012 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 30/34 2024-02-10T07:23:30.102 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 30/34 2024-02-10T07:23:30.102 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-10T07:23:30.103 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-10T07:23:30.103 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:23:30.138 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T07:23:31.054 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T07:23:31.054 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:31.055 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2024-02-10T07:23:31.055 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.055 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.055 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:23:31.055 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-10T07:23:31.055 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.055 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.055 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.055 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.055 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-10T07:23:31.055 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-10T07:23:31.055 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-10T07:23:31.055 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.056 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.057 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:31.057 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-23.11.0-1.el9.x86_64 2024-02-10T07:23:31.057 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-02-10T07:23:31.057 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-02-10T07:23:31.057 INFO:teuthology.orchestra.run.smithi170.stdout: re2-1:20211101-3.el9.x86_64 2024-02-10T07:23:31.057 INFO:teuthology.orchestra.run.smithi170.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-10T07:23:31.057 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:31.057 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:23:31.395 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-test 2024-02-10T07:23:31.973 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:29 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:23:32.131 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:23:32.131 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:23:32.131 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-02-10T07:23:32.132 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:23:32.132 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:23:32.132 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:19.0.0-1361.g290212a1.el9 ceph 47 M 2024-02-10T07:23:32.132 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-02-10T07:23:32.132 INFO:teuthology.orchestra.run.smithi170.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-10T07:23:32.132 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:19.0.0-1361.g290212a1.el9 ceph 163 k 2024-02-10T07:23:32.132 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-10T07:23:32.132 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-10T07:23:32.132 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-10T07:23:32.132 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:32.132 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:23:32.132 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:23:32.132 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2024-02-10T07:23:32.132 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:32.133 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 48 M 2024-02-10T07:23:32.133 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 204 M 2024-02-10T07:23:32.133 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:23:32.290 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.9 MB/s | 300 kB 00:00 2024-02-10T07:23:32.324 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-02-10T07:23:32.366 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-02-10T07:23:32.408 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-02-10T07:23:32.433 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): libcephsqlite-19.0.0-1361.g290212a1.el9. 545 kB/s | 163 kB 00:00 2024-02-10T07:23:34.500 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): ceph-test-19.0.0-1361.g290212a1.el9.x86_ 20 MB/s | 47 MB 00:02 2024-02-10T07:23:34.503 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:23:34.503 INFO:teuthology.orchestra.run.smithi170.stdout:Total 20 MB/s | 48 MB 00:02 2024-02-10T07:23:34.504 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:23:34.523 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:23:34.523 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:23:34.918 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:23:34.918 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:23:35.496 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:23:35.555 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-10T07:23:35.598 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-10T07:23:35.643 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-10T07:23:35.698 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-10T07:23:35.720 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 5/6 2024-02-10T07:23:40.108 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T07:23:40.108 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /sys 2024-02-10T07:23:40.108 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /proc 2024-02-10T07:23:40.108 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /mnt 2024-02-10T07:23:40.108 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /var/tmp 2024-02-10T07:23:40.108 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /home 2024-02-10T07:23:40.108 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /root 2024-02-10T07:23:40.108 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /tmp 2024-02-10T07:23:40.108 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:23:40.246 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 5/6 2024-02-10T07:23:40.327 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 6/6 2024-02-10T07:23:41.037 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 32/34 2024-02-10T07:23:41.062 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 32/34 2024-02-10T07:23:41.062 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:23:41.062 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-10T07:23:41.062 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T07:23:41.062 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-10T07:23:41.062 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:23:41.064 INFO:teuthology.orchestra.run.smithi008.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-02-10T07:23:41.148 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-02-10T07:23:41.148 INFO:teuthology.orchestra.run.smithi008.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T07:23:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T07:23:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 1/34 2024-02-10T07:23:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2/34 2024-02-10T07:23:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 3/34 2024-02-10T07:23:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 4/34 2024-02-10T07:23:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 5/34 2024-02-10T07:23:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_6 6/34 2024-02-10T07:23:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 7/34 2024-02-10T07:23:42.484 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9. 8/34 2024-02-10T07:23:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x8 9/34 2024-02-10T07:23:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 10/34 2024-02-10T07:23:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 11/34 2024-02-10T07:23:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 12/34 2024-02-10T07:23:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 13/34 2024-02-10T07:23:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-02-10T07:23:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-02-10T07:23:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-02-10T07:23:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-02-10T07:23:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-02-10T07:23:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-02-10T07:23:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-02-10T07:23:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-02-10T07:23:42.485 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-02-10T07:23:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-02-10T07:23:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-02-10T07:23:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-02-10T07:23:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-02-10T07:23:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-02-10T07:23:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-02-10T07:23:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-02-10T07:23:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-02-10T07:23:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 31/34 2024-02-10T07:23:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-02-10T07:23:42.486 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 33/34 2024-02-10T07:23:42.813 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 6/6 2024-02-10T07:23:42.813 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 1/6 2024-02-10T07:23:42.813 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2/6 2024-02-10T07:23:42.813 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-10T07:23:42.813 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-10T07:23:42.813 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-10T07:23:43.218 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-10T07:23:43.218 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:43.218 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:23:43.218 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.218 INFO:teuthology.orchestra.run.smithi170.stdout: jq-1.6-16.el9.x86_64 2024-02-10T07:23:43.218 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.218 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-10T07:23:43.218 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-10T07:23:43.218 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-10T07:23:43.218 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:43.218 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:23:43.499 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-02-10T07:23:43.500 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:23:43.500 INFO:teuthology.orchestra.run.smithi008.stdout:Upgraded: 2024-02-10T07:23:43.500 INFO:teuthology.orchestra.run.smithi008.stdout: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.500 INFO:teuthology.orchestra.run.smithi008.stdout: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.500 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:23:43.500 INFO:teuthology.orchestra.run.smithi008.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-10T07:23:43.500 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.500 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.500 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.500 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.500 INFO:teuthology.orchestra.run.smithi008.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.501 INFO:teuthology.orchestra.run.smithi008.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-10T07:23:43.502 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.502 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.502 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:43.502 INFO:teuthology.orchestra.run.smithi008.stdout: qatlib-23.11.0-1.el9.x86_64 2024-02-10T07:23:43.502 INFO:teuthology.orchestra.run.smithi008.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-02-10T07:23:43.502 INFO:teuthology.orchestra.run.smithi008.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-02-10T07:23:43.502 INFO:teuthology.orchestra.run.smithi008.stdout: re2-1:20211101-3.el9.x86_64 2024-02-10T07:23:43.502 INFO:teuthology.orchestra.run.smithi008.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-10T07:23:43.502 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:23:43.502 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:23:43.522 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph 2024-02-10T07:23:43.766 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-test 2024-02-10T07:23:44.120 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:42 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:23:44.304 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:23:44.306 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:19.0.0-1361.g290212a1.el9 ceph 6.6 k 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:19.0.0-1361.g290212a1.el9 ceph 2.1 M 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:19.0.0-1361.g290212a1.el9 ceph 1.5 M 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 247 k 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:19.0.0-1361.g290212a1.el9 ceph 4.7 M 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 17 M 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-10T07:23:44.307 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-10T07:23:44.308 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-10T07:23:44.308 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-10T07:23:44.308 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-10T07:23:44.308 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-10T07:23:44.308 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-10T07:23:44.308 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-10T07:23:44.308 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-10T07:23:44.308 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-10T07:23:44.308 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-10T07:23:44.308 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-10T07:23:44.308 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-10T07:23:44.308 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-10T07:23:44.308 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-10T07:23:44.308 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout:========================================================================================= 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout:Install 36 Packages 2024-02-10T07:23:44.309 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:44.310 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 30 M 2024-02-10T07:23:44.310 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 107 M 2024-02-10T07:23:44.310 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:23:44.356 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:00:29 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:23:44.529 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:23:44.529 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:23:44.529 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T07:23:44.530 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:23:44.530 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:23:44.530 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-test x86_64 2:19.0.0-1361.g290212a1.el9 ceph 47 M 2024-02-10T07:23:44.530 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T07:23:44.530 INFO:teuthology.orchestra.run.smithi008.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-10T07:23:44.530 INFO:teuthology.orchestra.run.smithi008.stdout: libcephsqlite x86_64 2:19.0.0-1361.g290212a1.el9 ceph 163 k 2024-02-10T07:23:44.530 INFO:teuthology.orchestra.run.smithi008.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-10T07:23:44.530 INFO:teuthology.orchestra.run.smithi008.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-10T07:23:44.530 INFO:teuthology.orchestra.run.smithi008.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-10T07:23:44.530 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:23:44.530 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:23:44.530 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:23:44.531 INFO:teuthology.orchestra.run.smithi008.stdout:Install 6 Packages 2024-02-10T07:23:44.531 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:23:44.531 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 48 M 2024-02-10T07:23:44.531 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 204 M 2024-02-10T07:23:44.531 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:23:44.701 INFO:teuthology.orchestra.run.smithi008.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 300 kB 00:00 2024-02-10T07:23:44.734 INFO:teuthology.orchestra.run.smithi008.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-02-10T07:23:44.776 INFO:teuthology.orchestra.run.smithi008.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-02-10T07:23:44.818 INFO:teuthology.orchestra.run.smithi008.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-02-10T07:23:44.835 INFO:teuthology.orchestra.run.smithi008.stdout:(5/6): libcephsqlite-19.0.0-1361.g290212a1.el9. 539 kB/s | 163 kB 00:00 2024-02-10T07:23:45.019 INFO:teuthology.orchestra.run.smithi170.stdout:(1/36): ceph-19.0.0-1361.g290212a1.el9.x86_64.r 25 kB/s | 6.6 kB 00:00 2024-02-10T07:23:46.120 INFO:teuthology.orchestra.run.smithi170.stdout:(2/36): ceph-mgr-19.0.0-1361.g290212a1.el9.x86_ 1.1 MB/s | 1.5 MB 00:01 2024-02-10T07:23:46.378 INFO:teuthology.orchestra.run.smithi170.stdout:(3/36): ceph-mds-19.0.0-1361.g290212a1.el9.x86_ 1.3 MB/s | 2.1 MB 00:01 2024-02-10T07:23:46.553 INFO:teuthology.orchestra.run.smithi170.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1361.g2902 1.4 MB/s | 247 kB 00:00 2024-02-10T07:23:46.662 INFO:teuthology.orchestra.run.smithi170.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-02-10T07:23:46.704 INFO:teuthology.orchestra.run.smithi170.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-02-10T07:23:46.738 INFO:teuthology.orchestra.run.smithi170.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-02-10T07:23:46.771 INFO:teuthology.orchestra.run.smithi170.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-02-10T07:23:46.805 INFO:teuthology.orchestra.run.smithi170.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-02-10T07:23:46.848 INFO:teuthology.orchestra.run.smithi170.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-02-10T07:23:46.914 INFO:teuthology.orchestra.run.smithi170.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-02-10T07:23:46.947 INFO:teuthology.orchestra.run.smithi170.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-02-10T07:23:46.981 INFO:teuthology.orchestra.run.smithi170.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-02-10T07:23:47.015 INFO:teuthology.orchestra.run.smithi170.stdout:(14/36): python3-requests-2.25.1-8.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2024-02-10T07:23:47.049 INFO:teuthology.orchestra.run.smithi170.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-02-10T07:23:47.093 INFO:teuthology.orchestra.run.smithi008.stdout:(6/6): ceph-test-19.0.0-1361.g290212a1.el9.x86_ 18 MB/s | 47 MB 00:02 2024-02-10T07:23:47.097 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:23:47.097 INFO:teuthology.orchestra.run.smithi008.stdout:Total 19 MB/s | 48 MB 00:02 2024-02-10T07:23:47.097 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:23:47.119 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:23:47.120 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:23:47.166 INFO:teuthology.orchestra.run.smithi170.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-02-10T07:23:47.299 INFO:teuthology.orchestra.run.smithi170.stdout:(17/36): ceph-mon-19.0.0-1361.g290212a1.el9.x86 2.1 MB/s | 4.7 MB 00:02 2024-02-10T07:23:47.400 INFO:teuthology.orchestra.run.smithi170.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 647 kB/s | 151 kB 00:00 2024-02-10T07:23:47.492 INFO:teuthology.orchestra.run.smithi170.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-02-10T07:23:47.508 INFO:teuthology.orchestra.run.smithi170.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 208 kB/s | 43 kB 00:00 2024-02-10T07:23:47.529 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:23:47.530 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:23:47.550 INFO:teuthology.orchestra.run.smithi170.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-02-10T07:23:47.601 INFO:teuthology.orchestra.run.smithi170.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-02-10T07:23:47.643 INFO:teuthology.orchestra.run.smithi170.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.3 MB/s | 358 kB 00:00 2024-02-10T07:23:47.662 INFO:teuthology.orchestra.run.smithi170.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 382 kB/s | 23 kB 00:00 2024-02-10T07:23:47.679 INFO:teuthology.orchestra.run.smithi170.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 542 kB/s | 19 kB 00:00 2024-02-10T07:23:47.721 INFO:teuthology.orchestra.run.smithi170.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 335 kB/s | 20 kB 00:00 2024-02-10T07:23:47.746 INFO:teuthology.orchestra.run.smithi170.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 690 kB/s | 46 kB 00:00 2024-02-10T07:23:47.955 INFO:teuthology.orchestra.run.smithi170.stdout:(28/36): ceph-osd-19.0.0-1361.g290212a1.el9.x86 9.1 MB/s | 17 MB 00:01 2024-02-10T07:23:47.989 INFO:teuthology.orchestra.run.smithi170.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 295 kB/s | 79 kB 00:00 2024-02-10T07:23:48.038 INFO:teuthology.orchestra.run.smithi170.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 932 kB/s | 272 kB 00:00 2024-02-10T07:23:48.081 INFO:teuthology.orchestra.run.smithi170.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-02-10T07:23:48.106 INFO:teuthology.orchestra.run.smithi170.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 543 kB/s | 36 kB 00:00 2024-02-10T07:23:48.148 INFO:teuthology.orchestra.run.smithi170.stdout:(33/36): python3-portend-3.1.0-2.el9.noarch.rpm 85 kB/s | 16 kB 00:00 2024-02-10T07:23:48.172 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:23:48.190 INFO:teuthology.orchestra.run.smithi170.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-02-10T07:23:48.215 INFO:teuthology.orchestra.run.smithi170.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 298 kB/s | 20 kB 00:00 2024-02-10T07:23:48.247 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-10T07:23:48.285 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-10T07:23:48.324 INFO:teuthology.orchestra.run.smithi170.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.9 MB/s | 427 kB 00:00 2024-02-10T07:23:48.329 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:23:48.329 INFO:teuthology.orchestra.run.smithi170.stdout:Total 7.4 MB/s | 30 MB 00:04 2024-02-10T07:23:48.329 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-10T07:23:48.350 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:23:48.370 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-10T07:23:48.390 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:23:48.390 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:23:48.397 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 5/6 2024-02-10T07:23:48.808 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:23:48.808 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:23:50.129 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:23:50.193 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-10T07:23:50.242 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-10T07:23:50.270 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-10T07:23:50.340 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-10T07:23:50.385 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-10T07:23:50.419 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-10T07:23:50.510 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-10T07:23:50.635 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-10T07:23:50.688 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-10T07:23:50.760 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-10T07:23:50.814 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-10T07:23:50.852 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-10T07:23:50.901 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-10T07:23:50.964 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-10T07:23:51.158 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-10T07:23:51.207 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-10T07:23:51.255 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-10T07:23:51.288 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-10T07:23:51.318 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-10T07:23:51.403 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-10T07:23:51.445 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-10T07:23:51.479 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-10T07:23:51.519 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-10T07:23:51.554 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-10T07:23:51.589 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-10T07:23:51.678 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-10T07:23:51.711 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-10T07:23:51.792 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-10T07:23:51.937 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-10T07:23:52.062 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-10T07:23:52.222 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9. 31/36 2024-02-10T07:23:52.250 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 32/36 2024-02-10T07:23:52.275 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 32/36 2024-02-10T07:23:52.275 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:23:52.275 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-10T07:23:52.275 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T07:23:52.275 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T07:23:52.275 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:53.161 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 5/6 2024-02-10T07:23:53.257 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 6/6 2024-02-10T07:23:53.709 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 33/36 2024-02-10T07:23:53.735 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 33/36 2024-02-10T07:23:53.735 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:23:53.735 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-10T07:23:53.735 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T07:23:53.735 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T07:23:53.735 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:54.193 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 34/36 2024-02-10T07:23:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 34/36 2024-02-10T07:23:54.216 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:23:54.216 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-10T07:23:54.216 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T07:23:54.216 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T07:23:54.216 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:54.427 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 35/36 2024-02-10T07:23:54.450 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 35/36 2024-02-10T07:23:54.450 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:23:54.450 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-10T07:23:54.450 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T07:23:54.450 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T07:23:54.450 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:54.515 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 36/36 2024-02-10T07:23:55.704 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 6/6 2024-02-10T07:23:55.704 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 1/6 2024-02-10T07:23:55.704 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2/6 2024-02-10T07:23:55.704 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-10T07:23:55.705 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-10T07:23:55.705 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-10T07:23:56.212 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-10T07:23:56.212 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:23:56.213 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:23:56.213 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:56.213 INFO:teuthology.orchestra.run.smithi008.stdout: jq-1.6-16.el9.x86_64 2024-02-10T07:23:56.213 INFO:teuthology.orchestra.run.smithi008.stdout: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:56.213 INFO:teuthology.orchestra.run.smithi008.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-10T07:23:56.213 INFO:teuthology.orchestra.run.smithi008.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-10T07:23:56.213 INFO:teuthology.orchestra.run.smithi008.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-10T07:23:56.213 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:23:56.213 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:23:56.353 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 36/36 2024-02-10T07:23:56.353 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 1/36 2024-02-10T07:23:56.353 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2/36 2024-02-10T07:23:56.353 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 3/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 4/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 5/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9. 6/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-10T07:23:56.354 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-10T07:23:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-10T07:23:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-10T07:23:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-10T07:23:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-10T07:23:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-10T07:23:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-10T07:23:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-10T07:23:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-10T07:23:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-10T07:23:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-10T07:23:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-10T07:23:56.355 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-10T07:23:56.356 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-10T07:23:56.356 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-10T07:23:56.356 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-10T07:23:56.357 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-10T07:23:56.357 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-10T07:23:56.516 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph 2024-02-10T07:23:57.109 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:00:42 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:23:57.279 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:23:57.281 INFO:teuthology.orchestra.run.smithi008.stdout:========================================================================================= 2024-02-10T07:23:57.282 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T07:23:57.282 INFO:teuthology.orchestra.run.smithi008.stdout:========================================================================================= 2024-02-10T07:23:57.282 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:23:57.282 INFO:teuthology.orchestra.run.smithi008.stdout: ceph x86_64 2:19.0.0-1361.g290212a1.el9 ceph 6.6 k 2024-02-10T07:23:57.282 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T07:23:57.282 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mds x86_64 2:19.0.0-1361.g290212a1.el9 ceph 2.1 M 2024-02-10T07:23:57.282 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr x86_64 2:19.0.0-1361.g290212a1.el9 ceph 1.5 M 2024-02-10T07:23:57.282 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 247 k 2024-02-10T07:23:57.282 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mon x86_64 2:19.0.0-1361.g290212a1.el9 ceph 4.7 M 2024-02-10T07:23:57.282 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-osd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 17 M 2024-02-10T07:23:57.282 INFO:teuthology.orchestra.run.smithi008.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-10T07:23:57.282 INFO:teuthology.orchestra.run.smithi008.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-10T07:23:57.283 INFO:teuthology.orchestra.run.smithi008.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-10T07:23:57.283 INFO:teuthology.orchestra.run.smithi008.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-10T07:23:57.283 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-10T07:23:57.283 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-10T07:23:57.283 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-10T07:23:57.283 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-10T07:23:57.283 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-10T07:23:57.283 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-10T07:23:57.283 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-10T07:23:57.283 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-10T07:23:57.283 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-10T07:23:57.283 INFO:teuthology.orchestra.run.smithi008.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-10T07:23:57.283 INFO:teuthology.orchestra.run.smithi008.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-10T07:23:57.284 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:23:57.285 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:23:57.285 INFO:teuthology.orchestra.run.smithi008.stdout:========================================================================================= 2024-02-10T07:23:57.285 INFO:teuthology.orchestra.run.smithi008.stdout:Install 36 Packages 2024-02-10T07:23:57.285 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:23:57.285 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 30 M 2024-02-10T07:23:57.285 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 107 M 2024-02-10T07:23:57.285 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:23:57.313 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-10T07:23:57.314 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:57.314 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:23:57.314 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:57.314 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:57.314 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:57.314 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:23:57.314 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:57.314 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:23:57.314 INFO:teuthology.orchestra.run.smithi170.stdout: lua-5.4.4-4.el9.x86_64 2024-02-10T07:23:57.314 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-10T07:23:57.314 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-10T07:23:57.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-10T07:23:57.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-10T07:23:57.314 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply-3.11-14.el9.noarch 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-10T07:23:57.315 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-10T07:23:57.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-10T07:23:57.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-10T07:23:57.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-10T07:23:57.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-10T07:23:57.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-10T07:23:57.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-10T07:23:57.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-10T07:23:57.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-10T07:23:57.316 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-10T07:23:57.316 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:57.316 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:23:57.597 INFO:teuthology.orchestra.run.smithi008.stdout:(1/36): ceph-19.0.0-1361.g290212a1.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-02-10T07:23:57.598 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-base 2024-02-10T07:23:57.847 INFO:teuthology.orchestra.run.smithi008.stdout:(2/36): ceph-mds-19.0.0-1361.g290212a1.el9.x86_ 5.0 MB/s | 2.1 MB 00:00 2024-02-10T07:23:57.981 INFO:teuthology.orchestra.run.smithi008.stdout:(3/36): ceph-mgr-19.0.0-1361.g290212a1.el9.x86_ 2.7 MB/s | 1.5 MB 00:00 2024-02-10T07:23:58.056 INFO:teuthology.orchestra.run.smithi008.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1361.g2902 3.2 MB/s | 247 kB 00:00 2024-02-10T07:23:58.164 INFO:teuthology.orchestra.run.smithi008.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-02-10T07:23:58.178 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:56 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:23:58.198 INFO:teuthology.orchestra.run.smithi008.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-02-10T07:23:58.265 INFO:teuthology.orchestra.run.smithi008.stdout:(7/36): ceph-mon-19.0.0-1361.g290212a1.el9.x86_ 7.0 MB/s | 4.7 MB 00:00 2024-02-10T07:23:58.287 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:23:58.290 INFO:teuthology.orchestra.run.smithi008.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 343 kB/s | 32 kB 00:00 2024-02-10T07:23:58.316 INFO:teuthology.orchestra.run.smithi008.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2024-02-10T07:23:58.339 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:23:58.340 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-02-10T07:23:58.340 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:23:58.341 INFO:teuthology.orchestra.run.smithi008.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-02-10T07:23:58.358 INFO:teuthology.orchestra.run.smithi008.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-02-10T07:23:58.396 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install cephadm 2024-02-10T07:23:58.408 INFO:teuthology.orchestra.run.smithi008.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-02-10T07:23:58.425 INFO:teuthology.orchestra.run.smithi008.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-02-10T07:23:58.442 INFO:teuthology.orchestra.run.smithi008.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-02-10T07:23:58.467 INFO:teuthology.orchestra.run.smithi008.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 2.6 MB/s | 113 kB 00:00 2024-02-10T07:23:58.492 INFO:teuthology.orchestra.run.smithi008.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-02-10T07:23:58.585 INFO:teuthology.orchestra.run.smithi008.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-02-10T07:23:58.835 INFO:teuthology.orchestra.run.smithi008.stdout:(18/36): ceph-osd-19.0.0-1361.g290212a1.el9.x86 17 MB/s | 17 MB 00:00 2024-02-10T07:23:58.852 INFO:teuthology.orchestra.run.smithi008.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 163 kB/s | 43 kB 00:00 2024-02-10T07:23:58.885 INFO:teuthology.orchestra.run.smithi008.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 385 kB/s | 151 kB 00:00 2024-02-10T07:23:58.984 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:56 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:23:59.002 INFO:teuthology.orchestra.run.smithi008.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-02-10T07:23:59.019 INFO:teuthology.orchestra.run.smithi008.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 80 kB/s | 11 kB 00:00 2024-02-10T07:23:59.044 INFO:teuthology.orchestra.run.smithi008.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-02-10T07:23:59.062 INFO:teuthology.orchestra.run.smithi008.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 301 kB/s | 18 kB 00:00 2024-02-10T07:23:59.078 INFO:teuthology.orchestra.run.smithi008.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 392 kB/s | 23 kB 00:00 2024-02-10T07:23:59.144 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:23:59.145 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:23:59.145 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-02-10T07:23:59.145 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:23:59.145 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:23:59.145 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 748 k 2024-02-10T07:23:59.145 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:59.145 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:23:59.145 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:23:59.145 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-02-10T07:23:59.146 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:23:59.146 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 748 k 2024-02-10T07:23:59.146 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 754 k 2024-02-10T07:23:59.146 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:23:59.162 INFO:teuthology.orchestra.run.smithi008.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 165 kB/s | 19 kB 00:00 2024-02-10T07:23:59.646 INFO:teuthology.orchestra.run.smithi008.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 34 kB/s | 20 kB 00:00 2024-02-10T07:23:59.663 INFO:teuthology.orchestra.run.smithi008.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 79 kB/s | 46 kB 00:00 2024-02-10T07:23:59.688 INFO:teuthology.orchestra.run.smithi008.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 150 kB/s | 79 kB 00:00 2024-02-10T07:23:59.818 INFO:teuthology.orchestra.run.smithi170.stdout:cephadm-19.0.0-1361.g290212a1.el9.noarch.rpm 1.1 MB/s | 748 kB 00:00 2024-02-10T07:23:59.818 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:23:59.819 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.1 MB/s | 748 kB 00:00 2024-02-10T07:23:59.819 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:23:59.825 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:23:59.825 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:23:59.831 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:23:59.831 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:23:59.909 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:24:00.400 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T07:24:00.484 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T07:24:00.663 INFO:teuthology.orchestra.run.smithi008.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 272 kB/s | 272 kB 00:00 2024-02-10T07:24:00.697 INFO:teuthology.orchestra.run.smithi008.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 16 kB/s | 16 kB 00:01 2024-02-10T07:24:00.722 INFO:teuthology.orchestra.run.smithi008.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 87 kB/s | 90 kB 00:01 2024-02-10T07:24:00.739 INFO:teuthology.orchestra.run.smithi008.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 476 kB/s | 36 kB 00:00 2024-02-10T07:24:00.831 INFO:teuthology.orchestra.run.smithi008.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-02-10T07:24:00.873 INFO:teuthology.orchestra.run.smithi008.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 149 kB/s | 20 kB 00:00 2024-02-10T07:24:00.893 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T07:24:00.906 INFO:teuthology.orchestra.run.smithi008.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2024-02-10T07:24:00.912 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:00.912 INFO:teuthology.orchestra.run.smithi008.stdout:Total 8.2 MB/s | 30 MB 00:03 2024-02-10T07:24:00.930 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:24:00.977 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:24:00.977 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:24:01.241 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T07:24:01.242 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:01.242 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:24:01.242 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:01.242 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:01.242 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:01.420 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:24:01.420 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:24:01.467 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T07:24:02.056 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:00 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:02.219 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:02.220 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:02.220 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-02-10T07:24:02.220 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:02.220 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:24:02.220 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1361.g290212a1.el9 ceph 145 k 2024-02-10T07:24:02.220 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:02.220 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:24:02.220 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:02.220 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-02-10T07:24:02.220 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:02.221 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 145 k 2024-02-10T07:24:02.221 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 436 k 2024-02-10T07:24:02.221 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:24:02.836 INFO:teuthology.orchestra.run.smithi170.stdout:ceph-immutable-object-cache-19.0.0-1361.g290212 237 kB/s | 145 kB 00:00 2024-02-10T07:24:02.837 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:02.837 INFO:teuthology.orchestra.run.smithi170.stdout:Total 236 kB/s | 145 kB 00:00 2024-02-10T07:24:02.837 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:24:02.845 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:24:02.845 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:24:02.897 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:24:02.897 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:24:02.898 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:24:02.950 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-10T07:24:02.991 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-10T07:24:03.020 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-10T07:24:03.040 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:24:03.092 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-10T07:24:03.103 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T07:24:03.128 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T07:24:03.128 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:24:03.128 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-10T07:24:03.128 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:03.260 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-10T07:24:03.293 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-10T07:24:03.389 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-10T07:24:03.474 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-10T07:24:03.521 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-10T07:24:03.595 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-10T07:24:03.664 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-10T07:24:03.702 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-10T07:24:03.769 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-10T07:24:03.832 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-10T07:24:04.031 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-10T07:24:04.081 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-10T07:24:04.120 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-10T07:24:04.146 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T07:24:04.146 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:04.147 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:24:04.147 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:24:04.147 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:04.147 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:04.154 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-10T07:24:04.184 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-10T07:24:04.264 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-10T07:24:04.378 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr 2024-02-10T07:24:04.420 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-10T07:24:04.453 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-10T07:24:04.493 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-10T07:24:04.529 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-10T07:24:04.563 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-10T07:24:04.647 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-10T07:24:04.694 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-10T07:24:04.770 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-10T07:24:04.920 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-10T07:24:04.975 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:02 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:05.049 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-10T07:24:05.082 INFO:teuthology.orchestra.run.smithi170.stdout:Package ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:24:05.134 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:05.135 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-02-10T07:24:05.135 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:05.195 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T07:24:05.223 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9. 31/36 2024-02-10T07:24:05.248 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 32/36 2024-02-10T07:24:05.273 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 32/36 2024-02-10T07:24:05.273 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:24:05.273 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-10T07:24:05.273 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T07:24:05.273 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-10T07:24:05.273 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:05.771 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:03 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:05.929 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:05.930 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================== 2024-02-10T07:24:05.930 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-02-10T07:24:05.930 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================== 2024-02-10T07:24:05.930 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:24:05.930 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 1.8 M 2024-02-10T07:24:05.930 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-02-10T07:24:05.930 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-10T07:24:05.930 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 28 k 2024-02-10T07:24:05.930 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 15 k 2024-02-10T07:24:05.931 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-10T07:24:05.931 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-10T07:24:05.931 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-10T07:24:05.931 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-10T07:24:05.931 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-10T07:24:05.931 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-10T07:24:05.931 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-10T07:24:05.931 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-10T07:24:05.931 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:05.931 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:24:05.931 INFO:teuthology.orchestra.run.smithi170.stdout:====================================================================================== 2024-02-10T07:24:05.931 INFO:teuthology.orchestra.run.smithi170.stdout:Install 12 Packages 2024-02-10T07:24:05.931 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:05.932 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 6.8 M 2024-02-10T07:24:05.932 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 26 M 2024-02-10T07:24:05.932 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:24:06.232 INFO:teuthology.orchestra.run.smithi170.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1361.g29 132 kB/s | 28 kB 00:00 2024-02-10T07:24:06.257 INFO:teuthology.orchestra.run.smithi170.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1361.g290 64 kB/s | 15 kB 00:00 2024-02-10T07:24:06.399 INFO:teuthology.orchestra.run.smithi170.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-02-10T07:24:06.433 INFO:teuthology.orchestra.run.smithi170.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-02-10T07:24:06.708 INFO:teuthology.orchestra.run.smithi170.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.9 MB/s | 548 kB 00:00 2024-02-10T07:24:06.742 INFO:teuthology.orchestra.run.smithi170.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 581 kB/s | 19 kB 00:00 2024-02-10T07:24:06.798 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 33/36 2024-02-10T07:24:06.809 INFO:teuthology.orchestra.run.smithi170.stdout:(7/12): ceph-mgr-dashboard-19.0.0-1361.g290212a 2.2 MB/s | 1.8 MB 00:00 2024-02-10T07:24:06.825 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 33/36 2024-02-10T07:24:06.825 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:24:06.825 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-10T07:24:06.825 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T07:24:06.825 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-10T07:24:06.826 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:07.101 INFO:teuthology.orchestra.run.smithi170.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 495 kB/s | 144 kB 00:00 2024-02-10T07:24:07.151 INFO:teuthology.orchestra.run.smithi170.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 615 kB/s | 31 kB 00:00 2024-02-10T07:24:07.276 INFO:teuthology.orchestra.run.smithi170.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-02-10T07:24:07.291 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 34/36 2024-02-10T07:24:07.314 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 34/36 2024-02-10T07:24:07.314 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:24:07.314 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-10T07:24:07.314 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T07:24:07.314 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-10T07:24:07.314 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:07.533 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 35/36 2024-02-10T07:24:07.555 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 35/36 2024-02-10T07:24:07.556 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:24:07.556 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-10T07:24:07.556 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T07:24:07.556 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-10T07:24:07.556 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:07.624 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 36/36 2024-02-10T07:24:07.701 INFO:teuthology.orchestra.run.smithi170.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.1 MB/s | 2.0 MB 00:00 2024-02-10T07:24:08.327 INFO:teuthology.orchestra.run.smithi170.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 448 kB/s | 863 kB 00:01 2024-02-10T07:24:08.333 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:08.333 INFO:teuthology.orchestra.run.smithi170.stdout:Total 2.8 MB/s | 6.8 MB 00:02 2024-02-10T07:24:08.352 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:24:08.371 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:24:08.371 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:24:08.560 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:24:08.560 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:24:09.081 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:24:09.152 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-10T07:24:09.187 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-10T07:24:09.233 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-10T07:24:09.264 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-10T07:24:09.325 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 36/36 2024-02-10T07:24:09.325 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 1/36 2024-02-10T07:24:09.325 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2/36 2024-02-10T07:24:09.325 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 3/36 2024-02-10T07:24:09.326 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 4/36 2024-02-10T07:24:09.326 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 5/36 2024-02-10T07:24:09.326 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9. 6/36 2024-02-10T07:24:09.326 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-10T07:24:09.326 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-10T07:24:09.326 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-10T07:24:09.326 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-10T07:24:09.326 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-10T07:24:09.326 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-10T07:24:09.326 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-10T07:24:09.326 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-10T07:24:09.326 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-10T07:24:09.326 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-10T07:24:09.327 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-10T07:24:09.328 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-10T07:24:09.328 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-10T07:24:09.328 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-10T07:24:09.328 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-10T07:24:09.362 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-10T07:24:09.488 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-10T07:24:09.584 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-10T07:24:09.628 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-10T07:24:09.656 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-10T07:24:09.686 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9 10/12 2024-02-10T07:24:09.926 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el 11/12 2024-02-10T07:24:09.963 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 12/12 2024-02-10T07:24:10.377 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-10T07:24:10.377 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:10.377 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:24:10.377 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:24:10.378 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:24:10.378 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:24:10.378 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:10.378 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:24:10.378 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:24:10.378 INFO:teuthology.orchestra.run.smithi008.stdout: lua-5.4.4-4.el9.x86_64 2024-02-10T07:24:10.378 INFO:teuthology.orchestra.run.smithi008.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-10T07:24:10.378 INFO:teuthology.orchestra.run.smithi008.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-10T07:24:10.378 INFO:teuthology.orchestra.run.smithi008.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-10T07:24:10.378 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-10T07:24:10.378 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-10T07:24:10.378 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ply-3.11-14.el9.noarch 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-10T07:24:10.379 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-10T07:24:10.380 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-10T07:24:10.380 INFO:teuthology.orchestra.run.smithi008.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-10T07:24:10.380 INFO:teuthology.orchestra.run.smithi008.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-10T07:24:10.380 INFO:teuthology.orchestra.run.smithi008.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-10T07:24:10.380 INFO:teuthology.orchestra.run.smithi008.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-10T07:24:10.380 INFO:teuthology.orchestra.run.smithi008.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-10T07:24:10.380 INFO:teuthology.orchestra.run.smithi008.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-10T07:24:10.380 INFO:teuthology.orchestra.run.smithi008.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-10T07:24:10.380 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:10.380 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:24:10.706 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-base 2024-02-10T07:24:10.803 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 12/12 2024-02-10T07:24:10.803 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el 1/12 2024-02-10T07:24:10.803 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 2/12 2024-02-10T07:24:10.803 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9 3/12 2024-02-10T07:24:10.803 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-10T07:24:10.803 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-10T07:24:10.804 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-10T07:24:10.804 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-10T07:24:10.804 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-10T07:24:10.804 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-10T07:24:10.804 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-10T07:24:10.804 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-10T07:24:11.179 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-10T07:24:11.179 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:11.180 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:24:11.180 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-10T07:24:11.180 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:11.180 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:11.180 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:11.180 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-10T07:24:11.180 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-10T07:24:11.180 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-10T07:24:11.180 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-10T07:24:11.180 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-10T07:24:11.181 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-10T07:24:11.181 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-10T07:24:11.181 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-10T07:24:11.181 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:11.181 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:11.299 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:00:56 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:24:11.410 INFO:teuthology.orchestra.run.smithi008.stdout:Package ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:24:11.424 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T07:24:11.464 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:24:11.466 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T07:24:11.466 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:24:11.529 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install cephadm 2024-02-10T07:24:12.006 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:10 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:12.116 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:00:57 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:24:12.165 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:12.166 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-02-10T07:24:12.166 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-02-10T07:24:12.166 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-02-10T07:24:12.167 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:24:12.167 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 7.4 M 2024-02-10T07:24:12.167 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-02-10T07:24:12.167 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-10T07:24:12.167 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-10T07:24:12.167 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-10T07:24:12.167 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-10T07:24:12.167 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-10T07:24:12.167 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-10T07:24:12.167 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-10T07:24:12.167 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-10T07:24:12.168 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-10T07:24:12.168 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-10T07:24:12.168 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-10T07:24:12.168 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:12.168 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:24:12.168 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-02-10T07:24:12.168 INFO:teuthology.orchestra.run.smithi170.stdout:Install 12 Packages 2024-02-10T07:24:12.168 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:12.169 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 37 M 2024-02-10T07:24:12.169 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 209 M 2024-02-10T07:24:12.169 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:24:12.281 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:24:12.282 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:24:12.282 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T07:24:12.283 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:24:12.283 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:24:12.283 INFO:teuthology.orchestra.run.smithi008.stdout: cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 748 k 2024-02-10T07:24:12.283 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:12.283 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:24:12.283 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:24:12.283 INFO:teuthology.orchestra.run.smithi008.stdout:Install 1 Package 2024-02-10T07:24:12.283 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:12.283 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 748 k 2024-02-10T07:24:12.283 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 754 k 2024-02-10T07:24:12.283 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:24:12.567 INFO:teuthology.orchestra.run.smithi170.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 250 kB/s | 32 kB 00:00 2024-02-10T07:24:12.600 INFO:teuthology.orchestra.run.smithi170.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 534 kB/s | 17 kB 00:00 2024-02-10T07:24:12.606 INFO:teuthology.orchestra.run.smithi008.stdout:cephadm-19.0.0-1361.g290212a1.el9.noarch.rpm 2.3 MB/s | 748 kB 00:00 2024-02-10T07:24:12.607 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:12.607 INFO:teuthology.orchestra.run.smithi008.stdout:Total 2.2 MB/s | 748 kB 00:00 2024-02-10T07:24:12.607 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:24:12.613 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:24:12.613 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:24:12.619 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:24:12.619 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:24:12.676 INFO:teuthology.orchestra.run.smithi170.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-02-10T07:24:12.697 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:24:12.701 INFO:teuthology.orchestra.run.smithi170.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 350 kB/s | 34 kB 00:00 2024-02-10T07:24:12.877 INFO:teuthology.orchestra.run.smithi170.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2024-02-10T07:24:12.943 INFO:teuthology.orchestra.run.smithi170.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-02-10T07:24:12.968 INFO:teuthology.orchestra.run.smithi170.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-02-10T07:24:13.144 INFO:teuthology.orchestra.run.smithi170.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 802 kB 00:00 2024-02-10T07:24:13.207 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T07:24:13.289 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T07:24:13.294 INFO:teuthology.orchestra.run.smithi170.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-02-10T07:24:13.319 INFO:teuthology.orchestra.run.smithi170.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 194 kB 00:00 2024-02-10T07:24:13.528 INFO:teuthology.orchestra.run.smithi170.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-02-10T07:24:13.670 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T07:24:13.736 INFO:teuthology.orchestra.run.smithi170.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 5.7 MB/s | 7.4 MB 00:01 2024-02-10T07:24:13.739 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:13.740 INFO:teuthology.orchestra.run.smithi170.stdout:Total 24 MB/s | 37 MB 00:01 2024-02-10T07:24:13.752 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:24:13.776 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:24:13.777 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:24:14.134 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/1 2024-02-10T07:24:14.134 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:14.134 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:24:14.135 INFO:teuthology.orchestra.run.smithi008.stdout: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:14.135 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:14.135 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:24:14.142 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:24:14.142 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:24:14.339 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T07:24:14.849 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:24:14.926 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-10T07:24:14.936 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:00:59 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:24:14.958 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-10T07:24:15.051 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-10T07:24:15.086 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-10T07:24:15.100 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:24:15.101 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:24:15.101 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repo Size 2024-02-10T07:24:15.101 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:24:15.101 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:24:15.101 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1361.g290212a1.el9 ceph 145 k 2024-02-10T07:24:15.101 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:15.101 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:24:15.101 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:24:15.101 INFO:teuthology.orchestra.run.smithi008.stdout:Install 1 Package 2024-02-10T07:24:15.101 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:15.102 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 145 k 2024-02-10T07:24:15.102 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 436 k 2024-02-10T07:24:15.102 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:24:15.263 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T07:24:15.293 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T07:24:15.340 INFO:teuthology.orchestra.run.smithi008.stdout:ceph-immutable-object-cache-19.0.0-1361.g290212 612 kB/s | 145 kB 00:00 2024-02-10T07:24:15.341 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:15.341 INFO:teuthology.orchestra.run.smithi008.stdout:Total 608 kB/s | 145 kB 00:00 2024-02-10T07:24:15.341 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:24:15.349 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:24:15.349 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:24:15.369 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-10T07:24:15.403 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:24:15.403 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:24:15.546 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:24:15.616 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T07:24:15.640 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T07:24:15.640 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:24:15.640 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-10T07:24:15.640 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-10T07:24:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-10T07:24:16.749 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1361.g290212a1. 1/1 2024-02-10T07:24:16.749 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:16.749 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:24:16.749 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:24:16.749 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:16.749 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:24:16.976 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-mgr 2024-02-10T07:24:17.372 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-10T07:24:17.572 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:02 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:24:17.682 INFO:teuthology.orchestra.run.smithi008.stdout:Package ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:24:17.737 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:24:17.738 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T07:24:17.738 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:24:17.797 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T07:24:18.395 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:03 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:24:18.401 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-10T07:24:18.426 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 12/12 2024-02-10T07:24:18.562 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:24:18.563 INFO:teuthology.orchestra.run.smithi008.stdout:====================================================================================== 2024-02-10T07:24:18.564 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T07:24:18.564 INFO:teuthology.orchestra.run.smithi008.stdout:====================================================================================== 2024-02-10T07:24:18.564 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:24:18.564 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 1.8 M 2024-02-10T07:24:18.564 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T07:24:18.564 INFO:teuthology.orchestra.run.smithi008.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-10T07:24:18.564 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 28 k 2024-02-10T07:24:18.564 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 15 k 2024-02-10T07:24:18.564 INFO:teuthology.orchestra.run.smithi008.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-10T07:24:18.564 INFO:teuthology.orchestra.run.smithi008.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-10T07:24:18.565 INFO:teuthology.orchestra.run.smithi008.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-10T07:24:18.565 INFO:teuthology.orchestra.run.smithi008.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-10T07:24:18.565 INFO:teuthology.orchestra.run.smithi008.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-10T07:24:18.565 INFO:teuthology.orchestra.run.smithi008.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-10T07:24:18.565 INFO:teuthology.orchestra.run.smithi008.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-10T07:24:18.565 INFO:teuthology.orchestra.run.smithi008.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-10T07:24:18.565 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:18.565 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:24:18.565 INFO:teuthology.orchestra.run.smithi008.stdout:====================================================================================== 2024-02-10T07:24:18.565 INFO:teuthology.orchestra.run.smithi008.stdout:Install 12 Packages 2024-02-10T07:24:18.565 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:18.566 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 6.8 M 2024-02-10T07:24:18.566 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 26 M 2024-02-10T07:24:18.566 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:24:19.202 INFO:teuthology.orchestra.run.smithi008.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1361.g290 75 kB/s | 15 kB 00:00 2024-02-10T07:24:19.228 INFO:teuthology.orchestra.run.smithi008.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1361.g29 123 kB/s | 28 kB 00:00 2024-02-10T07:24:19.370 INFO:teuthology.orchestra.run.smithi008.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-02-10T07:24:19.420 INFO:teuthology.orchestra.run.smithi008.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-02-10T07:24:19.570 INFO:teuthology.orchestra.run.smithi008.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1361.g290212a 3.1 MB/s | 1.8 MB 00:00 2024-02-10T07:24:19.620 INFO:teuthology.orchestra.run.smithi008.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.4 MB/s | 863 kB 00:00 2024-02-10T07:24:19.937 INFO:teuthology.orchestra.run.smithi008.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 53 kB/s | 19 kB 00:00 2024-02-10T07:24:21.183 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 12/12 2024-02-10T07:24:21.184 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 1/12 2024-02-10T07:24:21.184 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-10T07:24:21.184 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-10T07:24:21.184 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-10T07:24:21.184 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T07:24:21.184 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T07:24:21.184 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-10T07:24:21.184 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-10T07:24:21.184 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-10T07:24:21.185 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-10T07:24:21.185 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-10T07:24:21.640 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-10T07:24:21.640 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:21.640 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:24:21.640 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:21.640 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-10T07:24:21.640 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-10T07:24:21.640 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-10T07:24:21.640 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-10T07:24:21.640 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-10T07:24:21.640 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-10T07:24:21.641 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-10T07:24:21.641 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-10T07:24:21.641 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-10T07:24:21.641 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-10T07:24:21.641 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-10T07:24:21.641 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:21.641 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:22.018 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-rook 2024-02-10T07:24:22.614 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:20 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:22.776 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:22.777 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-02-10T07:24:22.777 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-02-10T07:24:22.777 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-02-10T07:24:22.777 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:24:22.777 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 52 k 2024-02-10T07:24:22.777 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-02-10T07:24:22.777 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-10T07:24:22.777 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-10T07:24:22.778 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-10T07:24:22.778 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-10T07:24:22.778 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-10T07:24:22.778 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-10T07:24:22.778 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-10T07:24:22.778 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-10T07:24:22.778 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-10T07:24:22.778 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-10T07:24:22.778 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-10T07:24:22.778 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-10T07:24:22.778 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:22.778 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:24:22.778 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-02-10T07:24:22.779 INFO:teuthology.orchestra.run.smithi170.stdout:Install 13 Packages 2024-02-10T07:24:22.779 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:22.779 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 2.0 M 2024-02-10T07:24:22.779 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 26 M 2024-02-10T07:24:22.779 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:24:22.888 INFO:teuthology.orchestra.run.smithi008.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 158 kB/s | 548 kB 00:03 2024-02-10T07:24:22.954 INFO:teuthology.orchestra.run.smithi170.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 186 kB/s | 24 kB 00:00 2024-02-10T07:24:22.979 INFO:teuthology.orchestra.run.smithi170.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 115 kB/s | 18 kB 00:00 2024-02-10T07:24:23.021 INFO:teuthology.orchestra.run.smithi170.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-02-10T07:24:23.046 INFO:teuthology.orchestra.run.smithi170.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-02-10T07:24:23.071 INFO:teuthology.orchestra.run.smithi170.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-10T07:24:23.088 INFO:teuthology.orchestra.run.smithi170.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-02-10T07:24:23.106 INFO:teuthology.orchestra.run.smithi170.stdout:(7/13): ceph-mgr-rook-19.0.0-1361.g290212a1.el9 184 kB/s | 52 kB 00:00 2024-02-10T07:24:23.230 INFO:teuthology.orchestra.run.smithi008.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 44 kB/s | 144 kB 00:03 2024-02-10T07:24:23.373 INFO:teuthology.orchestra.run.smithi170.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-02-10T07:24:23.389 INFO:teuthology.orchestra.run.smithi170.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2024-02-10T07:24:23.673 INFO:teuthology.orchestra.run.smithi170.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 208 kB/s | 59 kB 00:00 2024-02-10T07:24:23.874 INFO:teuthology.orchestra.run.smithi170.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 264 kB/s | 203 kB 00:00 2024-02-10T07:24:24.040 INFO:teuthology.orchestra.run.smithi170.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 244 kB/s | 90 kB 00:00 2024-02-10T07:24:24.530 INFO:teuthology.orchestra.run.smithi008.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 19 kB/s | 31 kB 00:01 2024-02-10T07:24:25.499 INFO:teuthology.orchestra.run.smithi170.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 492 kB/s | 1.0 MB 00:02 2024-02-10T07:24:25.501 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:25.501 INFO:teuthology.orchestra.run.smithi170.stdout:Total 771 kB/s | 2.0 MB 00:02 2024-02-10T07:24:25.502 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:24:25.518 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:24:25.518 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:24:25.624 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:24:25.625 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:24:26.121 INFO:teuthology.orchestra.run.smithi008.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 65 kB/s | 188 kB 00:02 2024-02-10T07:24:26.151 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:24:26.311 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-10T07:24:26.367 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-10T07:24:26.422 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-10T07:24:26.459 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-10T07:24:26.500 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-10T07:24:26.595 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-10T07:24:26.704 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-10T07:24:26.751 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-10T07:24:27.470 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-10T07:24:27.516 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-10T07:24:27.549 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-10T07:24:27.598 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-10T07:24:27.619 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 13/13 2024-02-10T07:24:28.088 INFO:teuthology.orchestra.run.smithi008.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 247 kB/s | 2.0 MB 00:08 2024-02-10T07:24:28.094 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:28.094 INFO:teuthology.orchestra.run.smithi008.stdout:Total 734 kB/s | 6.8 MB 00:09 2024-02-10T07:24:28.114 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:24:28.134 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:24:28.134 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:24:28.314 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:24:28.314 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:24:28.490 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 13/13 2024-02-10T07:24:28.490 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 1/13 2024-02-10T07:24:28.490 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-10T07:24:28.490 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-10T07:24:28.490 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-10T07:24:28.490 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-10T07:24:28.490 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-10T07:24:28.490 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-10T07:24:28.490 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-10T07:24:28.491 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-10T07:24:28.491 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-10T07:24:28.491 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-10T07:24:28.491 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-10T07:24:28.866 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:24:28.888 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-10T07:24:28.888 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:28.888 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:24:28.888 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:28.888 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-10T07:24:28.888 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-10T07:24:28.888 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-10T07:24:28.888 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-10T07:24:28.889 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-10T07:24:28.889 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-10T07:24:28.889 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-10T07:24:28.889 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-10T07:24:28.889 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-10T07:24:28.889 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-10T07:24:28.889 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-10T07:24:28.889 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-10T07:24:28.889 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:28.889 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:28.964 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-10T07:24:29.007 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-10T07:24:29.055 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-10T07:24:29.092 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-10T07:24:29.141 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T07:24:29.185 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-10T07:24:29.322 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-10T07:24:29.415 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-10T07:24:29.465 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-10T07:24:29.492 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-10T07:24:29.522 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9 10/12 2024-02-10T07:24:29.723 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:27 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:29.776 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el 11/12 2024-02-10T07:24:29.807 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 12/12 2024-02-10T07:24:29.885 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:29.886 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-02-10T07:24:29.886 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-02-10T07:24:29.886 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-02-10T07:24:29.886 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:24:29.886 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 138 k 2024-02-10T07:24:29.886 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-02-10T07:24:29.886 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-10T07:24:29.886 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-10T07:24:29.886 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-10T07:24:29.887 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-10T07:24:29.887 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-10T07:24:29.887 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:29.887 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:24:29.887 INFO:teuthology.orchestra.run.smithi170.stdout:======================================================================================== 2024-02-10T07:24:29.887 INFO:teuthology.orchestra.run.smithi170.stdout:Install 6 Packages 2024-02-10T07:24:29.887 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:29.887 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 6.8 M 2024-02-10T07:24:29.887 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 33 M 2024-02-10T07:24:29.887 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:24:30.092 INFO:teuthology.orchestra.run.smithi170.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 227 kB 00:00 2024-02-10T07:24:30.251 INFO:teuthology.orchestra.run.smithi170.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-02-10T07:24:30.385 INFO:teuthology.orchestra.run.smithi170.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1361.g290212a1.e 303 kB/s | 138 kB 00:00 2024-02-10T07:24:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 12/12 2024-02-10T07:24:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el 1/12 2024-02-10T07:24:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noa 2/12 2024-02-10T07:24:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9 3/12 2024-02-10T07:24:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-10T07:24:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-10T07:24:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-10T07:24:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-10T07:24:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-10T07:24:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-10T07:24:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-10T07:24:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-10T07:24:30.710 INFO:teuthology.orchestra.run.smithi170.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 126 kB/s | 58 kB 00:00 2024-02-10T07:24:30.785 INFO:teuthology.orchestra.run.smithi170.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 127 kB/s | 51 kB 00:00 2024-02-10T07:24:30.919 INFO:teuthology.orchestra.run.smithi170.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 664 kB/s | 548 kB 00:00 2024-02-10T07:24:30.921 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:30.921 INFO:teuthology.orchestra.run.smithi170.stdout:Total 6.5 MB/s | 6.8 MB 00:01 2024-02-10T07:24:30.921 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:24:30.930 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:24:30.930 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:24:30.998 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-10T07:24:30.998 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:30.998 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:24:30.998 INFO:teuthology.orchestra.run.smithi008.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-10T07:24:30.998 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:30.998 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:30.998 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:30.998 INFO:teuthology.orchestra.run.smithi008.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-10T07:24:30.999 INFO:teuthology.orchestra.run.smithi008.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-10T07:24:30.999 INFO:teuthology.orchestra.run.smithi008.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-10T07:24:30.999 INFO:teuthology.orchestra.run.smithi008.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-10T07:24:30.999 INFO:teuthology.orchestra.run.smithi008.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-10T07:24:30.999 INFO:teuthology.orchestra.run.smithi008.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-10T07:24:30.999 INFO:teuthology.orchestra.run.smithi008.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-10T07:24:30.999 INFO:teuthology.orchestra.run.smithi008.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-10T07:24:30.999 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:30.999 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:24:31.020 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:24:31.020 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:24:31.257 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T07:24:31.362 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:24:31.590 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-10T07:24:31.645 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-10T07:24:31.850 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:16 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:24:32.019 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:24:32.021 INFO:teuthology.orchestra.run.smithi008.stdout:============================================================================================ 2024-02-10T07:24:32.021 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T07:24:32.021 INFO:teuthology.orchestra.run.smithi008.stdout:============================================================================================ 2024-02-10T07:24:32.021 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:24:32.021 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 7.4 M 2024-02-10T07:24:32.021 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T07:24:32.021 INFO:teuthology.orchestra.run.smithi008.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-10T07:24:32.021 INFO:teuthology.orchestra.run.smithi008.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-10T07:24:32.021 INFO:teuthology.orchestra.run.smithi008.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-10T07:24:32.021 INFO:teuthology.orchestra.run.smithi008.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-10T07:24:32.021 INFO:teuthology.orchestra.run.smithi008.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-10T07:24:32.021 INFO:teuthology.orchestra.run.smithi008.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-10T07:24:32.022 INFO:teuthology.orchestra.run.smithi008.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-10T07:24:32.022 INFO:teuthology.orchestra.run.smithi008.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-10T07:24:32.022 INFO:teuthology.orchestra.run.smithi008.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-10T07:24:32.022 INFO:teuthology.orchestra.run.smithi008.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-10T07:24:32.022 INFO:teuthology.orchestra.run.smithi008.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-10T07:24:32.022 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:32.022 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:24:32.022 INFO:teuthology.orchestra.run.smithi008.stdout:============================================================================================ 2024-02-10T07:24:32.022 INFO:teuthology.orchestra.run.smithi008.stdout:Install 12 Packages 2024-02-10T07:24:32.022 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:32.023 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 37 M 2024-02-10T07:24:32.023 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 209 M 2024-02-10T07:24:32.023 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:24:32.164 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-10T07:24:32.247 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-10T07:24:32.317 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-10T07:24:32.350 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 6/6 2024-02-10T07:24:32.429 INFO:teuthology.orchestra.run.smithi008.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 234 kB/s | 32 kB 00:00 2024-02-10T07:24:32.471 INFO:teuthology.orchestra.run.smithi008.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 431 kB/s | 17 kB 00:00 2024-02-10T07:24:32.529 INFO:teuthology.orchestra.run.smithi008.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-02-10T07:24:32.555 INFO:teuthology.orchestra.run.smithi008.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 418 kB/s | 34 kB 00:00 2024-02-10T07:24:32.847 INFO:teuthology.orchestra.run.smithi008.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-02-10T07:24:32.914 INFO:teuthology.orchestra.run.smithi008.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 4.8 MB 00:00 2024-02-10T07:24:32.939 INFO:teuthology.orchestra.run.smithi008.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-02-10T07:24:33.123 INFO:teuthology.orchestra.run.smithi008.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 802 kB 00:00 2024-02-10T07:24:33.191 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 6/6 2024-02-10T07:24:33.192 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/6 2024-02-10T07:24:33.192 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-10T07:24:33.192 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-10T07:24:33.192 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-10T07:24:33.192 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-10T07:24:33.315 INFO:teuthology.orchestra.run.smithi008.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-02-10T07:24:33.340 INFO:teuthology.orchestra.run.smithi008.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 904 kB/s | 194 kB 00:00 2024-02-10T07:24:33.527 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-10T07:24:33.527 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:33.527 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:24:33.527 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:33.528 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-10T07:24:33.528 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-10T07:24:33.528 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-10T07:24:33.528 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-10T07:24:33.528 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-10T07:24:33.528 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:33.528 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:33.632 INFO:teuthology.orchestra.run.smithi008.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 776 kB/s | 245 kB 00:00 2024-02-10T07:24:33.785 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-fuse 2024-02-10T07:24:34.365 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:32 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:34.529 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:34.530 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:34.530 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-02-10T07:24:34.530 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:34.530 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:24:34.530 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 885 k 2024-02-10T07:24:34.530 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-02-10T07:24:34.530 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-10T07:24:34.530 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:34.530 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:24:34.530 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:34.531 INFO:teuthology.orchestra.run.smithi170.stdout:Install 2 Packages 2024-02-10T07:24:34.531 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:34.531 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 964 k 2024-02-10T07:24:34.531 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 2.8 M 2024-02-10T07:24:34.531 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:24:34.671 INFO:teuthology.orchestra.run.smithi170.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 564 kB/s | 79 kB 00:00 2024-02-10T07:24:35.488 INFO:teuthology.orchestra.run.smithi170.stdout:(2/2): ceph-fuse-19.0.0-1361.g290212a1.el9.x86_ 926 kB/s | 885 kB 00:00 2024-02-10T07:24:35.489 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:35.490 INFO:teuthology.orchestra.run.smithi170.stdout:Total 1.0 MB/s | 964 kB 00:00 2024-02-10T07:24:35.490 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:24:35.511 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:24:35.512 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:24:35.589 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:24:35.590 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:24:35.787 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:24:35.832 INFO:teuthology.orchestra.run.smithi008.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.1 MB/s | 7.4 MB 00:03 2024-02-10T07:24:35.835 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:35.835 INFO:teuthology.orchestra.run.smithi008.stdout:Total 9.7 MB/s | 37 MB 00:03 2024-02-10T07:24:35.849 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:24:35.877 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:24:35.877 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:24:35.930 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-10T07:24:36.004 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2/2 2024-02-10T07:24:36.244 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:24:36.244 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:24:36.731 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2/2 2024-02-10T07:24:36.731 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/2 2024-02-10T07:24:36.973 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:24:37.047 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-10T07:24:37.077 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-10T07:24:37.091 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-10T07:24:37.092 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:37.092 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:24:37.092 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-10T07:24:37.092 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:37.092 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:37.175 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-10T07:24:37.214 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-10T07:24:37.372 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-volume 2024-02-10T07:24:37.398 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T07:24:37.430 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T07:24:37.508 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-10T07:24:37.975 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:35 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:38.133 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:38.133 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:38.134 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-02-10T07:24:38.134 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:38.134 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:24:38.134 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 264 k 2024-02-10T07:24:38.134 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-02-10T07:24:38.134 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-10T07:24:38.134 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:38.134 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:24:38.134 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:38.134 INFO:teuthology.orchestra.run.smithi170.stdout:Install 2 Packages 2024-02-10T07:24:38.134 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:38.135 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 332 k 2024-02-10T07:24:38.135 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 1.5 M 2024-02-10T07:24:38.135 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:24:38.216 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-10T07:24:38.259 INFO:teuthology.orchestra.run.smithi170.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 554 kB/s | 69 kB 00:00 2024-02-10T07:24:38.347 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-10T07:24:38.651 INFO:teuthology.orchestra.run.smithi170.stdout:(2/2): ceph-volume-19.0.0-1361.g290212a1.el9.no 511 kB/s | 264 kB 00:00 2024-02-10T07:24:38.652 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:38.653 INFO:teuthology.orchestra.run.smithi170.stdout:Total 641 kB/s | 332 kB 00:00 2024-02-10T07:24:38.653 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:24:38.661 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:24:38.661 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:24:38.703 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:24:38.703 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:24:38.860 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:24:39.056 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-10T07:24:39.096 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2/2 2024-02-10T07:24:39.111 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2/2 2024-02-10T07:24:39.111 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:24:39.111 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-10T07:24:39.111 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:39.651 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-10T07:24:39.863 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 1/2 2024-02-10T07:24:40.251 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-10T07:24:40.251 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:40.252 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:24:40.252 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:40.252 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-10T07:24:40.252 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:40.252 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:40.581 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados-devel 2024-02-10T07:24:40.735 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-10T07:24:40.763 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 12/12 2024-02-10T07:24:41.179 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:39 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:41.340 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:41.341 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:41.341 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-02-10T07:24:41.341 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:41.341 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:24:41.341 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 127 k 2024-02-10T07:24:41.341 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:41.341 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:24:41.341 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:41.341 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-02-10T07:24:41.342 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:41.342 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 127 k 2024-02-10T07:24:41.342 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 456 k 2024-02-10T07:24:41.342 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:24:41.566 INFO:teuthology.orchestra.run.smithi170.stdout:librados-devel-19.0.0-1361.g290212a1.el9.x86_64 566 kB/s | 127 kB 00:00 2024-02-10T07:24:41.566 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:41.567 INFO:teuthology.orchestra.run.smithi170.stdout:Total 562 kB/s | 127 kB 00:00 2024-02-10T07:24:41.567 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:24:41.572 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:24:41.572 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:24:41.627 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:24:41.627 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:24:41.775 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:24:41.890 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:42.393 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:42.726 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:42.726 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:42.726 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:24:42.726 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:24:42.726 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:42.726 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:42.970 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs2 2024-02-10T07:24:43.355 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 12/12 2024-02-10T07:24:43.355 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1361.g29021 1/12 2024-02-10T07:24:43.355 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-10T07:24:43.355 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-10T07:24:43.355 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-10T07:24:43.355 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-10T07:24:43.355 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-10T07:24:43.355 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-10T07:24:43.355 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-10T07:24:43.355 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-10T07:24:43.355 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-10T07:24:43.356 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-10T07:24:43.575 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:41 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:43.685 INFO:teuthology.orchestra.run.smithi170.stdout:Package libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:24:43.738 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:43.739 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-02-10T07:24:43.739 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:43.792 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-10T07:24:43.792 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:43.792 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:24:43.792 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:43.792 INFO:teuthology.orchestra.run.smithi008.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-10T07:24:43.792 INFO:teuthology.orchestra.run.smithi008.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-10T07:24:43.792 INFO:teuthology.orchestra.run.smithi008.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-10T07:24:43.792 INFO:teuthology.orchestra.run.smithi008.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-10T07:24:43.792 INFO:teuthology.orchestra.run.smithi008.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-10T07:24:43.792 INFO:teuthology.orchestra.run.smithi008.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-10T07:24:43.792 INFO:teuthology.orchestra.run.smithi008.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-10T07:24:43.793 INFO:teuthology.orchestra.run.smithi008.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-10T07:24:43.793 INFO:teuthology.orchestra.run.smithi008.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-10T07:24:43.793 INFO:teuthology.orchestra.run.smithi008.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-10T07:24:43.793 INFO:teuthology.orchestra.run.smithi008.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-10T07:24:43.793 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:43.793 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:24:43.803 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install libcephfs-devel 2024-02-10T07:24:44.123 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-mgr-rook 2024-02-10T07:24:44.386 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:42 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:44.547 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:44.548 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:44.548 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-02-10T07:24:44.548 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:44.548 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:24:44.548 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 31 k 2024-02-10T07:24:44.548 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:44.548 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:24:44.549 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:44.549 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-02-10T07:24:44.549 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:44.549 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 31 k 2024-02-10T07:24:44.549 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 140 k 2024-02-10T07:24:44.549 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:24:44.718 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:29 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:24:44.755 INFO:teuthology.orchestra.run.smithi170.stdout:libcephfs-devel-19.0.0-1361.g290212a1.el9.x86_6 151 kB/s | 31 kB 00:00 2024-02-10T07:24:44.755 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:44.756 INFO:teuthology.orchestra.run.smithi170.stdout:Total 150 kB/s | 31 kB 00:00 2024-02-10T07:24:44.756 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:24:44.759 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:24:44.759 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:24:44.789 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:24:44.789 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:24:44.884 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:24:44.885 INFO:teuthology.orchestra.run.smithi008.stdout:======================================================================================== 2024-02-10T07:24:44.885 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T07:24:44.885 INFO:teuthology.orchestra.run.smithi008.stdout:======================================================================================== 2024-02-10T07:24:44.885 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:24:44.885 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-rook noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 52 k 2024-02-10T07:24:44.886 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T07:24:44.886 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-10T07:24:44.886 INFO:teuthology.orchestra.run.smithi008.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-10T07:24:44.886 INFO:teuthology.orchestra.run.smithi008.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-10T07:24:44.886 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-10T07:24:44.886 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-10T07:24:44.886 INFO:teuthology.orchestra.run.smithi008.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-10T07:24:44.886 INFO:teuthology.orchestra.run.smithi008.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-10T07:24:44.886 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-10T07:24:44.886 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-10T07:24:44.886 INFO:teuthology.orchestra.run.smithi008.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-10T07:24:44.886 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-10T07:24:44.886 INFO:teuthology.orchestra.run.smithi008.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-10T07:24:44.886 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:44.887 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:24:44.887 INFO:teuthology.orchestra.run.smithi008.stdout:======================================================================================== 2024-02-10T07:24:44.887 INFO:teuthology.orchestra.run.smithi008.stdout:Install 13 Packages 2024-02-10T07:24:44.887 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:44.887 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 2.0 M 2024-02-10T07:24:44.887 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 26 M 2024-02-10T07:24:44.887 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:24:44.914 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:24:45.012 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:45.205 INFO:teuthology.orchestra.run.smithi008.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 167 kB/s | 24 kB 00:00 2024-02-10T07:24:45.231 INFO:teuthology.orchestra.run.smithi008.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 106 kB/s | 18 kB 00:00 2024-02-10T07:24:45.314 INFO:teuthology.orchestra.run.smithi008.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 174 kB 00:00 2024-02-10T07:24:45.339 INFO:teuthology.orchestra.run.smithi008.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-02-10T07:24:45.364 INFO:teuthology.orchestra.run.smithi008.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-10T07:24:45.390 INFO:teuthology.orchestra.run.smithi008.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-02-10T07:24:45.400 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:45.415 INFO:teuthology.orchestra.run.smithi008.stdout:(7/13): ceph-mgr-rook-19.0.0-1361.g290212a1.el9 147 kB/s | 52 kB 00:00 2024-02-10T07:24:45.707 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:45.707 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:45.708 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:24:45.708 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:24:45.708 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:45.708 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:45.874 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librados2 2024-02-10T07:24:46.133 INFO:teuthology.orchestra.run.smithi008.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 20 kB/s | 15 kB 00:00 2024-02-10T07:24:46.299 INFO:teuthology.orchestra.run.smithi008.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 34 kB/s | 32 kB 00:00 2024-02-10T07:24:46.458 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:44 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:46.563 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:24:46.614 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:46.615 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-02-10T07:24:46.615 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:46.662 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install librbd1 2024-02-10T07:24:46.883 INFO:teuthology.orchestra.run.smithi008.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 138 kB/s | 203 kB 00:01 2024-02-10T07:24:46.958 INFO:teuthology.orchestra.run.smithi008.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 90 kB/s | 59 kB 00:00 2024-02-10T07:24:47.249 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:45 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:47.342 INFO:teuthology.orchestra.run.smithi008.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 865 kB/s | 1.0 MB 00:01 2024-02-10T07:24:47.355 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:24:47.367 INFO:teuthology.orchestra.run.smithi008.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 185 kB/s | 90 kB 00:00 2024-02-10T07:24:47.375 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:47.376 INFO:teuthology.orchestra.run.smithi008.stdout:Total 843 kB/s | 2.0 MB 00:02 2024-02-10T07:24:47.376 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:24:47.391 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:24:47.391 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:24:47.406 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:47.408 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-02-10T07:24:47.408 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:47.454 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rados 2024-02-10T07:24:47.500 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:24:47.502 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:24:48.029 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:24:48.037 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:46 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:48.144 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:24:48.195 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:48.196 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-02-10T07:24:48.196 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:48.203 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-10T07:24:48.240 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rgw 2024-02-10T07:24:48.269 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-10T07:24:48.333 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-10T07:24:48.369 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-10T07:24:48.411 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-10T07:24:48.516 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-10T07:24:48.634 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-10T07:24:48.687 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-10T07:24:48.841 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:46 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:48.954 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:24:49.009 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:49.010 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-02-10T07:24:49.010 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:49.058 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-cephfs 2024-02-10T07:24:49.448 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-10T07:24:49.493 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-10T07:24:49.526 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-10T07:24:49.575 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-10T07:24:49.604 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 13/13 2024-02-10T07:24:49.671 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:47 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:49.783 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:24:49.838 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:49.839 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-02-10T07:24:49.839 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:49.894 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install python3-rbd 2024-02-10T07:24:50.389 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 13/13 2024-02-10T07:24:50.389 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 1/13 2024-02-10T07:24:50.389 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-10T07:24:50.390 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-10T07:24:50.390 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-10T07:24:50.390 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-10T07:24:50.390 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-10T07:24:50.390 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-10T07:24:50.390 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-10T07:24:50.390 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-10T07:24:50.390 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-10T07:24:50.390 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-10T07:24:50.390 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-10T07:24:50.487 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:48 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:50.593 INFO:teuthology.orchestra.run.smithi170.stdout:Package python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:24:50.645 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:50.645 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-02-10T07:24:50.646 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:50.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-fuse 2024-02-10T07:24:50.773 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-10T07:24:50.773 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:50.773 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:24:50.773 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:50.773 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-10T07:24:50.773 INFO:teuthology.orchestra.run.smithi008.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-10T07:24:50.773 INFO:teuthology.orchestra.run.smithi008.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-10T07:24:50.773 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-10T07:24:50.773 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-10T07:24:50.774 INFO:teuthology.orchestra.run.smithi008.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-10T07:24:50.774 INFO:teuthology.orchestra.run.smithi008.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-10T07:24:50.774 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-10T07:24:50.774 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-10T07:24:50.774 INFO:teuthology.orchestra.run.smithi008.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-10T07:24:50.774 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-10T07:24:50.774 INFO:teuthology.orchestra.run.smithi008.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-10T07:24:50.774 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:50.774 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:24:51.023 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T07:24:51.282 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:49 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:51.439 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:51.440 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:51.440 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-02-10T07:24:51.440 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:51.440 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:24:51.440 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 85 k 2024-02-10T07:24:51.440 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:51.440 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:24:51.440 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:51.440 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-02-10T07:24:51.440 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:51.441 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 85 k 2024-02-10T07:24:51.441 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 231 k 2024-02-10T07:24:51.441 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:24:51.614 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:36 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:24:51.675 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-fuse-19.0.0-1361.g290212a1.el9.x86_64.rpm 364 kB/s | 85 kB 00:00 2024-02-10T07:24:51.676 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:51.676 INFO:teuthology.orchestra.run.smithi170.stdout:Total 362 kB/s | 85 kB 00:00 2024-02-10T07:24:51.676 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:24:51.683 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:24:51.683 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:24:51.739 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:24:51.739 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:24:51.783 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:24:51.784 INFO:teuthology.orchestra.run.smithi008.stdout:======================================================================================== 2024-02-10T07:24:51.784 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T07:24:51.784 INFO:teuthology.orchestra.run.smithi008.stdout:======================================================================================== 2024-02-10T07:24:51.784 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:24:51.784 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 138 k 2024-02-10T07:24:51.784 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T07:24:51.784 INFO:teuthology.orchestra.run.smithi008.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-10T07:24:51.784 INFO:teuthology.orchestra.run.smithi008.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-10T07:24:51.784 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-10T07:24:51.784 INFO:teuthology.orchestra.run.smithi008.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-10T07:24:51.785 INFO:teuthology.orchestra.run.smithi008.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-10T07:24:51.785 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:51.785 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:24:51.785 INFO:teuthology.orchestra.run.smithi008.stdout:======================================================================================== 2024-02-10T07:24:51.785 INFO:teuthology.orchestra.run.smithi008.stdout:Install 6 Packages 2024-02-10T07:24:51.785 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:51.785 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 6.8 M 2024-02-10T07:24:51.785 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 33 M 2024-02-10T07:24:51.786 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:24:51.876 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:24:51.974 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:52.119 INFO:teuthology.orchestra.run.smithi008.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 227 kB 00:00 2024-02-10T07:24:52.269 INFO:teuthology.orchestra.run.smithi008.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-02-10T07:24:52.336 INFO:teuthology.orchestra.run.smithi008.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1361.g290212a1.e 368 kB/s | 138 kB 00:00 2024-02-10T07:24:52.408 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:52.427 INFO:teuthology.orchestra.run.smithi008.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 365 kB/s | 58 kB 00:00 2024-02-10T07:24:52.478 INFO:teuthology.orchestra.run.smithi008.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-02-10T07:24:52.503 INFO:teuthology.orchestra.run.smithi008.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 307 kB/s | 51 kB 00:00 2024-02-10T07:24:52.507 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:52.508 INFO:teuthology.orchestra.run.smithi008.stdout:Total 9.4 MB/s | 6.8 MB 00:00 2024-02-10T07:24:52.508 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:24:52.517 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:24:52.517 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:24:52.611 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:24:52.611 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:24:52.691 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:52.691 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:52.691 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:24:52.691 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:24:52.692 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:52.692 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:52.904 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-mirror 2024-02-10T07:24:52.913 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:24:53.083 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-10T07:24:53.131 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-10T07:24:53.502 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:51 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:53.669 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:53.670 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:53.670 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repo Size 2024-02-10T07:24:53.670 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:53.671 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:24:53.671 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-10T07:24:53.671 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:53.671 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:24:53.671 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:53.671 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-02-10T07:24:53.671 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:53.671 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 3.1 M 2024-02-10T07:24:53.671 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 13 M 2024-02-10T07:24:53.671 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:24:53.688 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-10T07:24:53.767 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-10T07:24:53.829 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-10T07:24:53.859 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 6/6 2024-02-10T07:24:54.137 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-mirror-19.0.0-1361.g290212a1.el9.x86_64.rpm 6.6 MB/s | 3.1 MB 00:00 2024-02-10T07:24:54.138 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:54.138 INFO:teuthology.orchestra.run.smithi170.stdout:Total 6.6 MB/s | 3.1 MB 00:00 2024-02-10T07:24:54.139 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:24:54.147 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:24:54.147 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:24:54.220 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:24:54.220 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:24:54.644 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 6/6 2024-02-10T07:24:54.645 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 1/6 2024-02-10T07:24:54.645 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-10T07:24:54.645 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-10T07:24:54.645 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-10T07:24:54.645 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-10T07:24:54.678 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:24:54.738 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:54.762 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:54.762 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:24:54.762 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-10T07:24:54.762 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T07:24:54.762 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T07:24:54.762 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:54.971 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-10T07:24:54.971 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:54.971 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:24:54.971 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:24:54.971 INFO:teuthology.orchestra.run.smithi008.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-10T07:24:54.971 INFO:teuthology.orchestra.run.smithi008.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-10T07:24:54.971 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-10T07:24:54.971 INFO:teuthology.orchestra.run.smithi008.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-10T07:24:54.971 INFO:teuthology.orchestra.run.smithi008.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-10T07:24:54.971 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:54.972 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:24:55.217 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-fuse 2024-02-10T07:24:55.811 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:40 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:24:55.972 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:55.972 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:55.972 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:24:55.972 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:24:55.972 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:55.972 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:55.978 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:24:55.978 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:24:55.978 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T07:24:55.979 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:24:55.979 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:24:55.979 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 885 k 2024-02-10T07:24:55.979 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T07:24:55.979 INFO:teuthology.orchestra.run.smithi008.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-10T07:24:55.979 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:55.979 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:24:55.979 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:24:55.979 INFO:teuthology.orchestra.run.smithi008.stdout:Install 2 Packages 2024-02-10T07:24:55.979 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:55.980 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 964 k 2024-02-10T07:24:55.980 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 2.8 M 2024-02-10T07:24:55.980 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:24:56.105 INFO:teuthology.orchestra.run.smithi008.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 629 kB/s | 79 kB 00:00 2024-02-10T07:24:56.226 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install rbd-nbd 2024-02-10T07:24:56.314 INFO:teuthology.orchestra.run.smithi008.stdout:(2/2): ceph-fuse-19.0.0-1361.g290212a1.el9.x86_ 2.6 MB/s | 885 kB 00:00 2024-02-10T07:24:56.315 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:56.315 INFO:teuthology.orchestra.run.smithi008.stdout:Total 2.8 MB/s | 964 kB 00:00 2024-02-10T07:24:56.315 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:24:56.336 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:24:56.337 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:24:56.415 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:24:56.416 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:24:56.602 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:24:56.752 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-10T07:24:56.825 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2/2 2024-02-10T07:24:56.827 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:01:54 ago on Sat 10 Feb 2024 07:23:02 AM UTC. 2024-02-10T07:24:56.985 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-02-10T07:24:56.985 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:56.986 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-02-10T07:24:56.986 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:56.986 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-02-10T07:24:56.986 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 171 k 2024-02-10T07:24:56.986 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:56.986 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-02-10T07:24:56.986 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-02-10T07:24:56.986 INFO:teuthology.orchestra.run.smithi170.stdout:Install 1 Package 2024-02-10T07:24:56.986 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:56.986 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 171 k 2024-02-10T07:24:56.986 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 494 k 2024-02-10T07:24:56.986 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-02-10T07:24:57.299 INFO:teuthology.orchestra.run.smithi170.stdout:rbd-nbd-19.0.0-1361.g290212a1.el9.x86_64.rpm 546 kB/s | 171 kB 00:00 2024-02-10T07:24:57.300 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:57.300 INFO:teuthology.orchestra.run.smithi170.stdout:Total 544 kB/s | 171 kB 00:00 2024-02-10T07:24:57.300 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-02-10T07:24:57.308 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-02-10T07:24:57.308 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-02-10T07:24:57.365 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-02-10T07:24:57.365 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-02-10T07:24:57.498 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2/2 2024-02-10T07:24:57.498 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/2 2024-02-10T07:24:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-02-10T07:24:57.631 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:57.818 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-10T07:24:57.818 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:57.818 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:24:57.818 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-10T07:24:57.818 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:57.818 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:24:58.091 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:58.110 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-volume 2024-02-10T07:24:58.416 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:24:58.416 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:58.416 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-02-10T07:24:58.416 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:24:58.416 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:24:58.416 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-02-10T07:24:58.651 DEBUG:teuthology.parallel:result is None 2024-02-10T07:24:58.713 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:43 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:24:58.879 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:24:58.880 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:24:58.880 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repository Size 2024-02-10T07:24:58.880 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:24:58.881 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:24:58.881 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-volume noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 264 k 2024-02-10T07:24:58.881 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2024-02-10T07:24:58.881 INFO:teuthology.orchestra.run.smithi008.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-10T07:24:58.881 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:58.881 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:24:58.881 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:24:58.881 INFO:teuthology.orchestra.run.smithi008.stdout:Install 2 Packages 2024-02-10T07:24:58.881 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:24:58.881 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 332 k 2024-02-10T07:24:58.881 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 1.5 M 2024-02-10T07:24:58.881 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:24:59.012 INFO:teuthology.orchestra.run.smithi008.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 527 kB/s | 69 kB 00:00 2024-02-10T07:24:59.179 INFO:teuthology.orchestra.run.smithi008.stdout:(2/2): ceph-volume-19.0.0-1361.g290212a1.el9.no 888 kB/s | 264 kB 00:00 2024-02-10T07:24:59.180 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:24:59.180 INFO:teuthology.orchestra.run.smithi008.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-02-10T07:24:59.180 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:24:59.189 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:24:59.189 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:24:59.230 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:24:59.230 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:24:59.392 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:24:59.578 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-10T07:24:59.615 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2/2 2024-02-10T07:24:59.630 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2/2 2024-02-10T07:24:59.630 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:24:59.630 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-10T07:24:59.630 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:00.404 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 1/2 2024-02-10T07:25:00.794 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-10T07:25:00.794 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:00.794 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:25:00.794 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-10T07:25:00.794 INFO:teuthology.orchestra.run.smithi008.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-10T07:25:00.794 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:00.794 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:25:01.173 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install librados-devel 2024-02-10T07:25:01.786 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:46 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:25:01.954 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:25:01.955 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:01.955 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repo Size 2024-02-10T07:25:01.955 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:01.955 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:25:01.955 INFO:teuthology.orchestra.run.smithi008.stdout: librados-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 127 k 2024-02-10T07:25:01.955 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:01.955 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:25:01.956 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:01.956 INFO:teuthology.orchestra.run.smithi008.stdout:Install 1 Package 2024-02-10T07:25:01.956 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:01.956 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 127 k 2024-02-10T07:25:01.956 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 456 k 2024-02-10T07:25:01.956 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:25:02.236 INFO:teuthology.orchestra.run.smithi008.stdout:librados-devel-19.0.0-1361.g290212a1.el9.x86_64 453 kB/s | 127 kB 00:00 2024-02-10T07:25:02.236 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:25:02.236 INFO:teuthology.orchestra.run.smithi008.stdout:Total 450 kB/s | 127 kB 00:00 2024-02-10T07:25:02.237 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:25:02.242 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:25:02.243 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:25:02.300 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:25:02.301 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:25:02.456 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:25:02.554 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:03.021 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:03.329 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:03.329 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:03.330 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:25:03.330 INFO:teuthology.orchestra.run.smithi008.stdout: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:25:03.330 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:03.330 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:25:03.563 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install libcephfs2 2024-02-10T07:25:04.165 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:49 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:25:04.275 INFO:teuthology.orchestra.run.smithi008.stdout:Package libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:25:04.330 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:25:04.331 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T07:25:04.331 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:25:04.410 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install libcephfs-devel 2024-02-10T07:25:05.006 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:50 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:25:05.173 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:25:05.174 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:05.174 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repo Size 2024-02-10T07:25:05.174 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:05.174 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:25:05.174 INFO:teuthology.orchestra.run.smithi008.stdout: libcephfs-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 31 k 2024-02-10T07:25:05.175 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:05.175 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:25:05.175 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:05.175 INFO:teuthology.orchestra.run.smithi008.stdout:Install 1 Package 2024-02-10T07:25:05.175 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:05.175 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 31 k 2024-02-10T07:25:05.175 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 140 k 2024-02-10T07:25:05.175 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:25:05.674 INFO:teuthology.orchestra.run.smithi008.stdout:libcephfs-devel-19.0.0-1361.g290212a1.el9.x86_6 63 kB/s | 31 kB 00:00 2024-02-10T07:25:05.674 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:25:05.674 INFO:teuthology.orchestra.run.smithi008.stdout:Total 62 kB/s | 31 kB 00:00 2024-02-10T07:25:05.675 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:25:05.677 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:25:05.678 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:25:05.698 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:25:05.698 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:25:05.826 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:25:05.924 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:06.335 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:06.636 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:06.636 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:06.637 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:25:06.637 INFO:teuthology.orchestra.run.smithi008.stdout: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:25:06.637 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:06.637 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:25:06.803 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install librados2 2024-02-10T07:25:07.397 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:52 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:25:07.505 INFO:teuthology.orchestra.run.smithi008.stdout:Package librados2-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:25:07.559 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:25:07.560 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T07:25:07.560 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:25:07.609 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install librbd1 2024-02-10T07:25:08.204 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:53 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:25:08.313 INFO:teuthology.orchestra.run.smithi008.stdout:Package librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:25:08.368 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:25:08.369 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T07:25:08.369 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:25:08.417 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install python3-rados 2024-02-10T07:25:09.032 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:54 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:25:09.143 INFO:teuthology.orchestra.run.smithi008.stdout:Package python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:25:09.196 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:25:09.197 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T07:25:09.197 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:25:09.250 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install python3-rgw 2024-02-10T07:25:09.849 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:54 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:25:09.959 INFO:teuthology.orchestra.run.smithi008.stdout:Package python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:25:10.012 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:25:10.013 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T07:25:10.013 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:25:10.067 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install python3-cephfs 2024-02-10T07:25:10.661 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:55 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:25:10.771 INFO:teuthology.orchestra.run.smithi008.stdout:Package python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:25:10.827 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:25:10.828 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T07:25:10.828 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:25:10.883 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install python3-rbd 2024-02-10T07:25:11.483 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:56 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:25:11.592 INFO:teuthology.orchestra.run.smithi008.stdout:Package python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 is already installed. 2024-02-10T07:25:11.646 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:25:11.647 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2024-02-10T07:25:11.647 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:25:11.706 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install rbd-fuse 2024-02-10T07:25:12.308 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:57 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:25:12.478 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:25:12.479 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:12.479 INFO:teuthology.orchestra.run.smithi008.stdout: Package Architecture Version Repository Size 2024-02-10T07:25:12.479 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:12.479 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:25:12.479 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 85 k 2024-02-10T07:25:12.479 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:12.479 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:25:12.479 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:12.479 INFO:teuthology.orchestra.run.smithi008.stdout:Install 1 Package 2024-02-10T07:25:12.479 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:12.480 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 85 k 2024-02-10T07:25:12.480 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 231 k 2024-02-10T07:25:12.480 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:25:12.710 INFO:teuthology.orchestra.run.smithi008.stdout:rbd-fuse-19.0.0-1361.g290212a1.el9.x86_64.rpm 372 kB/s | 85 kB 00:00 2024-02-10T07:25:12.710 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:25:12.710 INFO:teuthology.orchestra.run.smithi008.stdout:Total 370 kB/s | 85 kB 00:00 2024-02-10T07:25:12.711 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:25:12.717 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:25:12.718 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:25:12.775 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:25:12.775 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:25:12.913 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:25:13.011 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:13.425 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:13.759 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:13.759 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:13.759 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:25:13.759 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:25:13.760 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:13.760 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:25:14.015 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install rbd-mirror 2024-02-10T07:25:14.616 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:01:59 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:25:14.781 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:25:14.782 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:14.782 INFO:teuthology.orchestra.run.smithi008.stdout: Package Arch Version Repo Size 2024-02-10T07:25:14.782 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:14.782 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:25:14.783 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-mirror x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-10T07:25:14.783 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:14.783 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:25:14.783 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:14.783 INFO:teuthology.orchestra.run.smithi008.stdout:Install 1 Package 2024-02-10T07:25:14.783 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:14.783 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 3.1 M 2024-02-10T07:25:14.783 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 13 M 2024-02-10T07:25:14.783 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:25:15.255 INFO:teuthology.orchestra.run.smithi008.stdout:rbd-mirror-19.0.0-1361.g290212a1.el9.x86_64.rpm 6.6 MB/s | 3.1 MB 00:00 2024-02-10T07:25:15.255 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:25:15.255 INFO:teuthology.orchestra.run.smithi008.stdout:Total 6.5 MB/s | 3.1 MB 00:00 2024-02-10T07:25:15.256 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:25:15.265 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:25:15.265 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:25:15.342 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:25:15.342 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:25:15.785 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:25:15.839 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:15.862 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:15.862 INFO:teuthology.orchestra.run.smithi008.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-10T07:25:15.862 INFO:teuthology.orchestra.run.smithi008.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-10T07:25:15.863 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T07:25:15.863 INFO:teuthology.orchestra.run.smithi008.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-10T07:25:15.863 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:16.990 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:16.990 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:16.990 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:25:16.990 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:25:16.990 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:16.990 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:25:17.225 DEBUG:teuthology.orchestra.run.smithi008:> sudo yum -y install rbd-nbd 2024-02-10T07:25:17.824 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:02:02 ago on Sat 10 Feb 2024 07:23:15 AM UTC. 2024-02-10T07:25:17.989 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2024-02-10T07:25:17.990 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:17.990 INFO:teuthology.orchestra.run.smithi008.stdout: Package Architecture Version Repository Size 2024-02-10T07:25:17.990 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:17.990 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2024-02-10T07:25:17.990 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-nbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 171 k 2024-02-10T07:25:17.990 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:17.990 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2024-02-10T07:25:17.990 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2024-02-10T07:25:17.990 INFO:teuthology.orchestra.run.smithi008.stdout:Install 1 Package 2024-02-10T07:25:17.990 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:17.990 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 171 k 2024-02-10T07:25:17.991 INFO:teuthology.orchestra.run.smithi008.stdout:Installed size: 494 k 2024-02-10T07:25:17.991 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2024-02-10T07:25:18.916 INFO:teuthology.orchestra.run.smithi008.stdout:rbd-nbd-19.0.0-1361.g290212a1.el9.x86_64.rpm 184 kB/s | 171 kB 00:00 2024-02-10T07:25:18.917 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:25:18.917 INFO:teuthology.orchestra.run.smithi008.stdout:Total 184 kB/s | 171 kB 00:00 2024-02-10T07:25:18.918 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2024-02-10T07:25:18.928 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2024-02-10T07:25:18.928 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2024-02-10T07:25:18.990 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2024-02-10T07:25:18.991 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2024-02-10T07:25:19.141 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2024-02-10T07:25:19.244 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:19.667 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 1/1 2024-02-10T07:25:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:19.982 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2024-02-10T07:25:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-10T07:25:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:25:19.983 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2024-02-10T07:25:20.262 DEBUG:teuthology.parallel:result is None 2024-02-10T07:25:20.263 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:25:20.430 DEBUG:teuthology.orchestra.run.smithi008:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T07:25:20.451 INFO:teuthology.orchestra.run.smithi008.stdout:19.0.0-1361.g290212a1.el9 2024-02-10T07:25:20.451 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1361.g290212a1.el9 2024-02-10T07:25:20.451 INFO:teuthology.task.install:The correct ceph version 19.0.0-1361.g290212a1 is installed. 2024-02-10T07:25:20.453 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:25:20.616 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T07:25:20.641 INFO:teuthology.orchestra.run.smithi170.stdout:19.0.0-1361.g290212a1.el9 2024-02-10T07:25:20.641 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1361.g290212a1.el9 2024-02-10T07:25:20.641 INFO:teuthology.task.install:The correct ceph version 19.0.0-1361.g290212a1 is installed. 2024-02-10T07:25:20.643 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-10T07:25:20.643 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:25:20.643 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T07:25:20.672 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-02-10T07:25:20.672 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T07:25:20.712 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-10T07:25:20.713 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:25:20.713 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T07:25:20.739 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T07:25:20.806 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-02-10T07:25:20.806 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T07:25:20.835 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T07:25:20.903 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-10T07:25:20.904 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:25:20.904 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T07:25:20.931 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T07:25:20.998 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-02-10T07:25:20.998 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T07:25:21.026 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T07:25:21.093 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-10T07:25:21.094 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:25:21.094 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T07:25:21.120 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T07:25:21.186 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-02-10T07:25:21.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T07:25:21.213 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T07:25:21.280 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-10T07:25:21.384 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'cephadm_mode': 'root'} 2024-02-10T07:25:21.384 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:25:21.385 INFO:tasks.cephadm:Cluster fsid is 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:25:21.385 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-10T07:25:21.385 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.8', 'mon.c': '[v2:172.21.15.8:3301,v1:172.21.15.8:6790]', 'mon.b': '172.21.15.170'} 2024-02-10T07:25:21.385 INFO:tasks.cephadm:First mon is mon.a on smithi008 2024-02-10T07:25:21.385 INFO:tasks.cephadm:First mgr is y 2024-02-10T07:25:21.385 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-10T07:25:21.385 DEBUG:teuthology.orchestra.run.smithi008:> sudo hostname $(hostname -s) 2024-02-10T07:25:21.412 DEBUG:teuthology.orchestra.run.smithi170:> sudo hostname $(hostname -s) 2024-02-10T07:25:21.439 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-02-10T07:25:21.440 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:25:21.594 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/76638/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1361-g290212a1', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1361.g290212a1'}, 'url': 'https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-02-09 18:34:21.832169', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-02-10T07:25:21.750 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref main, sha1 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:25:21.751 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm 2024-02-10T07:25:21.751 DEBUG:teuthology.orchestra.run.smithi008:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T07:25:22.181 INFO:teuthology.orchestra.run.smithi008.stdout:-rw-r--r--. 1 ubuntu ubuntu 766809 Feb 10 07:25 /home/ubuntu/cephtest/cephadm 2024-02-10T07:25:22.181 DEBUG:teuthology.orchestra.run.smithi170:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T07:25:22.595 INFO:teuthology.orchestra.run.smithi170.stdout:-rw-r--r--. 1 ubuntu ubuntu 766809 Feb 10 07:25 /home/ubuntu/cephtest/cephadm 2024-02-10T07:25:22.595 DEBUG:teuthology.orchestra.run.smithi008:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T07:25:22.614 DEBUG:teuthology.orchestra.run.smithi170:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T07:25:22.640 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 on all hosts... 2024-02-10T07:25:22.640 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 pull 2024-02-10T07:25:22.657 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 pull 2024-02-10T07:25:23.491 INFO:teuthology.orchestra.run.smithi170.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T07:25:23.562 INFO:teuthology.orchestra.run.smithi008.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T07:25:51.062 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:25:51.062 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-10T07:25:51.062 INFO:teuthology.orchestra.run.smithi170.stdout: "image_id": "c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6", 2024-02-10T07:25:51.062 INFO:teuthology.orchestra.run.smithi170.stdout: "repo_digests": [ 2024-02-10T07:25:51.062 INFO:teuthology.orchestra.run.smithi170.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58" 2024-02-10T07:25:51.063 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:25:51.063 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:26:07.788 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:26:07.789 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-10T07:26:07.789 INFO:teuthology.orchestra.run.smithi008.stdout: "image_id": "c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6", 2024-02-10T07:26:07.789 INFO:teuthology.orchestra.run.smithi008.stdout: "repo_digests": [ 2024-02-10T07:26:07.789 INFO:teuthology.orchestra.run.smithi008.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58" 2024-02-10T07:26:07.789 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:26:07.789 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:26:07.818 DEBUG:teuthology.orchestra.run.smithi008:> sudo mkdir -p /etc/ceph 2024-02-10T07:26:07.853 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph 2024-02-10T07:26:07.889 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 777 /etc/ceph 2024-02-10T07:26:07.921 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /etc/ceph 2024-02-10T07:26:07.958 INFO:tasks.cephadm:Writing seed config... 2024-02-10T07:26:07.959 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-10T07:26:07.959 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-10T07:26:07.959 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [global] ms type = async 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-02-10T07:26:07.960 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-02-10T07:26:07.961 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-02-10T07:26:07.961 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-02-10T07:26:07.961 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-02-10T07:26:07.961 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-10T07:26:07.961 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-02-10T07:26:07.961 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:26:07.961 DEBUG:teuthology.orchestra.run.smithi008:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-10T07:26:07.976 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8c8244e6-c7e5-11ee-95b8-87774f69a715 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-02-10T07:26:07.977 DEBUG:teuthology.orchestra.run.smithi008:mon.a> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mon.a.service 2024-02-10T07:26:08.019 DEBUG:teuthology.orchestra.run.smithi008:mgr.y> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mgr.y.service 2024-02-10T07:26:08.061 INFO:tasks.cephadm:Bootstrapping... 2024-02-10T07:26:08.061 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -v bootstrap --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.8 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:26:08.258 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2024-02-10T07:26:08.258 INFO:teuthology.orchestra.run.smithi008.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', '-v', 'bootstrap', '--fsid', '8c8244e6-c7e5-11ee-95b8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.8', '--skip-admin-label'] 2024-02-10T07:26:08.283 INFO:teuthology.orchestra.run.smithi008.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T07:26:08.284 INFO:teuthology.orchestra.run.smithi008.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-10T07:26:08.284 INFO:teuthology.orchestra.run.smithi008.stdout:Verifying podman|docker is present... 2024-02-10T07:26:08.306 INFO:teuthology.orchestra.run.smithi008.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T07:26:08.306 INFO:teuthology.orchestra.run.smithi008.stdout:Verifying lvm2 is present... 2024-02-10T07:26:08.306 INFO:teuthology.orchestra.run.smithi008.stdout:Verifying time synchronization is in place... 2024-02-10T07:26:08.313 INFO:teuthology.orchestra.run.smithi008.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-10T07:26:08.313 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T07:26:08.319 INFO:teuthology.orchestra.run.smithi008.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-10T07:26:08.319 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stdout inactive 2024-02-10T07:26:08.325 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stdout enabled 2024-02-10T07:26:08.331 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stdout active 2024-02-10T07:26:08.331 INFO:teuthology.orchestra.run.smithi008.stdout:Unit chronyd.service is enabled and running 2024-02-10T07:26:08.331 INFO:teuthology.orchestra.run.smithi008.stdout:Repeating the final host check... 2024-02-10T07:26:08.354 INFO:teuthology.orchestra.run.smithi008.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T07:26:08.354 INFO:teuthology.orchestra.run.smithi008.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-10T07:26:08.354 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl is present 2024-02-10T07:26:08.354 INFO:teuthology.orchestra.run.smithi008.stdout:lvcreate is present 2024-02-10T07:26:08.360 INFO:teuthology.orchestra.run.smithi008.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-10T07:26:08.361 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T07:26:08.367 INFO:teuthology.orchestra.run.smithi008.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-10T07:26:08.367 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stdout inactive 2024-02-10T07:26:08.373 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stdout enabled 2024-02-10T07:26:08.379 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stdout active 2024-02-10T07:26:08.379 INFO:teuthology.orchestra.run.smithi008.stdout:Unit chronyd.service is enabled and running 2024-02-10T07:26:08.379 INFO:teuthology.orchestra.run.smithi008.stdout:Host looks OK 2024-02-10T07:26:08.380 INFO:teuthology.orchestra.run.smithi008.stdout:Cluster fsid: 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:26:08.380 INFO:teuthology.orchestra.run.smithi008.stdout:Acquiring lock 140315323385456 on /run/cephadm/8c8244e6-c7e5-11ee-95b8-87774f69a715.lock 2024-02-10T07:26:08.380 INFO:teuthology.orchestra.run.smithi008.stdout:Lock 140315323385456 acquired on /run/cephadm/8c8244e6-c7e5-11ee-95b8-87774f69a715.lock 2024-02-10T07:26:08.380 INFO:teuthology.orchestra.run.smithi008.stdout:Verifying IP 172.21.15.8 port 3300 ... 2024-02-10T07:26:08.380 INFO:teuthology.orchestra.run.smithi008.stdout:Verifying IP 172.21.15.8 port 6789 ... 2024-02-10T07:26:08.380 INFO:teuthology.orchestra.run.smithi008.stdout:Base mon IP(s) is [172.21.15.8:3300, 172.21.15.8:6789], mon addrv is [v2:172.21.15.8:3300,v1:172.21.15.8:6789] 2024-02-10T07:26:08.383 INFO:teuthology.orchestra.run.smithi008.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.8 metric 100 2024-02-10T07:26:08.383 INFO:teuthology.orchestra.run.smithi008.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.8 metric 100 2024-02-10T07:26:08.385 INFO:teuthology.orchestra.run.smithi008.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-10T07:26:08.386 INFO:teuthology.orchestra.run.smithi008.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-10T07:26:08.386 INFO:teuthology.orchestra.run.smithi008.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-10T07:26:08.388 INFO:teuthology.orchestra.run.smithi008.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-10T07:26:08.388 INFO:teuthology.orchestra.run.smithi008.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-10T07:26:08.388 INFO:teuthology.orchestra.run.smithi008.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-10T07:26:08.388 INFO:teuthology.orchestra.run.smithi008.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-10T07:26:08.388 INFO:teuthology.orchestra.run.smithi008.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:25fe/64 scope link noprefixroute 2024-02-10T07:26:08.388 INFO:teuthology.orchestra.run.smithi008.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-10T07:26:08.389 INFO:teuthology.orchestra.run.smithi008.stdout:Mon IP `172.21.15.8` is in CIDR network `172.21.0.0/20` 2024-02-10T07:26:08.389 INFO:teuthology.orchestra.run.smithi008.stdout:Mon IP `172.21.15.8` is in CIDR network `172.21.0.0/20` 2024-02-10T07:26:08.389 INFO:teuthology.orchestra.run.smithi008.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-10T07:26:08.389 INFO:teuthology.orchestra.run.smithi008.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-10T07:26:08.390 INFO:teuthology.orchestra.run.smithi008.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T07:26:11.450 INFO:teuthology.orchestra.run.smithi008.stdout:/bin/podman: stdout c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 2024-02-10T07:26:11.450 INFO:teuthology.orchestra.run.smithi008.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T07:26:11.450 INFO:teuthology.orchestra.run.smithi008.stdout:/bin/podman: stderr Getting image source signatures 2024-02-10T07:26:11.450 INFO:teuthology.orchestra.run.smithi008.stdout:/bin/podman: stderr Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-10T07:26:11.450 INFO:teuthology.orchestra.run.smithi008.stdout:/bin/podman: stderr Copying blob sha256:5fe0e97a267976e17c7dcb240cf9455472de6876101ae78af4bdf182158b6d8d 2024-02-10T07:26:11.450 INFO:teuthology.orchestra.run.smithi008.stdout:/bin/podman: stderr Copying config sha256:c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 2024-02-10T07:26:11.451 INFO:teuthology.orchestra.run.smithi008.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-10T07:26:12.354 INFO:teuthology.orchestra.run.smithi008.stdout:ceph: stdout ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T07:26:12.355 INFO:teuthology.orchestra.run.smithi008.stdout:Ceph version: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T07:26:12.355 INFO:teuthology.orchestra.run.smithi008.stdout:Extracting ceph user uid/gid from container image... 2024-02-10T07:26:13.385 INFO:teuthology.orchestra.run.smithi008.stdout:stat: stdout 167 167 2024-02-10T07:26:13.385 INFO:teuthology.orchestra.run.smithi008.stdout:Creating initial keys... 2024-02-10T07:26:14.242 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph-authtool: stdout AQAVJcdlwZBoJxAAU2YOU3iKGDqWexEMJQdIjA== 2024-02-10T07:26:15.155 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph-authtool: stdout AQAWJcdl3Z9hIRAA+a6mcrsNqOar2pPNEaGnEA== 2024-02-10T07:26:16.045 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph-authtool: stdout AQAXJcdl2EPMGhAA+3xvuzgnwnwSv7w8IxUeHg== 2024-02-10T07:26:16.046 INFO:teuthology.orchestra.run.smithi008.stdout:Creating initial monmap... 2024-02-10T07:26:19.094 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T07:26:19.094 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-02-10T07:26:19.094 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:26:19.094 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T07:26:19.095 INFO:teuthology.orchestra.run.smithi008.stdout:monmaptool for a [v2:172.21.15.8:3300,v1:172.21.15.8:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T07:26:19.095 INFO:teuthology.orchestra.run.smithi008.stdout:setting min_mon_release = quincy 2024-02-10T07:26:19.095 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/monmaptool: set fsid to 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:26:19.095 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T07:26:19.095 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:26:19.095 INFO:teuthology.orchestra.run.smithi008.stdout:Creating mon... 2024-02-10T07:26:19.950 INFO:teuthology.orchestra.run.smithi008.stdout:create mon.a on 2024-02-10T07:26:20.146 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-10T07:26:20.285 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-10T07:26:20.454 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715.target. 2024-02-10T07:26:20.454 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715.target. 2024-02-10T07:26:20.701 INFO:teuthology.orchestra.run.smithi008.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mon.a 2024-02-10T07:26:20.702 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mon.a.service: Unit ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mon.a.service not loaded. 2024-02-10T07:26:20.858 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715.target.wants/ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@.service. 2024-02-10T07:26:21.318 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:21 smithi008 systemd[1]: Started Ceph mon.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:26:21.325 INFO:teuthology.orchestra.run.smithi008.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T07:26:21.325 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stdout disabled 2024-02-10T07:26:21.332 INFO:teuthology.orchestra.run.smithi008.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T07:26:21.332 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stdout inactive 2024-02-10T07:26:21.332 INFO:teuthology.orchestra.run.smithi008.stdout:firewalld.service is not enabled 2024-02-10T07:26:21.332 INFO:teuthology.orchestra.run.smithi008.stdout:Not possible to enable service . firewalld.service is not available 2024-02-10T07:26:21.333 INFO:teuthology.orchestra.run.smithi008.stdout:Waiting for mon to start... 2024-02-10T07:26:21.333 INFO:teuthology.orchestra.run.smithi008.stdout:Waiting for mon... 2024-02-10T07:26:21.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:21 smithi008 ceph-mon[29434]: mkfs 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:26:21.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:21 smithi008 ceph-mon[29434]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T07:26:22.398 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:22 smithi008 ceph-mon[29434]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T07:26:22.398 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:22 smithi008 ceph-mon[29434]: monmap e1: 1 mons at {a=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:26:22.398 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:22 smithi008 ceph-mon[29434]: fsmap 2024-02-10T07:26:22.398 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:22 smithi008 ceph-mon[29434]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T07:26:22.399 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:22 smithi008 ceph-mon[29434]: mgrmap e1: no daemons active 2024-02-10T07:26:22.399 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:22 smithi008 ceph-mon[29434]: from='client.? 172.21.15.8:0/3670332512' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T07:26:22.428 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout cluster: 2024-02-10T07:26:22.430 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout id: 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:26:22.430 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-10T07:26:22.431 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 2024-02-10T07:26:22.431 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout services: 2024-02-10T07:26:22.431 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.564287s) 2024-02-10T07:26:22.431 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-10T07:26:22.431 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-10T07:26:22.431 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 2024-02-10T07:26:22.431 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout data: 2024-02-10T07:26:22.431 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-10T07:26:22.431 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-10T07:26:22.431 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-10T07:26:22.431 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout pgs: 2024-02-10T07:26:22.431 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 2024-02-10T07:26:22.431 INFO:teuthology.orchestra.run.smithi008.stdout:mon is available 2024-02-10T07:26:22.431 INFO:teuthology.orchestra.run.smithi008.stdout:Assimilating anything we can from ceph.conf... 2024-02-10T07:26:23.452 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 2024-02-10T07:26:23.452 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout [global] 2024-02-10T07:26:23.452 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout fsid = 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:26:23.452 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.8:3300,v1:172.21.15.8:6789] 2024-02-10T07:26:23.453 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-10T07:26:23.453 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-10T07:26:23.453 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-10T07:26:23.453 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-10T07:26:23.453 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 2024-02-10T07:26:23.453 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-10T07:26:23.453 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-10T07:26:23.453 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 2024-02-10T07:26:23.453 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout [osd] 2024-02-10T07:26:23.453 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-10T07:26:23.453 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-10T07:26:23.453 INFO:teuthology.orchestra.run.smithi008.stdout:Generating new minimal ceph.conf... 2024-02-10T07:26:23.573 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:23 smithi008 ceph-mon[29434]: from='client.? 172.21.15.8:0/2286632276' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T07:26:23.574 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:23 smithi008 ceph-mon[29434]: from='client.? 172.21.15.8:0/2286632276' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T07:26:24.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:24 smithi008 ceph-mon[29434]: from='client.? 172.21.15.8:0/3550880067' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:26:24.917 INFO:teuthology.orchestra.run.smithi008.stdout:Restarting the monitor... 2024-02-10T07:26:25.176 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:24 smithi008 systemd[1]: Stopping Ceph mon.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:26:25.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:25 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a[29429]: 2024-02-10T07:26:25.175+0000 7fbbad41d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:26:25.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:25 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a[29429]: 2024-02-10T07:26:25.175+0000 7fbbad41d700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T07:26:25.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:25 smithi008 podman[29689]: 2024-02-10 07:26:25.411133493 +0000 UTC m=+0.322295332 container died 3d418422a6d81de4ec7d31201b01db8ceb53659de73322063e76009f91518bfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2) 2024-02-10T07:26:25.943 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:25 smithi008 podman[29689]: 2024-02-10 07:26:25.716750393 +0000 UTC m=+0.627912232 container cleanup 3d418422a6d81de4ec7d31201b01db8ceb53659de73322063e76009f91518bfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-10T07:26:25.943 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:25 smithi008 bash[29689]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a 2024-02-10T07:26:26.278 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:25 smithi008 podman[29703]: 2024-02-10 07:26:25.942848988 +0000 UTC m=+0.531796264 container remove 3d418422a6d81de4ec7d31201b01db8ceb53659de73322063e76009f91518bfd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-10T07:26:26.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-10T07:26:26.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 systemd[1]: Stopped Ceph mon.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:26:26.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 systemd[1]: Starting Ceph mon.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:26:26.529 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 podman[29797]: 2024-02-10 07:26:26.182595378 +0000 UTC m=+0.018778409 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T07:26:26.529 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 podman[29797]: 2024-02-10 07:26:26.310336061 +0000 UTC m=+0.146519080 container create e36eae6aadf6ba90f7c33fe2c60e781372c02d434b497c4b5c70e8cc55434f70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-10T07:26:26.529 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 podman[29797]: 2024-02-10 07:26:26.504325472 +0000 UTC m=+0.340508511 container init e36eae6aadf6ba90f7c33fe2c60e781372c02d434b497c4b5c70e8cc55434f70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:26:26.529 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 podman[29797]: 2024-02-10 07:26:26.507114074 +0000 UTC m=+0.343297100 container start e36eae6aadf6ba90f7c33fe2c60e781372c02d434b497c4b5c70e8cc55434f70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:26:26.574 INFO:teuthology.orchestra.run.smithi008.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-10T07:26:26.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T07:26:26.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-10T07:26:26.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: pidfile_write: ignore empty --pid-file 2024-02-10T07:26:26.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: load: jerasure load: lrc 2024-02-10T07:26:26.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: RocksDB version: 7.9.2 2024-02-10T07:26:26.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Git sha 0 2024-02-10T07:26:26.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T07:26:26.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: DB SUMMARY 2024-02-10T07:26:26.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: DB Session ID: 1682PV3B9I06CMFFF9O5 2024-02-10T07:26:26.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: CURRENT file: CURRENT 2024-02-10T07:26:26.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T07:26:26.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-02-10T07:26:26.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-10T07:26:26.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 90789 ; 2024-02-10T07:26:26.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.error_if_exists: 0 2024-02-10T07:26:26.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.create_if_missing: 0 2024-02-10T07:26:26.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.paranoid_checks: 1 2024-02-10T07:26:26.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T07:26:26.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T07:26:26.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T07:26:26.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.env: 0x5623b2cecae0 2024-02-10T07:26:26.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T07:26:26.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.info_log: 0x5623b47eb8c0 2024-02-10T07:26:26.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T07:26:26.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.statistics: (nil) 2024-02-10T07:26:26.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.use_fsync: 0 2024-02-10T07:26:26.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_log_file_size: 0 2024-02-10T07:26:26.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T07:26:26.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T07:26:26.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.allow_fallocate: 1 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.use_direct_reads: 0 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.db_log_dir: 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.wal_dir: 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T07:26:26.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T07:26:26.836 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T07:26:26.836 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T07:26:26.836 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.write_buffer_manager: 0x5623b47bf400 2024-02-10T07:26:26.836 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T07:26:26.836 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T07:26:26.836 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T07:26:26.836 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T07:26:26.836 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T07:26:26.836 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T07:26:26.836 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T07:26:26.836 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T07:26:26.836 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.unordered_write: 0 2024-02-10T07:26:26.836 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T07:26:26.836 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.row_cache: None 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.wal_filter: None 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.two_write_queues: 0 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.wal_compression: 0 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.atomic_flush: 0 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.log_readahead_size: 0 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T07:26:26.837 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T07:26:26.838 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T07:26:26.838 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T07:26:26.838 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T07:26:26.838 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T07:26:26.839 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_background_jobs: 2 2024-02-10T07:26:26.839 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_background_compactions: -1 2024-02-10T07:26:26.839 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_subcompactions: 1 2024-02-10T07:26:26.839 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T07:26:26.839 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T07:26:26.839 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T07:26:26.839 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T07:26:26.839 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T07:26:26.839 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T07:26:26.839 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_open_files: -1 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_background_flushes: -1 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Compression algorithms supported: 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: kZSTD supported: 0 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: kXpressCompression supported: 0 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: kZlibCompression supported: 1 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: kSnappyCompression supported: 1 2024-02-10T07:26:26.840 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: kLZ4Compression supported: 1 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: kBZip2Compression supported: 0 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.merge_operator: 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compaction_filter: None 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5623b47eb9c0) 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout: cache_index_and_filter_blocks: 1 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T07:26:26.841 INFO:journalctl@ceph.mon.a.smithi008.stdout: pin_top_level_index_and_filter: 1 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: index_type: 0 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: data_block_index_type: 0 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: index_shortening: 1 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: checksum: 4 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: no_block_cache: 0 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: block_cache: 0x5623b47ef090 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: block_cache_name: BinnedLRUCache 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: block_cache_options: 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: capacity : 536870912 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: num_shard_bits : 4 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: strict_capacity_limit : 0 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: high_pri_pool_ratio: 0.000 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: block_cache_compressed: (nil) 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: persistent_cache: (nil) 2024-02-10T07:26:26.842 INFO:journalctl@ceph.mon.a.smithi008.stdout: block_size: 4096 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: block_size_deviation: 10 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: block_restart_interval: 16 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: index_block_restart_interval: 1 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: metadata_block_size: 4096 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: partition_filters: 0 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: use_delta_encoding: 1 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: filter_policy: bloomfilter 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: whole_key_filtering: 1 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: verify_compression: 0 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: read_amp_bytes_per_bit: 0 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: format_version: 5 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: enable_index_compression: 1 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: block_align: 0 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: max_auto_readahead_size: 262144 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: prepopulate_block_cache: 0 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: initial_auto_readahead_size: 8192 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T07:26:26.843 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T07:26:26.844 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T07:26:26.844 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compression: NoCompression 2024-02-10T07:26:26.844 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T07:26:26.844 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T07:26:26.844 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T07:26:26.844 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.num_levels: 7 2024-02-10T07:26:26.844 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T07:26:26.844 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T07:26:26.844 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T07:26:26.844 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T07:26:26.844 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T07:26:26.844 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T07:26:26.844 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T07:26:26.844 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:26:26.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T07:26:26.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T07:26:26.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T07:26:26.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T07:26:26.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T07:26:26.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T07:26:26.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T07:26:26.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T07:26:26.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:26:26.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T07:26:26.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T07:26:26.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T07:26:26.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T07:26:26.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T07:26:26.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T07:26:26.847 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.table_properties_collectors: 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.inplace_update_support: 0 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.bloom_locality: 0 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.max_successive_merges: 0 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.ttl: 2592000 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T07:26:26.848 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.enable_blob_files: false 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.min_blob_size: 0 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 81fe71c4-9950-4ba5-b4bf-f39245dd3858 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707549986535413, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-02-10T07:26:26.849 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707549986536293, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 86784, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 278, "table_properties": {"data_size": 84866, "index_size": 264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11580, "raw_average_key_size": 48, "raw_value_size": 78487, "raw_average_value_size": 331, "num_data_blocks": 11, "num_entries": 237, "num_filter_entries": 237, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707549986, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "81fe71c4-9950-4ba5-b4bf-f39245dd3858", "db_session_id": "1682PV3B9I06CMFFF9O5", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707549986536376, "job": 1, "event": "recovery_finished"} 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5623b48e0000 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: DB pointer 0x5623b48ca000 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout: ** DB Stats ** 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T07:26:26.850 INFO:journalctl@ceph.mon.a.smithi008.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: ** Compaction Stats [default] ** 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: L0 2/0 86.58 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: Sum 2/0 86.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: ** Compaction Stats [default] ** 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 146.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: 2024-02-10T07:26:26.851 INFO:journalctl@ceph.mon.a.smithi008.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T07:26:26.852 INFO:journalctl@ceph.mon.a.smithi008.stdout: 2024-02-10T07:26:26.852 INFO:journalctl@ceph.mon.a.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:26:26.852 INFO:journalctl@ceph.mon.a.smithi008.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T07:26:26.852 INFO:journalctl@ceph.mon.a.smithi008.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T07:26:26.852 INFO:journalctl@ceph.mon.a.smithi008.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T07:26:26.852 INFO:journalctl@ceph.mon.a.smithi008.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T07:26:26.852 INFO:journalctl@ceph.mon.a.smithi008.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T07:26:26.852 INFO:journalctl@ceph.mon.a.smithi008.stdout: Cumulative compaction: 0.00 GB write, 28.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:26:26.852 INFO:journalctl@ceph.mon.a.smithi008.stdout: Interval compaction: 0.00 GB write, 28.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:26:26.852 INFO:journalctl@ceph.mon.a.smithi008.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T07:26:26.852 INFO:journalctl@ceph.mon.a.smithi008.stdout: Block cache BinnedLRUCache@0x5623b47ef090#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-10T07:26:26.852 INFO:journalctl@ceph.mon.a.smithi008.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-02-10T07:26:26.852 INFO:journalctl@ceph.mon.a.smithi008.stdout: 2024-02-10T07:26:26.852 INFO:journalctl@ceph.mon.a.smithi008.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T07:26:26.853 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: starting mon.a rank 0 at public addrs [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] at bind addrs [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:26:26.853 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: mon.a@-1(???) e1 preinit fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:26:26.853 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: mon.a@-1(???).mds e1 new map 2024-02-10T07:26:26.853 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: mon.a@-1(???).mds e1 print_map 2024-02-10T07:26:26.853 INFO:journalctl@ceph.mon.a.smithi008.stdout: e1 2024-02-10T07:26:26.853 INFO:journalctl@ceph.mon.a.smithi008.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T07:26:26.853 INFO:journalctl@ceph.mon.a.smithi008.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-10T07:26:26.853 INFO:journalctl@ceph.mon.a.smithi008.stdout: legacy client fscid: -1 2024-02-10T07:26:26.853 INFO:journalctl@ceph.mon.a.smithi008.stdout: 2024-02-10T07:26:26.853 INFO:journalctl@ceph.mon.a.smithi008.stdout: No filesystems configured 2024-02-10T07:26:26.853 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T07:26:26.853 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:26:26.853 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:26:26.854 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:26:26.854 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-10T07:26:26.854 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 bash[29797]: e36eae6aadf6ba90f7c33fe2c60e781372c02d434b497c4b5c70e8cc55434f70 2024-02-10T07:26:26.854 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 systemd[1]: Started Ceph mon.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:26:26.854 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T07:26:26.854 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: monmap e1: 1 mons at {a=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:26:26.854 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: fsmap 2024-02-10T07:26:26.854 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T07:26:26.854 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:26 smithi008 ceph-mon[29816]: mgrmap e1: no daemons active 2024-02-10T07:26:27.663 INFO:teuthology.orchestra.run.smithi008.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-10T07:26:27.685 INFO:teuthology.orchestra.run.smithi008.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:26:27.685 INFO:teuthology.orchestra.run.smithi008.stdout:Creating mgr... 2024-02-10T07:26:27.686 INFO:teuthology.orchestra.run.smithi008.stdout:Verifying port 0.0.0.0:9283 ... 2024-02-10T07:26:27.686 INFO:teuthology.orchestra.run.smithi008.stdout:Verifying port 0.0.0.0:8765 ... 2024-02-10T07:26:27.915 INFO:teuthology.orchestra.run.smithi008.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mgr.y 2024-02-10T07:26:27.916 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mgr.y.service: Unit ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mgr.y.service not loaded. 2024-02-10T07:26:28.066 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715.target.wants/ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@.service. 2024-02-10T07:26:28.399 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:28 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1986433319' entity='client.admin' 2024-02-10T07:26:28.587 INFO:teuthology.orchestra.run.smithi008.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T07:26:28.587 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stdout disabled 2024-02-10T07:26:28.594 INFO:teuthology.orchestra.run.smithi008.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T07:26:28.594 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stdout inactive 2024-02-10T07:26:28.594 INFO:teuthology.orchestra.run.smithi008.stdout:firewalld.service is not enabled 2024-02-10T07:26:28.594 INFO:teuthology.orchestra.run.smithi008.stdout:Not possible to enable service . firewalld.service is not available 2024-02-10T07:26:28.602 INFO:teuthology.orchestra.run.smithi008.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T07:26:28.602 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stdout disabled 2024-02-10T07:26:28.609 INFO:teuthology.orchestra.run.smithi008.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T07:26:28.609 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stdout inactive 2024-02-10T07:26:28.609 INFO:teuthology.orchestra.run.smithi008.stdout:firewalld.service is not enabled 2024-02-10T07:26:28.609 INFO:teuthology.orchestra.run.smithi008.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-02-10T07:26:28.610 INFO:teuthology.orchestra.run.smithi008.stdout:Waiting for mgr to start... 2024-02-10T07:26:28.610 INFO:teuthology.orchestra.run.smithi008.stdout:Waiting for mgr... 2024-02-10T07:26:28.734 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:28 smithi008 systemd[1]: Started Ceph mgr.y for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:26:29.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:29 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2001253535' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T07:26:29.332 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:29 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:29.052+0000 7f539ff59200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:26:29.332 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:29 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:29.149+0000 7f539ff59200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:26:29.677 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 2024-02-10T07:26:29.677 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout { 2024-02-10T07:26:29.677 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "fsid": "8c8244e6-c7e5-11ee-95b8-87774f69a715", 2024-02-10T07:26:29.677 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T07:26:29.677 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T07:26:29.677 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T07:26:29.677 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T07:26:29.677 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:29.677 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T07:26:29.677 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T07:26:29.678 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 0 2024-02-10T07:26:29.678 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout ], 2024-02-10T07:26:29.678 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T07:26:29.678 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "a" 2024-02-10T07:26:29.678 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout ], 2024-02-10T07:26:29.678 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-02-10T07:26:29.678 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T07:26:29.678 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:29.678 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-10T07:26:29.678 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T07:26:29.678 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:29.678 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T07:26:29.678 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:29.679 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T07:26:29.679 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T07:26:29.679 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T07:26:29.679 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T07:26:29.679 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T07:26:29.679 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T07:26:29.682 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:29.683 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T07:26:29.684 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T07:26:29.684 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T07:26:29.684 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T07:26:29.684 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T07:26:29.684 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T07:26:29.684 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T07:26:29.684 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout ], 2024-02-10T07:26:29.684 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T07:26:29.684 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:29.684 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T07:26:29.684 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:29.684 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T07:26:21.316796+0000", 2024-02-10T07:26:29.684 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T07:26:29.685 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:29.685 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T07:26:29.685 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout } 2024-02-10T07:26:29.685 INFO:teuthology.orchestra.run.smithi008.stdout:mgr not available, waiting (1/15)... 2024-02-10T07:26:30.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:29 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:29.822+0000 7f539ff59200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:26:30.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:29 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:29.917+0000 7f539ff59200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:26:30.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:30 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:30.065+0000 7f539ff59200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:26:30.511 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:30 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:30.162+0000 7f539ff59200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:26:30.511 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:30 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:30.246+0000 7f539ff59200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:26:30.511 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:30 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:30.510+0000 7f539ff59200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T07:26:30.804 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:30 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:30.621+0000 7f539ff59200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:26:30.804 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:30 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:30.708+0000 7f539ff59200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:26:30.804 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:30 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:30.803+0000 7f539ff59200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:26:31.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:30 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:30.979+0000 7f539ff59200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:26:32.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:31 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:31.982+0000 7f539ff59200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:26:32.402 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:32 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:32.124+0000 7f539ff59200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:26:32.688 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:32 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1729032083' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T07:26:32.689 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:32 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:32.475+0000 7f539ff59200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:26:32.689 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:32 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:32.566+0000 7f539ff59200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:26:32.952 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:32 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:32.765+0000 7f539ff59200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:26:32.952 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:32 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:32.951+0000 7f539ff59200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:26:32.996 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 2024-02-10T07:26:32.996 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout { 2024-02-10T07:26:32.996 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "fsid": "8c8244e6-c7e5-11ee-95b8-87774f69a715", 2024-02-10T07:26:32.996 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T07:26:32.996 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T07:26:32.996 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 0 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout ], 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "a" 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout ], 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T07:26:32.998 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout ], 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:32.999 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T07:26:33.000 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:33.000 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T07:26:21.316796+0000", 2024-02-10T07:26:33.000 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T07:26:33.000 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:33.000 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T07:26:33.000 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout } 2024-02-10T07:26:33.000 INFO:teuthology.orchestra.run.smithi008.stdout:mgr not available, waiting (2/15)... 2024-02-10T07:26:33.497 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:33 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:33.237+0000 7f539ff59200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:26:33.497 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:33 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:33.496+0000 7f539ff59200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:26:33.831 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:33 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:33.734+0000 7f539ff59200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:26:34.831 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:34 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:34.547+0000 7f539ff59200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:26:35.439 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:35.130+0000 7f539ff59200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:26:35.439 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:35.228+0000 7f539ff59200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:26:35.831 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:35.448+0000 7f539ff59200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:26:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-mon[29816]: Activating manager daemon y 2024-02-10T07:26:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-mon[29816]: mgrmap e2: y(active, starting, since 0.0029574s) 2024-02-10T07:26:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-mon[29816]: from='mgr.14100 172.21.15.8:0/4261578415' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:26:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-mon[29816]: from='mgr.14100 172.21.15.8:0/4261578415' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:26:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-mon[29816]: from='mgr.14100 172.21.15.8:0/4261578415' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:26:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-mon[29816]: from='mgr.14100 172.21.15.8:0/4261578415' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:26:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-mon[29816]: from='mgr.14100 172.21.15.8:0/4261578415' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:26:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-mon[29816]: Manager daemon y is now available 2024-02-10T07:26:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-mon[29816]: from='mgr.14100 172.21.15.8:0/4261578415' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:26:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-mon[29816]: from='mgr.14100 172.21.15.8:0/4261578415' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:26:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-mon[29816]: from='mgr.14100 172.21.15.8:0/4261578415' entity='mgr.y' 2024-02-10T07:26:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-mon[29816]: from='mgr.14100 172.21.15.8:0/4261578415' entity='mgr.y' 2024-02-10T07:26:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:35 smithi008 ceph-mon[29816]: from='mgr.14100 172.21.15.8:0/4261578415' entity='mgr.y' 2024-02-10T07:26:36.340 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 2024-02-10T07:26:36.340 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout { 2024-02-10T07:26:36.340 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "fsid": "8c8244e6-c7e5-11ee-95b8-87774f69a715", 2024-02-10T07:26:36.340 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T07:26:36.340 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T07:26:36.340 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T07:26:36.340 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T07:26:36.340 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:36.340 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T07:26:36.340 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 0 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout ], 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "a" 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout ], 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T07:26:36.341 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T07:26:36.342 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T07:26:36.345 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:36.345 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T07:26:36.345 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T07:26:36.345 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T07:26:36.345 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T07:26:36.345 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T07:26:36.345 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T07:26:36.345 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T07:26:36.345 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T07:26:36.345 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T07:26:36.345 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:36.345 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T07:26:36.345 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:36.345 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout ], 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T07:26:21.316796+0000", 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T07:26:36.346 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:36.347 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T07:26:36.347 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout } 2024-02-10T07:26:36.347 INFO:teuthology.orchestra.run.smithi008.stdout:mgr not available, waiting (3/15)... 2024-02-10T07:26:36.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:36 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2849096650' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T07:26:36.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:36 smithi008 ceph-mon[29816]: mgrmap e3: y(active, since 1.00743s) 2024-02-10T07:26:38.780 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:38 smithi008 ceph-mon[29816]: mgrmap e4: y(active, since 2s) 2024-02-10T07:26:39.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:39 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2377666057' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T07:26:39.864 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 2024-02-10T07:26:39.864 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout { 2024-02-10T07:26:39.864 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "fsid": "8c8244e6-c7e5-11ee-95b8-87774f69a715", 2024-02-10T07:26:39.864 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T07:26:39.864 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T07:26:39.864 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T07:26:39.864 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T07:26:39.864 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:39.864 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T07:26:39.864 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T07:26:39.865 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 0 2024-02-10T07:26:39.865 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout ], 2024-02-10T07:26:39.865 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T07:26:39.865 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "a" 2024-02-10T07:26:39.865 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout ], 2024-02-10T07:26:39.865 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-02-10T07:26:39.865 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T07:26:39.865 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:39.865 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-10T07:26:39.865 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T07:26:39.865 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:39.865 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T07:26:39.865 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:39.866 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T07:26:39.866 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T07:26:39.866 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T07:26:39.866 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T07:26:39.866 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T07:26:39.866 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T07:26:39.870 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:39.870 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T07:26:39.870 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T07:26:39.870 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T07:26:39.870 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T07:26:39.870 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T07:26:39.870 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T07:26:39.871 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T07:26:39.871 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T07:26:39.871 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T07:26:39.871 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:39.871 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T07:26:39.871 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:39.871 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T07:26:39.871 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T07:26:39.871 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:39.871 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T07:26:39.871 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T07:26:39.871 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T07:26:39.872 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T07:26:39.872 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T07:26:39.872 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T07:26:39.872 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T07:26:39.872 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout ], 2024-02-10T07:26:39.872 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T07:26:39.872 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:39.872 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T07:26:39.872 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T07:26:39.872 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T07:26:21.316796+0000", 2024-02-10T07:26:39.872 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T07:26:39.872 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout }, 2024-02-10T07:26:39.873 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T07:26:39.873 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout } 2024-02-10T07:26:39.873 INFO:teuthology.orchestra.run.smithi008.stdout:mgr is available 2024-02-10T07:26:40.755 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:40 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3703235612' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T07:26:41.043 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 2024-02-10T07:26:41.043 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout [global] 2024-02-10T07:26:41.043 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout fsid = 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:26:41.043 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-10T07:26:41.043 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-10T07:26:41.043 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-10T07:26:41.043 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-10T07:26:41.043 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 2024-02-10T07:26:41.043 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-10T07:26:41.043 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-10T07:26:41.043 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 2024-02-10T07:26:41.044 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout [osd] 2024-02-10T07:26:41.044 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-10T07:26:41.044 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-10T07:26:41.044 INFO:teuthology.orchestra.run.smithi008.stdout:Enabling cephadm module... 2024-02-10T07:26:42.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:42 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1808214145' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-10T07:26:42.832 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:42 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ignoring --setuser ceph since I am not root 2024-02-10T07:26:42.832 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:42 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ignoring --setgroup ceph since I am not root 2024-02-10T07:26:43.456 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:43 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:43.132+0000 7f3aeac53200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:26:43.456 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:43 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:43.228+0000 7f3aeac53200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:26:43.748 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:43 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1808214145' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-10T07:26:43.748 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:43 smithi008 ceph-mon[29816]: mgrmap e5: y(active, since 7s) 2024-02-10T07:26:44.076 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:43 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:43.896+0000 7f3aeac53200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:26:44.076 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:43 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:43.991+0000 7f3aeac53200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:26:44.331 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:44 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:44.134+0000 7f3aeac53200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:26:44.331 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:44 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:44.231+0000 7f3aeac53200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:26:44.332 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:44 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:44.316+0000 7f3aeac53200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:26:44.338 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout { 2024-02-10T07:26:44.338 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-10T07:26:44.338 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T07:26:44.338 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-10T07:26:44.338 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-10T07:26:44.338 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout } 2024-02-10T07:26:44.338 INFO:teuthology.orchestra.run.smithi008.stdout:Waiting for the mgr to restart... 2024-02-10T07:26:44.338 INFO:teuthology.orchestra.run.smithi008.stdout:Waiting for mgr epoch 5... 2024-02-10T07:26:44.593 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:44 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4155959099' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T07:26:44.891 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:44 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:44.592+0000 7f3aeac53200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T07:26:44.891 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:44 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:44.704+0000 7f3aeac53200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:26:44.891 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:44 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:44.794+0000 7f3aeac53200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:26:44.891 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:44 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:44.890+0000 7f3aeac53200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:26:45.331 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:45 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:45.067+0000 7f3aeac53200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:26:46.331 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:46 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:46.046+0000 7f3aeac53200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:26:46.332 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:46 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:46.179+0000 7f3aeac53200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:26:46.805 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:46 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:46.514+0000 7f3aeac53200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:26:46.805 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:46 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:46.606+0000 7f3aeac53200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:26:46.805 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:46 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:46.804+0000 7f3aeac53200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:26:47.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:46 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:46.990+0000 7f3aeac53200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:26:47.538 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:47 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:47.278+0000 7f3aeac53200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:26:47.538 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:47 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:47.537+0000 7f3aeac53200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:26:47.831 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:47 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:47.775+0000 7f3aeac53200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:26:49.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:48.591+0000 7f3aeac53200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:26:49.487 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:49.168+0000 7f3aeac53200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:26:49.488 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:49.268+0000 7f3aeac53200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:26:49.488 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:26:49.486+0000 7f3aeac53200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:26:49.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-mon[29816]: Active manager daemon y restarted 2024-02-10T07:26:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-mon[29816]: Activating manager daemon y 2024-02-10T07:26:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-mon[29816]: osdmap e2: 0 total, 0 up, 0 in 2024-02-10T07:26:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-mon[29816]: mgrmap e6: y(active, starting, since 0.0031394s) 2024-02-10T07:26:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:26:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:26:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:26:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:26:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:26:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-mon[29816]: Manager daemon y is now available 2024-02-10T07:26:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' 2024-02-10T07:26:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' 2024-02-10T07:26:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:49 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:50.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:50 smithi008 ceph-mon[29816]: Found migration_current of "None". Setting to last migration. 2024-02-10T07:26:50.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:50 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:50.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:50 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:26:50.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:50 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:26:50.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:50 smithi008 ceph-mon[29816]: mgrmap e7: y(active, since 1.0056s) 2024-02-10T07:26:51.087 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout { 2024-02-10T07:26:51.088 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-10T07:26:51.088 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-10T07:26:51.088 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout } 2024-02-10T07:26:51.088 INFO:teuthology.orchestra.run.smithi008.stdout:mgr epoch 5 is available 2024-02-10T07:26:51.088 INFO:teuthology.orchestra.run.smithi008.stdout:Setting orchestrator backend to cephadm... 2024-02-10T07:26:51.723 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:51 smithi008 ceph-mon[29816]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T07:26:51.723 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:51 smithi008 ceph-mon[29816]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T07:26:52.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:52 smithi008 ceph-mon[29816]: mgrmap e8: y(active, since 2s) 2024-02-10T07:26:52.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:52 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' 2024-02-10T07:26:52.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:52 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:52.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:52 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' 2024-02-10T07:26:52.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:52 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' 2024-02-10T07:26:52.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:52 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:26:53.533 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-10T07:26:53.533 INFO:teuthology.orchestra.run.smithi008.stdout:Generating ssh key... 2024-02-10T07:26:53.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:53 smithi008 ceph-mon[29816]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:26:53.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:53 smithi008 ceph-mon[29816]: [10/Feb/2024:07:26:52] ENGINE Bus STARTING 2024-02-10T07:26:53.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:53 smithi008 ceph-mon[29816]: [10/Feb/2024:07:26:52] ENGINE Serving on https://172.21.15.8:7150 2024-02-10T07:26:53.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:53 smithi008 ceph-mon[29816]: [10/Feb/2024:07:26:52] ENGINE Serving on http://172.21.15.8:8765 2024-02-10T07:26:53.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:53 smithi008 ceph-mon[29816]: [10/Feb/2024:07:26:52] ENGINE Bus STARTED 2024-02-10T07:26:54.703 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-mon[29816]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:26:54.703 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' 2024-02-10T07:26:54.703 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' 2024-02-10T07:26:54.703 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: Generating public/private rsa key pair. 2024-02-10T07:26:54.703 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: Your identification has been saved in /tmp/tmpv52dnhce/key. 2024-02-10T07:26:54.704 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: Your public key has been saved in /tmp/tmpv52dnhce/key.pub. 2024-02-10T07:26:54.704 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: The key fingerprint is: 2024-02-10T07:26:54.704 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: SHA256:4A969fLrEQSBz2w6tllINSqODADak6mbm5Jh47wm/w0 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:26:54.704 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: The key's randomart image is: 2024-02-10T07:26:54.704 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: +---[RSA 3072]----+ 2024-02-10T07:26:54.704 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: |o .o. | 2024-02-10T07:26:54.704 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: |o. o . o. | 2024-02-10T07:26:54.704 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: |o = .* .. | 2024-02-10T07:26:54.704 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: |.. ...o.=. | 2024-02-10T07:26:54.704 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: |.o o oo+S . | 2024-02-10T07:26:54.704 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: |.=o ..=+.. . | 2024-02-10T07:26:54.704 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: |*o. E..=o o | 2024-02-10T07:26:54.704 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: |+* +o o . | 2024-02-10T07:26:54.705 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: |*oo.. . .+. | 2024-02-10T07:26:54.705 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:26:54 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: +----[SHA256]-----+ 2024-02-10T07:26:55.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:55 smithi008 ceph-mon[29816]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:26:55.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:55 smithi008 ceph-mon[29816]: Generating ssh key... 2024-02-10T07:26:56.121 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQColwn3GhoVgh5YOgzJlGrEKfZpUgN2iMin/rfu1C9PG/gstSCuTeee83PKKUfsHBT6/2Rl3fBxL/7aQF1sxt8hCqFQ4FaP9o7fm7eUL1srFr20XFnmRqubs48FMgOrVavZJdrlW1Sk9rU5odHBAhyMkuqvVwuiZ8EWja0SC+KR8G5S+mpGe7/biG1rJ3hV28H362WzX6GGEb4EpFcCSWbyDrzZrJPEP5o19Z6MJDof6fx1i5d7fw2JMwjgdCSOfRaWhOchugHn/MdyLDX8qI71I7u/FU07bVAqLeFYAJELdiqVC7tJBkZbO9zwt8gvljJcJEgirHON4eLdlaQomA4LYYboLrR1dxOa9Y3zMuxuYTSYjRyaO1sEFo9L/24Iw5bZaqEKI+lqeKnAisKW1B31ekWL5ZDO2pnrTKl6+Jz+8MBv7fyLWuwr0AzQVKf1AiRcI3+X1npgv0ELa70Gp4I+oJUq8XyQbbDO85yuZE3dr7e2NethguzN34f/RIMysZU= ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:26:56.121 INFO:teuthology.orchestra.run.smithi008.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-10T07:26:56.122 INFO:teuthology.orchestra.run.smithi008.stdout:Adding key to root@localhost authorized_keys... 2024-02-10T07:26:56.122 INFO:teuthology.orchestra.run.smithi008.stdout:Adding host smithi008... 2024-02-10T07:26:56.794 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:56 smithi008 ceph-mon[29816]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:26:57.740 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:57 smithi008 ceph-mon[29816]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi008", "addr": "172.21.15.8", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:26:57.740 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:26:57 smithi008 ceph-mon[29816]: Deploying cephadm binary to smithi008 2024-02-10T07:27:01.621 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout Added host 'smithi008' with addr '172.21.15.8' 2024-02-10T07:27:01.622 INFO:teuthology.orchestra.run.smithi008.stdout:Deploying unmanaged mon service... 2024-02-10T07:27:02.250 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:02 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' 2024-02-10T07:27:02.250 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:02 smithi008 ceph-mon[29816]: Added host smithi008 2024-02-10T07:27:02.250 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:02 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:02.813 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-10T07:27:02.813 INFO:teuthology.orchestra.run.smithi008.stdout:Deploying unmanaged mgr service... 2024-02-10T07:27:03.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:03 smithi008 ceph-mon[29816]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:03.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:03 smithi008 ceph-mon[29816]: Saving service mon spec with placement count:5 2024-02-10T07:27:03.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:03 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' 2024-02-10T07:27:04.927 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-10T07:27:05.244 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:05 smithi008 ceph-mon[29816]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:05.244 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:05 smithi008 ceph-mon[29816]: Saving service mgr spec with placement count:2 2024-02-10T07:27:05.244 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:05 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' 2024-02-10T07:27:05.244 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:05 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' 2024-02-10T07:27:05.245 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:05 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' 2024-02-10T07:27:06.759 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:06 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3223393592' entity='client.admin' 2024-02-10T07:27:06.760 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:06 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' 2024-02-10T07:27:07.835 INFO:teuthology.orchestra.run.smithi008.stdout:Enabling the dashboard module... 2024-02-10T07:27:08.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:08 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/49023150' entity='client.admin' 2024-02-10T07:27:08.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:08 smithi008 ceph-mon[29816]: from='mgr.14120 172.21.15.8:0/1979055847' entity='mgr.y' 2024-02-10T07:27:09.288 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:09 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/440236736' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-10T07:27:09.581 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:09 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ignoring --setuser ceph since I am not root 2024-02-10T07:27:09.582 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:09 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ignoring --setgroup ceph since I am not root 2024-02-10T07:27:10.015 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:09 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:09.863+0000 7f0fd4802200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:27:10.016 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:09 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:09.959+0000 7f0fd4802200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:27:10.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:10 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/440236736' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-10T07:27:10.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:10 smithi008 ceph-mon[29816]: mgrmap e9: y(active, since 19s) 2024-02-10T07:27:10.969 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:10 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:10.627+0000 7f0fd4802200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:27:10.969 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:10 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:10.725+0000 7f0fd4802200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:27:10.969 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:10 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:10.870+0000 7f0fd4802200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:27:10.970 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:10 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:10.968+0000 7f0fd4802200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:27:11.003 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout { 2024-02-10T07:27:11.004 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-10T07:27:11.004 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T07:27:11.004 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-10T07:27:11.004 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-10T07:27:11.004 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout } 2024-02-10T07:27:11.004 INFO:teuthology.orchestra.run.smithi008.stdout:Waiting for the mgr to restart... 2024-02-10T07:27:11.005 INFO:teuthology.orchestra.run.smithi008.stdout:Waiting for mgr epoch 9... 2024-02-10T07:27:11.250 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:11 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:11.059+0000 7f0fd4802200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:27:11.542 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:11 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2711002683' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T07:27:11.543 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:11 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:11.337+0000 7f0fd4802200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T07:27:11.543 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:11 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:11.451+0000 7f0fd4802200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:27:11.543 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:11 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:11.541+0000 7f0fd4802200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:27:11.819 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:11 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:11.640+0000 7f0fd4802200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:27:11.819 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:11 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:11.818+0000 7f0fd4802200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:27:13.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:12 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:12.808+0000 7f0fd4802200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:27:13.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:12 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:12.943+0000 7f0fd4802200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:27:13.581 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:13 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:13.301+0000 7f0fd4802200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:27:13.581 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:13 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:13.396+0000 7f0fd4802200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:27:14.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:13 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:13.599+0000 7f0fd4802200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:27:14.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:13 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:13.795+0000 7f0fd4802200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:27:14.370 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:14 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:14.091+0000 7f0fd4802200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:27:14.370 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:14 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:14.369+0000 7f0fd4802200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:27:14.831 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:14 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:14.615+0000 7f0fd4802200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:27:15.831 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:15 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:15.448+0000 7f0fd4802200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:27:16.331 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:16.031+0000 7f0fd4802200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:27:16.332 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:16.133+0000 7f0fd4802200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:27:16.831 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:27:16.356+0000 7f0fd4802200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:27:16.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-mon[29816]: Active manager daemon y restarted 2024-02-10T07:27:16.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-mon[29816]: Activating manager daemon y 2024-02-10T07:27:16.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-mon[29816]: osdmap e3: 0 total, 0 up, 0 in 2024-02-10T07:27:16.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-mon[29816]: mgrmap e10: y(active, starting, since 0.00313471s) 2024-02-10T07:27:16.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:27:16.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:27:16.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:27:16.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:27:16.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:27:16.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-mon[29816]: Manager daemon y is now available 2024-02-10T07:27:16.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:16.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:16 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:17.694 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:17 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:27:17.695 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:17 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:27:17.695 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:17 smithi008 ceph-mon[29816]: [10/Feb/2024:07:27:17] ENGINE Bus STARTING 2024-02-10T07:27:17.695 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:17 smithi008 ceph-mon[29816]: [10/Feb/2024:07:27:17] ENGINE Serving on http://172.21.15.8:8765 2024-02-10T07:27:17.695 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:17 smithi008 ceph-mon[29816]: mgrmap e11: y(active, since 1.00583s) 2024-02-10T07:27:18.024 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout { 2024-02-10T07:27:18.024 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-10T07:27:18.024 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-10T07:27:18.024 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout } 2024-02-10T07:27:18.024 INFO:teuthology.orchestra.run.smithi008.stdout:mgr epoch 9 is available 2024-02-10T07:27:18.024 INFO:teuthology.orchestra.run.smithi008.stdout:Generating a dashboard self-signed certificate... 2024-02-10T07:27:18.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:18 smithi008 ceph-mon[29816]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T07:27:18.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:18 smithi008 ceph-mon[29816]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T07:27:18.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:18 smithi008 ceph-mon[29816]: [10/Feb/2024:07:27:17] ENGINE Serving on https://172.21.15.8:7150 2024-02-10T07:27:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:18 smithi008 ceph-mon[29816]: [10/Feb/2024:07:27:17] ENGINE Bus STARTED 2024-02-10T07:27:19.428 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:19 smithi008 ceph-mon[29816]: mgrmap e12: y(active, since 2s) 2024-02-10T07:27:19.428 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:19 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:19.428 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:19 smithi008 ceph-mon[29816]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:19.428 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:19 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:19.428 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:19 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:19.723 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-10T07:27:19.723 INFO:teuthology.orchestra.run.smithi008.stdout:Creating initial admin user... 2024-02-10T07:27:21.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:21.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:21.417 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$M2dDXhpIlLHtnNi9eBusyev.x9xrOcTfUgLK28oMAHV30IMvAUUUW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707550040, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-10T07:27:21.418 INFO:teuthology.orchestra.run.smithi008.stdout:Fetching dashboard port number... 2024-02-10T07:27:22.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:21 smithi008 ceph-mon[29816]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:22.877 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stdout 8443 2024-02-10T07:27:22.886 INFO:teuthology.orchestra.run.smithi008.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T07:27:22.886 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stdout disabled 2024-02-10T07:27:22.894 INFO:teuthology.orchestra.run.smithi008.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T07:27:22.894 INFO:teuthology.orchestra.run.smithi008.stdout:systemctl: stdout inactive 2024-02-10T07:27:22.894 INFO:teuthology.orchestra.run.smithi008.stdout:firewalld.service is not enabled 2024-02-10T07:27:22.894 INFO:teuthology.orchestra.run.smithi008.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-10T07:27:22.895 INFO:teuthology.orchestra.run.smithi008.stdout:Ceph Dashboard is now available at: 2024-02-10T07:27:22.895 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:27:22.896 INFO:teuthology.orchestra.run.smithi008.stdout: URL: https://smithi008.front.sepia.ceph.com:8443/ 2024-02-10T07:27:22.896 INFO:teuthology.orchestra.run.smithi008.stdout: User: admin 2024-02-10T07:27:22.896 INFO:teuthology.orchestra.run.smithi008.stdout: Password: gsqnml91yg 2024-02-10T07:27:22.896 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:27:22.896 INFO:teuthology.orchestra.run.smithi008.stdout:Saving cluster configuration to /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config directory 2024-02-10T07:27:22.896 INFO:teuthology.orchestra.run.smithi008.stdout:Enabling autotune for osd_memory_target 2024-02-10T07:27:23.008 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:22 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1932140494' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-10T07:27:24.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:23 smithi008 ceph-mon[29816]: mgrmap e13: y(active, since 6s) 2024-02-10T07:27:25.390 INFO:teuthology.orchestra.run.smithi008.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-10T07:27:25.390 INFO:teuthology.orchestra.run.smithi008.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-10T07:27:25.390 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout:Or, if you are only running a single cluster on this host: 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout: ceph telemetry on 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout:For more information see: 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:27:25.391 INFO:teuthology.orchestra.run.smithi008.stdout:Bootstrap complete. 2024-02-10T07:27:25.433 INFO:tasks.cephadm:Fetching config... 2024-02-10T07:27:25.434 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:27:25.434 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-10T07:27:25.450 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-10T07:27:25.450 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:27:25.450 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-10T07:27:25.505 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-10T07:27:25.505 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:27:25.505 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-10T07:27:25.572 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-10T07:27:25.572 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:27:25.572 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-10T07:27:25.627 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-10T07:27:25.627 DEBUG:teuthology.orchestra.run.smithi008:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQColwn3GhoVgh5YOgzJlGrEKfZpUgN2iMin/rfu1C9PG/gstSCuTeee83PKKUfsHBT6/2Rl3fBxL/7aQF1sxt8hCqFQ4FaP9o7fm7eUL1srFr20XFnmRqubs48FMgOrVavZJdrlW1Sk9rU5odHBAhyMkuqvVwuiZ8EWja0SC+KR8G5S+mpGe7/biG1rJ3hV28H362WzX6GGEb4EpFcCSWbyDrzZrJPEP5o19Z6MJDof6fx1i5d7fw2JMwjgdCSOfRaWhOchugHn/MdyLDX8qI71I7u/FU07bVAqLeFYAJELdiqVC7tJBkZbO9zwt8gvljJcJEgirHON4eLdlaQomA4LYYboLrR1dxOa9Y3zMuxuYTSYjRyaO1sEFo9L/24Iw5bZaqEKI+lqeKnAisKW1B31ekWL5ZDO2pnrTKl6+Jz+8MBv7fyLWuwr0AzQVKf1AiRcI3+X1npgv0ELa70Gp4I+oJUq8XyQbbDO85yuZE3dr7e2NethguzN34f/RIMysZU= ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T07:27:25.707 INFO:teuthology.orchestra.run.smithi008.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQColwn3GhoVgh5YOgzJlGrEKfZpUgN2iMin/rfu1C9PG/gstSCuTeee83PKKUfsHBT6/2Rl3fBxL/7aQF1sxt8hCqFQ4FaP9o7fm7eUL1srFr20XFnmRqubs48FMgOrVavZJdrlW1Sk9rU5odHBAhyMkuqvVwuiZ8EWja0SC+KR8G5S+mpGe7/biG1rJ3hV28H362WzX6GGEb4EpFcCSWbyDrzZrJPEP5o19Z6MJDof6fx1i5d7fw2JMwjgdCSOfRaWhOchugHn/MdyLDX8qI71I7u/FU07bVAqLeFYAJELdiqVC7tJBkZbO9zwt8gvljJcJEgirHON4eLdlaQomA4LYYboLrR1dxOa9Y3zMuxuYTSYjRyaO1sEFo9L/24Iw5bZaqEKI+lqeKnAisKW1B31ekWL5ZDO2pnrTKl6+Jz+8MBv7fyLWuwr0AzQVKf1AiRcI3+X1npgv0ELa70Gp4I+oJUq8XyQbbDO85yuZE3dr7e2NethguzN34f/RIMysZU= ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:27:25.722 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQColwn3GhoVgh5YOgzJlGrEKfZpUgN2iMin/rfu1C9PG/gstSCuTeee83PKKUfsHBT6/2Rl3fBxL/7aQF1sxt8hCqFQ4FaP9o7fm7eUL1srFr20XFnmRqubs48FMgOrVavZJdrlW1Sk9rU5odHBAhyMkuqvVwuiZ8EWja0SC+KR8G5S+mpGe7/biG1rJ3hV28H362WzX6GGEb4EpFcCSWbyDrzZrJPEP5o19Z6MJDof6fx1i5d7fw2JMwjgdCSOfRaWhOchugHn/MdyLDX8qI71I7u/FU07bVAqLeFYAJELdiqVC7tJBkZbO9zwt8gvljJcJEgirHON4eLdlaQomA4LYYboLrR1dxOa9Y3zMuxuYTSYjRyaO1sEFo9L/24Iw5bZaqEKI+lqeKnAisKW1B31ekWL5ZDO2pnrTKl6+Jz+8MBv7fyLWuwr0AzQVKf1AiRcI3+X1npgv0ELa70Gp4I+oJUq8XyQbbDO85yuZE3dr7e2NethguzN34f/RIMysZU= ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T07:27:25.767 INFO:teuthology.orchestra.run.smithi170.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQColwn3GhoVgh5YOgzJlGrEKfZpUgN2iMin/rfu1C9PG/gstSCuTeee83PKKUfsHBT6/2Rl3fBxL/7aQF1sxt8hCqFQ4FaP9o7fm7eUL1srFr20XFnmRqubs48FMgOrVavZJdrlW1Sk9rU5odHBAhyMkuqvVwuiZ8EWja0SC+KR8G5S+mpGe7/biG1rJ3hV28H362WzX6GGEb4EpFcCSWbyDrzZrJPEP5o19Z6MJDof6fx1i5d7fw2JMwjgdCSOfRaWhOchugHn/MdyLDX8qI71I7u/FU07bVAqLeFYAJELdiqVC7tJBkZbO9zwt8gvljJcJEgirHON4eLdlaQomA4LYYboLrR1dxOa9Y3zMuxuYTSYjRyaO1sEFo9L/24Iw5bZaqEKI+lqeKnAisKW1B31ekWL5ZDO2pnrTKl6+Jz+8MBv7fyLWuwr0AzQVKf1AiRcI3+X1npgv0ELa70Gp4I+oJUq8XyQbbDO85yuZE3dr7e2NethguzN34f/RIMysZU= ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:27:25.782 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-10T07:27:25.957 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:25 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/895381043' entity='client.admin' 2024-02-10T07:27:26.027 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:27:28.391 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:28 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:28.391 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:28 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:28.391 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:28 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi008", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:28.391 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:28 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:28.392 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:28 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:28.392 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:28 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:28.392 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:28 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/217055145' entity='client.admin' 2024-02-10T07:27:29.315 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-10T07:27:29.315 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-10T07:27:29.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:29 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:29.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:29 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:30.050 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:27:32.503 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi170 2024-02-10T07:27:32.503 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-02-10T07:27:32.503 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.conf 2024-02-10T07:27:32.522 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-02-10T07:27:32.522 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:27:32.579 INFO:tasks.cephadm:Adding host smithi170 to orchestrator... 2024-02-10T07:27:32.579 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch host add smithi170 2024-02-10T07:27:32.824 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:27:33.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:32 smithi008 ceph-mon[29816]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:33.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:32 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:33.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:32 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:33.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:32 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:33.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:32 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:33.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:32 smithi008 ceph-mon[29816]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T07:27:33.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:32 smithi008 ceph-mon[29816]: Updating smithi008:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:27:33.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:32 smithi008 ceph-mon[29816]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:27:33.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:32 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:33.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:32 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:33.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:32 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:33.855 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:33 smithi008 ceph-mon[29816]: Updating smithi008:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:27:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:35 smithi008 ceph-mon[29816]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:35 smithi008 ceph-mon[29816]: Deploying cephadm binary to smithi170 2024-02-10T07:27:37.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:37 smithi008 ceph-mon[29816]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:27:39.158 INFO:teuthology.orchestra.run.smithi008.stdout:Added host 'smithi170' with addr '172.21.15.170' 2024-02-10T07:27:39.813 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T07:27:40.058 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:27:40.440 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:40 smithi008 ceph-mon[29816]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:27:40.440 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:40 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:40.440 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:40 smithi008 ceph-mon[29816]: Added host smithi170 2024-02-10T07:27:40.440 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:40 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:40.440 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:40 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:41.522 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:41 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:41.739 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:27:41.739 INFO:teuthology.orchestra.run.smithi008.stdout:[{"addr": "172.21.15.8", "hostname": "smithi008", "labels": [], "status": ""}, {"addr": "172.21.15.170", "hostname": "smithi170", "labels": [], "status": ""}] 2024-02-10T07:27:42.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:42 smithi008 ceph-mon[29816]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:27:42.318 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:42 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:42.345 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-10T07:27:42.345 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-10T07:27:42.580 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:27:43.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:43 smithi008 ceph-mon[29816]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:27:44.219 INFO:teuthology.orchestra.run.smithi008.stderr:adjusted tunables profile to default 2024-02-10T07:27:44.505 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:44 smithi008 ceph-mon[29816]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:27:44.505 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:44 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1389053312' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T07:27:44.810 INFO:tasks.cephadm:Adding mon.a on smithi008 2024-02-10T07:27:44.810 INFO:tasks.cephadm:Adding mon.c on smithi008 2024-02-10T07:27:44.810 INFO:tasks.cephadm:Adding mon.b on smithi170 2024-02-10T07:27:44.810 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch apply mon '3;smithi008:172.21.15.8=a;smithi008:[v2:172.21.15.8:3301,v1:172.21.15.8:6790]=c;smithi170:172.21.15.170=b' 2024-02-10T07:27:45.182 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T07:27:45.229 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T07:27:45.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:45 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1389053312' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T07:27:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:45 smithi008 ceph-mon[29816]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:27:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:45 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:45 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:45 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:45 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:45 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:27:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:45 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:45 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:46 smithi008 ceph-mon[29816]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:27:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:46 smithi008 ceph-mon[29816]: Updating smithi170:/etc/ceph/ceph.conf 2024-02-10T07:27:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:46 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:46 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:46 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:47.132 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mon update... 2024-02-10T07:27:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:47 smithi008 ceph-mon[29816]: Updating smithi170:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:27:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:47 smithi008 ceph-mon[29816]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:27:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:47 smithi008 ceph-mon[29816]: Updating smithi170:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:27:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:27:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:47.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:27:47.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:47.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:27:47.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:47.660 DEBUG:teuthology.orchestra.run.smithi008:mon.c> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mon.c.service 2024-02-10T07:27:47.663 DEBUG:teuthology.orchestra.run.smithi170:mon.b> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mon.b.service 2024-02-10T07:27:47.665 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T07:27:47.666 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T07:27:48.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:48 smithi008 ceph-mon[29816]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:27:48.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:48 smithi008 ceph-mon[29816]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi008:172.21.15.8=a;smithi008:[v2:172.21.15.8:3301,v1:172.21.15.8:6790]=c;smithi170:172.21.15.170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:27:48.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:48 smithi008 ceph-mon[29816]: Saving service mon spec with placement smithi008:172.21.15.8=a;smithi008:[v2:172.21.15.8:3301,v1:172.21.15.8:6790]=c;smithi170:172.21.15.170=b;count:3 2024-02-10T07:27:48.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:48 smithi008 ceph-mon[29816]: Deploying daemon mon.b on smithi170 2024-02-10T07:27:49.295 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:27:50.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:50 smithi008 ceph-mon[29816]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:27:51.815 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:27:51.815 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"8c8244e6-c7e5-11ee-95b8-87774f69a715","modified":"2024-02-10T07:26:16.360438Z","created":"2024-02-10T07:26:16.360438Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T07:27:51.817 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2024-02-10T07:27:52.119 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:51 smithi170 ceph-mon[33394]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-10T07:27:52.119 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:52 smithi170 bash[33355]: 8c90b2d2d7b7ffb7b54b42b015a8ecd0d98d071bc1fcb01353935474e464d60d 2024-02-10T07:27:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:52 smithi170 systemd[1]: Started Ceph mon.b for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:27:53.525 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T07:27:53.526 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T07:27:53.772 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:27:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:27:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: mon.a calling monitor election 2024-02-10T07:27:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:27:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: mon.b calling monitor election 2024-02-10T07:27:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:27:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:57.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T07:27:57.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: monmap e2: 2 mons at {a=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:27:57.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: fsmap 2024-02-10T07:27:57.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:27:57.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: mgrmap e13: y(active, since 40s) 2024-02-10T07:27:57.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: overall HEALTH_OK 2024-02-10T07:27:57.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:57.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:57.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:57.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:27:57.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:56 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:27:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: mon.a calling monitor election 2024-02-10T07:27:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:27:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: mon.b calling monitor election 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: monmap e2: 2 mons at {a=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: fsmap 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: mgrmap e13: y(active, since 40s) 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: overall HEALTH_OK 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:27:57.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:27:57.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:56 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:27:58.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:58 smithi008 ceph-mon[29816]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:27:58.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:58 smithi008 ceph-mon[29816]: Deploying daemon mon.c on smithi008 2024-02-10T07:27:58.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:58 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:58.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:58 smithi170 ceph-mon[33394]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:27:58.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:58 smithi170 ceph-mon[33394]: Deploying daemon mon.c on smithi008 2024-02-10T07:27:58.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:58 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:27:58.609 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:27:58.609 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":2,"fsid":"8c8244e6-c7e5-11ee-95b8-87774f69a715","modified":"2024-02-10T07:27:51.900195Z","created":"2024-02-10T07:26:16.360438Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T07:27:58.611 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 2 2024-02-10T07:27:59.258 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:27:59 smithi170 ceph-mon[33394]: from='client.? 172.21.15.170:0/966266411' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:27:59.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:27:59 smithi008 ceph-mon[29816]: from='client.? 172.21.15.170:0/966266411' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:28:00.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[29816]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:00.287 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T07:28:00.287 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T07:28:00.317 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:00 smithi170 ceph-mon[33394]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:00.467 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 podman[40722]: 2024-02-10 07:28:00.384491307 +0000 UTC m=+0.343863711 container start c714c53367e5b61e6483c0842ba3f2f23d2232636b40b3ca53ae4bfce429c47e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-c, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-10T07:28:00.467 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T07:28:00.467 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-10T07:28:00.467 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: pidfile_write: ignore empty --pid-file 2024-02-10T07:28:00.533 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:28:00.724 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: load: jerasure load: lrc 2024-02-10T07:28:00.725 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: RocksDB version: 7.9.2 2024-02-10T07:28:00.725 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Git sha 0 2024-02-10T07:28:00.725 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T07:28:00.725 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: DB SUMMARY 2024-02-10T07:28:00.725 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: DB Session ID: D55T7Q30DBXCU54QLARV 2024-02-10T07:28:00.725 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: CURRENT file: CURRENT 2024-02-10T07:28:00.725 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T07:28:00.725 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-10T07:28:00.725 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-10T07:28:00.725 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-02-10T07:28:00.725 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.error_if_exists: 0 2024-02-10T07:28:00.725 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.create_if_missing: 0 2024-02-10T07:28:00.725 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.paranoid_checks: 1 2024-02-10T07:28:00.726 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T07:28:00.726 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T07:28:00.726 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T07:28:00.726 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.env: 0x55fca093dae0 2024-02-10T07:28:00.726 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T07:28:00.726 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.info_log: 0x55fca1dc18c0 2024-02-10T07:28:00.726 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T07:28:00.726 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.statistics: (nil) 2024-02-10T07:28:00.726 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.use_fsync: 0 2024-02-10T07:28:00.726 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_log_file_size: 0 2024-02-10T07:28:00.726 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T07:28:00.726 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T07:28:00.727 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T07:28:00.727 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T07:28:00.727 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.allow_fallocate: 1 2024-02-10T07:28:00.727 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T07:28:00.727 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T07:28:00.727 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.use_direct_reads: 0 2024-02-10T07:28:00.727 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T07:28:00.727 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T07:28:00.727 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.db_log_dir: 2024-02-10T07:28:00.727 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.wal_dir: 2024-02-10T07:28:00.727 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T07:28:00.727 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T07:28:00.727 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T07:28:00.727 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T07:28:00.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T07:28:00.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T07:28:00.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T07:28:00.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T07:28:00.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.write_buffer_manager: 0x55fca1d95400 2024-02-10T07:28:00.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T07:28:00.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T07:28:00.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T07:28:00.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T07:28:00.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T07:28:00.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T07:28:00.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T07:28:00.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.unordered_write: 0 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.row_cache: None 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.wal_filter: None 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.two_write_queues: 0 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.wal_compression: 0 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.atomic_flush: 0 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T07:28:00.729 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T07:28:00.730 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T07:28:00.730 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.log_readahead_size: 0 2024-02-10T07:28:00.730 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T07:28:00.730 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T07:28:00.730 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T07:28:00.730 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T07:28:00.730 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T07:28:00.730 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T07:28:00.730 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T07:28:00.730 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_background_jobs: 2 2024-02-10T07:28:00.730 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_background_compactions: -1 2024-02-10T07:28:00.730 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_subcompactions: 1 2024-02-10T07:28:00.732 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T07:28:00.732 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T07:28:00.732 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T07:28:00.732 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T07:28:00.732 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T07:28:00.732 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T07:28:00.732 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T07:28:00.732 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T07:28:00.732 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_open_files: -1 2024-02-10T07:28:00.732 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T07:28:00.732 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T07:28:00.732 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_background_flushes: -1 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Compression algorithms supported: 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: kZSTD supported: 0 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: kXpressCompression supported: 0 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: kZlibCompression supported: 1 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: kSnappyCompression supported: 1 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: kLZ4Compression supported: 1 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: kBZip2Compression supported: 0 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T07:28:00.733 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T07:28:00.734 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.merge_operator: 2024-02-10T07:28:00.734 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compaction_filter: None 2024-02-10T07:28:00.734 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T07:28:00.734 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T07:28:00.734 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T07:28:00.734 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T07:28:00.734 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fca1dc19c0) 2024-02-10T07:28:00.734 INFO:journalctl@ceph.mon.c.smithi008.stdout: cache_index_and_filter_blocks: 1 2024-02-10T07:28:00.734 INFO:journalctl@ceph.mon.c.smithi008.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T07:28:00.734 INFO:journalctl@ceph.mon.c.smithi008.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T07:28:00.734 INFO:journalctl@ceph.mon.c.smithi008.stdout: pin_top_level_index_and_filter: 1 2024-02-10T07:28:00.734 INFO:journalctl@ceph.mon.c.smithi008.stdout: index_type: 0 2024-02-10T07:28:00.734 INFO:journalctl@ceph.mon.c.smithi008.stdout: data_block_index_type: 0 2024-02-10T07:28:00.734 INFO:journalctl@ceph.mon.c.smithi008.stdout: index_shortening: 1 2024-02-10T07:28:00.735 INFO:journalctl@ceph.mon.c.smithi008.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T07:28:00.735 INFO:journalctl@ceph.mon.c.smithi008.stdout: checksum: 4 2024-02-10T07:28:00.735 INFO:journalctl@ceph.mon.c.smithi008.stdout: no_block_cache: 0 2024-02-10T07:28:00.735 INFO:journalctl@ceph.mon.c.smithi008.stdout: block_cache: 0x55fca1dc5090 2024-02-10T07:28:00.735 INFO:journalctl@ceph.mon.c.smithi008.stdout: block_cache_name: BinnedLRUCache 2024-02-10T07:28:00.735 INFO:journalctl@ceph.mon.c.smithi008.stdout: block_cache_options: 2024-02-10T07:28:00.735 INFO:journalctl@ceph.mon.c.smithi008.stdout: capacity : 536870912 2024-02-10T07:28:00.735 INFO:journalctl@ceph.mon.c.smithi008.stdout: num_shard_bits : 4 2024-02-10T07:28:00.735 INFO:journalctl@ceph.mon.c.smithi008.stdout: strict_capacity_limit : 0 2024-02-10T07:28:00.735 INFO:journalctl@ceph.mon.c.smithi008.stdout: high_pri_pool_ratio: 0.000 2024-02-10T07:28:00.735 INFO:journalctl@ceph.mon.c.smithi008.stdout: block_cache_compressed: (nil) 2024-02-10T07:28:00.735 INFO:journalctl@ceph.mon.c.smithi008.stdout: persistent_cache: (nil) 2024-02-10T07:28:00.735 INFO:journalctl@ceph.mon.c.smithi008.stdout: block_size: 4096 2024-02-10T07:28:00.735 INFO:journalctl@ceph.mon.c.smithi008.stdout: block_size_deviation: 10 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: block_restart_interval: 16 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: index_block_restart_interval: 1 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: metadata_block_size: 4096 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: partition_filters: 0 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: use_delta_encoding: 1 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: filter_policy: bloomfilter 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: whole_key_filtering: 1 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: verify_compression: 0 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: read_amp_bytes_per_bit: 0 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: format_version: 5 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: enable_index_compression: 1 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: block_align: 0 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: max_auto_readahead_size: 262144 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: prepopulate_block_cache: 0 2024-02-10T07:28:00.736 INFO:journalctl@ceph.mon.c.smithi008.stdout: initial_auto_readahead_size: 8192 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compression: NoCompression 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.num_levels: 7 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T07:28:00.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T07:28:00.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T07:28:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T07:28:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T07:28:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T07:28:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T07:28:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T07:28:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T07:28:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T07:28:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T07:28:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T07:28:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T07:28:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T07:28:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T07:28:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T07:28:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T07:28:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.table_properties_collectors: 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.inplace_update_support: 0 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.bloom_locality: 0 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.max_successive_merges: 0 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.ttl: 2592000 2024-02-10T07:28:00.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.enable_blob_files: false 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.min_blob_size: 0 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2140aa1e-56b8-4099-981d-11ab18516b2d 2024-02-10T07:28:00.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707550080473265, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707550080473652, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707550080, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2140aa1e-56b8-4099-981d-11ab18516b2d", "db_session_id": "D55T7Q30DBXCU54QLARV", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707550080473733, "job": 1, "event": "recovery_finished"} 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fca1eb6000 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: DB pointer 0x55fca1ea0000 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout: ** DB Stats ** 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T07:28:00.743 INFO:journalctl@ceph.mon.c.smithi008.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T07:28:00.744 INFO:journalctl@ceph.mon.c.smithi008.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T07:28:00.744 INFO:journalctl@ceph.mon.c.smithi008.stdout: 2024-02-10T07:28:00.744 INFO:journalctl@ceph.mon.c.smithi008.stdout: ** Compaction Stats [default] ** 2024-02-10T07:28:00.744 INFO:journalctl@ceph.mon.c.smithi008.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T07:28:00.744 INFO:journalctl@ceph.mon.c.smithi008.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T07:28:00.744 INFO:journalctl@ceph.mon.c.smithi008.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T07:28:00.744 INFO:journalctl@ceph.mon.c.smithi008.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T07:28:00.744 INFO:journalctl@ceph.mon.c.smithi008.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T07:28:00.744 INFO:journalctl@ceph.mon.c.smithi008.stdout: 2024-02-10T07:28:00.744 INFO:journalctl@ceph.mon.c.smithi008.stdout: ** Compaction Stats [default] ** 2024-02-10T07:28:00.744 INFO:journalctl@ceph.mon.c.smithi008.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T07:28:00.744 INFO:journalctl@ceph.mon.c.smithi008.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T07:28:00.744 INFO:journalctl@ceph.mon.c.smithi008.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: Block cache BinnedLRUCache@0x55fca1dc5090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-10T07:28:00.745 INFO:journalctl@ceph.mon.c.smithi008.stdout: 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: using public_addrv [v2:172.21.15.8:3301/0,v1:172.21.15.8:6790/0] 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: starting mon.c rank -1 at public addrs [v2:172.21.15.8:3301/0,v1:172.21.15.8:6790/0] at bind addrs [v2:172.21.15.8:3301/0,v1:172.21.15.8:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c@-1(???) e0 preinit fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c@-1(synchronizing).mds e1 new map 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c@-1(synchronizing).mds e1 print_map 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout: e1 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout: legacy client fscid: -1 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout: 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout: No filesystems configured 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T07:28:00.746 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T07:28:00.747 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T07:28:00.747 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T07:28:00.747 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T07:28:00.747 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T07:28:00.747 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:28:00.747 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:28:00.747 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T07:28:00.747 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:00.747 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:00.747 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:28:00.747 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:00.747 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1389053312' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T07:28:00.747 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1389053312' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T07:28:00.747 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:28:00.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:00.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:00.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:00.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:00.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:28:00.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:00.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:00.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:00.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: Updating smithi170:/etc/ceph/ceph.conf 2024-02-10T07:28:00.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:00.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:00.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:00.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: Updating smithi170:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:28:00.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: Updating smithi170:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi008:172.21.15.8=a;smithi008:[v2:172.21.15.8:3301,v1:172.21.15.8:6790]=c;smithi170:172.21.15.170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: Saving service mon spec with placement smithi008:172.21.15.8=a;smithi008:[v2:172.21.15.8:3301,v1:172.21.15.8:6790]=c;smithi170:172.21.15.170=b;count:3 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: Deploying daemon mon.b on smithi170 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.a calling monitor election 2024-02-10T07:28:00.749 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.b calling monitor election 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: monmap e2: 2 mons at {a=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: fsmap 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mgrmap e13: y(active, since 40s) 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: overall HEALTH_OK 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:00.750 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:00.751 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:28:00.751 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:00.751 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:00.751 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: Deploying daemon mon.c on smithi008 2024-02-10T07:28:00.751 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:28:00.751 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: from='client.? 172.21.15.170:0/966266411' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:28:00.751 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:00.751 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 ceph-mon[40743]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-10T07:28:00.751 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 bash[40722]: c714c53367e5b61e6483c0842ba3f2f23d2232636b40b3ca53ae4bfce429c47e 2024-02-10T07:28:00.751 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:00 smithi008 systemd[1]: Started Ceph mon.c for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:28:05.880 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-02-10T07:28:05.880 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":3,"fsid":"8c8244e6-c7e5-11ee-95b8-87774f69a715","modified":"2024-02-10T07:28:00.492932Z","created":"2024-02-10T07:26:16.360438Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3300","nonce":0},{"type":"v1","addr":"172.21.15.8:6789","nonce":0}]},"addr":"172.21.15.8:6789/0","public_addr":"172.21.15.8:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:3301","nonce":0},{"type":"v1","addr":"172.21.15.8:6790","nonce":0}]},"addr":"172.21.15.8:6790/0","public_addr":"172.21.15.8:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-02-10T07:28:05.880 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:05.880 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:28:05.880 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: mon.a calling monitor election 2024-02-10T07:28:05.880 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: mon.b calling monitor election 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: mon.c calling monitor election 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: monmap e3: 3 mons at {a=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.8:3301/0,v1:172.21.15.8:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: fsmap 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: mgrmap e13: y(active, since 49s) 2024-02-10T07:28:05.881 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: overall HEALTH_OK 2024-02-10T07:28:05.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:05.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:05.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:05.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:05.882 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 3 2024-02-10T07:28:06.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:06.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:28:06.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: mon.a calling monitor election 2024-02-10T07:28:06.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: mon.b calling monitor election 2024-02-10T07:28:06.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:28:06.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:06.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:06.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:06.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:06.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: mon.c calling monitor election 2024-02-10T07:28:06.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:06.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:06.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:06.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:06.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-10T07:28:06.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: monmap e3: 3 mons at {a=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.8:3301/0,v1:172.21.15.8:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:28:06.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: fsmap 2024-02-10T07:28:06.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:28:06.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: mgrmap e13: y(active, since 49s) 2024-02-10T07:28:06.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: overall HEALTH_OK 2024-02-10T07:28:06.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:06.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:06.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:06.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:06.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: mon.a calling monitor election 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: mon.b calling monitor election 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: mon.c calling monitor election 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-10T07:28:06.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: monmap e3: 3 mons at {a=[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.8:3301/0,v1:172.21.15.8:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T07:28:06.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: fsmap 2024-02-10T07:28:06.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T07:28:06.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: mgrmap e13: y(active, since 49s) 2024-02-10T07:28:06.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: overall HEALTH_OK 2024-02-10T07:28:06.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:06.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:06.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:06.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:06.487 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-10T07:28:06.488 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph config generate-minimal-conf 2024-02-10T07:28:06.751 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:06 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:06.751 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:06 smithi008 ceph-mon[29816]: from='client.? 172.21.15.170:0/3108827867' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:28:06.751 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:06 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:06.751 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:06 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:06.751 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:06 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:06.752 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:06 smithi008 ceph-mon[40743]: from='client.? 172.21.15.170:0/3108827867' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:28:06.752 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:06 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:06.752 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:06 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:06.814 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:06 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:06.815 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:06 smithi170 ceph-mon[33394]: from='client.? 172.21.15.170:0/3108827867' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:28:06.815 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:06 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:06.815 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:06 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:28:07.063 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:08.328 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:08.328 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.328 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.328 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.328 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: Updating smithi170:/etc/ceph/ceph.conf 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:28:08.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: Updating smithi170:/etc/ceph/ceph.conf 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:28:08.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:28:08.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:08.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:08.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:08.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:08.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T07:28:08.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: Updating smithi170:/etc/ceph/ceph.conf 2024-02-10T07:28:08.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:08.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:28:08.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:28:08.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:08.914 INFO:teuthology.orchestra.run.smithi008.stdout:# minimal ceph.conf for 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:28:08.914 INFO:teuthology.orchestra.run.smithi008.stdout:[global] 2024-02-10T07:28:08.915 INFO:teuthology.orchestra.run.smithi008.stdout: fsid = 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:28:08.915 INFO:teuthology.orchestra.run.smithi008.stdout: mon_host = [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] [v2:172.21.15.8:3301/0,v1:172.21.15.8:6790/0] 2024-02-10T07:28:09.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:09 smithi008 ceph-mon[40743]: Updating smithi170:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:28:09.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:09 smithi008 ceph-mon[40743]: Updating smithi008:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:28:09.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:09 smithi008 ceph-mon[40743]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T07:28:09.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:09 smithi008 ceph-mon[40743]: Reconfiguring daemon mon.a on smithi008 2024-02-10T07:28:09.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:09 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2506741895' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:09.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:09 smithi008 ceph-mon[29816]: Updating smithi170:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:28:09.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:09 smithi008 ceph-mon[29816]: Updating smithi008:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:28:09.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:09 smithi008 ceph-mon[29816]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T07:28:09.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:09 smithi008 ceph-mon[29816]: Reconfiguring daemon mon.a on smithi008 2024-02-10T07:28:09.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:09 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2506741895' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:09 smithi170 ceph-mon[33394]: Updating smithi170:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:28:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:09 smithi170 ceph-mon[33394]: Updating smithi008:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:28:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:09 smithi170 ceph-mon[33394]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T07:28:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:09 smithi170 ceph-mon[33394]: Reconfiguring daemon mon.a on smithi008 2024-02-10T07:28:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:09 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2506741895' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:09.909 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-10T07:28:09.910 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:28:09.910 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T07:28:09.942 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:28:09.942 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:28:10.012 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-02-10T07:28:10.012 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T07:28:10.046 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-02-10T07:28:10.047 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:28:10.119 INFO:tasks.cephadm:Adding mgr.y on smithi008 2024-02-10T07:28:10.119 INFO:tasks.cephadm:Adding mgr.x on smithi170 2024-02-10T07:28:10.119 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch apply mgr '2;smithi008=y;smithi170=x' 2024-02-10T07:28:10.290 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:10 smithi008 ceph-mon[29816]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:10.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:10 smithi008 ceph-mon[40743]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:10.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:10 smithi170 ceph-mon[33394]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:10.396 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:28:11.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:11.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:11.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[29816]: Reconfiguring mon.c (monmap changed)... 2024-02-10T07:28:11.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:28:11.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:28:11.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:11.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[29816]: Reconfiguring daemon mon.c on smithi008 2024-02-10T07:28:11.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[29816]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:11.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:11.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:11.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[40743]: Reconfiguring mon.c (monmap changed)... 2024-02-10T07:28:11.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:28:11.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:28:11.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:11.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[40743]: Reconfiguring daemon mon.c on smithi008 2024-02-10T07:28:11.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:11 smithi008 ceph-mon[40743]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:11.623 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:11 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:11.623 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:11 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:11.623 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:11 smithi170 ceph-mon[33394]: Reconfiguring mon.c (monmap changed)... 2024-02-10T07:28:11.623 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:11 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:28:11.623 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:11 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:28:11.623 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:11 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:11.624 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:11 smithi170 ceph-mon[33394]: Reconfiguring daemon mon.c on smithi008 2024-02-10T07:28:11.624 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:11 smithi170 ceph-mon[33394]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:12.211 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mgr update... 2024-02-10T07:28:12.882 DEBUG:teuthology.orchestra.run.smithi170:mgr.x> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mgr.x.service 2024-02-10T07:28:12.884 INFO:tasks.cephadm:Deploying OSDs... 2024-02-10T07:28:12.884 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:28:12.884 DEBUG:teuthology.orchestra.run.smithi008:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T07:28:12.901 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T07:28:12.901 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_1 2024-02-10T07:28:12.956 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T07:28:12.957 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:28:12.957 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-10T07:28:12.957 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:28:12.957 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:28:12.957 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-02-10 07:27:59.890041623 +0000 2024-02-10T07:28:12.957 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-02-10 07:22:01.865972523 +0000 2024-02-10T07:28:12.957 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-02-10 07:22:01.865972523 +0000 2024-02-10T07:28:12.957 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-02-10 07:22:01.865972523 +0000 2024-02-10T07:28:12.957 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T07:28:13.023 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-02-10T07:28:13.024 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-02-10T07:28:13.024 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000106143 s, 4.8 MB/s 2024-02-10T07:28:13.025 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T07:28:13.081 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_2 2024-02-10T07:28:13.129 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:12 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:13.129 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:12 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:13.129 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:12 smithi170 ceph-mon[33394]: Reconfiguring mon.b (monmap changed)... 2024-02-10T07:28:13.129 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:12 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:28:13.129 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:12 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:28:13.130 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:12 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:13.130 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:12 smithi170 ceph-mon[33394]: Reconfiguring daemon mon.b on smithi170 2024-02-10T07:28:13.130 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:12 smithi170 ceph-mon[33394]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi008=y;smithi170=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:28:13.130 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:12 smithi170 ceph-mon[33394]: Saving service mgr spec with placement smithi008=y;smithi170=x;count:2 2024-02-10T07:28:13.130 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:12 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:13.136 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T07:28:13.136 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:28:13.136 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-10T07:28:13.136 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:28:13.136 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:28:13.136 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-02-10 07:27:59.890041623 +0000 2024-02-10T07:28:13.136 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-02-10 07:22:02.204968911 +0000 2024-02-10T07:28:13.136 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-02-10 07:22:02.204968911 +0000 2024-02-10T07:28:13.136 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-02-10 07:22:02.204968911 +0000 2024-02-10T07:28:13.137 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T07:28:13.199 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:13.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:13.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[29816]: Reconfiguring mon.b (monmap changed)... 2024-02-10T07:28:13.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:28:13.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:28:13.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:13.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[29816]: Reconfiguring daemon mon.b on smithi170 2024-02-10T07:28:13.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[29816]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi008=y;smithi170=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:28:13.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[29816]: Saving service mgr spec with placement smithi008=y;smithi170=x;count:2 2024-02-10T07:28:13.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:13.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:13.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:13.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[40743]: Reconfiguring mon.b (monmap changed)... 2024-02-10T07:28:13.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T07:28:13.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T07:28:13.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:13.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[40743]: Reconfiguring daemon mon.b on smithi170 2024-02-10T07:28:13.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[40743]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi008=y;smithi170=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:28:13.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[40743]: Saving service mgr spec with placement smithi008=y;smithi170=x;count:2 2024-02-10T07:28:13.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:12 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:13.202 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-02-10T07:28:13.202 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-02-10T07:28:13.202 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 8.277e-05 s, 6.2 MB/s 2024-02-10T07:28:13.203 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T07:28:13.258 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_3 2024-02-10T07:28:13.312 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T07:28:13.313 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:28:13.313 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-02-10T07:28:13.313 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:28:13.313 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:28:13.313 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-02-10 07:27:59.891041612 +0000 2024-02-10T07:28:13.313 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-02-10 07:22:02.548965247 +0000 2024-02-10T07:28:13.313 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-02-10 07:22:02.548965247 +0000 2024-02-10T07:28:13.313 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-02-10 07:22:02.548965247 +0000 2024-02-10T07:28:13.313 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T07:28:13.378 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-02-10T07:28:13.378 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-02-10T07:28:13.378 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.00010143 s, 5.0 MB/s 2024-02-10T07:28:13.380 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T07:28:13.435 DEBUG:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_4 2024-02-10T07:28:13.489 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T07:28:13.489 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:28:13.490 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-10T07:28:13.490 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:28:13.490 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:28:13.490 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2024-02-10 07:27:59.891041612 +0000 2024-02-10T07:28:13.490 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2024-02-10 07:22:02.871961807 +0000 2024-02-10T07:28:13.490 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2024-02-10 07:22:02.871961807 +0000 2024-02-10T07:28:13.490 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: 2024-02-10 07:22:02.871961807 +0000 2024-02-10T07:28:13.490 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T07:28:13.554 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2024-02-10T07:28:13.554 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2024-02-10T07:28:13.554 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 8.8687e-05 s, 5.8 MB/s 2024-02-10T07:28:13.555 DEBUG:teuthology.orchestra.run.smithi008:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T07:28:13.610 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-02-10T07:28:13.610 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T07:28:13.626 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T07:28:13.626 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2024-02-10T07:28:13.683 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T07:28:13.683 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:28:13.683 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-10T07:28:13.683 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:28:13.683 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:28:13.683 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-02-10 07:27:50.643946443 +0000 2024-02-10T07:28:13.683 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-02-10 07:21:58.921591375 +0000 2024-02-10T07:28:13.683 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-02-10 07:21:58.921591375 +0000 2024-02-10T07:28:13.683 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-02-10 07:21:58.921591375 +0000 2024-02-10T07:28:13.683 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T07:28:13.750 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-02-10T07:28:13.750 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-02-10T07:28:13.750 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000107408 s, 4.8 MB/s 2024-02-10T07:28:13.751 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T07:28:13.810 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2024-02-10T07:28:13.865 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T07:28:13.865 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:28:13.865 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-10T07:28:13.865 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:28:13.865 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:28:13.866 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-02-10 07:27:50.643946443 +0000 2024-02-10T07:28:13.866 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-02-10 07:21:59.267583697 +0000 2024-02-10T07:28:13.866 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-02-10 07:21:59.267583697 +0000 2024-02-10T07:28:13.866 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-02-10 07:21:59.267583697 +0000 2024-02-10T07:28:13.866 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T07:28:13.930 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-02-10T07:28:13.930 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-02-10T07:28:13.930 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 9.2437e-05 s, 5.5 MB/s 2024-02-10T07:28:13.931 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T07:28:13.987 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2024-02-10T07:28:14.041 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T07:28:14.042 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:28:14.042 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-10T07:28:14.042 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:28:14.042 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:28:14.042 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-02-10 07:27:50.644946421 +0000 2024-02-10T07:28:14.042 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-02-10 07:21:59.596576395 +0000 2024-02-10T07:28:14.042 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-02-10 07:21:59.596576395 +0000 2024-02-10T07:28:14.042 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-02-10 07:21:59.596576395 +0000 2024-02-10T07:28:14.042 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T07:28:14.107 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-02-10T07:28:14.108 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-02-10T07:28:14.108 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000110537 s, 4.6 MB/s 2024-02-10T07:28:14.109 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T07:28:14.165 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2024-02-10T07:28:14.222 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T07:28:14.222 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T07:28:14.222 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-10T07:28:14.222 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T07:28:14.223 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T07:28:14.223 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-02-10 07:27:50.644946421 +0000 2024-02-10T07:28:14.223 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-02-10 07:21:59.924569116 +0000 2024-02-10T07:28:14.223 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-02-10 07:21:59.924569116 +0000 2024-02-10T07:28:14.223 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-02-10 07:21:59.924569116 +0000 2024-02-10T07:28:14.223 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T07:28:14.290 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-02-10T07:28:14.290 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-02-10T07:28:14.290 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000115007 s, 4.5 MB/s 2024-02-10T07:28:14.292 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T07:28:14.348 INFO:tasks.cephadm:Deploying osd.0 on smithi008 with /dev/vg_nvme/lv_4... 2024-02-10T07:28:14.348 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T07:28:14.471 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:14 smithi170 ceph-mon[33394]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:14.471 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:14 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:14.471 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:14 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:14.471 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:14 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:14.471 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:14 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:14.471 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:14 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:14.471 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:14 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:14.471 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:14 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:28:14.472 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:14 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T07:28:14.472 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:14 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:28:14.472 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:14 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:14.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[29816]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:28:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T07:28:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:28:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:14.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[40743]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:14.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:14.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:14.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:14.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:14.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:14.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:14.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:28:14.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T07:28:14.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:28:14.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:14 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:14.597 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:15.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:15 smithi008 ceph-mon[29816]: Deploying daemon mgr.x on smithi170 2024-02-10T07:28:15.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:15 smithi008 ceph-mon[40743]: Deploying daemon mgr.x on smithi170 2024-02-10T07:28:15.348 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:15 smithi170 ceph-mon[33394]: Deploying daemon mgr.x on smithi170 2024-02-10T07:28:16.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:16 smithi170 ceph-mon[33394]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:16.383 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:16 smithi170 systemd[1]: Started Ceph mgr.x for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:28:16.459 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T07:28:16.460 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T07:28:16.460 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-02-10T07:28:16.460 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-02-10T07:28:16.460 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00845085 s, 1.2 GB/s 2024-02-10T07:28:16.460 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-02-10T07:28:16.484 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:16 smithi008 ceph-mon[29816]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:16.484 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:16 smithi008 ceph-mon[40743]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:17.081 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi008:vg_nvme/lv_4 2024-02-10T07:28:17.133 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:16 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:16.733+0000 7f051594d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:28:17.133 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:16 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:16.829+0000 7f051594d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:28:17.320 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:17.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:17 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:17 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:17 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:17 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:17 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:17.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:17 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:17 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:17 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:17 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:17 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:17 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:17.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:17 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:17 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:17 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:17 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:17 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:17 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:17 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:17.880 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:17 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:17.533+0000 7f051594d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:28:17.880 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:17 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:17.630+0000 7f051594d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:28:17.880 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:17 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:17.783+0000 7f051594d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:28:17.880 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:17 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:17.878+0000 7f051594d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:28:18.133 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:17 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:17.961+0000 7f051594d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:28:18.518 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:18 smithi170 ceph-mon[33394]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:18.518 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:18 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:18.518 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:18 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:18.518 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:18 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:18.518 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:18 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:18.519 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:18 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:18.519 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:18 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:28:18.519 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:18 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:28:18.519 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:18 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:18.519 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:18 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:18.228+0000 7f051594d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T07:28:18.519 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:18 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:18.338+0000 7f051594d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:28:18.519 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:18 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:18.423+0000 7f051594d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:28:18.519 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:18 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:18.517+0000 7f051594d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:28:18.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[29816]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:18.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:28:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:28:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:18.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[40743]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:18.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:18.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:18.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:18.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:18.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:18.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T07:28:18.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T07:28:18.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:18 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:18.883 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:18 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:18.693+0000 7f051594d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:28:19.526 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:19 smithi008 ceph-mon[29816]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T07:28:19.526 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:19 smithi008 ceph-mon[29816]: Reconfiguring daemon mgr.y on smithi008 2024-02-10T07:28:19.527 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:19 smithi008 ceph-mon[40743]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T07:28:19.527 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:19 smithi008 ceph-mon[40743]: Reconfiguring daemon mgr.y on smithi008 2024-02-10T07:28:19.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:19 smithi170 ceph-mon[33394]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T07:28:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:19 smithi170 ceph-mon[33394]: Reconfiguring daemon mgr.y on smithi008 2024-02-10T07:28:20.127 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:19 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:19.664+0000 7f051594d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:28:20.128 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:19 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:19.794+0000 7f051594d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:28:20.128 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:20.127+0000 7f051594d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:28:20.383 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:20.218+0000 7f051594d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:28:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-mon[33394]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:28:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:28:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:20.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:20.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:20.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:20.686 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[29816]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:20.686 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:20.686 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:20.686 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:20.687 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:28:20.687 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:28:20.687 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:20.687 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:20.687 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:20.687 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:20.687 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[40743]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:20.687 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:20.687 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:20.687 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:20.687 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:28:20.687 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:28:20.687 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:20.687 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:20.688 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:20.688 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:20 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:20.882 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:20.413+0000 7f051594d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:28:20.882 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:20.598+0000 7f051594d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:28:21.133 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:20.882+0000 7f051594d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:28:21.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:21 smithi008 ceph-mon[29816]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:28:21.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:21 smithi008 ceph-mon[40743]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:28:21.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:21 smithi170 ceph-mon[33394]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:28:21.633 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:21 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:21.143+0000 7f051594d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:28:21.633 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:21 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:21.376+0000 7f051594d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:28:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:22 smithi170 ceph-mon[33394]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:22.633 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:22.171+0000 7f051594d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:28:22.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:22 smithi008 ceph-mon[29816]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:22.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:22 smithi008 ceph-mon[40743]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:23.054 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:22.742+0000 7f051594d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:28:23.055 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:22.839+0000 7f051594d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:28:23.055 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:28:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:28:23.054+0000 7f051594d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:28:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:23 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2178214086' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63abc4f1-1c83-4e03-8592-959c51d10847"}]: dispatch 2024-02-10T07:28:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:23 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2178214086' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63abc4f1-1c83-4e03-8592-959c51d10847"}]': finished 2024-02-10T07:28:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:23 smithi170 ceph-mon[33394]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T07:28:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:23 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:23 smithi170 ceph-mon[33394]: Standby manager daemon x started 2024-02-10T07:28:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:23 smithi170 ceph-mon[33394]: from='mgr.? 172.21.15.170:0/106024408' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:28:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:23 smithi170 ceph-mon[33394]: from='mgr.? 172.21.15.170:0/106024408' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:28:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:23 smithi170 ceph-mon[33394]: from='mgr.? 172.21.15.170:0/106024408' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:28:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:23 smithi170 ceph-mon[33394]: from='mgr.? 172.21.15.170:0/106024408' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:28:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2178214086' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63abc4f1-1c83-4e03-8592-959c51d10847"}]: dispatch 2024-02-10T07:28:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2178214086' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63abc4f1-1c83-4e03-8592-959c51d10847"}]': finished 2024-02-10T07:28:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[29816]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T07:28:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[29816]: Standby manager daemon x started 2024-02-10T07:28:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[29816]: from='mgr.? 172.21.15.170:0/106024408' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:28:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[29816]: from='mgr.? 172.21.15.170:0/106024408' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:28:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[29816]: from='mgr.? 172.21.15.170:0/106024408' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:28:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[29816]: from='mgr.? 172.21.15.170:0/106024408' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:28:23.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2178214086' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63abc4f1-1c83-4e03-8592-959c51d10847"}]: dispatch 2024-02-10T07:28:23.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2178214086' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63abc4f1-1c83-4e03-8592-959c51d10847"}]': finished 2024-02-10T07:28:23.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[40743]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T07:28:23.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:23.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[40743]: Standby manager daemon x started 2024-02-10T07:28:23.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[40743]: from='mgr.? 172.21.15.170:0/106024408' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:28:23.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[40743]: from='mgr.? 172.21.15.170:0/106024408' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:28:23.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[40743]: from='mgr.? 172.21.15.170:0/106024408' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:28:23.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:23 smithi008 ceph-mon[40743]: from='mgr.? 172.21.15.170:0/106024408' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:28:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:24 smithi170 ceph-mon[33394]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:24 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/226120051' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:28:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:24 smithi170 ceph-mon[33394]: mgrmap e14: y(active, since 67s), standbys: x 2024-02-10T07:28:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:24 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:28:24.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:24 smithi008 ceph-mon[29816]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:24.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:24 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/226120051' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:28:24.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:24 smithi008 ceph-mon[29816]: mgrmap e14: y(active, since 67s), standbys: x 2024-02-10T07:28:24.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:24 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:28:24.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:24 smithi008 ceph-mon[40743]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:24.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:24 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/226120051' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:28:24.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:24 smithi008 ceph-mon[40743]: mgrmap e14: y(active, since 67s), standbys: x 2024-02-10T07:28:24.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:24 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:28:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:26 smithi170 ceph-mon[33394]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:26.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:26 smithi008 ceph-mon[29816]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:26.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:26 smithi008 ceph-mon[40743]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:28.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:28 smithi008 ceph-mon[29816]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:28 smithi008 ceph-mon[40743]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:28.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:28 smithi170 ceph-mon[33394]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:29.351 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:29 smithi008 ceph-mon[29816]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:29.351 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:29 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T07:28:29.351 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:29 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:29.351 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:29 smithi008 ceph-mon[29816]: Deploying daemon osd.0 on smithi008 2024-02-10T07:28:29.352 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:29 smithi008 ceph-mon[40743]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:29.352 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:29 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T07:28:29.352 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:29 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:29.352 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:29 smithi008 ceph-mon[40743]: Deploying daemon osd.0 on smithi008 2024-02-10T07:28:29.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:29 smithi170 ceph-mon[33394]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:29.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:29 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T07:28:29.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:29 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:29.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:29 smithi170 ceph-mon[33394]: Deploying daemon osd.0 on smithi008 2024-02-10T07:28:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:31 smithi008 ceph-mon[29816]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:31.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:31 smithi008 ceph-mon[40743]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:31.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:31 smithi170 ceph-mon[33394]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:33.742 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:33 smithi008 ceph-mon[29816]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:33.743 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:33 smithi008 ceph-mon[40743]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:33.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:33 smithi170 ceph-mon[33394]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:36.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:35 smithi008 ceph-mon[29816]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:36.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:36.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:36.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:35 smithi008 ceph-mon[40743]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:35 smithi170 ceph-mon[33394]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:38.065 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:37 smithi008 ceph-mon[40743]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:38.065 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:38.065 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:38.065 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:38.065 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:38.066 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:38.066 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:37 smithi008 ceph-mon[29816]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:38.066 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:38.066 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:38.066 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:38.066 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:38.066 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:37 smithi170 ceph-mon[33394]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:38.494 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 0 on host 'smithi008' 2024-02-10T07:28:39.063 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:38 smithi008 ceph-mon[40743]: from='osd.0 [v2:172.21.15.8:6802/1442795455,v1:172.21.15.8:6803/1442795455]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:28:39.063 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:38 smithi008 ceph-mon[40743]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:28:39.063 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:38 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:39.063 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:38 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:39.064 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:38 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:39.064 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:38 smithi008 ceph-mon[29816]: from='osd.0 [v2:172.21.15.8:6802/1442795455,v1:172.21.15.8:6803/1442795455]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:28:39.064 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:38 smithi008 ceph-mon[29816]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:28:39.064 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:38 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:39.064 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:38 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:39.064 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:38 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:39.088 DEBUG:teuthology.orchestra.run.smithi008:osd.0> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.0.service 2024-02-10T07:28:39.091 INFO:tasks.cephadm:Deploying osd.1 on smithi008 with /dev/vg_nvme/lv_3... 2024-02-10T07:28:39.091 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T07:28:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:38 smithi170 ceph-mon[33394]: from='osd.0 [v2:172.21.15.8:6802/1442795455,v1:172.21.15.8:6803/1442795455]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:28:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:38 smithi170 ceph-mon[33394]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T07:28:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:38 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:38 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:38 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:28:39.792 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:40.187 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:39 smithi008 ceph-mon[40743]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:40.187 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:39 smithi008 ceph-mon[40743]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T07:28:40.187 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:39 smithi008 ceph-mon[40743]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T07:28:40.187 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:40.187 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:39 smithi008 ceph-mon[40743]: from='osd.0 [v2:172.21.15.8:6802/1442795455,v1:172.21.15.8:6803/1442795455]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:28:40.187 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:39 smithi008 ceph-mon[40743]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:28:40.188 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:39 smithi008 ceph-mon[29816]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:40.188 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:39 smithi008 ceph-mon[29816]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T07:28:40.188 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:39 smithi008 ceph-mon[29816]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T07:28:40.188 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:40.188 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:39 smithi008 ceph-mon[29816]: from='osd.0 [v2:172.21.15.8:6802/1442795455,v1:172.21.15.8:6803/1442795455]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:28:40.188 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:39 smithi008 ceph-mon[29816]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:28:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:39 smithi170 ceph-mon[33394]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:39 smithi170 ceph-mon[33394]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T07:28:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:39 smithi170 ceph-mon[33394]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T07:28:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:39 smithi170 ceph-mon[33394]: from='osd.0 [v2:172.21.15.8:6802/1442795455,v1:172.21.15.8:6803/1442795455]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:28:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:39 smithi170 ceph-mon[33394]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:28:40.582 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:28:40 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-0[44764]: 2024-02-10T07:28:40.242+0000 7fd18f57b700 -1 osd.0 0 waiting for initial osdmap 2024-02-10T07:28:40.619 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:28:40 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-0[44764]: 2024-02-10T07:28:40.244+0000 7fd187366700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:28:41.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:40 smithi008 ceph-mon[29816]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T07:28:41.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:40 smithi008 ceph-mon[29816]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T07:28:41.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:40 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:41.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:40 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:41.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:40 smithi008 ceph-mon[29816]: from='osd.0 ' entity='osd.0' 2024-02-10T07:28:41.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:40 smithi008 ceph-mon[40743]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T07:28:41.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:40 smithi008 ceph-mon[40743]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T07:28:41.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:40 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:41.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:40 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:41.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:40 smithi008 ceph-mon[40743]: from='osd.0 ' entity='osd.0' 2024-02-10T07:28:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:40 smithi170 ceph-mon[33394]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T07:28:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:40 smithi170 ceph-mon[33394]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T07:28:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:40 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:40 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:40 smithi170 ceph-mon[33394]: from='osd.0 ' entity='osd.0' 2024-02-10T07:28:42.176 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:41 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:28:42.176 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:41 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:28:42.176 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:41 smithi008 ceph-mon[40743]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:42.176 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:41 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:42.176 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:41 smithi008 ceph-mon[40743]: osd.0 [v2:172.21.15.8:6802/1442795455,v1:172.21.15.8:6803/1442795455] boot 2024-02-10T07:28:42.176 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:41 smithi008 ceph-mon[40743]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T07:28:42.176 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:41 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:42.177 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:41 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:28:42.177 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:41 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:28:42.177 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:41 smithi008 ceph-mon[29816]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:42.177 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:41 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:42.177 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:41 smithi008 ceph-mon[29816]: osd.0 [v2:172.21.15.8:6802/1442795455,v1:172.21.15.8:6803/1442795455] boot 2024-02-10T07:28:42.177 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:41 smithi008 ceph-mon[29816]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T07:28:42.177 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:41 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:41 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:28:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:41 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:28:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:41 smithi170 ceph-mon[33394]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T07:28:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:41 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:41 smithi170 ceph-mon[33394]: osd.0 [v2:172.21.15.8:6802/1442795455,v1:172.21.15.8:6803/1442795455] boot 2024-02-10T07:28:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:41 smithi170 ceph-mon[33394]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T07:28:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:41 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:28:42.528 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T07:28:42.529 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T07:28:42.529 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-02-10T07:28:42.529 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-02-10T07:28:42.529 INFO:teuthology.orchestra.run.smithi008.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109552 s, 957 MB/s 2024-02-10T07:28:42.529 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-02-10T07:28:43.675 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[40743]: pgmap v41: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:43.675 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[40743]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T07:28:43.675 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[40743]: Detected new or changed devices on smithi008 2024-02-10T07:28:43.676 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:43.676 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:43.676 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:28:43.676 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[40743]: Adjusting osd_memory_target on smithi008 to 16069M 2024-02-10T07:28:43.676 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:43.676 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:43.676 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:43.676 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:43.676 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[29816]: pgmap v41: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:43.676 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[29816]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T07:28:43.677 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[29816]: Detected new or changed devices on smithi008 2024-02-10T07:28:43.677 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:43.677 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:43.677 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:28:43.677 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[29816]: Adjusting osd_memory_target on smithi008 to 16069M 2024-02-10T07:28:43.677 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:43.677 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:43.677 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:43.677 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:43 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:43.704 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi008:vg_nvme/lv_3 2024-02-10T07:28:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:43 smithi170 ceph-mon[33394]: pgmap v41: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:43 smithi170 ceph-mon[33394]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T07:28:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:43 smithi170 ceph-mon[33394]: Detected new or changed devices on smithi008 2024-02-10T07:28:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:43 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:43 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:43 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:28:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:43 smithi170 ceph-mon[33394]: Adjusting osd_memory_target on smithi008 to 16069M 2024-02-10T07:28:43.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:43 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:43.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:43 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:43.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:43 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:28:43.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:43 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:28:43.945 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:28:45.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:45 smithi008 ceph-mon[29816]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:45 smithi008 ceph-mon[40743]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:45.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:45 smithi170 ceph-mon[33394]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:46.696 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:46 smithi008 ceph-mon[29816]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:28:46.696 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:46 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:28:46.696 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:46 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:28:46.697 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:46 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:46.697 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:46 smithi008 ceph-mon[40743]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:28:46.697 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:46 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:28:46.697 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:46 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:28:46.697 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:46 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:46.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:46 smithi170 ceph-mon[33394]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:28:46.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:46 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:28:46.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:46 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:28:46.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:46 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:47.626 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:47 smithi008 ceph-mon[29816]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:47.626 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:47 smithi008 ceph-mon[40743]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:47 smithi170 ceph-mon[33394]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:48.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:48 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/638511727' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88307389-bfa8-402e-92b9-9fa5094262c4"}]: dispatch 2024-02-10T07:28:48.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:48 smithi008 ceph-mon[29816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88307389-bfa8-402e-92b9-9fa5094262c4"}]: dispatch 2024-02-10T07:28:48.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:48 smithi008 ceph-mon[29816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88307389-bfa8-402e-92b9-9fa5094262c4"}]': finished 2024-02-10T07:28:48.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:48 smithi008 ceph-mon[29816]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T07:28:48.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:48 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:28:48.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:48 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/638511727' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88307389-bfa8-402e-92b9-9fa5094262c4"}]: dispatch 2024-02-10T07:28:48.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:48 smithi008 ceph-mon[40743]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88307389-bfa8-402e-92b9-9fa5094262c4"}]: dispatch 2024-02-10T07:28:48.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:48 smithi008 ceph-mon[40743]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88307389-bfa8-402e-92b9-9fa5094262c4"}]': finished 2024-02-10T07:28:48.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:48 smithi008 ceph-mon[40743]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T07:28:48.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:48 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:28:48.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:48 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/638511727' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88307389-bfa8-402e-92b9-9fa5094262c4"}]: dispatch 2024-02-10T07:28:48.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:48 smithi170 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88307389-bfa8-402e-92b9-9fa5094262c4"}]: dispatch 2024-02-10T07:28:48.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:48 smithi170 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88307389-bfa8-402e-92b9-9fa5094262c4"}]': finished 2024-02-10T07:28:48.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:48 smithi170 ceph-mon[33394]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T07:28:48.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:48 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:28:49.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:49 smithi008 ceph-mon[29816]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:49 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3273013672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:28:49.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:49 smithi008 ceph-mon[40743]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:49.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:49 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3273013672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:28:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:49 smithi170 ceph-mon[33394]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:49 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3273013672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:28:51.700 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:51 smithi008 ceph-mon[29816]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:51.700 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:51 smithi008 ceph-mon[40743]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:51.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:51 smithi170 ceph-mon[33394]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:53.636 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:53 smithi008 ceph-mon[29816]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:53.636 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:53 smithi008 ceph-mon[40743]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:53.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:53 smithi170 ceph-mon[33394]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:55.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:55 smithi008 ceph-mon[29816]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:55.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:55 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T07:28:55.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:55 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:55.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:55 smithi008 ceph-mon[29816]: Deploying daemon osd.1 on smithi008 2024-02-10T07:28:55.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:55 smithi008 ceph-mon[40743]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:55.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:55 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T07:28:55.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:55 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:55.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:55 smithi008 ceph-mon[40743]: Deploying daemon osd.1 on smithi008 2024-02-10T07:28:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:55 smithi170 ceph-mon[33394]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:55 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T07:28:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:55 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:28:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:55 smithi170 ceph-mon[33394]: Deploying daemon osd.1 on smithi008 2024-02-10T07:28:57.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:57 smithi170 ceph-mon[33394]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:57.986 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:57 smithi008 ceph-mon[29816]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:57.986 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:57 smithi008 ceph-mon[40743]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:59.880 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:28:59 smithi008 ceph-mon[29816]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:59.881 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:28:59 smithi008 ceph-mon[40743]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:28:59.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:28:59 smithi170 ceph-mon[33394]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:29:00.682 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:00.682 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:00.682 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:00.682 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:00.682 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:00.682 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:00.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:00.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:00.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:02.306 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:02 smithi008 ceph-mon[40743]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:29:02.306 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:02 smithi008 ceph-mon[29816]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:29:02.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:02 smithi170 ceph-mon[33394]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:29:03.997 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 1 on host 'smithi008' 2024-02-10T07:29:04.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[29816]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:29:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[29816]: from='osd.1 [v2:172.21.15.8:6810/3278758893,v1:172.21.15.8:6811/3278758893]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:29:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[29816]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:29:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:04.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[40743]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:29:04.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:04.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:04.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[40743]: from='osd.1 [v2:172.21.15.8:6810/3278758893,v1:172.21.15.8:6811/3278758893]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:29:04.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[40743]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:29:04.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:04 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:04 smithi170 ceph-mon[33394]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:29:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:04 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:04 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:04 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:04 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:04 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:04 smithi170 ceph-mon[33394]: from='osd.1 [v2:172.21.15.8:6810/3278758893,v1:172.21.15.8:6811/3278758893]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:29:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:04 smithi170 ceph-mon[33394]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T07:29:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:04 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:04 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:04 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:04.662 DEBUG:teuthology.orchestra.run.smithi008:osd.1> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.1.service 2024-02-10T07:29:04.664 INFO:tasks.cephadm:Deploying osd.2 on smithi008 with /dev/vg_nvme/lv_2... 2024-02-10T07:29:04.664 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T07:29:05.206 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:05 smithi008 ceph-mon[29816]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T07:29:05.206 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:05 smithi008 ceph-mon[29816]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T07:29:05.206 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:05.207 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:05 smithi008 ceph-mon[29816]: from='osd.1 [v2:172.21.15.8:6810/3278758893,v1:172.21.15.8:6811/3278758893]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:05.207 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:05 smithi008 ceph-mon[29816]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:05.207 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:05 smithi008 ceph-mon[40743]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T07:29:05.207 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:05 smithi008 ceph-mon[40743]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T07:29:05.208 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:05.208 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:05 smithi008 ceph-mon[40743]: from='osd.1 [v2:172.21.15.8:6810/3278758893,v1:172.21.15.8:6811/3278758893]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:05.208 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:05 smithi008 ceph-mon[40743]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:05 smithi170 ceph-mon[33394]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T07:29:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:05 smithi170 ceph-mon[33394]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T07:29:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:05 smithi170 ceph-mon[33394]: from='osd.1 [v2:172.21.15.8:6810/3278758893,v1:172.21.15.8:6811/3278758893]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:05.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:05 smithi170 ceph-mon[33394]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:05.479 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:29:05.575 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:29:05 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-1[50308]: 2024-02-10T07:29:05.304+0000 7ff867743700 -1 osd.1 0 waiting for initial osdmap 2024-02-10T07:29:05.576 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:29:05 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-1[50308]: 2024-02-10T07:29:05.306+0000 7ff85f52e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:29:06.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:06 smithi008 ceph-mon[29816]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:29:06.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:06 smithi008 ceph-mon[29816]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T07:29:06.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:06 smithi008 ceph-mon[29816]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T07:29:06.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:06 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:06.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:06 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:06.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:06 smithi008 ceph-mon[29816]: from='osd.1 ' entity='osd.1' 2024-02-10T07:29:06.162 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:06 smithi008 ceph-mon[40743]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:29:06.163 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:06 smithi008 ceph-mon[40743]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T07:29:06.163 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:06 smithi008 ceph-mon[40743]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T07:29:06.163 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:06 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:06.163 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:06 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:06.163 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:06 smithi008 ceph-mon[40743]: from='osd.1 ' entity='osd.1' 2024-02-10T07:29:06.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:06 smithi170 ceph-mon[33394]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:29:06.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:06 smithi170 ceph-mon[33394]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T07:29:06.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:06 smithi170 ceph-mon[33394]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T07:29:06.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:06 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:06.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:06 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:06.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:06 smithi170 ceph-mon[33394]: from='osd.1 ' entity='osd.1' 2024-02-10T07:29:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:07 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:29:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:07 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:29:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:07 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:07 smithi008 ceph-mon[40743]: osd.1 [v2:172.21.15.8:6810/3278758893,v1:172.21.15.8:6811/3278758893] boot 2024-02-10T07:29:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:07 smithi008 ceph-mon[40743]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T07:29:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:07 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:07 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:29:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:07 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:29:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:07 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:07.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:07 smithi008 ceph-mon[29816]: osd.1 [v2:172.21.15.8:6810/3278758893,v1:172.21.15.8:6811/3278758893] boot 2024-02-10T07:29:07.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:07 smithi008 ceph-mon[29816]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T07:29:07.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:07 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:07.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:07 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:29:07.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:07 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:29:07.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:07 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:07.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:07 smithi170 ceph-mon[33394]: osd.1 [v2:172.21.15.8:6810/3278758893,v1:172.21.15.8:6811/3278758893] boot 2024-02-10T07:29:07.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:07 smithi170 ceph-mon[33394]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T07:29:07.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:07 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:29:08.480 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T07:29:08.480 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T07:29:08.481 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-02-10T07:29:08.481 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-02-10T07:29:08.481 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00821789 s, 1.3 GB/s 2024-02-10T07:29:08.481 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-02-10T07:29:08.569 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:08 smithi008 ceph-mon[29816]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:29:08.569 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:08 smithi008 ceph-mon[29816]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T07:29:08.570 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:08 smithi008 ceph-mon[40743]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:29:08.570 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:08 smithi008 ceph-mon[40743]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T07:29:08.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:08 smithi170 ceph-mon[33394]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T07:29:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:08 smithi170 ceph-mon[33394]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T07:29:09.135 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi008:vg_nvme/lv_2 2024-02-10T07:29:09.378 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:29:09.827 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[29816]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:09.827 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[29816]: Detected new or changed devices on smithi008 2024-02-10T07:29:09.827 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:09.827 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:09.827 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:09.828 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:09.828 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[29816]: Adjusting osd_memory_target on smithi008 to 8034M 2024-02-10T07:29:09.828 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:09.828 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:09.828 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:09.828 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:09.828 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[40743]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:09.828 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[40743]: Detected new or changed devices on smithi008 2024-02-10T07:29:09.828 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:09.828 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:09.829 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:09.829 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:09.829 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[40743]: Adjusting osd_memory_target on smithi008 to 8034M 2024-02-10T07:29:09.829 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:09.829 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:09.829 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:09.829 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:10.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:09 smithi170 ceph-mon[33394]: pgmap v60: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:10.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:09 smithi170 ceph-mon[33394]: Detected new or changed devices on smithi008 2024-02-10T07:29:10.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:10.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:10.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:10.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:10.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:09 smithi170 ceph-mon[33394]: Adjusting osd_memory_target on smithi008 to 8034M 2024-02-10T07:29:10.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:10.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:10.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:10.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:11.899 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:11 smithi008 ceph-mon[29816]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:11.900 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:11 smithi008 ceph-mon[29816]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:11.900 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:11 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:29:11.900 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:11 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:29:11.900 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:11 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:11.900 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:11 smithi008 ceph-mon[40743]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:11.900 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:11 smithi008 ceph-mon[40743]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:11.900 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:11 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:29:11.900 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:11 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:29:11.900 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:11 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:12.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:11 smithi170 ceph-mon[33394]: pgmap v61: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:12.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:11 smithi170 ceph-mon[33394]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:12.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:11 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:29:12.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:11 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:29:12.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:11 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:14.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:13 smithi008 ceph-mon[29816]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:14.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:13 smithi008 ceph-mon[40743]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:14.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:13 smithi170 ceph-mon[33394]: pgmap v62: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:15.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:14 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4206874445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6120003-9ab5-4cf5-87ac-6351ed2c6226"}]: dispatch 2024-02-10T07:29:15.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:14 smithi008 ceph-mon[29816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6120003-9ab5-4cf5-87ac-6351ed2c6226"}]: dispatch 2024-02-10T07:29:15.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:14 smithi008 ceph-mon[29816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6120003-9ab5-4cf5-87ac-6351ed2c6226"}]': finished 2024-02-10T07:29:15.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:14 smithi008 ceph-mon[29816]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T07:29:15.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:14 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:15.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:14 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1861848968' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:29:15.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:14 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4206874445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6120003-9ab5-4cf5-87ac-6351ed2c6226"}]: dispatch 2024-02-10T07:29:15.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:14 smithi008 ceph-mon[40743]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6120003-9ab5-4cf5-87ac-6351ed2c6226"}]: dispatch 2024-02-10T07:29:15.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:14 smithi008 ceph-mon[40743]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6120003-9ab5-4cf5-87ac-6351ed2c6226"}]': finished 2024-02-10T07:29:15.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:14 smithi008 ceph-mon[40743]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T07:29:15.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:14 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:15.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:14 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1861848968' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:29:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:14 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4206874445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6120003-9ab5-4cf5-87ac-6351ed2c6226"}]: dispatch 2024-02-10T07:29:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:14 smithi170 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c6120003-9ab5-4cf5-87ac-6351ed2c6226"}]: dispatch 2024-02-10T07:29:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:14 smithi170 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c6120003-9ab5-4cf5-87ac-6351ed2c6226"}]': finished 2024-02-10T07:29:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:14 smithi170 ceph-mon[33394]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T07:29:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:14 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:14 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1861848968' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:29:16.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:15 smithi170 ceph-mon[33394]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:16.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:15 smithi008 ceph-mon[29816]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:16.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:15 smithi008 ceph-mon[40743]: pgmap v64: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:18.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:17 smithi170 ceph-mon[33394]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:18.177 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:17 smithi008 ceph-mon[29816]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:18.177 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:17 smithi008 ceph-mon[40743]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:20.318 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:19 smithi008 ceph-mon[29816]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:20.319 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:19 smithi008 ceph-mon[40743]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:19 smithi170 ceph-mon[33394]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:21.148 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T07:29:21.148 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:21.148 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:20 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T07:29:21.148 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:20 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:21.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:20 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T07:29:21.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:20 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:22.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:21 smithi008 ceph-mon[29816]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:22.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:21 smithi008 ceph-mon[29816]: Deploying daemon osd.2 on smithi008 2024-02-10T07:29:22.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:21 smithi008 ceph-mon[40743]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:22.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:21 smithi008 ceph-mon[40743]: Deploying daemon osd.2 on smithi008 2024-02-10T07:29:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:21 smithi170 ceph-mon[33394]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:21 smithi170 ceph-mon[33394]: Deploying daemon osd.2 on smithi008 2024-02-10T07:29:24.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:23 smithi008 ceph-mon[29816]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:24.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:23 smithi008 ceph-mon[40743]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:23 smithi170 ceph-mon[33394]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:26.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:25 smithi008 ceph-mon[29816]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:26.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:25 smithi008 ceph-mon[40743]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:26.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:25 smithi170 ceph-mon[33394]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:27.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:27 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:27.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:27 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:27.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:27 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:27 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:27 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:27 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:27 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:27 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:27 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:28.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:28 smithi008 ceph-mon[29816]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:28.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:28 smithi008 ceph-mon[40743]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:28 smithi170 ceph-mon[33394]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:29.789 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 2 on host 'smithi008' 2024-02-10T07:29:30.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:29 smithi008 ceph-mon[29816]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:29 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:29 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:29 smithi008 ceph-mon[29816]: from='osd.2 [v2:172.21.15.8:6818/481314802,v1:172.21.15.8:6819/481314802]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:29:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:29 smithi008 ceph-mon[29816]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:29:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:29 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:29 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:30.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:29 smithi008 ceph-mon[40743]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:30.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:29 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:29 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:29 smithi008 ceph-mon[40743]: from='osd.2 [v2:172.21.15.8:6818/481314802,v1:172.21.15.8:6819/481314802]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:29:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:29 smithi008 ceph-mon[40743]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:29:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:29 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:29 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:29 smithi170 ceph-mon[33394]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:29 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:29 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:29 smithi170 ceph-mon[33394]: from='osd.2 [v2:172.21.15.8:6818/481314802,v1:172.21.15.8:6819/481314802]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:29:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:29 smithi170 ceph-mon[33394]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T07:29:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:29 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:29 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:30.428 DEBUG:teuthology.orchestra.run.smithi008:osd.2> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.2.service 2024-02-10T07:29:30.431 INFO:tasks.cephadm:Deploying osd.3 on smithi008 with /dev/vg_nvme/lv_1... 2024-02-10T07:29:30.431 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T07:29:31.355 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:31 smithi008 ceph-mon[40743]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T07:29:31.355 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:31 smithi008 ceph-mon[40743]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T07:29:31.356 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:31 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:31.356 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:31 smithi008 ceph-mon[40743]: from='osd.2 [v2:172.21.15.8:6818/481314802,v1:172.21.15.8:6819/481314802]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:31.356 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:31 smithi008 ceph-mon[40743]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:31.356 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:31 smithi008 ceph-mon[29816]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T07:29:31.356 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:31 smithi008 ceph-mon[29816]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T07:29:31.356 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:31 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:31.356 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:31 smithi008 ceph-mon[29816]: from='osd.2 [v2:172.21.15.8:6818/481314802,v1:172.21.15.8:6819/481314802]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:31.356 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:31 smithi008 ceph-mon[29816]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:31 smithi170 ceph-mon[33394]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T07:29:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:31 smithi170 ceph-mon[33394]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T07:29:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:31 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:31 smithi170 ceph-mon[33394]: from='osd.2 [v2:172.21.15.8:6818/481314802,v1:172.21.15.8:6819/481314802]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:31 smithi170 ceph-mon[33394]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:31.497 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:29:31.620 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:29:31 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-2[56371]: 2024-02-10T07:29:31.355+0000 7fee48735700 -1 osd.2 0 waiting for initial osdmap 2024-02-10T07:29:31.620 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:29:31 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-2[56371]: 2024-02-10T07:29:31.358+0000 7fee40520700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:29:32.277 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:32 smithi008 ceph-mon[29816]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:32.277 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:32 smithi008 ceph-mon[29816]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T07:29:32.277 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:32 smithi008 ceph-mon[29816]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T07:29:32.277 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:32 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:32.277 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:32 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:32.278 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:32 smithi008 ceph-mon[29816]: from='osd.2 ' entity='osd.2' 2024-02-10T07:29:32.278 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:32 smithi008 ceph-mon[40743]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:32.278 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:32 smithi008 ceph-mon[40743]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T07:29:32.278 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:32 smithi008 ceph-mon[40743]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T07:29:32.278 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:32 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:32.278 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:32 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:32.278 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:32 smithi008 ceph-mon[40743]: from='osd.2 ' entity='osd.2' 2024-02-10T07:29:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:32 smithi170 ceph-mon[33394]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:32 smithi170 ceph-mon[33394]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T07:29:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:32 smithi170 ceph-mon[33394]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T07:29:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:32 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:32 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:32 smithi170 ceph-mon[33394]: from='osd.2 ' entity='osd.2' 2024-02-10T07:29:33.376 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:33 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:29:33.377 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:33 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:29:33.377 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:33 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:33.377 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:33 smithi008 ceph-mon[29816]: osd.2 [v2:172.21.15.8:6818/481314802,v1:172.21.15.8:6819/481314802] boot 2024-02-10T07:29:33.377 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:33 smithi008 ceph-mon[29816]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T07:29:33.377 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:33 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:33.377 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:33 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:29:33.377 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:33 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:29:33.377 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:33 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:29:33.377 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:33 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:33.377 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:33 smithi008 ceph-mon[40743]: osd.2 [v2:172.21.15.8:6818/481314802,v1:172.21.15.8:6819/481314802] boot 2024-02-10T07:29:33.377 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:33 smithi008 ceph-mon[40743]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T07:29:33.377 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:33 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:33.378 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:33 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:29:33.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:33 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:29:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:33 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:29:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:33 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:33 smithi170 ceph-mon[33394]: osd.2 [v2:172.21.15.8:6818/481314802,v1:172.21.15.8:6819/481314802] boot 2024-02-10T07:29:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:33 smithi170 ceph-mon[33394]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T07:29:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:33 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:29:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:33 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:29:33.873 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T07:29:33.873 INFO:teuthology.orchestra.run.smithi008.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T07:29:33.873 INFO:teuthology.orchestra.run.smithi008.stderr: stderr: 10+0 records in 2024-02-10T07:29:33.873 INFO:teuthology.orchestra.run.smithi008.stderr:10+0 records out 2024-02-10T07:29:33.874 INFO:teuthology.orchestra.run.smithi008.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00947905 s, 1.1 GB/s 2024-02-10T07:29:33.874 INFO:teuthology.orchestra.run.smithi008.stderr:--> Zapping successful for: 2024-02-10T07:29:34.481 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:34 smithi008 ceph-mon[29816]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:34.481 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:34 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-10T07:29:34.481 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:34 smithi008 ceph-mon[29816]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T07:29:34.481 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:34 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:29:34.481 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:34 smithi008 ceph-mon[40743]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:34.481 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:34 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-10T07:29:34.482 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:34 smithi008 ceph-mon[40743]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T07:29:34.482 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:34 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:29:34.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:34 smithi170 ceph-mon[33394]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-02-10T07:29:34.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:34 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-10T07:29:34.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:34 smithi170 ceph-mon[33394]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T07:29:34.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:34 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:29:34.796 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi008:vg_nvme/lv_1 2024-02-10T07:29:35.283 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:29:35.461 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:29:35 smithi008 sudo[60620]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T07:29:35.461 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:29:35 smithi008 sudo[60620]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T07:29:35.461 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:29:35 smithi008 sudo[60620]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:29:35.461 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:29:35 smithi008 sudo[60620]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:29:35.461 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:29:35.461 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[40743]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T07:29:35.461 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:35.461 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:35.462 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:35.462 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:35.462 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:35.462 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:35.462 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:35.462 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:35.462 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:35.462 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:29:35.462 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[29816]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T07:29:35.462 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:35.462 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:35.462 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:35.462 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:35.463 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:35.463 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:35.463 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:35.463 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:35.463 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:35.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:29:35.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:35 smithi170 ceph-mon[33394]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T07:29:35.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:35.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:35.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:35.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:35.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:29:35.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:35.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:35.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:35.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:35.713 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:29:35 smithi008 sudo[60660]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T07:29:35.713 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:29:35 smithi008 sudo[60660]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T07:29:35.713 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:29:35 smithi008 sudo[60660]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:29:35.713 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:29:35 smithi008 sudo[60660]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:29:35.967 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:29:35 smithi008 sudo[60677]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T07:29:35.967 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:29:35 smithi008 sudo[60677]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T07:29:35.967 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:29:35 smithi008 sudo[60677]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:29:35.967 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:29:35 smithi008 sudo[60677]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:29:36.478 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:29:36 smithi008 sudo[60683]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T07:29:36.478 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:29:36 smithi008 sudo[60683]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T07:29:36.479 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:29:36 smithi008 sudo[60683]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:29:36.479 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:29:36 smithi008 sudo[60683]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:29:36.479 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[29816]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:36.479 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[29816]: Detected new or changed devices on smithi008 2024-02-10T07:29:36.479 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[29816]: Adjusting osd_memory_target on smithi008 to 5356M 2024-02-10T07:29:36.479 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[29816]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T07:29:36.479 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[29816]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T07:29:36.480 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:36.480 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:36.480 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:36.480 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:36.480 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:36.480 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[40743]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:36.480 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[40743]: Detected new or changed devices on smithi008 2024-02-10T07:29:36.480 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[40743]: Adjusting osd_memory_target on smithi008 to 5356M 2024-02-10T07:29:36.480 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[40743]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T07:29:36.480 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[40743]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T07:29:36.480 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:36.481 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:36.481 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:36.481 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:36.481 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:36 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:36.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:36 smithi170 ceph-mon[33394]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:36.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:36 smithi170 ceph-mon[33394]: Detected new or changed devices on smithi008 2024-02-10T07:29:36.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:36 smithi170 ceph-mon[33394]: Adjusting osd_memory_target on smithi008 to 5356M 2024-02-10T07:29:36.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:36 smithi170 ceph-mon[33394]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T07:29:36.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:36 smithi170 ceph-mon[33394]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T07:29:36.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:36 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:36.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:36 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:36.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:36 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:29:36.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:36 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:36.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:36 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:36.731 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:29:36 smithi008 sudo[60704]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T07:29:36.731 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:29:36 smithi008 sudo[60704]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T07:29:36.731 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:29:36 smithi008 sudo[60704]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:29:36.731 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:29:36 smithi008 sudo[60704]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:29:37.003 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:29:36 smithi008 sudo[60713]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T07:29:37.003 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:29:36 smithi008 sudo[60713]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T07:29:37.003 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:29:36 smithi008 sudo[60713]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:29:37.003 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:29:36 smithi008 sudo[60713]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:29:37.269 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 sudo[60740]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T07:29:37.269 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 sudo[60740]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T07:29:37.269 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 sudo[60740]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:29:37.269 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 sudo[60740]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:29:37.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 sudo[60744]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T07:29:37.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 sudo[60744]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T07:29:37.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 sudo[60744]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:29:37.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 sudo[60744]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:29:37.764 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 sudo[35967]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T07:29:37.764 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 sudo[35967]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T07:29:37.764 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 sudo[35967]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:29:37.764 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 sudo[35967]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:29:38.046 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[40743]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:38.046 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[40743]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:29:38.046 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[40743]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:38.047 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[40743]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:29:38.047 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:29:38.047 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:29:38.047 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:29:38.047 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:29:38.047 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:29:38.047 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:29:38.047 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:29:38.047 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:29:38.047 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:38.047 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[29816]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:38.047 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[29816]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:29:38.047 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[29816]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:38.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[29816]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:29:38.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:29:38.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:29:38.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:29:38.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:29:38.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:29:38.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:29:38.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:29:38.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:29:38.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:38.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 sudo[35970]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T07:29:38.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 sudo[35970]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T07:29:38.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 sudo[35970]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:29:38.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 sudo[35970]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:29:38.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 ceph-mon[33394]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:38.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 ceph-mon[33394]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:29:38.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 ceph-mon[33394]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi008:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:29:38.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 ceph-mon[33394]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:29:38.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:29:38.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:29:38.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:29:38.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:29:38.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:29:38.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:29:38.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:29:38.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:29:38.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:38.310 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 sudo[60871]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T07:29:38.311 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 sudo[60871]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T07:29:38.311 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 sudo[60871]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:29:38.311 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 sudo[60871]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:29:38.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 sudo[60887]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T07:29:38.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 sudo[60887]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T07:29:38.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 sudo[60887]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T07:29:38.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 sudo[60887]: pam_unix(sudo:session): session closed for user root 2024-02-10T07:29:39.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[29816]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:29:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[29816]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:29:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[29816]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:29:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:29:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:29:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:29:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[29816]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:29:39.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[40743]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:29:39.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[40743]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:29:39.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[40743]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:29:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:29:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:29:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:29:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:38 smithi008 ceph-mon[40743]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:29:39.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:38 smithi170 ceph-mon[33394]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:29:39.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:38 smithi170 ceph-mon[33394]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:29:39.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:38 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:39.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:38 smithi170 ceph-mon[33394]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T07:29:39.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:38 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:29:39.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:38 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:29:39.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:38 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:29:39.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:38 smithi170 ceph-mon[33394]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T07:29:40.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:39 smithi008 ceph-mon[29816]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:40.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:39 smithi008 ceph-mon[40743]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:40.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:39 smithi170 ceph-mon[33394]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:41.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:40 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/388386560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ca8f6f3-013a-48cb-9083-be37c1767489"}]: dispatch 2024-02-10T07:29:41.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:40 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/388386560' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ca8f6f3-013a-48cb-9083-be37c1767489"}]': finished 2024-02-10T07:29:41.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:40 smithi008 ceph-mon[29816]: osdmap e22: 4 total, 3 up, 4 in 2024-02-10T07:29:41.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:40 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:41.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:40 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/388386560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ca8f6f3-013a-48cb-9083-be37c1767489"}]: dispatch 2024-02-10T07:29:41.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:40 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/388386560' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ca8f6f3-013a-48cb-9083-be37c1767489"}]': finished 2024-02-10T07:29:41.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:40 smithi008 ceph-mon[40743]: osdmap e22: 4 total, 3 up, 4 in 2024-02-10T07:29:41.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:40 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:41.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:40 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/388386560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ca8f6f3-013a-48cb-9083-be37c1767489"}]: dispatch 2024-02-10T07:29:41.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:40 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/388386560' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ca8f6f3-013a-48cb-9083-be37c1767489"}]': finished 2024-02-10T07:29:41.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:40 smithi170 ceph-mon[33394]: osdmap e22: 4 total, 3 up, 4 in 2024-02-10T07:29:41.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:40 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:42.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:41 smithi008 ceph-mon[40743]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:42.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:41 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:29:42.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:41 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:29:42.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:41 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2391336497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:29:42.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:41 smithi008 ceph-mon[40743]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T07:29:42.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:41 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:42.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:41 smithi008 ceph-mon[29816]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:42.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:41 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:29:42.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:41 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:29:42.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:41 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2391336497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:29:42.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:41 smithi008 ceph-mon[29816]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T07:29:42.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:41 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:42.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:41 smithi170 ceph-mon[33394]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:42.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:41 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:29:42.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:41 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:29:42.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:41 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2391336497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:29:42.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:41 smithi170 ceph-mon[33394]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T07:29:42.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:41 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:44.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:43 smithi008 ceph-mon[29816]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-02-10T07:29:44.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:43 smithi008 ceph-mon[40743]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-02-10T07:29:44.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:43 smithi170 ceph-mon[33394]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-02-10T07:29:46.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:45 smithi008 ceph-mon[29816]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-10T07:29:46.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:45 smithi008 ceph-mon[40743]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-10T07:29:46.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:45 smithi170 ceph-mon[33394]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-10T07:29:47.705 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:47 smithi008 ceph-mon[29816]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-10T07:29:47.705 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T07:29:47.705 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:47.705 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:47 smithi008 ceph-mon[29816]: Deploying daemon osd.3 on smithi008 2024-02-10T07:29:47.705 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:47 smithi008 ceph-mon[40743]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-10T07:29:47.705 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:47 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T07:29:47.705 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:47 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:47.705 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:47 smithi008 ceph-mon[40743]: Deploying daemon osd.3 on smithi008 2024-02-10T07:29:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:47 smithi170 ceph-mon[33394]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-10T07:29:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:47 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T07:29:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:47 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:29:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:47 smithi170 ceph-mon[33394]: Deploying daemon osd.3 on smithi008 2024-02-10T07:29:49.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:49 smithi008 ceph-mon[29816]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 96 KiB/s wr, 6 op/s 2024-02-10T07:29:49.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:49 smithi008 ceph-mon[40743]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 96 KiB/s wr, 6 op/s 2024-02-10T07:29:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:49 smithi170 ceph-mon[33394]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 96 KiB/s wr, 6 op/s 2024-02-10T07:29:51.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:51 smithi008 ceph-mon[29816]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-10T07:29:51.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:51 smithi008 ceph-mon[40743]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-10T07:29:51.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:51 smithi170 ceph-mon[33394]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-10T07:29:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:53 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:53 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:53 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:53.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:53 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:53.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:53 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:53.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:53 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:53 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:53 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:53 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:29:54.407 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:54 smithi008 ceph-mon[29816]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-02-10T07:29:54.408 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:54 smithi008 ceph-mon[40743]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-02-10T07:29:54.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:54 smithi170 ceph-mon[33394]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-02-10T07:29:55.674 INFO:teuthology.orchestra.run.smithi008.stdout:Created osd(s) 3 on host 'smithi008' 2024-02-10T07:29:56.248 DEBUG:teuthology.orchestra.run.smithi008:osd.3> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.3.service 2024-02-10T07:29:56.251 INFO:tasks.cephadm:Deploying osd.4 on smithi170 with /dev/vg_nvme/lv_4... 2024-02-10T07:29:56.251 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-10T07:29:56.496 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:29:56.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:56 smithi008 ceph-mon[29816]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:56 smithi008 ceph-mon[29816]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:29:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:56 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:56 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:56.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:56 smithi008 ceph-mon[40743]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:56.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:56 smithi008 ceph-mon[40743]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:29:56.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:56 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:56.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:56 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:56.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:56 smithi170 ceph-mon[33394]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:56.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:56 smithi170 ceph-mon[33394]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:29:56.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:56 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:56.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:56 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:57.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:57 smithi008 ceph-mon[29816]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T07:29:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:57 smithi008 ceph-mon[29816]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T07:29:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:57 smithi008 ceph-mon[29816]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:57 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:57 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:57 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:57.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:57 smithi008 ceph-mon[40743]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T07:29:57.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:57 smithi008 ceph-mon[40743]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T07:29:57.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:57 smithi008 ceph-mon[40743]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:57.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:57 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:57.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:57 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:57.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:57 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:57 smithi170 ceph-mon[33394]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T07:29:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:57 smithi170 ceph-mon[33394]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T07:29:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:57 smithi170 ceph-mon[33394]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:29:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:57 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:57 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:57 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:29:57.832 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:29:57 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3[62633]: 2024-02-10T07:29:57.497+0000 7f3835a4a700 -1 osd.3 0 waiting for initial osdmap 2024-02-10T07:29:57.832 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:29:57 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3[62633]: 2024-02-10T07:29:57.501+0000 7f382d835700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:29:58.268 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-10T07:29:58.268 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-10T07:29:58.268 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-02-10T07:29:58.268 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-02-10T07:29:58.268 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00879077 s, 1.2 GB/s 2024-02-10T07:29:58.268 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-02-10T07:29:58.293 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:58 smithi008 ceph-mon[29816]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:58.293 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:58 smithi008 ceph-mon[29816]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T07:29:58.293 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:58 smithi008 ceph-mon[29816]: osdmap e25: 4 total, 3 up, 4 in 2024-02-10T07:29:58.293 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:58 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:58.293 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:58 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:58.293 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:58 smithi008 ceph-mon[29816]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' 2024-02-10T07:29:58.293 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:58 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:58.293 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:58 smithi008 ceph-mon[40743]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:58.294 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:58 smithi008 ceph-mon[40743]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T07:29:58.294 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:58 smithi008 ceph-mon[40743]: osdmap e25: 4 total, 3 up, 4 in 2024-02-10T07:29:58.294 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:58 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:58.294 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:58 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:58.294 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:58 smithi008 ceph-mon[40743]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' 2024-02-10T07:29:58.294 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:58 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:58.561 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:58 smithi170 ceph-mon[33394]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:29:58.561 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:58 smithi170 ceph-mon[33394]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]': finished 2024-02-10T07:29:58.561 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:58 smithi170 ceph-mon[33394]: osdmap e25: 4 total, 3 up, 4 in 2024-02-10T07:29:58.561 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:58 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:58.561 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:58 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:58.561 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:58 smithi170 ceph-mon[33394]: from='osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651]' entity='osd.3' 2024-02-10T07:29:58.562 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:58 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:58.864 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_4 2024-02-10T07:29:59.104 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:29:59.513 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:59 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:29:59.513 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:59 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:29:59.513 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:59 smithi008 ceph-mon[29816]: osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651] boot 2024-02-10T07:29:59.513 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:59 smithi008 ceph-mon[29816]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T07:29:59.513 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:29:59 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:59.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:59 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:29:59.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:59 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:29:59.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:59 smithi008 ceph-mon[40743]: osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651] boot 2024-02-10T07:29:59.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:59 smithi008 ceph-mon[40743]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T07:29:59.514 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:29:59 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:29:59.518 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:59 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:29:59.518 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:59 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:29:59.518 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:59 smithi170 ceph-mon[33394]: osd.3 [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651] boot 2024-02-10T07:29:59.518 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:59 smithi170 ceph-mon[33394]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T07:29:59.518 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:29:59 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:30:00.427 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:30:00.427 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.427 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:00.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:00.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:00.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:00.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:00.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:00.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:00.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:00.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:00.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:00 smithi170 ceph-mon[33394]: overall HEALTH_OK 2024-02-10T07:30:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:30:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:00.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:00.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:00.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:00.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:00.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:00.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:00.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[29816]: overall HEALTH_OK 2024-02-10T07:30:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-02-10T07:30:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:00.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:00.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:00.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:00.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:00.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:00.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:00.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:00.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:00.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:00.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:00 smithi008 ceph-mon[40743]: overall HEALTH_OK 2024-02-10T07:30:01.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:01 smithi170 ceph-mon[33394]: Detected new or changed devices on smithi008 2024-02-10T07:30:01.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:01 smithi170 ceph-mon[33394]: Adjusting osd_memory_target on smithi008 to 4017M 2024-02-10T07:30:01.507 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:01 smithi170 ceph-mon[33394]: osdmap e27: 4 total, 4 up, 4 in 2024-02-10T07:30:01.507 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:01 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:30:01.507 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:01 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:30:01.507 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:01 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:01.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:01 smithi008 ceph-mon[40743]: Detected new or changed devices on smithi008 2024-02-10T07:30:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:01 smithi008 ceph-mon[40743]: Adjusting osd_memory_target on smithi008 to 4017M 2024-02-10T07:30:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:01 smithi008 ceph-mon[40743]: osdmap e27: 4 total, 4 up, 4 in 2024-02-10T07:30:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:01 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:30:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:01 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:30:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:01 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:01 smithi008 ceph-mon[29816]: Detected new or changed devices on smithi008 2024-02-10T07:30:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:01 smithi008 ceph-mon[29816]: Adjusting osd_memory_target on smithi008 to 4017M 2024-02-10T07:30:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:01 smithi008 ceph-mon[29816]: osdmap e27: 4 total, 4 up, 4 in 2024-02-10T07:30:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:01 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:30:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:01 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:30:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:01 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:02.502 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:02 smithi008 ceph-mon[29816]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:02.502 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:02 smithi008 ceph-mon[29816]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:30:02.502 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:02 smithi008 ceph-mon[40743]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:02.502 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:02 smithi008 ceph-mon[40743]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:30:02.623 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:02 smithi170 ceph-mon[33394]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:02.623 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:02 smithi170 ceph-mon[33394]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:30:04.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:04 smithi008 ceph-mon[29816]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:04.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:04 smithi008 ceph-mon[29816]: from='client.? 172.21.15.170:0/286510319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa667766-e57f-447b-9a31-e3f3e61ac346"}]: dispatch 2024-02-10T07:30:04.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:04 smithi008 ceph-mon[29816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa667766-e57f-447b-9a31-e3f3e61ac346"}]: dispatch 2024-02-10T07:30:04.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:04 smithi008 ceph-mon[29816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa667766-e57f-447b-9a31-e3f3e61ac346"}]': finished 2024-02-10T07:30:04.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:04 smithi008 ceph-mon[29816]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T07:30:04.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:04 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:04.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:04 smithi008 ceph-mon[29816]: from='client.? 172.21.15.170:0/3968439789' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:30:04.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:04 smithi008 ceph-mon[40743]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:04.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:04 smithi008 ceph-mon[40743]: from='client.? 172.21.15.170:0/286510319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa667766-e57f-447b-9a31-e3f3e61ac346"}]: dispatch 2024-02-10T07:30:04.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:04 smithi008 ceph-mon[40743]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa667766-e57f-447b-9a31-e3f3e61ac346"}]: dispatch 2024-02-10T07:30:04.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:04 smithi008 ceph-mon[40743]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa667766-e57f-447b-9a31-e3f3e61ac346"}]': finished 2024-02-10T07:30:04.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:04 smithi008 ceph-mon[40743]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T07:30:04.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:04 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:04.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:04 smithi008 ceph-mon[40743]: from='client.? 172.21.15.170:0/3968439789' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:30:04.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:04 smithi170 ceph-mon[33394]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:04 smithi170 ceph-mon[33394]: from='client.? 172.21.15.170:0/286510319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa667766-e57f-447b-9a31-e3f3e61ac346"}]: dispatch 2024-02-10T07:30:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:04 smithi170 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa667766-e57f-447b-9a31-e3f3e61ac346"}]: dispatch 2024-02-10T07:30:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:04 smithi170 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa667766-e57f-447b-9a31-e3f3e61ac346"}]': finished 2024-02-10T07:30:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:04 smithi170 ceph-mon[33394]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T07:30:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:04 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:04 smithi170 ceph-mon[33394]: from='client.? 172.21.15.170:0/3968439789' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:30:06.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:06 smithi008 ceph-mon[29816]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:06.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:06 smithi008 ceph-mon[40743]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:06 smithi170 ceph-mon[33394]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:07.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:07 smithi170 ceph-mon[33394]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:07.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:07 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:30:07.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:07 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:30:08.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:07 smithi008 ceph-mon[40743]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:08.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:07 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:30:08.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:07 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:30:08.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:07 smithi008 ceph-mon[29816]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:08.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:07 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:30:08.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:07 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:30:08.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:08 smithi170 ceph-mon[33394]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T07:30:08.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:09.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:08 smithi008 ceph-mon[40743]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T07:30:09.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:09.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:08 smithi008 ceph-mon[29816]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T07:30:09.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:09.858 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:09 smithi170 ceph-mon[33394]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:10.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:09 smithi008 ceph-mon[29816]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:10.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:09 smithi008 ceph-mon[40743]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:10.786 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:10 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T07:30:10.787 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:10 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:10.787 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:10 smithi170 ceph-mon[33394]: Deploying daemon osd.4 on smithi170 2024-02-10T07:30:11.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:10 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T07:30:11.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:10 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:11.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:10 smithi008 ceph-mon[29816]: Deploying daemon osd.4 on smithi170 2024-02-10T07:30:11.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:10 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T07:30:11.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:10 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:11.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:10 smithi008 ceph-mon[40743]: Deploying daemon osd.4 on smithi170 2024-02-10T07:30:11.702 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:11 smithi170 ceph-mon[33394]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:12.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:11 smithi008 ceph-mon[29816]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:12.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:11 smithi008 ceph-mon[40743]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:13 smithi170 ceph-mon[33394]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:14.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:13 smithi008 ceph-mon[29816]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:14.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:13 smithi008 ceph-mon[40743]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:16.364 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:16 smithi170 ceph-mon[33394]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:16.364 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:16 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:16.364 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:16 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:16.364 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:16 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:16.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:16 smithi008 ceph-mon[29816]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:16.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:16 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:16.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:16 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:16.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:16 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:16.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:16 smithi008 ceph-mon[40743]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:16.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:16 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:16.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:16 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:16.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:16 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:18.432 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:18 smithi170 ceph-mon[33394]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:18.519 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 4 on host 'smithi170' 2024-02-10T07:30:18.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:18 smithi008 ceph-mon[29816]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:18.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:18 smithi008 ceph-mon[40743]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:19.206 DEBUG:teuthology.orchestra.run.smithi170:osd.4> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.4.service 2024-02-10T07:30:19.209 INFO:tasks.cephadm:Deploying osd.5 on smithi170 with /dev/vg_nvme/lv_3... 2024-02-10T07:30:19.209 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-10T07:30:19.490 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:30:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:19 smithi008 ceph-mon[29816]: from='osd.4 [v2:172.21.15.170:6800/3997132504,v1:172.21.15.170:6801/3997132504]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:30:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:19 smithi008 ceph-mon[29816]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:30:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:19 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:19 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:19 smithi008 ceph-mon[40743]: from='osd.4 [v2:172.21.15.170:6800/3997132504,v1:172.21.15.170:6801/3997132504]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:30:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:19 smithi008 ceph-mon[40743]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:30:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:19 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:19 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:19.611 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:19 smithi170 ceph-mon[33394]: from='osd.4 [v2:172.21.15.170:6800/3997132504,v1:172.21.15.170:6801/3997132504]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:30:19.612 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:19 smithi170 ceph-mon[33394]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T07:30:19.612 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:19 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:19.612 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:19 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:20.447 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:20 smithi170 ceph-mon[33394]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:20.447 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:20 smithi170 ceph-mon[33394]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T07:30:20.447 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:20 smithi170 ceph-mon[33394]: from='osd.4 [v2:172.21.15.170:6800/3997132504,v1:172.21.15.170:6801/3997132504]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:30:20.447 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:20 smithi170 ceph-mon[33394]: osdmap e30: 5 total, 4 up, 5 in 2024-02-10T07:30:20.447 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:20 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:20.447 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:20 smithi170 ceph-mon[33394]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:30:20.447 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:20 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:20.448 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:20 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:20.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[29816]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:20.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[29816]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T07:30:20.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[29816]: from='osd.4 [v2:172.21.15.170:6800/3997132504,v1:172.21.15.170:6801/3997132504]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:30:20.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[29816]: osdmap e30: 5 total, 4 up, 5 in 2024-02-10T07:30:20.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:20.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[29816]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:30:20.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:20.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:20.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[40743]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:20.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[40743]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T07:30:20.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[40743]: from='osd.4 [v2:172.21.15.170:6800/3997132504,v1:172.21.15.170:6801/3997132504]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:30:20.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[40743]: osdmap e30: 5 total, 4 up, 5 in 2024-02-10T07:30:20.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:20.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[40743]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:30:20.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:20.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:20 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:20.712 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:30:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:30:20.446+0000 7f98b6fa1700 -1 osd.4 0 waiting for initial osdmap 2024-02-10T07:30:20.713 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:30:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:30:20.450+0000 7f98ae58b700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:30:21.295 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:21 smithi170 ceph-mon[33394]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-02-10T07:30:21.295 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:21 smithi170 ceph-mon[33394]: osdmap e31: 5 total, 4 up, 5 in 2024-02-10T07:30:21.296 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:21 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:21.296 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:21 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:21.296 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:21 smithi170 ceph-mon[33394]: from='osd.4 ' entity='osd.4' 2024-02-10T07:30:21.412 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-10T07:30:21.412 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-10T07:30:21.412 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-02-10T07:30:21.412 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-02-10T07:30:21.412 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00977193 s, 1.1 GB/s 2024-02-10T07:30:21.412 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-02-10T07:30:21.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:21 smithi008 ceph-mon[29816]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-02-10T07:30:21.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:21 smithi008 ceph-mon[29816]: osdmap e31: 5 total, 4 up, 5 in 2024-02-10T07:30:21.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:21 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:21.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:21 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:21.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:21 smithi008 ceph-mon[29816]: from='osd.4 ' entity='osd.4' 2024-02-10T07:30:21.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:21 smithi008 ceph-mon[40743]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-02-10T07:30:21.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:21 smithi008 ceph-mon[40743]: osdmap e31: 5 total, 4 up, 5 in 2024-02-10T07:30:21.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:21 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:21.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:21 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:21.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:21 smithi008 ceph-mon[40743]: from='osd.4 ' entity='osd.4' 2024-02-10T07:30:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:22 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:30:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:22 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:30:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:22 smithi170 ceph-mon[33394]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:22 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:22 smithi170 ceph-mon[33394]: osd.4 [v2:172.21.15.170:6800/3997132504,v1:172.21.15.170:6801/3997132504] boot 2024-02-10T07:30:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:22 smithi170 ceph-mon[33394]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T07:30:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:22 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:22 smithi170 ceph-mon[33394]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T07:30:22.415 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_3 2024-02-10T07:30:22.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:30:22.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:30:22.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[29816]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:22.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:22.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[29816]: osd.4 [v2:172.21.15.170:6800/3997132504,v1:172.21.15.170:6801/3997132504] boot 2024-02-10T07:30:22.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[29816]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T07:30:22.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:22.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[29816]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T07:30:22.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:30:22.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:30:22.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[40743]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-02-10T07:30:22.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:22.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[40743]: osd.4 [v2:172.21.15.170:6800/3997132504,v1:172.21.15.170:6801/3997132504] boot 2024-02-10T07:30:22.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[40743]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T07:30:22.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:30:22.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:22 smithi008 ceph-mon[40743]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T07:30:23.307 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:30:23.762 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:23 smithi170 ceph-mon[33394]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:23.762 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:23 smithi170 ceph-mon[33394]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T07:30:24.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:23 smithi008 ceph-mon[29816]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:24.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:23 smithi008 ceph-mon[29816]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T07:30:24.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:23 smithi008 ceph-mon[40743]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:24.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:23 smithi008 ceph-mon[40743]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T07:30:24.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:24 smithi170 ceph-mon[33394]: osdmap e35: 5 total, 5 up, 5 in 2024-02-10T07:30:25.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:24 smithi008 ceph-mon[29816]: osdmap e35: 5 total, 5 up, 5 in 2024-02-10T07:30:25.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:24 smithi008 ceph-mon[40743]: osdmap e35: 5 total, 5 up, 5 in 2024-02-10T07:30:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: osdmap e36: 5 total, 5 up, 5 in 2024-02-10T07:30:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: Detected new or changed devices on smithi170 2024-02-10T07:30:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:25.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: Adjusting osd_memory_target on smithi170 to 17088M 2024-02-10T07:30:25.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:25.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:25.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:25.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:25.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:25.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:25.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:25.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:25 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: osdmap e36: 5 total, 5 up, 5 in 2024-02-10T07:30:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: Detected new or changed devices on smithi170 2024-02-10T07:30:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: Adjusting osd_memory_target on smithi170 to 17088M 2024-02-10T07:30:26.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:26.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:26.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:26.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:26.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:26.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:26.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:26.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: osdmap e36: 5 total, 5 up, 5 in 2024-02-10T07:30:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: Detected new or changed devices on smithi170 2024-02-10T07:30:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:26.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:26.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: Adjusting osd_memory_target on smithi170 to 17088M 2024-02-10T07:30:26.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:26.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:26.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:26.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:26.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:26.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:26.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:26.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:25 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:26 smithi170 ceph-mon[33394]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:30:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:26 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:30:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:26 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:30:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:26 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:27.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:26 smithi008 ceph-mon[29816]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:30:27.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:26 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:30:27.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:26 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:30:27.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:26 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:27.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:26 smithi008 ceph-mon[40743]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:30:27.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:26 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:30:27.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:26 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:30:27.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:26 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:27.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:27 smithi170 ceph-mon[33394]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:28.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:27 smithi008 ceph-mon[29816]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:28.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:27 smithi008 ceph-mon[40743]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:28.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:28 smithi170 ceph-mon[33394]: from='client.? 172.21.15.170:0/881786363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e9ab14d-1aff-4638-9f41-311f6b5f6770"}]: dispatch 2024-02-10T07:30:28.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:28 smithi170 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e9ab14d-1aff-4638-9f41-311f6b5f6770"}]: dispatch 2024-02-10T07:30:28.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:28 smithi170 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e9ab14d-1aff-4638-9f41-311f6b5f6770"}]': finished 2024-02-10T07:30:28.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:28 smithi170 ceph-mon[33394]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T07:30:28.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:28 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:29.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:28 smithi008 ceph-mon[29816]: from='client.? 172.21.15.170:0/881786363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e9ab14d-1aff-4638-9f41-311f6b5f6770"}]: dispatch 2024-02-10T07:30:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:28 smithi008 ceph-mon[29816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e9ab14d-1aff-4638-9f41-311f6b5f6770"}]: dispatch 2024-02-10T07:30:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:28 smithi008 ceph-mon[29816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e9ab14d-1aff-4638-9f41-311f6b5f6770"}]': finished 2024-02-10T07:30:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:28 smithi008 ceph-mon[29816]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T07:30:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:28 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:28 smithi008 ceph-mon[40743]: from='client.? 172.21.15.170:0/881786363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e9ab14d-1aff-4638-9f41-311f6b5f6770"}]: dispatch 2024-02-10T07:30:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:28 smithi008 ceph-mon[40743]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e9ab14d-1aff-4638-9f41-311f6b5f6770"}]: dispatch 2024-02-10T07:30:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:28 smithi008 ceph-mon[40743]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e9ab14d-1aff-4638-9f41-311f6b5f6770"}]': finished 2024-02-10T07:30:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:28 smithi008 ceph-mon[40743]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T07:30:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:28 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:29.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:29 smithi170 ceph-mon[33394]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-02-10T07:30:29.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:29 smithi170 ceph-mon[33394]: from='client.? 172.21.15.170:0/1233585305' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:30:30.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:29 smithi008 ceph-mon[29816]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-02-10T07:30:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:29 smithi008 ceph-mon[29816]: from='client.? 172.21.15.170:0/1233585305' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:30:30.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:29 smithi008 ceph-mon[40743]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-02-10T07:30:30.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:29 smithi008 ceph-mon[40743]: from='client.? 172.21.15.170:0/1233585305' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:30:31.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:31 smithi170 ceph-mon[33394]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-10T07:30:31.883 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:31 smithi008 ceph-mon[29816]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-10T07:30:31.884 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:31 smithi008 ceph-mon[40743]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-10T07:30:33.831 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:33 smithi170 ceph-mon[33394]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T07:30:34.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:33 smithi008 ceph-mon[29816]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T07:30:34.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:33 smithi008 ceph-mon[40743]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T07:30:35.855 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:35 smithi170 ceph-mon[33394]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-02-10T07:30:35.856 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T07:30:35.856 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:35.856 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:35 smithi170 ceph-mon[33394]: Deploying daemon osd.5 on smithi170 2024-02-10T07:30:36.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:35 smithi008 ceph-mon[29816]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-02-10T07:30:36.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T07:30:36.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:36.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:35 smithi008 ceph-mon[29816]: Deploying daemon osd.5 on smithi170 2024-02-10T07:30:36.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:35 smithi008 ceph-mon[40743]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-02-10T07:30:36.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T07:30:36.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:36.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:35 smithi008 ceph-mon[40743]: Deploying daemon osd.5 on smithi170 2024-02-10T07:30:37.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:37 smithi170 ceph-mon[33394]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T07:30:38.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:37 smithi008 ceph-mon[29816]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T07:30:38.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:37 smithi008 ceph-mon[40743]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T07:30:39.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:39 smithi170 ceph-mon[33394]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:40.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:39 smithi008 ceph-mon[29816]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:40.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:39 smithi008 ceph-mon[40743]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:41.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:41 smithi008 ceph-mon[29816]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:41.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:41 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:41.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:41 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:41.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:41 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:41.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:41 smithi008 ceph-mon[40743]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:41.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:41 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:41.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:41 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:41.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:41 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:41.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:41 smithi170 ceph-mon[33394]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:41.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:41 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:41.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:41 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:41.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:41 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:44.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:43 smithi170 ceph-mon[33394]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:44.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:43 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:44.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:43 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:44.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:43 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:44.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:43 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:44.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:43 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:44.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:43 smithi170 ceph-mon[33394]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:30:44.119 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 5 on host 'smithi170' 2024-02-10T07:30:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:43 smithi008 ceph-mon[29816]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:43 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:43 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:43 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:43 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:44.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:43 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:44.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:43 smithi008 ceph-mon[29816]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:30:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:43 smithi008 ceph-mon[40743]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:43 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:43 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:43 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:44.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:43 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:44.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:43 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:44.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:43 smithi008 ceph-mon[40743]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:30:44.776 DEBUG:teuthology.orchestra.run.smithi170:osd.5> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.5.service 2024-02-10T07:30:44.778 INFO:tasks.cephadm:Deploying osd.6 on smithi170 with /dev/vg_nvme/lv_2... 2024-02-10T07:30:44.778 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-10T07:30:45.022 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:44 smithi170 ceph-mon[33394]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T07:30:45.022 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:44 smithi170 ceph-mon[33394]: osdmap e38: 6 total, 5 up, 6 in 2024-02-10T07:30:45.022 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:44 smithi170 ceph-mon[33394]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:30:45.022 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:44 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:45.022 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:44 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:45.022 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:44 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:45.022 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:44 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:44 smithi008 ceph-mon[29816]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T07:30:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:44 smithi008 ceph-mon[29816]: osdmap e38: 6 total, 5 up, 6 in 2024-02-10T07:30:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:44 smithi008 ceph-mon[29816]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:30:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:44 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:44 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:44 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:44 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:44 smithi008 ceph-mon[40743]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T07:30:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:44 smithi008 ceph-mon[40743]: osdmap e38: 6 total, 5 up, 6 in 2024-02-10T07:30:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:44 smithi008 ceph-mon[40743]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:30:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:44 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:44 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:44 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:44 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:30:45.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:30:45 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[44249]: 2024-02-10T07:30:45.239+0000 7f8ec3209700 -1 osd.5 0 waiting for initial osdmap 2024-02-10T07:30:45.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:30:45 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[44249]: 2024-02-10T07:30:45.243+0000 7f8ebaff4700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:30:45.826 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:30:46.055 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:45 smithi170 ceph-mon[33394]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:46.055 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:45 smithi170 ceph-mon[33394]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-02-10T07:30:46.055 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:45 smithi170 ceph-mon[33394]: osdmap e39: 6 total, 5 up, 6 in 2024-02-10T07:30:46.056 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:45 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:46.056 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:45 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:46.056 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:45 smithi170 ceph-mon[33394]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' 2024-02-10T07:30:46.056 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:45 smithi170 ceph-mon[33394]: osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245] boot 2024-02-10T07:30:46.056 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:45 smithi170 ceph-mon[33394]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T07:30:46.056 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:45 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[29816]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[29816]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-02-10T07:30:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[29816]: osdmap e39: 6 total, 5 up, 6 in 2024-02-10T07:30:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[29816]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' 2024-02-10T07:30:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[29816]: osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245] boot 2024-02-10T07:30:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[29816]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T07:30:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:46.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[40743]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-02-10T07:30:46.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[40743]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-02-10T07:30:46.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[40743]: osdmap e39: 6 total, 5 up, 6 in 2024-02-10T07:30:46.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:46.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:46.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[40743]: from='osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245]' entity='osd.5' 2024-02-10T07:30:46.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[40743]: osd.5 [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245] boot 2024-02-10T07:30:46.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[40743]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T07:30:46.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:45 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:30:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:47 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:30:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:47 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:30:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:47 smithi170 ceph-mon[33394]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:30:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:47 smithi170 ceph-mon[33394]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T07:30:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:47 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:48.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:47 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:30:48.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:47 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:30:48.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:47 smithi008 ceph-mon[29816]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:30:48.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:47 smithi008 ceph-mon[29816]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T07:30:48.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:48.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:47 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:30:48.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:47 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:30:48.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:47 smithi008 ceph-mon[40743]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:30:48.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:47 smithi008 ceph-mon[40743]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T07:30:48.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:47 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:48.811 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:48 smithi170 ceph-mon[33394]: Detected new or changed devices on smithi170 2024-02-10T07:30:48.811 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:48 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:48.811 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:48 smithi170 ceph-mon[33394]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T07:30:48.811 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:48 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:48.811 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:48 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:48.811 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:48 smithi170 ceph-mon[33394]: Adjusting osd_memory_target on smithi170 to 8544M 2024-02-10T07:30:48.811 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:48 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:48.811 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:48 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:48.811 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:48 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:48.811 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:48 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:48.812 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-10T07:30:48.812 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-10T07:30:48.812 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-02-10T07:30:48.812 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-02-10T07:30:48.812 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00894777 s, 1.2 GB/s 2024-02-10T07:30:48.812 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-02-10T07:30:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[29816]: Detected new or changed devices on smithi170 2024-02-10T07:30:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[29816]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T07:30:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[29816]: Adjusting osd_memory_target on smithi170 to 8544M 2024-02-10T07:30:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:49.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[40743]: Detected new or changed devices on smithi170 2024-02-10T07:30:49.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:49.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[40743]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T07:30:49.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:49.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:30:49.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[40743]: Adjusting osd_memory_target on smithi170 to 8544M 2024-02-10T07:30:49.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:49.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:49.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:30:49.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:48 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:30:49.437 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_2 2024-02-10T07:30:49.677 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:30:49.793 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:49 smithi170 ceph-mon[33394]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:30:49.793 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:49 smithi170 ceph-mon[33394]: osdmap e43: 6 total, 6 up, 6 in 2024-02-10T07:30:49.793 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:49 smithi170 ceph-mon[33394]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T07:30:50.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:49 smithi008 ceph-mon[29816]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:30:50.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:49 smithi008 ceph-mon[29816]: osdmap e43: 6 total, 6 up, 6 in 2024-02-10T07:30:50.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:49 smithi008 ceph-mon[29816]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T07:30:50.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:49 smithi008 ceph-mon[40743]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:30:50.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:49 smithi008 ceph-mon[40743]: osdmap e43: 6 total, 6 up, 6 in 2024-02-10T07:30:50.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:49 smithi008 ceph-mon[40743]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T07:30:50.866 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:50 smithi170 ceph-mon[33394]: osdmap e44: 6 total, 6 up, 6 in 2024-02-10T07:30:51.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:50 smithi008 ceph-mon[29816]: osdmap e44: 6 total, 6 up, 6 in 2024-02-10T07:30:51.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:50 smithi008 ceph-mon[40743]: osdmap e44: 6 total, 6 up, 6 in 2024-02-10T07:30:51.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:51 smithi170 ceph-mon[33394]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:30:51.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:51 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:30:51.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:51 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:30:51.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:51 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:52.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:51 smithi008 ceph-mon[29816]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:30:52.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:51 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:30:52.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:51 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:30:52.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:51 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:52.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:51 smithi008 ceph-mon[40743]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:30:52.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:51 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:30:52.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:51 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:30:52.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:51 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:30:52.866 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:52 smithi170 ceph-mon[33394]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:30:53.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:52 smithi008 ceph-mon[29816]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:30:53.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:52 smithi008 ceph-mon[40743]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:30:53.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:53 smithi170 ceph-mon[33394]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:30:53.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:53 smithi170 ceph-mon[33394]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T07:30:53.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:53 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:30:54.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:53 smithi008 ceph-mon[29816]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:30:54.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:53 smithi008 ceph-mon[29816]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T07:30:54.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:53 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:30:54.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:53 smithi008 ceph-mon[40743]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:30:54.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:53 smithi008 ceph-mon[40743]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T07:30:54.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:53 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:30:54.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:54 smithi170 ceph-mon[33394]: from='client.? 172.21.15.170:0/643812097' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61970c81-4b79-4743-a379-50d5d102b373"}]: dispatch 2024-02-10T07:30:54.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:54 smithi170 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61970c81-4b79-4743-a379-50d5d102b373"}]: dispatch 2024-02-10T07:30:54.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:54 smithi170 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61970c81-4b79-4743-a379-50d5d102b373"}]': finished 2024-02-10T07:30:54.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:54 smithi170 ceph-mon[33394]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T07:30:54.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:54 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:30:54.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:54 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:30:54.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:54 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:30:54.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:54 smithi170 ceph-mon[33394]: from='client.? 172.21.15.170:0/3037019529' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:30:55.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[29816]: from='client.? 172.21.15.170:0/643812097' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61970c81-4b79-4743-a379-50d5d102b373"}]: dispatch 2024-02-10T07:30:55.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[29816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61970c81-4b79-4743-a379-50d5d102b373"}]: dispatch 2024-02-10T07:30:55.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[29816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61970c81-4b79-4743-a379-50d5d102b373"}]': finished 2024-02-10T07:30:55.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[29816]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T07:30:55.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:30:55.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:30:55.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:30:55.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[29816]: from='client.? 172.21.15.170:0/3037019529' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:30:55.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[40743]: from='client.? 172.21.15.170:0/643812097' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61970c81-4b79-4743-a379-50d5d102b373"}]: dispatch 2024-02-10T07:30:55.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[40743]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61970c81-4b79-4743-a379-50d5d102b373"}]: dispatch 2024-02-10T07:30:55.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[40743]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61970c81-4b79-4743-a379-50d5d102b373"}]': finished 2024-02-10T07:30:55.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[40743]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T07:30:55.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:30:55.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:30:55.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:30:55.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:54 smithi008 ceph-mon[40743]: from='client.? 172.21.15.170:0/3037019529' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:30:56.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:55 smithi008 ceph-mon[29816]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T07:30:56.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:55 smithi008 ceph-mon[29816]: osdmap e46: 7 total, 6 up, 7 in 2024-02-10T07:30:56.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:55 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:30:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:55 smithi008 ceph-mon[40743]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T07:30:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:55 smithi008 ceph-mon[40743]: osdmap e46: 7 total, 6 up, 7 in 2024-02-10T07:30:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:55 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:30:56.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:55 smithi170 ceph-mon[33394]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T07:30:56.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:55 smithi170 ceph-mon[33394]: osdmap e46: 7 total, 6 up, 7 in 2024-02-10T07:30:56.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:55 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:30:58.229 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:57 smithi170 ceph-mon[33394]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-10T07:30:58.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:57 smithi008 ceph-mon[29816]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-10T07:30:58.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:57 smithi008 ceph-mon[40743]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-10T07:30:59.157 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:58 smithi170 ceph-mon[33394]: 1.0 scrub starts 2024-02-10T07:30:59.158 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:58 smithi170 ceph-mon[33394]: 1.0 scrub ok 2024-02-10T07:30:59.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:58 smithi008 ceph-mon[29816]: 1.0 scrub starts 2024-02-10T07:30:59.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:58 smithi008 ceph-mon[29816]: 1.0 scrub ok 2024-02-10T07:30:59.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:58 smithi008 ceph-mon[40743]: 1.0 scrub starts 2024-02-10T07:30:59.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:58 smithi008 ceph-mon[40743]: 1.0 scrub ok 2024-02-10T07:31:00.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:30:59 smithi170 ceph-mon[33394]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T07:31:00.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:30:59 smithi008 ceph-mon[29816]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T07:31:00.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:30:59 smithi008 ceph-mon[40743]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T07:31:01.209 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T07:31:01.209 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:01.209 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:00 smithi170 ceph-mon[33394]: Deploying daemon osd.6 on smithi170 2024-02-10T07:31:01.233 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T07:31:01.233 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:01.233 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:00 smithi008 ceph-mon[40743]: Deploying daemon osd.6 on smithi170 2024-02-10T07:31:01.233 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T07:31:01.233 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:01.233 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:00 smithi008 ceph-mon[29816]: Deploying daemon osd.6 on smithi170 2024-02-10T07:31:02.121 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:01 smithi170 ceph-mon[33394]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:02.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:01 smithi008 ceph-mon[29816]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:02.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:01 smithi008 ceph-mon[40743]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:04.257 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:03 smithi170 ceph-mon[33394]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:04.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:03 smithi008 ceph-mon[29816]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:04.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:03 smithi008 ceph-mon[40743]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:06.235 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:05 smithi170 ceph-mon[33394]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:06.235 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:06.235 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:06.235 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:06.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:05 smithi008 ceph-mon[29816]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:06.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:06.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:06.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:06.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:05 smithi008 ceph-mon[40743]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:06.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:06.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:06.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:08.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:07 smithi170 ceph-mon[33394]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:08.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:07 smithi008 ceph-mon[29816]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:08.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:07 smithi008 ceph-mon[40743]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:09.536 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 6 on host 'smithi170' 2024-02-10T07:31:09.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:09.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:31:09.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:31:09.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:09.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:09.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:09.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:09.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:09.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: osdmap e47: 7 total, 6 up, 7 in 2024-02-10T07:31:09.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:09.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: from='osd.6 [v2:172.21.15.170:6816/1764436984,v1:172.21.15.170:6817/1764436984]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:31:09.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:31:09.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:09.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:09.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:09 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:10.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:10.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:31:10.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:31:10.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:10.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:10.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:10.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:10.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:10.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: osdmap e47: 7 total, 6 up, 7 in 2024-02-10T07:31:10.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:10.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: from='osd.6 [v2:172.21.15.170:6816/1764436984,v1:172.21.15.170:6817/1764436984]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:31:10.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:31:10.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:10.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:10.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:10.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:10.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:31:10.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:31:10.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:10.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:10.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:10.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:10.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:10.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: osdmap e47: 7 total, 6 up, 7 in 2024-02-10T07:31:10.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:10.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: from='osd.6 [v2:172.21.15.170:6816/1764436984,v1:172.21.15.170:6817/1764436984]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:31:10.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T07:31:10.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:10.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:10.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:09 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:10.192 DEBUG:teuthology.orchestra.run.smithi170:osd.6> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.6.service 2024-02-10T07:31:10.195 INFO:tasks.cephadm:Deploying osd.7 on smithi170 with /dev/vg_nvme/lv_1... 2024-02-10T07:31:10.195 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-10T07:31:11.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:10 smithi170 ceph-mon[33394]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T07:31:11.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:10 smithi170 ceph-mon[33394]: osdmap e48: 7 total, 6 up, 7 in 2024-02-10T07:31:11.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:10 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:11.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:10 smithi170 ceph-mon[33394]: from='osd.6 [v2:172.21.15.170:6816/1764436984,v1:172.21.15.170:6817/1764436984]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:31:11.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:10 smithi170 ceph-mon[33394]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:31:11.133 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:31:10 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:31:10.948+0000 7fc48a6a8700 -1 osd.6 0 waiting for initial osdmap 2024-02-10T07:31:11.134 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:31:10 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:31:10.955+0000 7fc48649b700 -1 osd.6 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:31:11.283 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:31:11.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:10 smithi008 ceph-mon[29816]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T07:31:11.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:10 smithi008 ceph-mon[29816]: osdmap e48: 7 total, 6 up, 7 in 2024-02-10T07:31:11.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:10 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:11.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:10 smithi008 ceph-mon[29816]: from='osd.6 [v2:172.21.15.170:6816/1764436984,v1:172.21.15.170:6817/1764436984]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:31:11.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:10 smithi008 ceph-mon[29816]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:31:11.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:10 smithi008 ceph-mon[40743]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T07:31:11.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:10 smithi008 ceph-mon[40743]: osdmap e48: 7 total, 6 up, 7 in 2024-02-10T07:31:11.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:10 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:11.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:10 smithi008 ceph-mon[40743]: from='osd.6 [v2:172.21.15.170:6816/1764436984,v1:172.21.15.170:6817/1764436984]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:31:11.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:10 smithi008 ceph-mon[40743]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:31:12.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:11 smithi170 ceph-mon[33394]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:12.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:11 smithi170 ceph-mon[33394]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-02-10T07:31:12.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:11 smithi170 ceph-mon[33394]: osdmap e49: 7 total, 6 up, 7 in 2024-02-10T07:31:12.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:11 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:12.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:11 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:12.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:11 smithi170 ceph-mon[33394]: osd.6 [v2:172.21.15.170:6816/1764436984,v1:172.21.15.170:6817/1764436984] boot 2024-02-10T07:31:12.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:11 smithi170 ceph-mon[33394]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T07:31:12.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:11 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:12.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[40743]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:12.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[40743]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-02-10T07:31:12.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[40743]: osdmap e49: 7 total, 6 up, 7 in 2024-02-10T07:31:12.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:12.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:12.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[40743]: osd.6 [v2:172.21.15.170:6816/1764436984,v1:172.21.15.170:6817/1764436984] boot 2024-02-10T07:31:12.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[40743]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T07:31:12.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:12.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[29816]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-02-10T07:31:12.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[29816]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-02-10T07:31:12.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[29816]: osdmap e49: 7 total, 6 up, 7 in 2024-02-10T07:31:12.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:12.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:12.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[29816]: osd.6 [v2:172.21.15.170:6816/1764436984,v1:172.21.15.170:6817/1764436984] boot 2024-02-10T07:31:12.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[29816]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T07:31:12.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:11 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:31:13.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:12 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:31:13.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:12 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:31:13.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:12 smithi170 ceph-mon[33394]: osdmap e51: 7 total, 7 up, 7 in 2024-02-10T07:31:13.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:12 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:31:13.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:12 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:31:13.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:12 smithi008 ceph-mon[29816]: osdmap e51: 7 total, 7 up, 7 in 2024-02-10T07:31:13.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:12 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:31:13.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:12 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:31:13.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:12 smithi008 ceph-mon[40743]: osdmap e51: 7 total, 7 up, 7 in 2024-02-10T07:31:13.837 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-10T07:31:13.837 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-10T07:31:13.837 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-02-10T07:31:13.837 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-02-10T07:31:13.837 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0107911 s, 972 MB/s 2024-02-10T07:31:13.837 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-02-10T07:31:14.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:13 smithi170 ceph-mon[33394]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:14.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:13 smithi170 ceph-mon[33394]: osdmap e52: 7 total, 7 up, 7 in 2024-02-10T07:31:14.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:13 smithi008 ceph-mon[29816]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:14.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:13 smithi008 ceph-mon[29816]: osdmap e52: 7 total, 7 up, 7 in 2024-02-10T07:31:14.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:13 smithi008 ceph-mon[40743]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:14.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:13 smithi008 ceph-mon[40743]: osdmap e52: 7 total, 7 up, 7 in 2024-02-10T07:31:14.591 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_1 2024-02-10T07:31:14.831 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:31:15.242 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:15 smithi170 ceph-mon[33394]: Detected new or changed devices on smithi170 2024-02-10T07:31:15.243 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:15 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:15.243 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:15 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:15.243 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:15 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:15.243 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:15 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:15.243 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:15 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:15.243 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:15 smithi170 ceph-mon[33394]: Adjusting osd_memory_target on smithi170 to 5696M 2024-02-10T07:31:15.243 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:15 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:15.243 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:15 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:15.243 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:15 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:15.243 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:15 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:15.243 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:15 smithi170 ceph-mon[33394]: osdmap e53: 7 total, 7 up, 7 in 2024-02-10T07:31:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[29816]: Detected new or changed devices on smithi170 2024-02-10T07:31:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[29816]: Adjusting osd_memory_target on smithi170 to 5696M 2024-02-10T07:31:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:15.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[29816]: osdmap e53: 7 total, 7 up, 7 in 2024-02-10T07:31:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[40743]: Detected new or changed devices on smithi170 2024-02-10T07:31:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[40743]: Adjusting osd_memory_target on smithi170 to 5696M 2024-02-10T07:31:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:15.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:15.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:15 smithi008 ceph-mon[40743]: osdmap e53: 7 total, 7 up, 7 in 2024-02-10T07:31:16.211 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:16 smithi170 ceph-mon[33394]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:16.211 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:16 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:31:16.211 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:16 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:31:16.211 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:16 smithi170 ceph-mon[33394]: osdmap e54: 7 total, 7 up, 7 in 2024-02-10T07:31:16.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:16 smithi008 ceph-mon[29816]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:16.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:16 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:31:16.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:16 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:31:16.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:16 smithi008 ceph-mon[29816]: osdmap e54: 7 total, 7 up, 7 in 2024-02-10T07:31:16.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:16 smithi008 ceph-mon[40743]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:16.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:16 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:31:16.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:16 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:31:16.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:16 smithi008 ceph-mon[40743]: osdmap e54: 7 total, 7 up, 7 in 2024-02-10T07:31:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:17 smithi170 ceph-mon[33394]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:17 smithi170 ceph-mon[33394]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:17 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:31:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:17 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:31:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:17 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:18.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:17 smithi008 ceph-mon[40743]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:18.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:17 smithi008 ceph-mon[40743]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:18.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:17 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:31:18.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:17 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:31:18.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:17 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:18.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:17 smithi008 ceph-mon[29816]: pgmap v164: 1 pgs: 1 remapped+peering; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:18.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:17 smithi008 ceph-mon[29816]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:18.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:17 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T07:31:18.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:17 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T07:31:18.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:17 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:19 smithi170 ceph-mon[33394]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-02-10T07:31:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:19 smithi170 ceph-mon[33394]: from='client.? 172.21.15.170:0/2583618423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e303441-b454-4750-8c5a-96c5348129a8"}]: dispatch 2024-02-10T07:31:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:19 smithi170 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e303441-b454-4750-8c5a-96c5348129a8"}]: dispatch 2024-02-10T07:31:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:19 smithi170 ceph-mon[33394]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e303441-b454-4750-8c5a-96c5348129a8"}]': finished 2024-02-10T07:31:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:19 smithi170 ceph-mon[33394]: osdmap e55: 8 total, 7 up, 8 in 2024-02-10T07:31:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:19 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:20.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:19 smithi008 ceph-mon[29816]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-02-10T07:31:20.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:19 smithi008 ceph-mon[29816]: from='client.? 172.21.15.170:0/2583618423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e303441-b454-4750-8c5a-96c5348129a8"}]: dispatch 2024-02-10T07:31:20.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:19 smithi008 ceph-mon[29816]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e303441-b454-4750-8c5a-96c5348129a8"}]: dispatch 2024-02-10T07:31:20.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:19 smithi008 ceph-mon[29816]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e303441-b454-4750-8c5a-96c5348129a8"}]': finished 2024-02-10T07:31:20.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:19 smithi008 ceph-mon[29816]: osdmap e55: 8 total, 7 up, 8 in 2024-02-10T07:31:20.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:19 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:20.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:19 smithi008 ceph-mon[40743]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-02-10T07:31:20.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:19 smithi008 ceph-mon[40743]: from='client.? 172.21.15.170:0/2583618423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e303441-b454-4750-8c5a-96c5348129a8"}]: dispatch 2024-02-10T07:31:20.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:19 smithi008 ceph-mon[40743]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e303441-b454-4750-8c5a-96c5348129a8"}]: dispatch 2024-02-10T07:31:20.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:19 smithi008 ceph-mon[40743]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e303441-b454-4750-8c5a-96c5348129a8"}]': finished 2024-02-10T07:31:20.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:19 smithi008 ceph-mon[40743]: osdmap e55: 8 total, 7 up, 8 in 2024-02-10T07:31:20.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:19 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:20.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:20 smithi170 ceph-mon[33394]: from='client.? 172.21.15.170:0/3001056754' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:31:21.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:20 smithi008 ceph-mon[29816]: from='client.? 172.21.15.170:0/3001056754' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:31:21.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:20 smithi008 ceph-mon[40743]: from='client.? 172.21.15.170:0/3001056754' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T07:31:21.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:21 smithi170 ceph-mon[33394]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T07:31:21.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:21 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:31:21.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:21 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:31:22.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:21 smithi008 ceph-mon[29816]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T07:31:22.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:21 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:31:22.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:21 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:31:22.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:21 smithi008 ceph-mon[40743]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 215 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T07:31:22.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:21 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:31:22.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:21 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:31:22.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:22 smithi170 ceph-mon[33394]: osdmap e56: 8 total, 7 up, 8 in 2024-02-10T07:31:22.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:22 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:23.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:22 smithi008 ceph-mon[29816]: osdmap e56: 8 total, 7 up, 8 in 2024-02-10T07:31:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:22 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:23.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:22 smithi008 ceph-mon[40743]: osdmap e56: 8 total, 7 up, 8 in 2024-02-10T07:31:23.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:22 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:23 smithi170 ceph-mon[33394]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-10T07:31:24.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:23 smithi008 ceph-mon[29816]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-10T07:31:24.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:23 smithi008 ceph-mon[40743]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-10T07:31:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:25 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:31:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:25 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:31:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:25 smithi170 ceph-mon[33394]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T07:31:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:25 smithi170 ceph-mon[33394]: osdmap e57: 8 total, 7 up, 8 in 2024-02-10T07:31:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:25 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:26.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:25 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:31:26.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:25 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:31:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:25 smithi008 ceph-mon[29816]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T07:31:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:25 smithi008 ceph-mon[29816]: osdmap e57: 8 total, 7 up, 8 in 2024-02-10T07:31:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:25 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:26.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:25 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:31:26.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:25 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:31:26.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:25 smithi008 ceph-mon[40743]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T07:31:26.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:25 smithi008 ceph-mon[40743]: osdmap e57: 8 total, 7 up, 8 in 2024-02-10T07:31:26.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:25 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:26.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:26 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T07:31:26.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:26 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:26.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:26 smithi170 ceph-mon[33394]: Deploying daemon osd.7 on smithi170 2024-02-10T07:31:27.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:26 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T07:31:27.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:26 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:27.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:26 smithi008 ceph-mon[29816]: Deploying daemon osd.7 on smithi170 2024-02-10T07:31:27.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:26 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T07:31:27.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:26 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:27.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:26 smithi008 ceph-mon[40743]: Deploying daemon osd.7 on smithi170 2024-02-10T07:31:27.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:27 smithi170 ceph-mon[33394]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:28.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:27 smithi008 ceph-mon[29816]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:28.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:27 smithi008 ceph-mon[40743]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:29.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:29 smithi170 ceph-mon[33394]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:30.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:29 smithi008 ceph-mon[29816]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:30.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:29 smithi008 ceph-mon[40743]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:32.470 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:32 smithi170 ceph-mon[33394]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:32.470 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:32 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:32.470 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:32 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:32.470 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:32 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:32.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:32 smithi008 ceph-mon[40743]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:32.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:32 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:32.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:32 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:32.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:32 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:32.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:32 smithi008 ceph-mon[29816]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:32.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:32 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:32.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:32 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:32.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:32 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:34 smithi008 ceph-mon[40743]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:34 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:34 smithi008 ceph-mon[29816]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:34 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:34.589 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:34 smithi170 ceph-mon[33394]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:34.589 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:34 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:34.938 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 7 on host 'smithi170' 2024-02-10T07:31:35.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:35.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:35 smithi170 ceph-mon[33394]: from='osd.7 [v2:172.21.15.170:6824/4048577717,v1:172.21.15.170:6825/4048577717]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:31:35.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:35 smithi170 ceph-mon[33394]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:31:35.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:35.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:35 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:35.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:35.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:35 smithi008 ceph-mon[29816]: from='osd.7 [v2:172.21.15.170:6824/4048577717,v1:172.21.15.170:6825/4048577717]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:31:35.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:35 smithi008 ceph-mon[29816]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:31:35.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:35.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:35 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:35.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:35.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:35 smithi008 ceph-mon[40743]: from='osd.7 [v2:172.21.15.170:6824/4048577717,v1:172.21.15.170:6825/4048577717]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:31:35.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:35 smithi008 ceph-mon[40743]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T07:31:35.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:35.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:35 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:35.597 DEBUG:teuthology.orchestra.run.smithi170:osd.7> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.7.service 2024-02-10T07:31:35.599 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-10T07:31:35.599 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T07:31:35.854 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:31:36.234 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:36 smithi008 ceph-mon[29816]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:36.234 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:36 smithi008 ceph-mon[29816]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T07:31:36.234 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:36 smithi008 ceph-mon[29816]: osdmap e58: 8 total, 7 up, 8 in 2024-02-10T07:31:36.234 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:36 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:36.234 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:36 smithi008 ceph-mon[29816]: from='osd.7 [v2:172.21.15.170:6824/4048577717,v1:172.21.15.170:6825/4048577717]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:31:36.234 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:36 smithi008 ceph-mon[29816]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:31:36.234 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:36 smithi008 ceph-mon[40743]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:36.235 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:36 smithi008 ceph-mon[40743]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T07:31:36.235 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:36 smithi008 ceph-mon[40743]: osdmap e58: 8 total, 7 up, 8 in 2024-02-10T07:31:36.235 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:36 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:36.235 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:36 smithi008 ceph-mon[40743]: from='osd.7 [v2:172.21.15.170:6824/4048577717,v1:172.21.15.170:6825/4048577717]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:31:36.235 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:36 smithi008 ceph-mon[40743]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:31:36.616 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:36 smithi170 ceph-mon[33394]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:36.617 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:36 smithi170 ceph-mon[33394]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T07:31:36.617 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:36 smithi170 ceph-mon[33394]: osdmap e58: 8 total, 7 up, 8 in 2024-02-10T07:31:36.617 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:36 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:36.617 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:36 smithi170 ceph-mon[33394]: from='osd.7 [v2:172.21.15.170:6824/4048577717,v1:172.21.15.170:6825/4048577717]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:31:36.617 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:36 smithi170 ceph-mon[33394]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:31:36.617 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:31:36 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:31:36.182+0000 7f082e5ca700 -1 osd.7 0 waiting for initial osdmap 2024-02-10T07:31:36.617 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:31:36 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:31:36.188+0000 7f082a3bd700 -1 osd.7 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:31:37.417 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:37 smithi008 ceph-mon[29816]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-02-10T07:31:37.417 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:37 smithi008 ceph-mon[29816]: osdmap e59: 8 total, 7 up, 8 in 2024-02-10T07:31:37.417 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:37.417 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:37.417 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:37 smithi008 ceph-mon[29816]: osd.7 [v2:172.21.15.170:6824/4048577717,v1:172.21.15.170:6825/4048577717] boot 2024-02-10T07:31:37.418 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:37 smithi008 ceph-mon[29816]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T07:31:37.418 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:37 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:37.418 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:37 smithi008 ceph-mon[40743]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-02-10T07:31:37.418 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:37 smithi008 ceph-mon[40743]: osdmap e59: 8 total, 7 up, 8 in 2024-02-10T07:31:37.418 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:37.418 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:37.418 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:37 smithi008 ceph-mon[40743]: osd.7 [v2:172.21.15.170:6824/4048577717,v1:172.21.15.170:6825/4048577717] boot 2024-02-10T07:31:37.418 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:37 smithi008 ceph-mon[40743]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T07:31:37.418 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:37 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:37 smithi170 ceph-mon[33394]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-02-10T07:31:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:37 smithi170 ceph-mon[33394]: osdmap e59: 8 total, 7 up, 8 in 2024-02-10T07:31:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:37 smithi170 ceph-mon[33394]: osd.7 [v2:172.21.15.170:6824/4048577717,v1:172.21.15.170:6825/4048577717] boot 2024-02-10T07:31:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:37 smithi170 ceph-mon[33394]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T07:31:37.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:37 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:31:37.862 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:31:38.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:38 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:31:38.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:38 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:31:38.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:38 smithi170 ceph-mon[33394]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:38.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:38 smithi170 ceph-mon[33394]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T07:31:38.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:38 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2098240247' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T07:31:38.451 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:38 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:31:38.452 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:38 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:31:38.452 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:38 smithi008 ceph-mon[40743]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:38.452 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:38 smithi008 ceph-mon[40743]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T07:31:38.452 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:38 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2098240247' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T07:31:38.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:38 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:31:38.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:38 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:31:38.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:38 smithi008 ceph-mon[29816]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-02-10T07:31:38.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:38 smithi008 ceph-mon[29816]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T07:31:38.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:38 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2098240247' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T07:31:38.479 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":61,"num_osds":8,"num_up_osds":8,"osd_up_since":1707550296,"num_in_osds":8,"osd_in_since":1707550279,"num_remapped_pgs":1} 2024-02-10T07:31:38.480 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T07:31:38.730 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:31:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: Detected new or changed devices on smithi170 2024-02-10T07:31:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: Adjusting osd_memory_target on smithi170 to 4272M 2024-02-10T07:31:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:39.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:39.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:39.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[40743]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T07:31:39.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: Detected new or changed devices on smithi170 2024-02-10T07:31:39.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:39.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:39.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:39.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:39.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: Adjusting osd_memory_target on smithi170 to 4272M 2024-02-10T07:31:39.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:39.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:39.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:39.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:39.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:39.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:39 smithi008 ceph-mon[29816]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T07:31:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: Detected new or changed devices on smithi170 2024-02-10T07:31:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:31:39.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: Adjusting osd_memory_target on smithi170 to 4272M 2024-02-10T07:31:39.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:39.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:39.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:39.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:39.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:39.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:39.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:39 smithi170 ceph-mon[33394]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T07:31:40.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:40 smithi008 ceph-mon[40743]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:31:40.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:40 smithi008 ceph-mon[40743]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T07:31:40.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:40 smithi008 ceph-mon[29816]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:31:40.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:40 smithi008 ceph-mon[29816]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T07:31:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:40 smithi170 ceph-mon[33394]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:31:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:40 smithi170 ceph-mon[33394]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T07:31:40.654 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:31:40.654 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":64,"fsid":"8c8244e6-c7e5-11ee-95b8-87774f69a715","created":"2024-02-10T07:26:21.316239+0000","modified":"2024-02-10T07:31:40.608328+0000","last_up_change":"2024-02-10T07:31:36.599653+0000","last_in_change":"2024-02-10T07:31:19.283633+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T07:29:32.704872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"63abc4f1-1c83-4e03-8592-959c51d10847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6803","nonce":1442795455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6805","nonce":1442795455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6809","nonce":1442795455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6807","nonce":1442795455}]},"public_addr":"172.21.15.8:6803/1442795455","cluster_addr":"172.21.15.8:6805/1442795455","heartbeat_back_addr":"172.21.15.8:6809/1442795455","heartbeat_front_addr":"172.21.15.8:6807/1442795455","state":["exists","up"]},{"osd":1,"uuid":"88307389-bfa8-402e-92b9-9fa5094262c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6811","nonce":3278758893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6813","nonce":3278758893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6817","nonce":3278758893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6815","nonce":3278758893}]},"public_addr":"172.21.15.8:6811/3278758893","cluster_addr":"172.21.15.8:6813/3278758893","heartbeat_back_addr":"172.21.15.8:6817/3278758893","heartbeat_front_addr":"172.21.15.8:6815/3278758893","state":["exists","up"]},{"osd":2,"uuid":"c6120003-9ab5-4cf5-87ac-6351ed2c6226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6819","nonce":481314802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6821","nonce":481314802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6825","nonce":481314802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6823","nonce":481314802}]},"public_addr":"172.21.15.8:6819/481314802","cluster_addr":"172.21.15.8:6821/481314802","heartbeat_back_addr":"172.21.15.8:6825/481314802","heartbeat_front_addr":"172.21.15.8:6823/481314802","state":["exists","up"]},{"osd":3,"uuid":"6ca8f6f3-013a-48cb-9083-be37c1767489","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":3692331651},{"type":"v1","addr":"172.21.15.8:6827","nonce":3692331651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":3692331651},{"type":"v1","addr":"172.21.15.8:6829","nonce":3692331651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":3692331651},{"type":"v1","addr":"172.21.15.8:6833","nonce":3692331651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":3692331651},{"type":"v1","addr":"172.21.15.8:6831","nonce":3692331651}]},"public_addr":"172.21.15.8:6827/3692331651","cluster_addr":"172.21.15.8:6829/3692331651","heartbeat_back_addr":"172.21.15.8:6833/3692331651","heartbeat_front_addr":"172.21.15.8:6831/3692331651","state":["exists","up"]},{"osd":4,"uuid":"aa667766-e57f-447b-9a31-e3f3e61ac346","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6801","nonce":3997132504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6803","nonce":3997132504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6807","nonce":3997132504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6805","nonce":3997132504}]},"public_addr":"172.21.15.170:6801/3997132504","cluster_addr":"172.21.15.170:6803/3997132504","heartbeat_back_addr":"172.21.15.170:6807/3997132504","heartbeat_front_addr":"172.21.15.170:6805/3997132504","state":["exists","up"]},{"osd":5,"uuid":"2e9ab14d-1aff-4638-9f41-311f6b5f6770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2364068245},{"type":"v1","addr":"172.21.15.170:6809","nonce":2364068245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2364068245},{"type":"v1","addr":"172.21.15.170:6811","nonce":2364068245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2364068245},{"type":"v1","addr":"172.21.15.170:6815","nonce":2364068245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2364068245},{"type":"v1","addr":"172.21.15.170:6813","nonce":2364068245}]},"public_addr":"172.21.15.170:6809/2364068245","cluster_addr":"172.21.15.170:6811/2364068245","heartbeat_back_addr":"172.21.15.170:6815/2364068245","heartbeat_front_addr":"172.21.15.170:6813/2364068245","state":["exists","up"]},{"osd":6,"uuid":"61970c81-4b79-4743-a379-50d5d102b373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764436984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764436984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764436984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764436984}]},"public_addr":"172.21.15.170:6817/1764436984","cluster_addr":"172.21.15.170:6819/1764436984","heartbeat_back_addr":"172.21.15.170:6823/1764436984","heartbeat_front_addr":"172.21.15.170:6821/1764436984","state":["exists","up"]},{"osd":7,"uuid":"2e303441-b454-4750-8c5a-96c5348129a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6825","nonce":4048577717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6827","nonce":4048577717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6831","nonce":4048577717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6829","nonce":4048577717}]},"public_addr":"172.21.15.170:6825/4048577717","cluster_addr":"172.21.15.170:6827/4048577717","heartbeat_back_addr":"172.21.15.170:6831/4048577717","heartbeat_front_addr":"172.21.15.170:6829/4048577717","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:08.446044+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:24.315254+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:30:54.067546+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:15.124689+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:20.860261+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:30:44.566346+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:09.904349+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:35.318623+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.8:0/2789652913":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/2381650878":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/1905866501":"2024-02-11T07:26:49.500037+0000","172.21.15.8:0/802075678":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6800/4001702773":"2024-02-11T07:26:49.500037+0000","172.21.15.8:0/1462056527":"2024-02-11T07:27:16.371197+0000","172.21.15.8:6801/4001702773":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6801/1412156979":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/2515892714":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6800/1412156979":"2024-02-11T07:27:16.371197+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T07:31:41.368 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-10T07:29:32.704872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-10T07:31:41.369 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-10T07:31:41.615 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:31:41.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:41 smithi170 ceph-mon[33394]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:31:41.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:41 smithi170 ceph-mon[33394]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T07:31:41.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:41 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2292426703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:31:41.898 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:41 smithi008 ceph-mon[40743]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:31:41.898 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:41 smithi008 ceph-mon[40743]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T07:31:41.898 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:41 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2292426703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:31:41.899 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:41 smithi008 ceph-mon[29816]: pgmap v185: 1 pgs: 1 remapped+peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:31:41.899 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:41 smithi008 ceph-mon[29816]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T07:31:41.899 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:41 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2292426703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:31:43.452 INFO:teuthology.orchestra.run.smithi008.stdout:pg_num: 1 2024-02-10T07:31:43.757 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:43 smithi008 ceph-mon[29816]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-02-10T07:31:43.757 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:43 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/634721810' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T07:31:43.758 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:43 smithi008 ceph-mon[40743]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-02-10T07:31:43.758 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:43 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/634721810' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T07:31:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:43 smithi170 ceph-mon[33394]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-02-10T07:31:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:43 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/634721810' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T07:31:44.122 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi008 2024-02-10T07:31:44.123 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi008=foo.a' 2024-02-10T07:31:44.373 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:31:45.859 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:45 smithi170 ceph-mon[33394]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T07:31:46.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:45 smithi008 ceph-mon[29816]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T07:31:46.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:45 smithi008 ceph-mon[40743]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-10T07:31:46.263 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled rgw.foo.a update... 2024-02-10T07:31:46.891 DEBUG:teuthology.orchestra.run.smithi008:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@rgw.foo.a.service 2024-02-10T07:31:46.893 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi170 2024-02-10T07:31:46.894 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-02-10T07:31:47.127 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:31:47.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[29816]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi008=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:47.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[29816]: Saving service rgw.foo.a spec with placement smithi008=foo.a;count:1 2024-02-10T07:31:47.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:47.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:47.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:47.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:47.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:47.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:31:47.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:31:47.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:47.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:47.524 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[40743]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi008=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:47.524 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[40743]: Saving service rgw.foo.a spec with placement smithi008=foo.a;count:1 2024-02-10T07:31:47.524 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:47.525 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:47.525 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:47.525 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:47.525 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:47.525 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:31:47.525 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:31:47.525 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:47.525 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:47 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:47.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:47 smithi170 ceph-mon[33394]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi008=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:47.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:47 smithi170 ceph-mon[33394]: Saving service rgw.foo.a spec with placement smithi008=foo.a;count:1 2024-02-10T07:31:47.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:47 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:47.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:47 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:47.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:47 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:47.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:47 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:47.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:47 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:47.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:47 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T07:31:47.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:47 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T07:31:47.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:47 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:47.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:47 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:48.502 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:48 smithi170 ceph-mon[33394]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-02-10T07:31:48.502 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:48 smithi170 ceph-mon[33394]: Deploying daemon rgw.foo.a on smithi008 2024-02-10T07:31:48.546 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:48 smithi008 ceph-mon[40743]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-02-10T07:31:48.546 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:48 smithi008 ceph-mon[40743]: Deploying daemon rgw.foo.a on smithi008 2024-02-10T07:31:48.546 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:48 smithi008 ceph-mon[29816]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-02-10T07:31:48.546 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:48 smithi008 ceph-mon[29816]: Deploying daemon rgw.foo.a on smithi008 2024-02-10T07:31:49.113 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:31:49 smithi008 systemd[1]: Starting Ceph rgw.foo.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:31:49.276 INFO:teuthology.orchestra.run.smithi170.stderr:pool 'datapool' created 2024-02-10T07:31:49.441 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:49 smithi008 ceph-mon[29816]: from='client.? 172.21.15.170:0/2571214878' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T07:31:49.442 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:49 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T07:31:49.442 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:49 smithi008 ceph-mon[40743]: from='client.? 172.21.15.170:0/2571214878' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T07:31:49.442 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:49 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T07:31:49.442 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:31:49 smithi008 podman[67608]: 2024-02-10 07:31:49.14816294 +0000 UTC m=+0.019499974 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:31:49.442 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:31:49 smithi008 podman[67608]: 2024-02-10 07:31:49.312762304 +0000 UTC m=+0.184099326 container create d8d2986f826582c7030e8f62dc19a7dd3b2266c70bed9bef59f60b3dee216164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True) 2024-02-10T07:31:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:49 smithi170 ceph-mon[33394]: from='client.? 172.21.15.170:0/2571214878' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T07:31:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:49 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T07:31:49.831 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:31:49 smithi008 podman[67608]: 2024-02-10 07:31:49.543790098 +0000 UTC m=+0.415127117 container init d8d2986f826582c7030e8f62dc19a7dd3b2266c70bed9bef59f60b3dee216164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-290212a) 2024-02-10T07:31:49.832 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:31:49 smithi008 podman[67608]: 2024-02-10 07:31:49.546461244 +0000 UTC m=+0.417798268 container start d8d2986f826582c7030e8f62dc19a7dd3b2266c70bed9bef59f60b3dee216164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-rgw-foo-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-10T07:31:49.832 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:31:49 smithi008 bash[67608]: d8d2986f826582c7030e8f62dc19a7dd3b2266c70bed9bef59f60b3dee216164 2024-02-10T07:31:49.832 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:31:49 smithi008 systemd[1]: Started Ceph rgw.foo.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:31:49.976 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- rbd pool init datapool 2024-02-10T07:31:50.216 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:31:50.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[40743]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T07:31:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-10T07:31:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[40743]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T07:31:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:50.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[29816]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T07:31:50.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-10T07:31:50.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[29816]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T07:31:50.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:50 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:50.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:50 smithi170 ceph-mon[33394]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-10T07:31:50.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:50 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-10T07:31:50.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:50 smithi170 ceph-mon[33394]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T07:31:50.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:50 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:50 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:50 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:50 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:50 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:50.588 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:50 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:31:51.518 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:51 smithi170 ceph-mon[33394]: Saving service rgw.foo.a spec with placement smithi008=foo.a;count:1 2024-02-10T07:31:51.519 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:51 smithi170 ceph-mon[33394]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T07:31:51.519 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:51 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3456391015' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T07:31:51.519 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:51 smithi170 ceph-mon[33394]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T07:31:51.519 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:51 smithi170 ceph-mon[33394]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T07:31:51.519 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:51 smithi170 ceph-mon[33394]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T07:31:51.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:51 smithi008 ceph-mon[29816]: Saving service rgw.foo.a spec with placement smithi008=foo.a;count:1 2024-02-10T07:31:51.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:51 smithi008 ceph-mon[29816]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T07:31:51.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:51 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3456391015' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T07:31:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:51 smithi008 ceph-mon[29816]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T07:31:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:51 smithi008 ceph-mon[29816]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T07:31:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:51 smithi008 ceph-mon[29816]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T07:31:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:51 smithi008 ceph-mon[40743]: Saving service rgw.foo.a spec with placement smithi008=foo.a;count:1 2024-02-10T07:31:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:51 smithi008 ceph-mon[40743]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T07:31:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:51 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3456391015' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T07:31:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:51 smithi008 ceph-mon[40743]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T07:31:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:51 smithi008 ceph-mon[40743]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T07:31:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:51 smithi008 ceph-mon[40743]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T07:31:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[29816]: pgmap v193: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:31:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[29816]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:31:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[29816]: from='client.? 172.21.15.170:0/2415151291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T07:31:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T07:31:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-10T07:31:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[29816]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T07:31:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:31:52.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1933562162' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:31:52.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[29816]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:31:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[40743]: pgmap v193: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:31:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[40743]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:31:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[40743]: from='client.? 172.21.15.170:0/2415151291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T07:31:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T07:31:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-10T07:31:52.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[40743]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T07:31:52.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:31:52.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1933562162' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:31:52.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:52 smithi008 ceph-mon[40743]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:31:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:52 smithi170 ceph-mon[33394]: pgmap v193: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:31:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:52 smithi170 ceph-mon[33394]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:31:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:52 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:52 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:52 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:52 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:52 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:31:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:52 smithi170 ceph-mon[33394]: from='client.? 172.21.15.170:0/2415151291' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T07:31:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:52 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T07:31:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:52 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-10T07:31:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:52 smithi170 ceph-mon[33394]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T07:31:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:52 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:31:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:52 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1933562162' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:31:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:52 smithi170 ceph-mon[33394]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T07:31:53.831 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:53 smithi008 ceph-mon[40743]: Checking dashboard <-> RGW credentials 2024-02-10T07:31:53.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:53 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T07:31:53.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:53 smithi008 ceph-mon[40743]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T07:31:53.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:53 smithi008 ceph-mon[40743]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T07:31:53.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:53 smithi008 ceph-mon[29816]: Checking dashboard <-> RGW credentials 2024-02-10T07:31:53.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:53 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T07:31:53.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:53 smithi008 ceph-mon[29816]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T07:31:53.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:53 smithi008 ceph-mon[29816]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T07:31:53.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:53 smithi170 ceph-mon[33394]: Checking dashboard <-> RGW credentials 2024-02-10T07:31:53.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:53 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T07:31:53.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:53 smithi170 ceph-mon[33394]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T07:31:53.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:53 smithi170 ceph-mon[33394]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T07:31:54.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:54 smithi008 ceph-mon[29816]: pgmap v196: 68 pgs: 5 creating+peering, 24 active+clean, 39 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-10T07:31:54.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:54 smithi008 ceph-mon[29816]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T07:31:54.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:54 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:31:54.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:54 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1933562162' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:31:54.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:54 smithi008 ceph-mon[29816]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:31:54.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:54 smithi008 ceph-mon[40743]: pgmap v196: 68 pgs: 5 creating+peering, 24 active+clean, 39 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-10T07:31:54.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:54 smithi008 ceph-mon[40743]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T07:31:54.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:54 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:31:54.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:54 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1933562162' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:31:54.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:54 smithi008 ceph-mon[40743]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:31:54.857 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:54 smithi170 ceph-mon[33394]: pgmap v196: 68 pgs: 5 creating+peering, 24 active+clean, 39 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-10T07:31:54.858 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:54 smithi170 ceph-mon[33394]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T07:31:54.858 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:54 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:31:54.858 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:54 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1933562162' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:31:54.858 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:54 smithi170 ceph-mon[33394]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T07:31:54.891 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.170 --placement '1;smithi170=iscsi.a' 2024-02-10T07:31:55.134 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:31:56.516 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:56 smithi170 ceph-mon[33394]: pgmap v199: 100 pgs: 6 creating+peering, 38 active+clean, 56 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-10T07:31:56.517 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:56 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T07:31:56.517 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:56 smithi170 ceph-mon[33394]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T07:31:56.517 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:56 smithi170 ceph-mon[33394]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T07:31:56.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:56 smithi008 ceph-mon[29816]: pgmap v199: 100 pgs: 6 creating+peering, 38 active+clean, 56 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-10T07:31:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:56 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T07:31:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:56 smithi008 ceph-mon[29816]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T07:31:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:56 smithi008 ceph-mon[29816]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T07:31:56.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:56 smithi008 ceph-mon[40743]: pgmap v199: 100 pgs: 6 creating+peering, 38 active+clean, 56 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-10T07:31:56.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:56 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T07:31:56.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:56 smithi008 ceph-mon[40743]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T07:31:56.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:56 smithi008 ceph-mon[40743]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T07:31:56.955 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled iscsi.datapool update... 2024-02-10T07:31:57.555 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:57 smithi170 ceph-mon[33394]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T07:31:57.556 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:57 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:31:57.556 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:57 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1933562162' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:31:57.556 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:57 smithi170 ceph-mon[33394]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:31:57.556 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:57 smithi170 ceph-mon[33394]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:31:57.556 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:57 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:57.582 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-02-10T07:31:57.582 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:31:57.582 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-02-10T07:31:57.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:57 smithi008 ceph-mon[29816]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T07:31:57.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:57 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:31:57.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:57 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1933562162' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:31:57.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:57 smithi008 ceph-mon[29816]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:31:57.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:57 smithi008 ceph-mon[29816]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:31:57.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:57 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:57.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:57 smithi008 ceph-mon[40743]: osdmap e72: 8 total, 8 up, 8 in 2024-02-10T07:31:57.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:57 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:31:57.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:57 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1933562162' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:31:57.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:57 smithi008 ceph-mon[40743]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T07:31:57.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:57 smithi008 ceph-mon[40743]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:31:57.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:57 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:57.618 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-02-10T07:31:57.618 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-02-10T07:31:57.647 DEBUG:teuthology.orchestra.run.smithi170:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@iscsi.iscsi.a.service 2024-02-10T07:31:57.690 INFO:tasks.cephadm:Adding prometheus.a on smithi170 2024-02-10T07:31:57.690 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch apply prometheus '1;smithi170=a' 2024-02-10T07:31:57.961 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:31:58.367 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:58 smithi170 ceph-mon[33394]: pgmap v202: 132 pgs: 1 creating+peering, 50 active+clean, 81 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-02-10T07:31:58.367 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:58 smithi170 ceph-mon[33394]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.170", "placement": "1;smithi170=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:58.368 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:58 smithi170 ceph-mon[33394]: Saving service iscsi.datapool spec with placement smithi170=iscsi.a;count:1 2024-02-10T07:31:58.368 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:58 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T07:31:58.368 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:58 smithi170 ceph-mon[33394]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T07:31:58.368 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:58 smithi170 ceph-mon[33394]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T07:31:58.368 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:58 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1933562162' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:31:58.368 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:58 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:31:58.368 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:58 smithi170 ceph-mon[33394]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:31:58.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[29816]: pgmap v202: 132 pgs: 1 creating+peering, 50 active+clean, 81 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-02-10T07:31:58.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[29816]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.170", "placement": "1;smithi170=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[29816]: Saving service iscsi.datapool spec with placement smithi170=iscsi.a;count:1 2024-02-10T07:31:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T07:31:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[29816]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T07:31:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[29816]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T07:31:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1933562162' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:31:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:31:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[29816]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:31:58.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[40743]: pgmap v202: 132 pgs: 1 creating+peering, 50 active+clean, 81 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-02-10T07:31:58.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[40743]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.170", "placement": "1;smithi170=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:31:58.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[40743]: Saving service iscsi.datapool spec with placement smithi170=iscsi.a;count:1 2024-02-10T07:31:58.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T07:31:58.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[40743]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T07:31:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[40743]: osdmap e73: 8 total, 8 up, 8 in 2024-02-10T07:31:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1933562162' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:31:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:31:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:58 smithi008 ceph-mon[40743]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T07:31:59.495 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:59 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T07:31:59.495 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:59 smithi170 ceph-mon[33394]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T07:31:59.495 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:59 smithi170 ceph-mon[33394]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T07:31:59.495 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:59 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:59.495 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:59 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:59.495 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:59 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:59.496 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:59 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:31:59.496 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:59 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T07:31:59.496 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:31:59 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:59.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T07:31:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[29816]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T07:31:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[29816]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T07:31:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:31:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T07:31:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3431741597' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T07:31:59.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[40743]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T07:31:59.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[40743]: osdmap e74: 8 total, 8 up, 8 in 2024-02-10T07:31:59.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:59.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:59.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:31:59.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T07:31:59.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T07:31:59.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:31:59 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:31:59.678 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled prometheus update... 2024-02-10T07:32:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:00 smithi170 ceph-mon[33394]: pgmap v205: 132 pgs: 19 creating+peering, 98 active+clean, 15 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-02-10T07:32:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:00 smithi170 ceph-mon[33394]: Deploying daemon iscsi.iscsi.a on smithi170 2024-02-10T07:32:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:00 smithi170 ceph-mon[33394]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:32:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:00 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:32:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:00 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:00 smithi008 ceph-mon[29816]: pgmap v205: 132 pgs: 19 creating+peering, 98 active+clean, 15 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-02-10T07:32:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:00 smithi008 ceph-mon[29816]: Deploying daemon iscsi.iscsi.a on smithi170 2024-02-10T07:32:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:00 smithi008 ceph-mon[29816]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:32:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:00 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:32:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:00 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:00 smithi008 ceph-mon[40743]: pgmap v205: 132 pgs: 19 creating+peering, 98 active+clean, 15 unknown; 578 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-02-10T07:32:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:00 smithi008 ceph-mon[40743]: Deploying daemon iscsi.iscsi.a on smithi170 2024-02-10T07:32:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:00 smithi008 ceph-mon[40743]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:32:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:00 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:32:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:00 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:01.359 DEBUG:teuthology.orchestra.run.smithi170:prometheus.a> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@prometheus.a.service 2024-02-10T07:32:01.362 INFO:tasks.cephadm:Adding node-exporter.a on smithi008 2024-02-10T07:32:01.362 INFO:tasks.cephadm:Adding node-exporter.b on smithi170 2024-02-10T07:32:01.362 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch apply node-exporter '2;smithi008=a;smithi170=b' 2024-02-10T07:32:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:01 smithi008 ceph-mon[29816]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:01 smithi008 ceph-mon[29816]: Saving service prometheus spec with placement smithi170=a;count:1 2024-02-10T07:32:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:01 smithi008 ceph-mon[40743]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:01 smithi008 ceph-mon[40743]: Saving service prometheus spec with placement smithi170=a;count:1 2024-02-10T07:32:01.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:01 smithi170 systemd[1]: Starting Ceph iscsi.iscsi.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:32:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:01 smithi170 ceph-mon[33394]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:01 smithi170 ceph-mon[33394]: Saving service prometheus spec with placement smithi170=a;count:1 2024-02-10T07:32:01.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:01 smithi170 podman[61079]: 2024-02-10 07:32:01.731210037 +0000 UTC m=+0.115280027 container create cf83e29530758e183eaffe0dbe745fb0aea6ce6bd0631843e0622e89d8e8407d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240131, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:32:01.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:01 smithi170 podman[61079]: 2024-02-10 07:32:01.63744178 +0000 UTC m=+0.021511784 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:32:01.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:01 smithi170 podman[61079]: 2024-02-10 07:32:01.916507387 +0000 UTC m=+0.300577387 container init cf83e29530758e183eaffe0dbe745fb0aea6ce6bd0631843e0622e89d8e8407d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-10T07:32:01.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:01 smithi170 podman[61079]: 2024-02-10 07:32:01.919591185 +0000 UTC m=+0.303661174 container start cf83e29530758e183eaffe0dbe745fb0aea6ce6bd0631843e0622e89d8e8407d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0) 2024-02-10T07:32:02.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:01 smithi170 bash[61079]: cf83e29530758e183eaffe0dbe745fb0aea6ce6bd0631843e0622e89d8e8407d 2024-02-10T07:32:02.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:02 smithi170 systemd[1]: Started Ceph iscsi.iscsi.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:32:02.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug Started the configuration object watcher 2024-02-10T07:32:02.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug Checking for config object changes every 1s 2024-02-10T07:32:02.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug Processing osd blocklist entries for this node 2024-02-10T07:32:02.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug Reading the configuration object to update local LIO configuration 2024-02-10T07:32:02.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug Configuration does not have an entry for this host(smithi170.front.sepia.ceph.com) - nothing to define to LIO 2024-02-10T07:32:02.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-02-10T07:32:02.564 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:02 smithi170 ceph-mon[33394]: pgmap v206: 132 pgs: 19 creating+peering, 109 active+clean, 4 unknown; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 2.1 KiB/s wr, 46 op/s 2024-02-10T07:32:02.564 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:02 smithi170 ceph-mon[33394]: osdmap e75: 8 total, 8 up, 8 in 2024-02-10T07:32:02.564 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:02 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:02.564 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:02 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:02.564 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:02 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:02.565 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:02 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:02.565 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:02 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:32:02.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:02 smithi008 ceph-mon[40743]: pgmap v206: 132 pgs: 19 creating+peering, 109 active+clean, 4 unknown; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 2.1 KiB/s wr, 46 op/s 2024-02-10T07:32:02.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:02 smithi008 ceph-mon[40743]: osdmap e75: 8 total, 8 up, 8 in 2024-02-10T07:32:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:02 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:02 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:02 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:02 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:02 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:32:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:02 smithi008 ceph-mon[29816]: pgmap v206: 132 pgs: 19 creating+peering, 109 active+clean, 4 unknown; 579 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 2.1 KiB/s wr, 46 op/s 2024-02-10T07:32:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:02 smithi008 ceph-mon[29816]: osdmap e75: 8 total, 8 up, 8 in 2024-02-10T07:32:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:02 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:02 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:02 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:02 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:02 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:32:02.688 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:32:03.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:03 smithi170 ceph-mon[33394]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-10T07:32:03.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:03 smithi170 ceph-mon[33394]: from='client.? 172.21.15.170:0/1828412466' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:32:03.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:03 smithi008 ceph-mon[29816]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-10T07:32:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:03 smithi008 ceph-mon[29816]: from='client.? 172.21.15.170:0/1828412466' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:32:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:03 smithi008 ceph-mon[40743]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-10T07:32:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:03 smithi008 ceph-mon[40743]: from='client.? 172.21.15.170:0/1828412466' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T07:32:04.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:04 smithi008 ceph-mon[29816]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 5.8 KiB/s wr, 180 op/s 2024-02-10T07:32:04.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:04 smithi008 ceph-mon[40743]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 5.8 KiB/s wr, 180 op/s 2024-02-10T07:32:04.590 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:04 smithi170 ceph-mon[33394]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 5.8 KiB/s wr, 180 op/s 2024-02-10T07:32:04.657 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled node-exporter update... 2024-02-10T07:32:05.538 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:05 smithi170 ceph-mon[33394]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T07:32:05.538 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:05 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:05.564 DEBUG:teuthology.orchestra.run.smithi008:node-exporter.a> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@node-exporter.a.service 2024-02-10T07:32:05.567 DEBUG:teuthology.orchestra.run.smithi170:node-exporter.b> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@node-exporter.b.service 2024-02-10T07:32:05.569 INFO:tasks.cephadm:Adding alertmanager.a on smithi008 2024-02-10T07:32:05.570 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch apply alertmanager '1;smithi008=a' 2024-02-10T07:32:05.593 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:05 smithi008 ceph-mon[29816]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T07:32:05.593 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:05 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:05.594 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:05 smithi008 ceph-mon[40743]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T07:32:05.594 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:05 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:06.271 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:32:06.627 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:06 smithi170 ceph-mon[33394]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 4.8 KiB/s wr, 156 op/s 2024-02-10T07:32:06.628 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:06 smithi170 ceph-mon[33394]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi008=a;smithi170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:06.628 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:06 smithi170 ceph-mon[33394]: Saving service node-exporter spec with placement smithi008=a;smithi170=b;count:2 2024-02-10T07:32:06.628 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:06 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:06.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:06 smithi008 ceph-mon[29816]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 4.8 KiB/s wr, 156 op/s 2024-02-10T07:32:06.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:06 smithi008 ceph-mon[29816]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi008=a;smithi170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:06.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:06 smithi008 ceph-mon[29816]: Saving service node-exporter spec with placement smithi008=a;smithi170=b;count:2 2024-02-10T07:32:06.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:06 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:06.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:06 smithi008 ceph-mon[40743]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 4.8 KiB/s wr, 156 op/s 2024-02-10T07:32:06.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:06 smithi008 ceph-mon[40743]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi008=a;smithi170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:06.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:06 smithi008 ceph-mon[40743]: Saving service node-exporter spec with placement smithi008=a;smithi170=b;count:2 2024-02-10T07:32:06.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:06 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:07.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:07 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:07.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:07 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:32:07.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:07 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:32:07.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:07 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:07.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:07 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:07.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:07 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:07.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:07 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:32:07.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:07 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:32:07.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:07 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:07.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:07 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:07.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:07 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:07.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:07 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:32:07.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:07 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:32:07.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:07 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:07.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:07 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:08.185 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled alertmanager update... 2024-02-10T07:32:08.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:08 smithi170 ceph-mon[33394]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.2 KiB/s wr, 137 op/s 2024-02-10T07:32:08.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:08 smithi170 ceph-mon[33394]: Deploying daemon prometheus.a on smithi170 2024-02-10T07:32:08.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:08 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:08.831 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:08 smithi008 ceph-mon[40743]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.2 KiB/s wr, 137 op/s 2024-02-10T07:32:08.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:08 smithi008 ceph-mon[40743]: Deploying daemon prometheus.a on smithi170 2024-02-10T07:32:08.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:08 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:08.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:08 smithi008 ceph-mon[29816]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.2 KiB/s wr, 137 op/s 2024-02-10T07:32:08.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:08 smithi008 ceph-mon[29816]: Deploying daemon prometheus.a on smithi170 2024-02-10T07:32:08.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:08 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:08.951 DEBUG:teuthology.orchestra.run.smithi008:alertmanager.a> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@alertmanager.a.service 2024-02-10T07:32:08.955 INFO:tasks.cephadm:Adding grafana.a on smithi170 2024-02-10T07:32:08.955 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph orch apply grafana '1;smithi170=a' 2024-02-10T07:32:09.193 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:32:09.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:09 smithi170 ceph-mon[33394]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi008=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:09 smithi170 ceph-mon[33394]: Saving service alertmanager spec with placement smithi008=a;count:1 2024-02-10T07:32:09.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:09 smithi008 ceph-mon[29816]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi008=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:09.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:09 smithi008 ceph-mon[29816]: Saving service alertmanager spec with placement smithi008=a;count:1 2024-02-10T07:32:09.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:09 smithi008 ceph-mon[40743]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi008=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:09.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:09 smithi008 ceph-mon[40743]: Saving service alertmanager spec with placement smithi008=a;count:1 2024-02-10T07:32:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:10 smithi170 ceph-mon[33394]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-02-10T07:32:10.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:10 smithi008 ceph-mon[29816]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-02-10T07:32:10.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:10 smithi008 ceph-mon[40743]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-02-10T07:32:11.711 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled grafana update... 2024-02-10T07:32:12.629 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:12 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:32:12.629 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:12 smithi170 ceph-mon[33394]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 2.9 KiB/s wr, 91 op/s 2024-02-10T07:32:12.629 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:12 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:12.629 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:12 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:12.655 DEBUG:teuthology.orchestra.run.smithi170:grafana.a> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@grafana.a.service 2024-02-10T07:32:12.657 INFO:tasks.cephadm:Setting up client nodes... 2024-02-10T07:32:12.657 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T07:32:12.685 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:12 smithi008 ceph-mon[29816]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 2.9 KiB/s wr, 91 op/s 2024-02-10T07:32:12.685 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:12 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:12.685 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:12 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:12.685 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:12 smithi008 ceph-mon[40743]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 2.9 KiB/s wr, 91 op/s 2024-02-10T07:32:12.686 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:12 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:12.686 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:12 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:12.914 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:13.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:13 smithi008 ceph-mon[29816]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:13.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:13 smithi008 ceph-mon[29816]: Saving service grafana spec with placement smithi170=a;count:1 2024-02-10T07:32:13.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:13 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:13.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:13 smithi008 ceph-mon[40743]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:13.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:13 smithi008 ceph-mon[40743]: Saving service grafana spec with placement smithi170=a;count:1 2024-02-10T07:32:13.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:13 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:13.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:13 smithi170 ceph-mon[33394]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:32:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:13 smithi170 ceph-mon[33394]: Saving service grafana spec with placement smithi170=a;count:1 2024-02-10T07:32:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:13 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:14.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:14 smithi170 ceph-mon[33394]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.4 KiB/s wr, 78 op/s 2024-02-10T07:32:14.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:14 smithi008 ceph-mon[29816]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.4 KiB/s wr, 78 op/s 2024-02-10T07:32:14.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:14 smithi008 ceph-mon[40743]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 2.4 KiB/s wr, 78 op/s 2024-02-10T07:32:14.928 INFO:teuthology.orchestra.run.smithi008.stdout:[client.0] 2024-02-10T07:32:14.928 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQB+Jsdl8HpENxAAvOpWuqlHNlPyxrecgL/gNg== 2024-02-10T07:32:15.496 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:15 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:32:15.496 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:15 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:32:15.496 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:15 smithi008 ceph-mon[29816]: osdmap e76: 8 total, 8 up, 8 in 2024-02-10T07:32:15.496 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:15 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/871663581' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:32:15.496 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:15 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/871663581' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T07:32:15.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:15 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:32:15.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:15 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:32:15.497 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:15 smithi008 ceph-mon[40743]: osdmap e76: 8 total, 8 up, 8 in 2024-02-10T07:32:15.497 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:15 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/871663581' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:32:15.497 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:15 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/871663581' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T07:32:15.522 DEBUG:teuthology.orchestra.run.smithi008:> set -ex 2024-02-10T07:32:15.522 DEBUG:teuthology.orchestra.run.smithi008:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-10T07:32:15.522 DEBUG:teuthology.orchestra.run.smithi008:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-10T07:32:15.606 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T07:32:15.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:15 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:32:15.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:15 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:32:15.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:15 smithi170 ceph-mon[33394]: osdmap e76: 8 total, 8 up, 8 in 2024-02-10T07:32:15.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:15 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/871663581' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:32:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:15 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/871663581' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T07:32:15.843 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:32:16.559 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:16 smithi170 ceph-mon[33394]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-02-10T07:32:16.752 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:16 smithi008 ceph-mon[29816]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-02-10T07:32:16.752 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:16 smithi008 ceph-mon[40743]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-02-10T07:32:17.617 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:17 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-10T07:32:17.618 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:17 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-10T07:32:17.618 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:17 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T07:32:17.618 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:17 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:32:17.618 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:17 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:32:17.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:17 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-10T07:32:17.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:17 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-10T07:32:17.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:17 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T07:32:17.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:17 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:32:17.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:17 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:32:17.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:17 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-10T07:32:17.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:17 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-10T07:32:17.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:17 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-02-10T07:32:17.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:17 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:32:17.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:17 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:32:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:18 smithi170 ceph-mon[33394]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-02-10T07:32:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:18 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-10T07:32:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:18 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-10T07:32:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:18 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-10T07:32:18.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:18 smithi170 ceph-mon[33394]: osdmap e77: 8 total, 8 up, 8 in 2024-02-10T07:32:18.728 INFO:teuthology.orchestra.run.smithi170.stdout:[client.1] 2024-02-10T07:32:18.728 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQCCJsdl8D8zKxAAgN659pSppDFYSRFi3kty8g== 2024-02-10T07:32:18.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:18 smithi008 ceph-mon[29816]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-02-10T07:32:18.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:18 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-10T07:32:18.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:18 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-10T07:32:18.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:18 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-10T07:32:18.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:18 smithi008 ceph-mon[29816]: osdmap e77: 8 total, 8 up, 8 in 2024-02-10T07:32:18.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:18 smithi008 ceph-mon[40743]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-02-10T07:32:18.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:18 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-10T07:32:18.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:18 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-10T07:32:18.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:18 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-02-10T07:32:18.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:18 smithi008 ceph-mon[40743]: osdmap e77: 8 total, 8 up, 8 in 2024-02-10T07:32:19.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:19 smithi008 ceph-mon[29816]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T07:32:19.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:19 smithi008 ceph-mon[29816]: from='client.? 172.21.15.170:0/726665862' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:32:19.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:19 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:32:19.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:19 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T07:32:19.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:19 smithi008 ceph-mon[40743]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T07:32:19.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:19 smithi008 ceph-mon[40743]: from='client.? 172.21.15.170:0/726665862' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:32:19.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:19 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:32:19.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:19 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T07:32:19.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:19 smithi170 ceph-mon[33394]: osdmap e78: 8 total, 8 up, 8 in 2024-02-10T07:32:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:19 smithi170 ceph-mon[33394]: from='client.? 172.21.15.170:0/726665862' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:32:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:19 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T07:32:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:19 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T07:32:20.650 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:20 smithi170 ceph-mon[33394]: pgmap v219: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:32:20.651 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-02-10T07:32:20.651 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-10T07:32:20.651 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-10T07:32:20.738 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-10T07:32:20.738 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-10T07:32:20.739 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph mgr dump --format=json 2024-02-10T07:32:20.769 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:20 smithi008 ceph-mon[29816]: pgmap v219: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:32:20.769 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:20 smithi008 ceph-mon[40743]: pgmap v219: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:32:20.996 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:22.490 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:22 smithi170 ceph-mon[33394]: pgmap v220: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 675 B/s rd, 0 op/s 2024-02-10T07:32:22.491 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:32:22.641 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:22 smithi008 ceph-mon[29816]: pgmap v220: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 675 B/s rd, 0 op/s 2024-02-10T07:32:22.641 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:22 smithi008 ceph-mon[40743]: pgmap v220: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 675 B/s rd, 0 op/s 2024-02-10T07:32:22.917 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:32:23.383 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:32:23 smithi170 systemd[1]: Started Ceph prometheus.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:32:23.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:23 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:23.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:23 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3291993627' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T07:32:23.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:23 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:23.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:23 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:23.563 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:23 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:23.563 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:23 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T07:32:23.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:23 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:23.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:23 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3291993627' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T07:32:23.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:23 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:23.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:23 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:23.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:23 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:23.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:23 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T07:32:23.589 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":2677792951},{"type":"v1","addr":"172.21.15.8:6801","nonce":2677792951}]},"active_addr":"172.21.15.8:6801/2677792951","active_change":"2024-02-10T07:27:16.371319+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.8:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":4074987889}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":1624888680}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":3905205183}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":2210906095}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.8:0","nonce":3314827874}]}]} 2024-02-10T07:32:23.593 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-10T07:32:23.593 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-10T07:32:23.593 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T07:32:23.871 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:23 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:23 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3291993627' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T07:32:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:23 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:23 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:23 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' 2024-02-10T07:32:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:23 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T07:32:24.269 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:24 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ignoring --setuser ceph since I am not root 2024-02-10T07:32:24.269 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:24 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ignoring --setgroup ceph since I am not root 2024-02-10T07:32:24.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:24 smithi008 ceph-mon[29816]: pgmap v221: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-02-10T07:32:24.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:24 smithi008 ceph-mon[29816]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T07:32:24.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:24 smithi008 ceph-mon[29816]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-10T07:32:24.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:24 smithi008 ceph-mon[40743]: pgmap v221: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-02-10T07:32:24.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:24 smithi008 ceph-mon[40743]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T07:32:24.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:24 smithi008 ceph-mon[40743]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-10T07:32:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:24 smithi170 ceph-mon[33394]: pgmap v221: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 41 B/s, 1 objects/s recovering 2024-02-10T07:32:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:24 smithi170 ceph-mon[33394]: from='mgr.14152 172.21.15.8:0/611149138' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T07:32:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:24 smithi170 ceph-mon[33394]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-10T07:32:24.633 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: ignoring --setuser ceph since I am not root 2024-02-10T07:32:24.633 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: ignoring --setgroup ceph since I am not root 2024-02-10T07:32:24.633 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:24 smithi170 ceph-mgr[35259]: -- 172.21.15.170:0/816464567 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d658e421a0 con 0x55d658e19000 2024-02-10T07:32:25.031 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:24 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:24.750+0000 7f408ef4c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:32:25.031 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:24 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:24.846+0000 7f408ef4c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:32:25.135 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:24.715+0000 7f0a1cadf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T07:32:25.135 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:24.806+0000 7f0a1cadf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T07:32:25.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:25 smithi008 ceph-mon[29816]: 1.0 scrub starts 2024-02-10T07:32:25.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:25 smithi008 ceph-mon[29816]: 1.0 scrub ok 2024-02-10T07:32:25.582 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:25 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:25.533+0000 7f408ef4c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:32:25.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:25 smithi008 ceph-mon[40743]: 1.0 scrub starts 2024-02-10T07:32:25.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:25 smithi008 ceph-mon[40743]: 1.0 scrub ok 2024-02-10T07:32:25.670 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:25 smithi170 ceph-mon[33394]: 1.0 scrub starts 2024-02-10T07:32:25.671 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:25 smithi170 ceph-mon[33394]: 1.0 scrub ok 2024-02-10T07:32:25.671 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:25.439+0000 7f0a1cadf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T07:32:25.671 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:25.529+0000 7f0a1cadf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:32:25.671 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:25.669+0000 7f0a1cadf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:32:25.743 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:32:25.744 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":79,"fsid":"8c8244e6-c7e5-11ee-95b8-87774f69a715","created":"2024-02-10T07:26:21.316239+0000","modified":"2024-02-10T07:32:25.395063+0000","last_up_change":"2024-02-10T07:31:36.599653+0000","last_in_change":"2024-02-10T07:31:19.283633+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T07:29:32.704872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-10T07:31:49.150741+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-10T07:31:49.831683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-10T07:31:51.622544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-10T07:31:53.550827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-10T07:31:55.600117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"63abc4f1-1c83-4e03-8592-959c51d10847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6803","nonce":1442795455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6805","nonce":1442795455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6809","nonce":1442795455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6807","nonce":1442795455}]},"public_addr":"172.21.15.8:6803/1442795455","cluster_addr":"172.21.15.8:6805/1442795455","heartbeat_back_addr":"172.21.15.8:6809/1442795455","heartbeat_front_addr":"172.21.15.8:6807/1442795455","state":["exists","up"]},{"osd":1,"uuid":"88307389-bfa8-402e-92b9-9fa5094262c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6811","nonce":3278758893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6813","nonce":3278758893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6817","nonce":3278758893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6815","nonce":3278758893}]},"public_addr":"172.21.15.8:6811/3278758893","cluster_addr":"172.21.15.8:6813/3278758893","heartbeat_back_addr":"172.21.15.8:6817/3278758893","heartbeat_front_addr":"172.21.15.8:6815/3278758893","state":["exists","up"]},{"osd":2,"uuid":"c6120003-9ab5-4cf5-87ac-6351ed2c6226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6819","nonce":481314802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6821","nonce":481314802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6825","nonce":481314802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6823","nonce":481314802}]},"public_addr":"172.21.15.8:6819/481314802","cluster_addr":"172.21.15.8:6821/481314802","heartbeat_back_addr":"172.21.15.8:6825/481314802","heartbeat_front_addr":"172.21.15.8:6823/481314802","state":["exists","up"]},{"osd":3,"uuid":"6ca8f6f3-013a-48cb-9083-be37c1767489","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":3692331651},{"type":"v1","addr":"172.21.15.8:6827","nonce":3692331651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":3692331651},{"type":"v1","addr":"172.21.15.8:6829","nonce":3692331651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":3692331651},{"type":"v1","addr":"172.21.15.8:6833","nonce":3692331651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":3692331651},{"type":"v1","addr":"172.21.15.8:6831","nonce":3692331651}]},"public_addr":"172.21.15.8:6827/3692331651","cluster_addr":"172.21.15.8:6829/3692331651","heartbeat_back_addr":"172.21.15.8:6833/3692331651","heartbeat_front_addr":"172.21.15.8:6831/3692331651","state":["exists","up"]},{"osd":4,"uuid":"aa667766-e57f-447b-9a31-e3f3e61ac346","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6801","nonce":3997132504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6803","nonce":3997132504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6807","nonce":3997132504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6805","nonce":3997132504}]},"public_addr":"172.21.15.170:6801/3997132504","cluster_addr":"172.21.15.170:6803/3997132504","heartbeat_back_addr":"172.21.15.170:6807/3997132504","heartbeat_front_addr":"172.21.15.170:6805/3997132504","state":["exists","up"]},{"osd":5,"uuid":"2e9ab14d-1aff-4638-9f41-311f6b5f6770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2364068245},{"type":"v1","addr":"172.21.15.170:6809","nonce":2364068245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2364068245},{"type":"v1","addr":"172.21.15.170:6811","nonce":2364068245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2364068245},{"type":"v1","addr":"172.21.15.170:6815","nonce":2364068245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2364068245},{"type":"v1","addr":"172.21.15.170:6813","nonce":2364068245}]},"public_addr":"172.21.15.170:6809/2364068245","cluster_addr":"172.21.15.170:6811/2364068245","heartbeat_back_addr":"172.21.15.170:6815/2364068245","heartbeat_front_addr":"172.21.15.170:6813/2364068245","state":["exists","up"]},{"osd":6,"uuid":"61970c81-4b79-4743-a379-50d5d102b373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764436984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764436984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764436984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764436984}]},"public_addr":"172.21.15.170:6817/1764436984","cluster_addr":"172.21.15.170:6819/1764436984","heartbeat_back_addr":"172.21.15.170:6823/1764436984","heartbeat_front_addr":"172.21.15.170:6821/1764436984","state":["exists","up"]},{"osd":7,"uuid":"2e303441-b454-4750-8c5a-96c5348129a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6825","nonce":4048577717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6827","nonce":4048577717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6831","nonce":4048577717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6829","nonce":4048577717}]},"public_addr":"172.21.15.170:6825/4048577717","cluster_addr":"172.21.15.170:6827/4048577717","heartbeat_back_addr":"172.21.15.170:6831/4048577717","heartbeat_front_addr":"172.21.15.170:6829/4048577717","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:32:25.292772+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:24.315254+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:32:16.961062+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:15.124689+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:20.860261+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:32:13.697472+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:09.904349+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:35.318623+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.8:0/2789652913":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/2381650878":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/1905866501":"2024-02-11T07:26:49.500037+0000","172.21.15.8:0/802075678":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6800/4001702773":"2024-02-11T07:26:49.500037+0000","172.21.15.8:0/1462056527":"2024-02-11T07:27:16.371197+0000","172.21.15.8:6801/4001702773":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6801/1412156979":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/2515892714":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6800/1412156979":"2024-02-11T07:27:16.371197+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T07:32:25.964 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:25 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:25.631+0000 7f408ef4c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T07:32:25.964 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:25 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:25.779+0000 7f408ef4c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T07:32:25.965 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:25 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:25.878+0000 7f408ef4c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:32:25.965 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:25 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:25.964+0000 7f408ef4c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:32:26.103 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:25.763+0000 7f0a1cadf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T07:32:26.104 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:25.844+0000 7f0a1cadf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T07:32:26.104 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:26 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:26.102+0000 7f0a1cadf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T07:32:26.383 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:26 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:26.212+0000 7f0a1cadf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:32:26.383 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:26 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:26.297+0000 7f0a1cadf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:32:26.497 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:26 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:32:26.497 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:26 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:32:26.497 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:26 smithi008 ceph-mon[29816]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T07:32:26.497 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:26 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4145244700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:32:26.498 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:26 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:26.240+0000 7f408ef4c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T07:32:26.498 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:26 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:26.354+0000 7f408ef4c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T07:32:26.498 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:26 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:26.445+0000 7f408ef4c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T07:32:26.498 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:26 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:32:26.498 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:26 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:32:26.498 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:26 smithi008 ceph-mon[40743]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T07:32:26.498 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:26 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4145244700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:32:26.524 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-10T07:32:26.525 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T07:32:26.781 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:26.829 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:26 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:26.548+0000 7f408ef4c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:32:26.829 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:26 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:26.738+0000 7f408ef4c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:32:26.883 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:26 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:26.392+0000 7f0a1cadf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T07:32:26.883 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:26 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:26.570+0000 7f0a1cadf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T07:32:26.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:26 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:32:26.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:26 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:32:26.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:26 smithi170 ceph-mon[33394]: osdmap e79: 8 total, 8 up, 8 in 2024-02-10T07:32:26.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:26 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4145244700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:32:27.799 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:27 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:27.539+0000 7f0a1cadf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:32:27.799 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:27 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:27.670+0000 7f0a1cadf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:32:27.800 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:27 smithi170 ceph-mon[33394]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T07:32:27.888 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:27 smithi008 ceph-mon[29816]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T07:32:27.888 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:27 smithi008 ceph-mon[40743]: osdmap e80: 8 total, 8 up, 8 in 2024-02-10T07:32:27.888 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:27 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:27.775+0000 7f408ef4c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T07:32:28.058 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:28 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:28.015+0000 7f0a1cadf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:32:28.166 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:27 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:27.913+0000 7f408ef4c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T07:32:28.319 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:28 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:28.111+0000 7f0a1cadf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:32:28.567 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:28.261+0000 7f408ef4c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T07:32:28.567 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:28.357+0000 7f408ef4c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T07:32:28.567 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:28.566+0000 7f408ef4c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:32:28.632 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:28 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:28.318+0000 7f0a1cadf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:32:28.633 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:28 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:28.510+0000 7f0a1cadf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:32:28.643 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:32:28.643 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":80,"fsid":"8c8244e6-c7e5-11ee-95b8-87774f69a715","created":"2024-02-10T07:26:21.316239+0000","modified":"2024-02-10T07:32:26.631254+0000","last_up_change":"2024-02-10T07:31:36.599653+0000","last_in_change":"2024-02-10T07:31:19.283633+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T07:29:32.704872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-10T07:31:49.150741+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-10T07:31:49.831683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-10T07:31:51.622544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-10T07:31:53.550827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-10T07:31:55.600117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"63abc4f1-1c83-4e03-8592-959c51d10847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6803","nonce":1442795455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6805","nonce":1442795455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6809","nonce":1442795455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6807","nonce":1442795455}]},"public_addr":"172.21.15.8:6803/1442795455","cluster_addr":"172.21.15.8:6805/1442795455","heartbeat_back_addr":"172.21.15.8:6809/1442795455","heartbeat_front_addr":"172.21.15.8:6807/1442795455","state":["exists","up"]},{"osd":1,"uuid":"88307389-bfa8-402e-92b9-9fa5094262c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6811","nonce":3278758893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6813","nonce":3278758893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6817","nonce":3278758893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6815","nonce":3278758893}]},"public_addr":"172.21.15.8:6811/3278758893","cluster_addr":"172.21.15.8:6813/3278758893","heartbeat_back_addr":"172.21.15.8:6817/3278758893","heartbeat_front_addr":"172.21.15.8:6815/3278758893","state":["exists","up"]},{"osd":2,"uuid":"c6120003-9ab5-4cf5-87ac-6351ed2c6226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6819","nonce":481314802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6821","nonce":481314802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6825","nonce":481314802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6823","nonce":481314802}]},"public_addr":"172.21.15.8:6819/481314802","cluster_addr":"172.21.15.8:6821/481314802","heartbeat_back_addr":"172.21.15.8:6825/481314802","heartbeat_front_addr":"172.21.15.8:6823/481314802","state":["exists","up"]},{"osd":3,"uuid":"6ca8f6f3-013a-48cb-9083-be37c1767489","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":3692331651},{"type":"v1","addr":"172.21.15.8:6827","nonce":3692331651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":3692331651},{"type":"v1","addr":"172.21.15.8:6829","nonce":3692331651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":3692331651},{"type":"v1","addr":"172.21.15.8:6833","nonce":3692331651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":3692331651},{"type":"v1","addr":"172.21.15.8:6831","nonce":3692331651}]},"public_addr":"172.21.15.8:6827/3692331651","cluster_addr":"172.21.15.8:6829/3692331651","heartbeat_back_addr":"172.21.15.8:6833/3692331651","heartbeat_front_addr":"172.21.15.8:6831/3692331651","state":["exists","up"]},{"osd":4,"uuid":"aa667766-e57f-447b-9a31-e3f3e61ac346","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6801","nonce":3997132504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6803","nonce":3997132504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6807","nonce":3997132504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6805","nonce":3997132504}]},"public_addr":"172.21.15.170:6801/3997132504","cluster_addr":"172.21.15.170:6803/3997132504","heartbeat_back_addr":"172.21.15.170:6807/3997132504","heartbeat_front_addr":"172.21.15.170:6805/3997132504","state":["exists","up"]},{"osd":5,"uuid":"2e9ab14d-1aff-4638-9f41-311f6b5f6770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":2364068245},{"type":"v1","addr":"172.21.15.170:6809","nonce":2364068245}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":2364068245},{"type":"v1","addr":"172.21.15.170:6811","nonce":2364068245}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":2364068245},{"type":"v1","addr":"172.21.15.170:6815","nonce":2364068245}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":2364068245},{"type":"v1","addr":"172.21.15.170:6813","nonce":2364068245}]},"public_addr":"172.21.15.170:6809/2364068245","cluster_addr":"172.21.15.170:6811/2364068245","heartbeat_back_addr":"172.21.15.170:6815/2364068245","heartbeat_front_addr":"172.21.15.170:6813/2364068245","state":["exists","up"]},{"osd":6,"uuid":"61970c81-4b79-4743-a379-50d5d102b373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764436984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764436984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764436984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764436984}]},"public_addr":"172.21.15.170:6817/1764436984","cluster_addr":"172.21.15.170:6819/1764436984","heartbeat_back_addr":"172.21.15.170:6823/1764436984","heartbeat_front_addr":"172.21.15.170:6821/1764436984","state":["exists","up"]},{"osd":7,"uuid":"2e303441-b454-4750-8c5a-96c5348129a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6825","nonce":4048577717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6827","nonce":4048577717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6831","nonce":4048577717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6829","nonce":4048577717}]},"public_addr":"172.21.15.170:6825/4048577717","cluster_addr":"172.21.15.170:6827/4048577717","heartbeat_back_addr":"172.21.15.170:6831/4048577717","heartbeat_front_addr":"172.21.15.170:6829/4048577717","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:32:25.292772+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:24.315254+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:32:16.961062+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:15.124689+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:20.860261+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:32:13.697472+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:09.904349+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:31:35.318623+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.8:0/2789652913":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/2381650878":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/1905866501":"2024-02-11T07:26:49.500037+0000","172.21.15.8:0/802075678":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6800/4001702773":"2024-02-11T07:26:49.500037+0000","172.21.15.8:0/1462056527":"2024-02-11T07:27:16.371197+0000","172.21.15.8:6801/4001702773":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6801/1412156979":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/2515892714":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6800/1412156979":"2024-02-11T07:27:16.371197+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T07:32:28.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:28 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2146664411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:32:28.832 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:28.764+0000 7f408ef4c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T07:32:28.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:28 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2146664411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:32:29.019 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:28 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:28.802+0000 7f0a1cadf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:32:29.020 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:28 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2146664411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:32:29.245 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:29 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:29.062+0000 7f408ef4c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T07:32:29.274 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-10T07:32:29.275 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-10T07:32:29.275 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-10T07:32:29.275 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-10T07:32:29.276 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-10T07:32:29.276 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-10T07:32:29.277 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-10T07:32:29.277 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-10T07:32:29.309 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:29.074+0000 7f0a1cadf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:32:29.581 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:29 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:29.366+0000 7f408ef4c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T07:32:29.633 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:29.308+0000 7f0a1cadf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:32:29.819 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:29.819 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:29.824 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:29.828 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:29.828 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:29.834 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:29.892 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:29.897 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:30.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:29 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:29.791+0000 7f408ef4c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T07:32:30.382 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:30.105+0000 7f0a1cadf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:32:30.991 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:30.677+0000 7f0a1cadf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:32:30.991 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:30.774+0000 7f0a1cadf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:32:31.008 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:30 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:30.698+0000 7f408ef4c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T07:32:31.301 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-mon[40743]: Standby manager daemon x restarted 2024-02-10T07:32:31.301 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-mon[40743]: Standby manager daemon x started 2024-02-10T07:32:31.301 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-mon[40743]: from='mgr.? 172.21.15.170:0/2976399678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:32:31.301 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-mon[40743]: from='mgr.? 172.21.15.170:0/2976399678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:32:31.301 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-mon[40743]: from='mgr.? 172.21.15.170:0/2976399678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:32:31.301 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-mon[40743]: from='mgr.? 172.21.15.170:0/2976399678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:32:31.302 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-mon[29816]: Standby manager daemon x restarted 2024-02-10T07:32:31.302 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-mon[29816]: Standby manager daemon x started 2024-02-10T07:32:31.302 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-mon[29816]: from='mgr.? 172.21.15.170:0/2976399678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:32:31.302 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-mon[29816]: from='mgr.? 172.21.15.170:0/2976399678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:32:31.302 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-mon[29816]: from='mgr.? 172.21.15.170:0/2976399678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:32:31.302 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-mon[29816]: from='mgr.? 172.21.15.170:0/2976399678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:32:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:31 smithi170 ceph-mon[33394]: Standby manager daemon x restarted 2024-02-10T07:32:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:31 smithi170 ceph-mon[33394]: Standby manager daemon x started 2024-02-10T07:32:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:31 smithi170 ceph-mon[33394]: from='mgr.? 172.21.15.170:0/2976399678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T07:32:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:31 smithi170 ceph-mon[33394]: from='mgr.? 172.21.15.170:0/2976399678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T07:32:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:31 smithi170 ceph-mon[33394]: from='mgr.? 172.21.15.170:0/2976399678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T07:32:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:31 smithi170 ceph-mon[33394]: from='mgr.? 172.21.15.170:0/2976399678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T07:32:31.384 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:30.990+0000 7f0a1cadf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:32:31.384 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:31 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: [10/Feb/2024:07:32:31] ENGINE Bus STARTING 2024-02-10T07:32:31.384 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:31 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: CherryPy Checker: 2024-02-10T07:32:31.384 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:31 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: The Application mounted at '' has an empty config. 2024-02-10T07:32:31.384 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:31 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: 2024-02-10T07:32:31.384 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:31 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: [10/Feb/2024:07:32:31] ENGINE Serving on http://:::9283 2024-02-10T07:32:31.384 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:32:31 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x[35255]: [10/Feb/2024:07:32:31] ENGINE Bus STARTED 2024-02-10T07:32:31.581 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:31.300+0000 7f408ef4c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T07:32:31.581 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:31.406+0000 7f408ef4c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T07:32:31.881 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:31.637+0000 7f408ef4c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T07:32:31.881 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:32:31] ENGINE Bus STARTING 2024-02-10T07:32:32.331 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: CherryPy Checker: 2024-02-10T07:32:32.332 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: The Application mounted at '' has an empty config. 2024-02-10T07:32:32.332 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:32:32.332 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:32:31] ENGINE Serving on http://:::9283 2024-02-10T07:32:32.332 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:31 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:32:31] ENGINE Bus STARTED 2024-02-10T07:32:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-10T07:32:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T07:32:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: Active manager daemon y restarted 2024-02-10T07:32:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: Activating manager daemon y 2024-02-10T07:32:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T07:32:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: mgrmap e19: y(active, starting, since 0.00607629s), standbys: x 2024-02-10T07:32:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:32:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:32:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:32:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:32:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:32:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:32:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:32:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:32:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:32:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:32:32.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:32:32.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:32:32.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:32:32.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:32:32.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:32:32.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:32:32.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: Manager daemon y is now available 2024-02-10T07:32:32.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:32:32.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:32:32.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:32.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:32:32.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:32:32.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:32:32.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-10T07:32:32.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T07:32:32.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: Active manager daemon y restarted 2024-02-10T07:32:32.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: Activating manager daemon y 2024-02-10T07:32:32.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T07:32:32.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: mgrmap e19: y(active, starting, since 0.00607629s), standbys: x 2024-02-10T07:32:32.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:32:32.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:32:32.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:32:32.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:32:32.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:32:32.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:32:32.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:32:32.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:32:32.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:32:32.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:32:32.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:32:32.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:32:32.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:32:32.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:32:32.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:32:32.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:32:32.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: Manager daemon y is now available 2024-02-10T07:32:32.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:32:32.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:32:32.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:32.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:32:32.337 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:32:32.337 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:32:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-10T07:32:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: osdmap e81: 8 total, 8 up, 8 in 2024-02-10T07:32:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: Active manager daemon y restarted 2024-02-10T07:32:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: Activating manager daemon y 2024-02-10T07:32:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: osdmap e82: 8 total, 8 up, 8 in 2024-02-10T07:32:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: mgrmap e19: y(active, starting, since 0.00607629s), standbys: x 2024-02-10T07:32:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T07:32:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T07:32:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: Manager daemon y is now available 2024-02-10T07:32:32.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:32:32.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:32:32.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:32.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:32:32.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T07:32:32.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T07:32:32.984 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:32 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:32:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:33 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:32:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:33 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:32:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:33 smithi008 ceph-mon[29816]: mgrmap e20: y(active, since 1.01069s), standbys: x 2024-02-10T07:32:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:33 smithi008 ceph-mon[29816]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T07:32:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:33 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:32:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:33 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:32:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:33 smithi008 ceph-mon[40743]: mgrmap e20: y(active, since 1.01069s), standbys: x 2024-02-10T07:32:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:33 smithi008 ceph-mon[40743]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T07:32:33.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:33 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:32:33.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:33 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:32:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:33 smithi170 ceph-mon[33394]: mgrmap e20: y(active, since 1.01069s), standbys: x 2024-02-10T07:32:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:33 smithi170 ceph-mon[33394]: osdmap e83: 8 total, 8 up, 8 in 2024-02-10T07:32:34.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:34 smithi008 ceph-mon[29816]: [10/Feb/2024:07:32:32] ENGINE Bus STARTING 2024-02-10T07:32:34.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:34 smithi008 ceph-mon[29816]: [10/Feb/2024:07:32:32] ENGINE Serving on http://172.21.15.8:8765 2024-02-10T07:32:34.163 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:34 smithi008 ceph-mon[29816]: [10/Feb/2024:07:32:33] ENGINE Serving on https://172.21.15.8:7150 2024-02-10T07:32:34.163 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:34 smithi008 ceph-mon[29816]: [10/Feb/2024:07:32:33] ENGINE Bus STARTED 2024-02-10T07:32:34.163 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:34 smithi008 ceph-mon[40743]: [10/Feb/2024:07:32:32] ENGINE Bus STARTING 2024-02-10T07:32:34.163 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:34 smithi008 ceph-mon[40743]: [10/Feb/2024:07:32:32] ENGINE Serving on http://172.21.15.8:8765 2024-02-10T07:32:34.163 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:34 smithi008 ceph-mon[40743]: [10/Feb/2024:07:32:33] ENGINE Serving on https://172.21.15.8:7150 2024-02-10T07:32:34.163 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:34 smithi008 ceph-mon[40743]: [10/Feb/2024:07:32:33] ENGINE Bus STARTED 2024-02-10T07:32:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:34 smithi170 ceph-mon[33394]: [10/Feb/2024:07:32:32] ENGINE Bus STARTING 2024-02-10T07:32:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:34 smithi170 ceph-mon[33394]: [10/Feb/2024:07:32:32] ENGINE Serving on http://172.21.15.8:8765 2024-02-10T07:32:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:34 smithi170 ceph-mon[33394]: [10/Feb/2024:07:32:33] ENGINE Serving on https://172.21.15.8:7150 2024-02-10T07:32:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:34 smithi170 ceph-mon[33394]: [10/Feb/2024:07:32:33] ENGINE Bus STARTED 2024-02-10T07:32:35.260 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:35 smithi008 ceph-mon[29816]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:32:35.260 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:35 smithi008 ceph-mon[29816]: mgrmap e21: y(active, since 2s), standbys: x 2024-02-10T07:32:35.260 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:35 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:35.260 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:35 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:35.260 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:35 smithi008 ceph-mon[40743]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:32:35.261 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:35 smithi008 ceph-mon[40743]: mgrmap e21: y(active, since 2s), standbys: x 2024-02-10T07:32:35.261 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:35 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:35.261 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:35 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:35 smithi170 ceph-mon[33394]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:32:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:35 smithi170 ceph-mon[33394]: mgrmap e21: y(active, since 2s), standbys: x 2024-02-10T07:32:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:35 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:35 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:36.198 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:32:36.199 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:32:36.199 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[29816]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T07:32:36.199 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:36.199 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:36.199 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:32:36.199 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:32:36.199 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:32:36.199 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:32:36.199 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:36.199 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:32:36.200 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:32:36.200 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[40743]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T07:32:36.200 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:36.200 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:36.200 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:32:36.200 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:32:36.200 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:32:36.200 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:32:36.200 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:36 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:36 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:32:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:36 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:32:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:36 smithi170 ceph-mon[33394]: osdmap e84: 8 total, 8 up, 8 in 2024-02-10T07:32:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:36 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:36 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:36 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:32:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:36 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:32:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:36 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:32:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:36 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:32:36.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:36 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:32:37.323 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:37 smithi008 ceph-mon[40743]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:32:37.324 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:37 smithi008 ceph-mon[40743]: Adjusting osd_memory_target on smithi170 to 3760M 2024-02-10T07:32:37.324 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:37 smithi008 ceph-mon[40743]: mgrmap e22: y(active, since 4s), standbys: x 2024-02-10T07:32:37.324 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:37 smithi008 ceph-mon[40743]: osdmap e85: 8 total, 8 up, 8 in 2024-02-10T07:32:37.324 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:37 smithi008 ceph-mon[29816]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:32:37.324 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:37 smithi008 ceph-mon[29816]: Adjusting osd_memory_target on smithi170 to 3760M 2024-02-10T07:32:37.324 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:37 smithi008 ceph-mon[29816]: mgrmap e22: y(active, since 4s), standbys: x 2024-02-10T07:32:37.324 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:37 smithi008 ceph-mon[29816]: osdmap e85: 8 total, 8 up, 8 in 2024-02-10T07:32:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:37 smithi170 ceph-mon[33394]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:32:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:37 smithi170 ceph-mon[33394]: Adjusting osd_memory_target on smithi170 to 3760M 2024-02-10T07:32:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:37 smithi170 ceph-mon[33394]: mgrmap e22: y(active, since 4s), standbys: x 2024-02-10T07:32:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:37 smithi170 ceph-mon[33394]: osdmap e85: 8 total, 8 up, 8 in 2024-02-10T07:32:39.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:32:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:32:39.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:39 smithi008 ceph-mon[29816]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:32:39.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:39 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:32:39.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:39 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:32:39.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:39 smithi008 ceph-mon[29816]: osdmap e86: 8 total, 8 up, 8 in 2024-02-10T07:32:39.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:39 smithi008 ceph-mon[40743]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:32:39.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:39 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:32:39.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:39 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:32:39.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:39 smithi008 ceph-mon[40743]: osdmap e86: 8 total, 8 up, 8 in 2024-02-10T07:32:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:39 smithi170 ceph-mon[33394]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-02-10T07:32:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:39 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:32:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:39 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:32:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:39 smithi170 ceph-mon[33394]: osdmap e86: 8 total, 8 up, 8 in 2024-02-10T07:32:40.216 INFO:teuthology.orchestra.run.smithi008.stdout:34359738418 2024-02-10T07:32:40.216 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-10T07:32:41.243 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:41 smithi008 ceph-mon[29816]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-02-10T07:32:41.244 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:41 smithi008 ceph-mon[40743]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-02-10T07:32:41.275 INFO:teuthology.orchestra.run.smithi008.stdout:77309411367 2024-02-10T07:32:41.276 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-10T07:32:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:41 smithi170 ceph-mon[33394]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-02-10T07:32:42.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:42 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:32:42.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:42 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:32:42.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:42 smithi008 ceph-mon[40743]: osdmap e87: 8 total, 8 up, 8 in 2024-02-10T07:32:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:42 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:32:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:42 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:32:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:42 smithi008 ceph-mon[29816]: osdmap e87: 8 total, 8 up, 8 in 2024-02-10T07:32:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:42 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:32:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:42 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:32:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:42 smithi170 ceph-mon[33394]: osdmap e87: 8 total, 8 up, 8 in 2024-02-10T07:32:42.791 INFO:teuthology.orchestra.run.smithi008.stdout:55834574893 2024-02-10T07:32:42.791 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-10T07:32:43.092 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:42 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:32:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:43 smithi170 ceph-mon[33394]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 21 op/s 2024-02-10T07:32:43.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:43 smithi008 ceph-mon[29816]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 21 op/s 2024-02-10T07:32:43.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:43 smithi008 ceph-mon[40743]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 21 op/s 2024-02-10T07:32:43.912 INFO:teuthology.orchestra.run.smithi008.stdout:257698037775 2024-02-10T07:32:43.912 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-10T07:32:44.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:44 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:44.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:44 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:44 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:44.424 INFO:teuthology.orchestra.run.smithi008.stdout:214748364818 2024-02-10T07:32:44.425 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-10T07:32:45.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:45 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:32:45.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:45 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:32:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:45 smithi008 ceph-mon[29816]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 18 op/s 2024-02-10T07:32:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:45 smithi008 ceph-mon[29816]: osdmap e88: 8 total, 8 up, 8 in 2024-02-10T07:32:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:45 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:32:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:45 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:32:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:45 smithi008 ceph-mon[40743]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 18 op/s 2024-02-10T07:32:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:45 smithi008 ceph-mon[40743]: osdmap e88: 8 total, 8 up, 8 in 2024-02-10T07:32:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:45 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:32:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:45 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:32:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:45 smithi170 ceph-mon[33394]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 18 op/s 2024-02-10T07:32:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:45 smithi170 ceph-mon[33394]: osdmap e88: 8 total, 8 up, 8 in 2024-02-10T07:32:46.390 INFO:teuthology.orchestra.run.smithi008.stdout:137438953502 2024-02-10T07:32:46.390 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-10T07:32:46.898 INFO:teuthology.orchestra.run.smithi008.stdout:111669149731 2024-02-10T07:32:46.898 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-10T07:32:47.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:47 smithi008 ceph-mon[29816]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-02-10T07:32:47.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:47 smithi008 ceph-mon[29816]: osdmap e89: 8 total, 8 up, 8 in 2024-02-10T07:32:47.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:47 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:32:47.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:47 smithi008 ceph-mon[40743]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-02-10T07:32:47.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:47 smithi008 ceph-mon[40743]: osdmap e89: 8 total, 8 up, 8 in 2024-02-10T07:32:47.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:47 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:32:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:47 smithi170 ceph-mon[33394]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-02-10T07:32:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:47 smithi170 ceph-mon[33394]: osdmap e89: 8 total, 8 up, 8 in 2024-02-10T07:32:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:47 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:32:47.882 INFO:teuthology.orchestra.run.smithi008.stdout:171798691865 2024-02-10T07:32:47.883 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-10T07:32:47.927 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:47.933 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:47.933 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:47.933 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:47.935 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:47.937 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:49.036 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:32:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:32:49.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:49 smithi008 ceph-mon[29816]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-02-10T07:32:49.317 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:49 smithi008 ceph-mon[40743]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-02-10T07:32:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:49 smithi170 ceph-mon[33394]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-02-10T07:32:50.279 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:32:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:51 smithi170 ceph-mon[33394]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:51.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:51 smithi008 ceph-mon[29816]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:51 smithi008 ceph-mon[40743]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:32:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:52 smithi170 ceph-mon[33394]: 3.1d scrub starts 2024-02-10T07:32:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:52 smithi170 ceph-mon[33394]: 3.1d scrub ok 2024-02-10T07:32:52.405 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:52 smithi008 ceph-mon[40743]: 3.1d scrub starts 2024-02-10T07:32:52.405 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:52 smithi008 ceph-mon[40743]: 3.1d scrub ok 2024-02-10T07:32:52.405 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:52 smithi008 ceph-mon[29816]: 3.1d scrub starts 2024-02-10T07:32:52.405 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:52 smithi008 ceph-mon[29816]: 3.1d scrub ok 2024-02-10T07:32:53.109 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:32:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:32:53.377 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:53 smithi008 ceph-mon[40743]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:32:53.377 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:53 smithi008 ceph-mon[29816]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:32:53.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:53 smithi170 ceph-mon[33394]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:32:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:54 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:54 smithi170 ceph-mon[33394]: 3.16 deep-scrub starts 2024-02-10T07:32:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:54 smithi170 ceph-mon[33394]: 3.16 deep-scrub ok 2024-02-10T07:32:54.402 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:54 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:54.402 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:54 smithi008 ceph-mon[29816]: 3.16 deep-scrub starts 2024-02-10T07:32:54.402 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:54 smithi008 ceph-mon[29816]: 3.16 deep-scrub ok 2024-02-10T07:32:54.403 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:54 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:32:54.403 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:54 smithi008 ceph-mon[40743]: 3.16 deep-scrub starts 2024-02-10T07:32:54.403 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:54 smithi008 ceph-mon[40743]: 3.16 deep-scrub ok 2024-02-10T07:32:55.228 INFO:teuthology.orchestra.run.smithi008.stdout:55834574896 2024-02-10T07:32:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:55 smithi170 ceph-mon[33394]: 4.2 scrub starts 2024-02-10T07:32:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:55 smithi170 ceph-mon[33394]: 4.2 scrub ok 2024-02-10T07:32:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:55 smithi170 ceph-mon[33394]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-02-10T07:32:55.552 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:55 smithi008 ceph-mon[29816]: 4.2 scrub starts 2024-02-10T07:32:55.552 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:55 smithi008 ceph-mon[29816]: 4.2 scrub ok 2024-02-10T07:32:55.552 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:55 smithi008 ceph-mon[29816]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-02-10T07:32:55.552 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:55 smithi008 ceph-mon[40743]: 4.2 scrub starts 2024-02-10T07:32:55.552 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:55 smithi008 ceph-mon[40743]: 4.2 scrub ok 2024-02-10T07:32:55.552 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:55 smithi008 ceph-mon[40743]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-02-10T07:32:56.248 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:56 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4066400048' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T07:32:56.249 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:56 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4066400048' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T07:32:56.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:56 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4066400048' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T07:32:56.412 INFO:teuthology.orchestra.run.smithi008.stdout:77309411370 2024-02-10T07:32:57.247 INFO:teuthology.orchestra.run.smithi008.stdout:34359738421 2024-02-10T07:32:57.247 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[29816]: 5.0 scrub starts 2024-02-10T07:32:57.248 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[29816]: 5.0 scrub ok 2024-02-10T07:32:57.248 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[29816]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T07:32:57.248 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/550887096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T07:32:57.248 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[29816]: 4.16 deep-scrub starts 2024-02-10T07:32:57.248 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[29816]: 4.16 deep-scrub ok 2024-02-10T07:32:57.248 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[29816]: 3.a deep-scrub starts 2024-02-10T07:32:57.248 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[29816]: 3.a deep-scrub ok 2024-02-10T07:32:57.248 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[40743]: 5.0 scrub starts 2024-02-10T07:32:57.248 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[40743]: 5.0 scrub ok 2024-02-10T07:32:57.248 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[40743]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T07:32:57.249 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/550887096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T07:32:57.249 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[40743]: 4.16 deep-scrub starts 2024-02-10T07:32:57.249 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[40743]: 4.16 deep-scrub ok 2024-02-10T07:32:57.249 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[40743]: 3.a deep-scrub starts 2024-02-10T07:32:57.249 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:57 smithi008 ceph-mon[40743]: 3.a deep-scrub ok 2024-02-10T07:32:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:57 smithi170 ceph-mon[33394]: 5.0 scrub starts 2024-02-10T07:32:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:57 smithi170 ceph-mon[33394]: 5.0 scrub ok 2024-02-10T07:32:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:57 smithi170 ceph-mon[33394]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-10T07:32:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:57 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/550887096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T07:32:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:57 smithi170 ceph-mon[33394]: 4.16 deep-scrub starts 2024-02-10T07:32:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:57 smithi170 ceph-mon[33394]: 4.16 deep-scrub ok 2024-02-10T07:32:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:57 smithi170 ceph-mon[33394]: 3.a deep-scrub starts 2024-02-10T07:32:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:57 smithi170 ceph-mon[33394]: 3.a deep-scrub ok 2024-02-10T07:32:58.084 INFO:teuthology.orchestra.run.smithi008.stdout:111669149733 2024-02-10T07:32:58.347 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:58 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/156133756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T07:32:58.347 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:58 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4276129275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T07:32:58.347 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:58 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/156133756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T07:32:58.347 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:58 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4276129275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T07:32:58.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:58 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/156133756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T07:32:58.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:58 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4276129275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T07:32:59.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:32:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:32:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:32:59.121 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738421 for osd.0 2024-02-10T07:32:59.122 DEBUG:teuthology.parallel:result is None 2024-02-10T07:32:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:59 smithi170 ceph-mon[33394]: 4.f scrub starts 2024-02-10T07:32:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:59 smithi170 ceph-mon[33394]: 4.f scrub ok 2024-02-10T07:32:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:59 smithi170 ceph-mon[33394]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 835 B/s rd, 0 op/s 2024-02-10T07:32:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:59 smithi170 ceph-mon[33394]: 5.18 scrub starts 2024-02-10T07:32:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:32:59 smithi170 ceph-mon[33394]: 5.18 scrub ok 2024-02-10T07:32:59.469 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:59 smithi008 ceph-mon[29816]: 4.f scrub starts 2024-02-10T07:32:59.469 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:59 smithi008 ceph-mon[29816]: 4.f scrub ok 2024-02-10T07:32:59.469 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:59 smithi008 ceph-mon[29816]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 835 B/s rd, 0 op/s 2024-02-10T07:32:59.469 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:59 smithi008 ceph-mon[29816]: 5.18 scrub starts 2024-02-10T07:32:59.469 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:32:59 smithi008 ceph-mon[29816]: 5.18 scrub ok 2024-02-10T07:32:59.469 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:59 smithi008 ceph-mon[40743]: 4.f scrub starts 2024-02-10T07:32:59.469 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:59 smithi008 ceph-mon[40743]: 4.f scrub ok 2024-02-10T07:32:59.469 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:59 smithi008 ceph-mon[40743]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 835 B/s rd, 0 op/s 2024-02-10T07:32:59.470 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:59 smithi008 ceph-mon[40743]: 5.18 scrub starts 2024-02-10T07:32:59.470 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:32:59 smithi008 ceph-mon[40743]: 5.18 scrub ok 2024-02-10T07:32:59.613 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149731 got 111669149733 for osd.3 2024-02-10T07:32:59.613 DEBUG:teuthology.parallel:result is None 2024-02-10T07:32:59.995 INFO:teuthology.orchestra.run.smithi008.stdout:214748364823 2024-02-10T07:33:00.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[29816]: 2.1 scrub starts 2024-02-10T07:33:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[29816]: 2.1 scrub ok 2024-02-10T07:33:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[29816]: 6.5 scrub starts 2024-02-10T07:33:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[29816]: 6.5 scrub ok 2024-02-10T07:33:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[29816]: 4.11 scrub starts 2024-02-10T07:33:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[29816]: 4.11 scrub ok 2024-02-10T07:33:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[29816]: 4.8 scrub starts 2024-02-10T07:33:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[29816]: 4.8 scrub ok 2024-02-10T07:33:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1189192657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T07:33:00.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[40743]: 2.1 scrub starts 2024-02-10T07:33:00.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[40743]: 2.1 scrub ok 2024-02-10T07:33:00.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[40743]: 6.5 scrub starts 2024-02-10T07:33:00.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[40743]: 6.5 scrub ok 2024-02-10T07:33:00.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[40743]: 4.11 scrub starts 2024-02-10T07:33:00.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[40743]: 4.11 scrub ok 2024-02-10T07:33:00.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[40743]: 4.8 scrub starts 2024-02-10T07:33:00.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[40743]: 4.8 scrub ok 2024-02-10T07:33:00.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:00 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1189192657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T07:33:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:00 smithi170 ceph-mon[33394]: 2.1 scrub starts 2024-02-10T07:33:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:00 smithi170 ceph-mon[33394]: 2.1 scrub ok 2024-02-10T07:33:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:00 smithi170 ceph-mon[33394]: 6.5 scrub starts 2024-02-10T07:33:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:00 smithi170 ceph-mon[33394]: 6.5 scrub ok 2024-02-10T07:33:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:00 smithi170 ceph-mon[33394]: 4.11 scrub starts 2024-02-10T07:33:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:00 smithi170 ceph-mon[33394]: 4.11 scrub ok 2024-02-10T07:33:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:00 smithi170 ceph-mon[33394]: 4.8 scrub starts 2024-02-10T07:33:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:00 smithi170 ceph-mon[33394]: 4.8 scrub ok 2024-02-10T07:33:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:00 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1189192657' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T07:33:00.550 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574896 for osd.1 2024-02-10T07:33:00.550 DEBUG:teuthology.parallel:result is None 2024-02-10T07:33:01.212 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364818 got 214748364823 for osd.6 2024-02-10T07:33:01.212 DEBUG:teuthology.parallel:result is None 2024-02-10T07:33:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:01 smithi170 ceph-mon[33394]: 5.8 scrub starts 2024-02-10T07:33:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:01 smithi170 ceph-mon[33394]: 5.8 scrub ok 2024-02-10T07:33:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:01 smithi170 ceph-mon[33394]: 5.13 scrub starts 2024-02-10T07:33:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:01 smithi170 ceph-mon[33394]: 5.13 scrub ok 2024-02-10T07:33:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:01 smithi170 ceph-mon[33394]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:01 smithi170 ceph-mon[33394]: 4.13 scrub starts 2024-02-10T07:33:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:01 smithi170 ceph-mon[33394]: 4.13 scrub ok 2024-02-10T07:33:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:01 smithi170 ceph-mon[33394]: 4.1f scrub starts 2024-02-10T07:33:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:01 smithi170 ceph-mon[33394]: 4.1f scrub ok 2024-02-10T07:33:01.425 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[29816]: 5.8 scrub starts 2024-02-10T07:33:01.425 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[29816]: 5.8 scrub ok 2024-02-10T07:33:01.425 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[29816]: 5.13 scrub starts 2024-02-10T07:33:01.425 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[29816]: 5.13 scrub ok 2024-02-10T07:33:01.425 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[29816]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:01.425 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[29816]: 4.13 scrub starts 2024-02-10T07:33:01.425 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[29816]: 4.13 scrub ok 2024-02-10T07:33:01.425 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[29816]: 4.1f scrub starts 2024-02-10T07:33:01.425 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[29816]: 4.1f scrub ok 2024-02-10T07:33:01.426 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[40743]: 5.8 scrub starts 2024-02-10T07:33:01.426 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[40743]: 5.8 scrub ok 2024-02-10T07:33:01.426 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[40743]: 5.13 scrub starts 2024-02-10T07:33:01.426 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[40743]: 5.13 scrub ok 2024-02-10T07:33:01.426 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[40743]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:01.426 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[40743]: 4.13 scrub starts 2024-02-10T07:33:01.426 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[40743]: 4.13 scrub ok 2024-02-10T07:33:01.426 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[40743]: 4.1f scrub starts 2024-02-10T07:33:01.426 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:01 smithi008 ceph-mon[40743]: 4.1f scrub ok 2024-02-10T07:33:01.704 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:01.876 INFO:teuthology.orchestra.run.smithi008.stdout:257698037778 2024-02-10T07:33:02.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[40743]: 4.19 scrub starts 2024-02-10T07:33:02.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[40743]: 4.19 scrub ok 2024-02-10T07:33:02.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[40743]: 5.1 scrub starts 2024-02-10T07:33:02.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[40743]: 5.1 scrub ok 2024-02-10T07:33:02.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[40743]: 3.12 scrub starts 2024-02-10T07:33:02.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[40743]: 3.12 scrub ok 2024-02-10T07:33:02.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:02.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:02.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:02.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3740672043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T07:33:02.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[29816]: 4.19 scrub starts 2024-02-10T07:33:02.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[29816]: 4.19 scrub ok 2024-02-10T07:33:02.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[29816]: 5.1 scrub starts 2024-02-10T07:33:02.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[29816]: 5.1 scrub ok 2024-02-10T07:33:02.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[29816]: 3.12 scrub starts 2024-02-10T07:33:02.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[29816]: 3.12 scrub ok 2024-02-10T07:33:02.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:02.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:02.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:02.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:02 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3740672043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T07:33:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:02 smithi170 ceph-mon[33394]: 4.19 scrub starts 2024-02-10T07:33:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:02 smithi170 ceph-mon[33394]: 4.19 scrub ok 2024-02-10T07:33:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:02 smithi170 ceph-mon[33394]: 5.1 scrub starts 2024-02-10T07:33:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:02 smithi170 ceph-mon[33394]: 5.1 scrub ok 2024-02-10T07:33:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:02 smithi170 ceph-mon[33394]: 3.12 scrub starts 2024-02-10T07:33:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:02 smithi170 ceph-mon[33394]: 3.12 scrub ok 2024-02-10T07:33:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:02 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:02 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:02.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:02 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:02.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:02 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3740672043' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T07:33:02.719 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037775 got 257698037778 for osd.7 2024-02-10T07:33:02.719 DEBUG:teuthology.parallel:result is None 2024-02-10T07:33:03.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:33:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:33:03.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:03 smithi008 ceph-mon[29816]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:03.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:03 smithi008 ceph-mon[29816]: 3.1a scrub starts 2024-02-10T07:33:03.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:03 smithi008 ceph-mon[29816]: 3.1a scrub ok 2024-02-10T07:33:03.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:03 smithi008 ceph-mon[40743]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:03.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:03 smithi008 ceph-mon[40743]: 3.1a scrub starts 2024-02-10T07:33:03.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:03 smithi008 ceph-mon[40743]: 3.1a scrub ok 2024-02-10T07:33:03.381 INFO:teuthology.orchestra.run.smithi008.stdout:137438953505 2024-02-10T07:33:03.414 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411370 for osd.2 2024-02-10T07:33:03.414 DEBUG:teuthology.parallel:result is None 2024-02-10T07:33:03.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:03 smithi170 ceph-mon[33394]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:03 smithi170 ceph-mon[33394]: 3.1a scrub starts 2024-02-10T07:33:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:03 smithi170 ceph-mon[33394]: 3.1a scrub ok 2024-02-10T07:33:04.138 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:04.139 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/18951480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T07:33:04.139 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:04.139 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[29816]: 4.1e scrub starts 2024-02-10T07:33:04.139 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[29816]: 4.1e scrub ok 2024-02-10T07:33:04.139 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:04.139 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:04.139 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:04.139 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:04.139 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:04.139 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:04.139 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:33:04.139 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:33:04.140 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:04.140 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/18951480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T07:33:04.140 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:04.140 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[40743]: 4.1e scrub starts 2024-02-10T07:33:04.140 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[40743]: 4.1e scrub ok 2024-02-10T07:33:04.140 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:04.140 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:04.140 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:04.140 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:04.158 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953502 got 137438953505 for osd.4 2024-02-10T07:33:04.158 DEBUG:teuthology.parallel:result is None 2024-02-10T07:33:04.455 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:04.455 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:04.455 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:33:04.455 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:33:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:04 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:04 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/18951480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T07:33:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:04 smithi170 ceph-mon[33394]: 4.1e scrub starts 2024-02-10T07:33:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:04 smithi170 ceph-mon[33394]: 4.1e scrub ok 2024-02-10T07:33:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:04.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:33:04.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:04.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:33:04.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:33:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: 3.0 scrub starts 2024-02-10T07:33:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: 3.0 scrub ok 2024-02-10T07:33:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: Adjusting osd_memory_target on smithi008 to 3761M 2024-02-10T07:33:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T07:33:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: Updating smithi170:/etc/ceph/ceph.conf 2024-02-10T07:33:05.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:05.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: Updating smithi170:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:33:05.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: Updating smithi008:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:33:05.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:33:05.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:33:05.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:05.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:05.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:05.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:05.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:05.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: 3.0 scrub starts 2024-02-10T07:33:05.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: 3.0 scrub ok 2024-02-10T07:33:05.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: Adjusting osd_memory_target on smithi008 to 3761M 2024-02-10T07:33:05.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T07:33:05.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: Updating smithi170:/etc/ceph/ceph.conf 2024-02-10T07:33:05.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:05.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: Updating smithi170:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:33:05.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: Updating smithi008:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:33:05.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:33:05.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:33:05.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:05.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:05.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:05.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:05.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:05 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: 3.0 scrub starts 2024-02-10T07:33:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: 3.0 scrub ok 2024-02-10T07:33:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: Adjusting osd_memory_target on smithi008 to 3761M 2024-02-10T07:33:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: Updating smithi008:/etc/ceph/ceph.conf 2024-02-10T07:33:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: Updating smithi170:/etc/ceph/ceph.conf 2024-02-10T07:33:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: Updating smithi170:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:33:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: Updating smithi008:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T07:33:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:33:05.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: Updating smithi008:/etc/ceph/ceph.client.admin.keyring 2024-02-10T07:33:05.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:05.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:05.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:05.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:05.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:05 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:06.157 INFO:teuthology.orchestra.run.smithi008.stdout:171798691869 2024-02-10T07:33:06.561 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[29816]: Updating smithi170:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:33:06.561 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[29816]: Updating smithi008:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:33:06.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[29816]: Deploying daemon alertmanager.a on smithi008 2024-02-10T07:33:06.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[29816]: 6.c scrub starts 2024-02-10T07:33:06.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[29816]: 6.c scrub ok 2024-02-10T07:33:06.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[29816]: 6.f deep-scrub starts 2024-02-10T07:33:06.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[29816]: 6.f deep-scrub ok 2024-02-10T07:33:06.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[29816]: 6.0 scrub starts 2024-02-10T07:33:06.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[29816]: 6.0 scrub ok 2024-02-10T07:33:06.562 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[40743]: Updating smithi170:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:33:06.562 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[40743]: Updating smithi008:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:33:06.562 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[40743]: Deploying daemon alertmanager.a on smithi008 2024-02-10T07:33:06.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[40743]: 6.c scrub starts 2024-02-10T07:33:06.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[40743]: 6.c scrub ok 2024-02-10T07:33:06.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[40743]: 6.f deep-scrub starts 2024-02-10T07:33:06.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[40743]: 6.f deep-scrub ok 2024-02-10T07:33:06.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[40743]: 6.0 scrub starts 2024-02-10T07:33:06.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:06 smithi008 ceph-mon[40743]: 6.0 scrub ok 2024-02-10T07:33:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:06 smithi170 ceph-mon[33394]: Updating smithi170:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:33:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:06 smithi170 ceph-mon[33394]: Updating smithi008:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T07:33:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:06 smithi170 ceph-mon[33394]: Deploying daemon alertmanager.a on smithi008 2024-02-10T07:33:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:06 smithi170 ceph-mon[33394]: 6.c scrub starts 2024-02-10T07:33:06.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:06 smithi170 ceph-mon[33394]: 6.c scrub ok 2024-02-10T07:33:06.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:06 smithi170 ceph-mon[33394]: 6.f deep-scrub starts 2024-02-10T07:33:06.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:06 smithi170 ceph-mon[33394]: 6.f deep-scrub ok 2024-02-10T07:33:06.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:06 smithi170 ceph-mon[33394]: 6.0 scrub starts 2024-02-10T07:33:06.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:06 smithi170 ceph-mon[33394]: 6.0 scrub ok 2024-02-10T07:33:06.998 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691865 got 171798691869 for osd.5 2024-02-10T07:33:06.998 DEBUG:teuthology.parallel:result is None 2024-02-10T07:33:06.998 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-10T07:33:06.998 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:33:07.259 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:07.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:07 smithi008 ceph-mon[29816]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:07 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2356982275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T07:33:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:07 smithi008 ceph-mon[29816]: 5.6 scrub starts 2024-02-10T07:33:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:07 smithi008 ceph-mon[29816]: 5.6 scrub ok 2024-02-10T07:33:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:07 smithi008 ceph-mon[29816]: 5.f scrub starts 2024-02-10T07:33:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:07 smithi008 ceph-mon[29816]: 5.f scrub ok 2024-02-10T07:33:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:07 smithi008 ceph-mon[40743]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:07 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2356982275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T07:33:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:07 smithi008 ceph-mon[40743]: 5.6 scrub starts 2024-02-10T07:33:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:07 smithi008 ceph-mon[40743]: 5.6 scrub ok 2024-02-10T07:33:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:07 smithi008 ceph-mon[40743]: 5.f scrub starts 2024-02-10T07:33:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:07 smithi008 ceph-mon[40743]: 5.f scrub ok 2024-02-10T07:33:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:07 smithi170 ceph-mon[33394]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:07 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2356982275' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T07:33:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:07 smithi170 ceph-mon[33394]: 5.6 scrub starts 2024-02-10T07:33:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:07 smithi170 ceph-mon[33394]: 5.6 scrub ok 2024-02-10T07:33:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:07 smithi170 ceph-mon[33394]: 5.f scrub starts 2024-02-10T07:33:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:07 smithi170 ceph-mon[33394]: 5.f scrub ok 2024-02-10T07:33:08.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:08 smithi008 ceph-mon[29816]: 3.4 scrub starts 2024-02-10T07:33:08.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:08 smithi008 ceph-mon[29816]: 3.4 scrub ok 2024-02-10T07:33:08.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:08 smithi008 ceph-mon[29816]: 3.13 deep-scrub starts 2024-02-10T07:33:08.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:08 smithi008 ceph-mon[29816]: 3.13 deep-scrub ok 2024-02-10T07:33:08.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:08 smithi008 ceph-mon[40743]: 3.4 scrub starts 2024-02-10T07:33:08.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:08 smithi008 ceph-mon[40743]: 3.4 scrub ok 2024-02-10T07:33:08.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:08 smithi008 ceph-mon[40743]: 3.13 deep-scrub starts 2024-02-10T07:33:08.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:08 smithi008 ceph-mon[40743]: 3.13 deep-scrub ok 2024-02-10T07:33:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:08 smithi170 ceph-mon[33394]: 3.4 scrub starts 2024-02-10T07:33:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:08 smithi170 ceph-mon[33394]: 3.4 scrub ok 2024-02-10T07:33:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:08 smithi170 ceph-mon[33394]: 3.13 deep-scrub starts 2024-02-10T07:33:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:08 smithi170 ceph-mon[33394]: 3.13 deep-scrub ok 2024-02-10T07:33:09.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:33:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:33:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:09 smithi170 ceph-mon[33394]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:09.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:09 smithi008 ceph-mon[29816]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:09.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:09 smithi008 ceph-mon[40743]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:10.236 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:10 smithi008 ceph-mon[29816]: 4.0 scrub starts 2024-02-10T07:33:10.236 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:10 smithi008 ceph-mon[29816]: 4.0 scrub ok 2024-02-10T07:33:10.236 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:10 smithi008 ceph-mon[29816]: 4.15 scrub starts 2024-02-10T07:33:10.236 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:10 smithi008 ceph-mon[29816]: 4.15 scrub ok 2024-02-10T07:33:10.236 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:10 smithi008 ceph-mon[29816]: 3.10 scrub starts 2024-02-10T07:33:10.236 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:10 smithi008 ceph-mon[29816]: 3.10 scrub ok 2024-02-10T07:33:10.236 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:10 smithi008 ceph-mon[40743]: 4.0 scrub starts 2024-02-10T07:33:10.236 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:10 smithi008 ceph-mon[40743]: 4.0 scrub ok 2024-02-10T07:33:10.237 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:10 smithi008 ceph-mon[40743]: 4.15 scrub starts 2024-02-10T07:33:10.237 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:10 smithi008 ceph-mon[40743]: 4.15 scrub ok 2024-02-10T07:33:10.237 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:10 smithi008 ceph-mon[40743]: 3.10 scrub starts 2024-02-10T07:33:10.237 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:10 smithi008 ceph-mon[40743]: 3.10 scrub ok 2024-02-10T07:33:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:10 smithi170 ceph-mon[33394]: 4.0 scrub starts 2024-02-10T07:33:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:10 smithi170 ceph-mon[33394]: 4.0 scrub ok 2024-02-10T07:33:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:10 smithi170 ceph-mon[33394]: 4.15 scrub starts 2024-02-10T07:33:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:10 smithi170 ceph-mon[33394]: 4.15 scrub ok 2024-02-10T07:33:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:10 smithi170 ceph-mon[33394]: 3.10 scrub starts 2024-02-10T07:33:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:10 smithi170 ceph-mon[33394]: 3.10 scrub ok 2024-02-10T07:33:10.929 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:33:10.932 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:33:11.234 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:11 smithi008 ceph-mon[29816]: 4.4 scrub starts 2024-02-10T07:33:11.235 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:11 smithi008 ceph-mon[29816]: 4.4 scrub ok 2024-02-10T07:33:11.235 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:11 smithi008 ceph-mon[29816]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:11.235 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:11 smithi008 ceph-mon[29816]: 6.14 deep-scrub starts 2024-02-10T07:33:11.235 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:11 smithi008 ceph-mon[29816]: 6.14 deep-scrub ok 2024-02-10T07:33:11.235 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:11 smithi008 ceph-mon[40743]: 4.4 scrub starts 2024-02-10T07:33:11.235 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:11 smithi008 ceph-mon[40743]: 4.4 scrub ok 2024-02-10T07:33:11.235 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:11 smithi008 ceph-mon[40743]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:11.235 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:11 smithi008 ceph-mon[40743]: 6.14 deep-scrub starts 2024-02-10T07:33:11.235 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:11 smithi008 ceph-mon[40743]: 6.14 deep-scrub ok 2024-02-10T07:33:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:11 smithi170 ceph-mon[33394]: 4.4 scrub starts 2024-02-10T07:33:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:11 smithi170 ceph-mon[33394]: 4.4 scrub ok 2024-02-10T07:33:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:11 smithi170 ceph-mon[33394]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:11 smithi170 ceph-mon[33394]: 6.14 deep-scrub starts 2024-02-10T07:33:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:11 smithi170 ceph-mon[33394]: 6.14 deep-scrub ok 2024-02-10T07:33:12.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:12 smithi008 ceph-mon[29816]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:33:12.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:12 smithi008 ceph-mon[29816]: 3.17 scrub starts 2024-02-10T07:33:12.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:12 smithi008 ceph-mon[29816]: 3.17 scrub ok 2024-02-10T07:33:12.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:12 smithi008 ceph-mon[29816]: 4.5 deep-scrub starts 2024-02-10T07:33:12.471 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:12 smithi008 ceph-mon[29816]: 4.5 deep-scrub ok 2024-02-10T07:33:12.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:12 smithi008 ceph-mon[40743]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:33:12.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:12 smithi008 ceph-mon[40743]: 3.17 scrub starts 2024-02-10T07:33:12.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:12 smithi008 ceph-mon[40743]: 3.17 scrub ok 2024-02-10T07:33:12.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:12 smithi008 ceph-mon[40743]: 4.5 deep-scrub starts 2024-02-10T07:33:12.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:12 smithi008 ceph-mon[40743]: 4.5 deep-scrub ok 2024-02-10T07:33:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:12 smithi170 ceph-mon[33394]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:33:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:12 smithi170 ceph-mon[33394]: 3.17 scrub starts 2024-02-10T07:33:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:12 smithi170 ceph-mon[33394]: 3.17 scrub ok 2024-02-10T07:33:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:12 smithi170 ceph-mon[33394]: 4.5 deep-scrub starts 2024-02-10T07:33:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:12 smithi170 ceph-mon[33394]: 4.5 deep-scrub ok 2024-02-10T07:33:13.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:33:12 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:33:13.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:13 smithi008 ceph-mon[40743]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:13.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:13 smithi008 ceph-mon[40743]: 2.2 scrub starts 2024-02-10T07:33:13.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:13 smithi008 ceph-mon[40743]: 2.2 scrub ok 2024-02-10T07:33:13.565 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:13 smithi008 ceph-mon[29816]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:13.565 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:13 smithi008 ceph-mon[29816]: 2.2 scrub starts 2024-02-10T07:33:13.565 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:13 smithi008 ceph-mon[29816]: 2.2 scrub ok 2024-02-10T07:33:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:13 smithi170 ceph-mon[33394]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:13 smithi170 ceph-mon[33394]: 2.2 scrub starts 2024-02-10T07:33:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:13 smithi170 ceph-mon[33394]: 2.2 scrub ok 2024-02-10T07:33:14.116 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-02-10T07:33:09.674465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1004,"num_read_kb":850,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":604,"ondisk_log_size":604,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":254076,"kb_used_data":8828,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749707140,"statfs":{"total":767960285184,"available":767700111360,"internally_reserved":0,"allocated":9039872,"data_stored":5156508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002753"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640262+0000","last_change":"2024-02-10T07:31:57.311638+0000","last_active":"2024-02-10T07:32:46.640262+0000","last_peered":"2024-02-10T07:32:46.640262+0000","last_clean":"2024-02-10T07:32:46.640262+0000","last_became_active":"2024-02-10T07:31:57.311360+0000","last_became_peered":"2024-02-10T07:31:57.311360+0000","last_unstale":"2024-02-10T07:32:46.640262+0000","last_undegraded":"2024-02-10T07:32:46.640262+0000","last_fullsized":"2024-02-10T07:32:46.640262+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.653308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694646+0000","last_change":"2024-02-10T07:31:55.300335+0000","last_active":"2024-02-10T07:32:47.694646+0000","last_peered":"2024-02-10T07:32:47.694646+0000","last_clean":"2024-02-10T07:32:47.694646+0000","last_became_active":"2024-02-10T07:31:55.299946+0000","last_became_peered":"2024-02-10T07:31:55.299946+0000","last_unstale":"2024-02-10T07:32:47.694646+0000","last_undegraded":"2024-02-10T07:32:47.694646+0000","last_fullsized":"2024-02-10T07:32:47.694646+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.036416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101374+0000","last_change":"2024-02-10T07:31:51.321177+0000","last_active":"2024-02-10T07:32:47.101374+0000","last_peered":"2024-02-10T07:32:47.101374+0000","last_clean":"2024-02-10T07:32:47.101374+0000","last_became_active":"2024-02-10T07:31:51.320873+0000","last_became_peered":"2024-02-10T07:31:51.320873+0000","last_unstale":"2024-02-10T07:32:47.101374+0000","last_undegraded":"2024-02-10T07:32:47.101374+0000","last_fullsized":"2024-02-10T07:32:47.101374+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:09.885451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640003+0000","last_change":"2024-02-10T07:31:53.301175+0000","last_active":"2024-02-10T07:32:46.640003+0000","last_peered":"2024-02-10T07:32:46.640003+0000","last_clean":"2024-02-10T07:32:46.640003+0000","last_became_active":"2024-02-10T07:31:53.300846+0000","last_became_peered":"2024-02-10T07:31:53.300846+0000","last_unstale":"2024-02-10T07:32:46.640003+0000","last_undegraded":"2024-02-10T07:32:46.640003+0000","last_fullsized":"2024-02-10T07:32:46.640003+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:15.839827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598212+0000","last_change":"2024-02-10T07:31:57.300388+0000","last_active":"2024-02-10T07:32:47.598212+0000","last_peered":"2024-02-10T07:32:47.598212+0000","last_clean":"2024-02-10T07:32:47.598212+0000","last_became_active":"2024-02-10T07:31:57.300056+0000","last_became_peered":"2024-02-10T07:31:57.300056+0000","last_unstale":"2024-02-10T07:32:47.598212+0000","last_undegraded":"2024-02-10T07:32:47.598212+0000","last_fullsized":"2024-02-10T07:32:47.598212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:00.126262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:58.168423+0000","last_change":"2024-02-10T07:32:58.168446+0000","last_active":"2024-02-10T07:32:58.168423+0000","last_peered":"2024-02-10T07:32:58.168423+0000","last_clean":"2024-02-10T07:32:58.168423+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:32:58.168423+0000","last_undegraded":"2024-02-10T07:32:58.168423+0000","last_fullsized":"2024-02-10T07:32:58.168423+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:58.168353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.168353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:11.296018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597637+0000","last_change":"2024-02-10T07:31:51.323657+0000","last_active":"2024-02-10T07:32:47.597637+0000","last_peered":"2024-02-10T07:32:47.597637+0000","last_clean":"2024-02-10T07:32:47.597637+0000","last_became_active":"2024-02-10T07:31:51.323331+0000","last_became_peered":"2024-02-10T07:31:51.323331+0000","last_unstale":"2024-02-10T07:32:47.597637+0000","last_undegraded":"2024-02-10T07:32:47.597637+0000","last_fullsized":"2024-02-10T07:32:47.597637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:07.093553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":84,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:00.565446+0000","last_change":"2024-02-10T07:33:00.565478+0000","last_active":"2024-02-10T07:33:00.565446+0000","last_peered":"2024-02-10T07:33:00.565446+0000","last_clean":"2024-02-10T07:33:00.565446+0000","last_became_active":"2024-02-10T07:31:53.286957+0000","last_became_peered":"2024-02-10T07:31:53.286957+0000","last_unstale":"2024-02-10T07:33:00.565446+0000","last_undegraded":"2024-02-10T07:33:00.565446+0000","last_fullsized":"2024-02-10T07:33:00.565446+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:33:00.565384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:00.565384+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:05.401431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.100632+0000","last_change":"2024-02-10T07:31:57.313731+0000","last_active":"2024-02-10T07:32:47.100632+0000","last_peered":"2024-02-10T07:32:47.100632+0000","last_clean":"2024-02-10T07:32:47.100632+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:32:47.100632+0000","last_undegraded":"2024-02-10T07:32:47.100632+0000","last_fullsized":"2024-02-10T07:32:47.100632+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:16.885483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639717+0000","last_change":"2024-02-10T07:31:55.298018+0000","last_active":"2024-02-10T07:32:46.639717+0000","last_peered":"2024-02-10T07:32:46.639717+0000","last_clean":"2024-02-10T07:32:46.639717+0000","last_became_active":"2024-02-10T07:31:55.297715+0000","last_became_peered":"2024-02-10T07:31:55.297715+0000","last_unstale":"2024-02-10T07:32:46.639717+0000","last_undegraded":"2024-02-10T07:32:46.639717+0000","last_fullsized":"2024-02-10T07:32:46.639717+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.150692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:51.545567+0000","last_change":"2024-02-10T07:32:51.545597+0000","last_active":"2024-02-10T07:32:51.545567+0000","last_peered":"2024-02-10T07:32:51.545567+0000","last_clean":"2024-02-10T07:32:51.545567+0000","last_became_active":"2024-02-10T07:31:51.283229+0000","last_became_peered":"2024-02-10T07:31:51.283229+0000","last_unstale":"2024-02-10T07:32:51.545567+0000","last_undegraded":"2024-02-10T07:32:51.545567+0000","last_fullsized":"2024-02-10T07:32:51.545567+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:32:51.545493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:32:51.545493+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:17.023144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642023+0000","last_change":"2024-02-10T07:31:53.288142+0000","last_active":"2024-02-10T07:32:46.642023+0000","last_peered":"2024-02-10T07:32:46.642023+0000","last_clean":"2024-02-10T07:32:46.642023+0000","last_became_active":"2024-02-10T07:31:53.287858+0000","last_became_peered":"2024-02-10T07:31:53.287858+0000","last_unstale":"2024-02-10T07:32:46.642023+0000","last_undegraded":"2024-02-10T07:32:46.642023+0000","last_fullsized":"2024-02-10T07:32:46.642023+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:20.421670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639021+0000","last_change":"2024-02-10T07:31:57.310358+0000","last_active":"2024-02-10T07:32:46.639021+0000","last_peered":"2024-02-10T07:32:46.639021+0000","last_clean":"2024-02-10T07:32:46.639021+0000","last_became_active":"2024-02-10T07:31:57.310088+0000","last_became_peered":"2024-02-10T07:31:57.310088+0000","last_unstale":"2024-02-10T07:32:46.639021+0000","last_undegraded":"2024-02-10T07:32:46.639021+0000","last_fullsized":"2024-02-10T07:32:46.639021+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:23.164310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639188+0000","last_change":"2024-02-10T07:31:55.303127+0000","last_active":"2024-02-10T07:32:46.639188+0000","last_peered":"2024-02-10T07:32:46.639188+0000","last_clean":"2024-02-10T07:32:46.639188+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:32:46.639188+0000","last_undegraded":"2024-02-10T07:32:46.639188+0000","last_fullsized":"2024-02-10T07:32:46.639188+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:08.495702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641300+0000","last_change":"2024-02-10T07:31:51.276682+0000","last_active":"2024-02-10T07:32:46.641300+0000","last_peered":"2024-02-10T07:32:46.641300+0000","last_clean":"2024-02-10T07:32:46.641300+0000","last_became_active":"2024-02-10T07:31:51.276350+0000","last_became_peered":"2024-02-10T07:31:51.276350+0000","last_unstale":"2024-02-10T07:32:46.641300+0000","last_undegraded":"2024-02-10T07:32:46.641300+0000","last_fullsized":"2024-02-10T07:32:46.641300+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:11.602006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641535+0000","last_change":"2024-02-10T07:31:53.292840+0000","last_active":"2024-02-10T07:32:46.641535+0000","last_peered":"2024-02-10T07:32:46.641535+0000","last_clean":"2024-02-10T07:32:46.641535+0000","last_became_active":"2024-02-10T07:31:53.292417+0000","last_became_peered":"2024-02-10T07:31:53.292417+0000","last_unstale":"2024-02-10T07:32:46.641535+0000","last_undegraded":"2024-02-10T07:32:46.641535+0000","last_fullsized":"2024-02-10T07:32:46.641535+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:22.210604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642273+0000","last_change":"2024-02-10T07:31:57.306629+0000","last_active":"2024-02-10T07:32:46.642273+0000","last_peered":"2024-02-10T07:32:46.642273+0000","last_clean":"2024-02-10T07:32:46.642273+0000","last_became_active":"2024-02-10T07:31:57.305613+0000","last_became_peered":"2024-02-10T07:31:57.305613+0000","last_unstale":"2024-02-10T07:32:46.642273+0000","last_undegraded":"2024-02-10T07:32:46.642273+0000","last_fullsized":"2024-02-10T07:32:46.642273+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:00.617793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.695149+0000","last_change":"2024-02-10T07:31:55.299041+0000","last_active":"2024-02-10T07:32:47.695149+0000","last_peered":"2024-02-10T07:32:47.695149+0000","last_clean":"2024-02-10T07:32:47.695149+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:32:47.695149+0000","last_undegraded":"2024-02-10T07:32:47.695149+0000","last_fullsized":"2024-02-10T07:32:47.695149+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:14.438305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101011+0000","last_change":"2024-02-10T07:31:51.287400+0000","last_active":"2024-02-10T07:32:47.101011+0000","last_peered":"2024-02-10T07:32:47.101011+0000","last_clean":"2024-02-10T07:32:47.101011+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:32:47.101011+0000","last_undegraded":"2024-02-10T07:32:47.101011+0000","last_fullsized":"2024-02-10T07:32:47.101011+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:17.607773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":75,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.195006+0000","last_change":"2024-02-10T07:31:53.295830+0000","last_active":"2024-02-10T07:32:47.195006+0000","last_peered":"2024-02-10T07:32:47.195006+0000","last_clean":"2024-02-10T07:32:47.195006+0000","last_became_active":"2024-02-10T07:31:53.295527+0000","last_became_peered":"2024-02-10T07:31:53.295527+0000","last_unstale":"2024-02-10T07:32:47.195006+0000","last_undegraded":"2024-02-10T07:32:47.195006+0000","last_fullsized":"2024-02-10T07:32:47.195006+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.171389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598595+0000","last_change":"2024-02-10T07:31:57.306606+0000","last_active":"2024-02-10T07:32:47.598595+0000","last_peered":"2024-02-10T07:32:47.598595+0000","last_clean":"2024-02-10T07:32:47.598595+0000","last_became_active":"2024-02-10T07:31:57.306315+0000","last_became_peered":"2024-02-10T07:31:57.306315+0000","last_unstale":"2024-02-10T07:32:47.598595+0000","last_undegraded":"2024-02-10T07:32:47.598595+0000","last_fullsized":"2024-02-10T07:32:47.598595+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:22.626690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641457+0000","last_change":"2024-02-10T07:31:55.298789+0000","last_active":"2024-02-10T07:32:46.641457+0000","last_peered":"2024-02-10T07:32:46.641457+0000","last_clean":"2024-02-10T07:32:46.641457+0000","last_became_active":"2024-02-10T07:31:55.298474+0000","last_became_peered":"2024-02-10T07:31:55.298474+0000","last_unstale":"2024-02-10T07:32:46.641457+0000","last_undegraded":"2024-02-10T07:32:46.641457+0000","last_fullsized":"2024-02-10T07:32:46.641457+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.404110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:02.198071+0000","last_change":"2024-02-10T07:33:02.198096+0000","last_active":"2024-02-10T07:33:02.198071+0000","last_peered":"2024-02-10T07:33:02.198071+0000","last_clean":"2024-02-10T07:33:02.198071+0000","last_became_active":"2024-02-10T07:31:51.321237+0000","last_became_peered":"2024-02-10T07:31:51.321237+0000","last_unstale":"2024-02-10T07:33:02.198071+0000","last_undegraded":"2024-02-10T07:33:02.198071+0000","last_fullsized":"2024-02-10T07:33:02.198071+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:02.197996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:02.197996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:08.678151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598463+0000","last_change":"2024-02-10T07:31:53.295233+0000","last_active":"2024-02-10T07:32:47.598463+0000","last_peered":"2024-02-10T07:32:47.598463+0000","last_clean":"2024-02-10T07:32:47.598463+0000","last_became_active":"2024-02-10T07:31:53.294914+0000","last_became_peered":"2024-02-10T07:31:53.294914+0000","last_unstale":"2024-02-10T07:32:47.598463+0000","last_undegraded":"2024-02-10T07:32:47.598463+0000","last_fullsized":"2024-02-10T07:32:47.598463+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:00.458485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640318+0000","last_change":"2024-02-10T07:31:57.307172+0000","last_active":"2024-02-10T07:32:46.640318+0000","last_peered":"2024-02-10T07:32:46.640318+0000","last_clean":"2024-02-10T07:32:46.640318+0000","last_became_active":"2024-02-10T07:31:57.306892+0000","last_became_peered":"2024-02-10T07:31:57.306892+0000","last_unstale":"2024-02-10T07:32:46.640318+0000","last_undegraded":"2024-02-10T07:32:46.640318+0000","last_fullsized":"2024-02-10T07:32:46.640318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:15.856701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.371904+0000","last_change":"2024-02-10T07:31:55.295914+0000","last_active":"2024-02-10T07:33:01.371904+0000","last_peered":"2024-02-10T07:33:01.371904+0000","last_clean":"2024-02-10T07:33:01.371904+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:33:01.371904+0000","last_undegraded":"2024-02-10T07:33:01.371904+0000","last_fullsized":"2024-02-10T07:33:01.371904+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:17.555191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694224+0000","last_change":"2024-02-10T07:31:51.282350+0000","last_active":"2024-02-10T07:32:47.694224+0000","last_peered":"2024-02-10T07:32:47.694224+0000","last_clean":"2024-02-10T07:32:47.694224+0000","last_became_active":"2024-02-10T07:31:51.282077+0000","last_became_peered":"2024-02-10T07:31:51.282077+0000","last_unstale":"2024-02-10T07:32:47.694224+0000","last_undegraded":"2024-02-10T07:32:47.694224+0000","last_fullsized":"2024-02-10T07:32:47.694224+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:13.165010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:03.462205+0000","last_change":"2024-02-10T07:33:03.462253+0000","last_active":"2024-02-10T07:33:03.462205+0000","last_peered":"2024-02-10T07:33:03.462205+0000","last_clean":"2024-02-10T07:33:03.462205+0000","last_became_active":"2024-02-10T07:31:53.292936+0000","last_became_peered":"2024-02-10T07:31:53.292936+0000","last_unstale":"2024-02-10T07:33:03.462205+0000","last_undegraded":"2024-02-10T07:33:03.462205+0000","last_fullsized":"2024-02-10T07:33:03.462205+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:33:03.462109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:03.462109+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:31.967818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.695230+0000","last_change":"2024-02-10T07:31:57.309137+0000","last_active":"2024-02-10T07:32:47.695230+0000","last_peered":"2024-02-10T07:32:47.695230+0000","last_clean":"2024-02-10T07:32:47.695230+0000","last_became_active":"2024-02-10T07:31:57.308820+0000","last_became_peered":"2024-02-10T07:31:57.308820+0000","last_unstale":"2024-02-10T07:32:47.695230+0000","last_undegraded":"2024-02-10T07:32:47.695230+0000","last_fullsized":"2024-02-10T07:32:47.695230+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:13.939044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101652+0000","last_change":"2024-02-10T07:31:55.297952+0000","last_active":"2024-02-10T07:32:47.101652+0000","last_peered":"2024-02-10T07:32:47.101652+0000","last_clean":"2024-02-10T07:32:47.101652+0000","last_became_active":"2024-02-10T07:31:55.297595+0000","last_became_peered":"2024-02-10T07:31:55.297595+0000","last_unstale":"2024-02-10T07:32:47.101652+0000","last_undegraded":"2024-02-10T07:32:47.101652+0000","last_fullsized":"2024-02-10T07:32:47.101652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.915423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597500+0000","last_change":"2024-02-10T07:31:51.285841+0000","last_active":"2024-02-10T07:32:47.597500+0000","last_peered":"2024-02-10T07:32:47.597500+0000","last_clean":"2024-02-10T07:32:47.597500+0000","last_became_active":"2024-02-10T07:31:51.285405+0000","last_became_peered":"2024-02-10T07:31:51.285405+0000","last_unstale":"2024-02-10T07:32:47.597500+0000","last_undegraded":"2024-02-10T07:32:47.597500+0000","last_fullsized":"2024-02-10T07:32:47.597500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:00.794907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":77,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:00.861580+0000","last_change":"2024-02-10T07:33:00.861634+0000","last_active":"2024-02-10T07:33:00.861580+0000","last_peered":"2024-02-10T07:33:00.861580+0000","last_clean":"2024-02-10T07:33:00.861580+0000","last_became_active":"2024-02-10T07:31:53.300871+0000","last_became_peered":"2024-02-10T07:31:53.300871+0000","last_unstale":"2024-02-10T07:33:00.861580+0000","last_undegraded":"2024-02-10T07:33:00.861580+0000","last_fullsized":"2024-02-10T07:33:00.861580+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:33:00.861482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:00.861482+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:09.147490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639149+0000","last_change":"2024-02-10T07:31:57.307069+0000","last_active":"2024-02-10T07:32:46.639149+0000","last_peered":"2024-02-10T07:32:46.639149+0000","last_clean":"2024-02-10T07:32:46.639149+0000","last_became_active":"2024-02-10T07:31:57.306776+0000","last_became_peered":"2024-02-10T07:31:57.306776+0000","last_unstale":"2024-02-10T07:32:46.639149+0000","last_undegraded":"2024-02-10T07:32:46.639149+0000","last_fullsized":"2024-02-10T07:32:46.639149+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:09.159262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.695257+0000","last_change":"2024-02-10T07:31:55.291538+0000","last_active":"2024-02-10T07:32:47.695257+0000","last_peered":"2024-02-10T07:32:47.695257+0000","last_clean":"2024-02-10T07:32:47.695257+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:32:47.695257+0000","last_undegraded":"2024-02-10T07:32:47.695257+0000","last_fullsized":"2024-02-10T07:32:47.695257+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:04.691663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101267+0000","last_change":"2024-02-10T07:31:51.284332+0000","last_active":"2024-02-10T07:32:47.101267+0000","last_peered":"2024-02-10T07:32:47.101267+0000","last_clean":"2024-02-10T07:32:47.101267+0000","last_became_active":"2024-02-10T07:31:51.284022+0000","last_became_peered":"2024-02-10T07:31:51.284022+0000","last_unstale":"2024-02-10T07:32:47.101267+0000","last_undegraded":"2024-02-10T07:32:47.101267+0000","last_fullsized":"2024-02-10T07:32:47.101267+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:02.226389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597927+0000","last_change":"2024-02-10T07:31:53.295829+0000","last_active":"2024-02-10T07:32:47.597927+0000","last_peered":"2024-02-10T07:32:47.597927+0000","last_clean":"2024-02-10T07:32:47.597927+0000","last_became_active":"2024-02-10T07:31:53.295520+0000","last_became_peered":"2024-02-10T07:31:53.295520+0000","last_unstale":"2024-02-10T07:32:47.597927+0000","last_undegraded":"2024-02-10T07:32:47.597927+0000","last_fullsized":"2024-02-10T07:32:47.597927+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.493903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598100+0000","last_change":"2024-02-10T07:31:57.315994+0000","last_active":"2024-02-10T07:32:47.598100+0000","last_peered":"2024-02-10T07:32:47.598100+0000","last_clean":"2024-02-10T07:32:47.598100+0000","last_became_active":"2024-02-10T07:31:57.315738+0000","last_became_peered":"2024-02-10T07:31:57.315738+0000","last_unstale":"2024-02-10T07:32:47.598100+0000","last_undegraded":"2024-02-10T07:32:47.598100+0000","last_fullsized":"2024-02-10T07:32:47.598100+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:07.663297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642118+0000","last_change":"2024-02-10T07:31:55.302310+0000","last_active":"2024-02-10T07:32:46.642118+0000","last_peered":"2024-02-10T07:32:46.642118+0000","last_clean":"2024-02-10T07:32:46.642118+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:32:46.642118+0000","last_undegraded":"2024-02-10T07:32:46.642118+0000","last_fullsized":"2024-02-10T07:32:46.642118+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:05.175419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:53.470063+0000","last_change":"2024-02-10T07:32:53.470087+0000","last_active":"2024-02-10T07:32:53.470063+0000","last_peered":"2024-02-10T07:32:53.470063+0000","last_clean":"2024-02-10T07:32:53.470063+0000","last_became_active":"2024-02-10T07:31:51.279438+0000","last_became_peered":"2024-02-10T07:31:51.279438+0000","last_unstale":"2024-02-10T07:32:53.470063+0000","last_undegraded":"2024-02-10T07:32:53.470063+0000","last_fullsized":"2024-02-10T07:32:53.470063+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:32:53.469958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:00.926321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:58.537802+0000","last_change":"2024-02-10T07:32:58.537831+0000","last_active":"2024-02-10T07:32:58.537802+0000","last_peered":"2024-02-10T07:32:58.537802+0000","last_clean":"2024-02-10T07:32:58.537802+0000","last_became_active":"2024-02-10T07:31:53.287823+0000","last_became_peered":"2024-02-10T07:31:53.287823+0000","last_unstale":"2024-02-10T07:32:58.537802+0000","last_undegraded":"2024-02-10T07:32:58.537802+0000","last_fullsized":"2024-02-10T07:32:58.537802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:32:58.537746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.537746+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:07.841413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.100440+0000","last_change":"2024-02-10T07:31:57.313822+0000","last_active":"2024-02-10T07:32:47.100440+0000","last_peered":"2024-02-10T07:32:47.100440+0000","last_clean":"2024-02-10T07:32:47.100440+0000","last_became_active":"2024-02-10T07:31:57.313529+0000","last_became_peered":"2024-02-10T07:31:57.313529+0000","last_unstale":"2024-02-10T07:32:47.100440+0000","last_undegraded":"2024-02-10T07:32:47.100440+0000","last_fullsized":"2024-02-10T07:32:47.100440+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:23.816714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:59.584337+0000","last_change":"2024-02-10T07:32:59.584373+0000","last_active":"2024-02-10T07:32:59.584337+0000","last_peered":"2024-02-10T07:32:59.584337+0000","last_clean":"2024-02-10T07:32:59.584337+0000","last_became_active":"2024-02-10T07:31:55.296670+0000","last_became_peered":"2024-02-10T07:31:55.296670+0000","last_unstale":"2024-02-10T07:32:59.584337+0000","last_undegraded":"2024-02-10T07:32:59.584337+0000","last_fullsized":"2024-02-10T07:32:59.584337+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:59.584219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:59.584219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:22.310522+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640519+0000","last_change":"2024-02-10T07:31:51.288447+0000","last_active":"2024-02-10T07:32:46.640519+0000","last_peered":"2024-02-10T07:32:46.640519+0000","last_clean":"2024-02-10T07:32:46.640519+0000","last_became_active":"2024-02-10T07:31:51.288165+0000","last_became_peered":"2024-02-10T07:31:51.288165+0000","last_unstale":"2024-02-10T07:32:46.640519+0000","last_undegraded":"2024-02-10T07:32:46.640519+0000","last_fullsized":"2024-02-10T07:32:46.640519+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:32:59.646262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694985+0000","last_change":"2024-02-10T07:31:53.292496+0000","last_active":"2024-02-10T07:32:47.694985+0000","last_peered":"2024-02-10T07:32:47.694985+0000","last_clean":"2024-02-10T07:32:47.694985+0000","last_became_active":"2024-02-10T07:31:53.292182+0000","last_became_peered":"2024-02-10T07:31:53.292182+0000","last_unstale":"2024-02-10T07:32:47.694985+0000","last_undegraded":"2024-02-10T07:32:47.694985+0000","last_fullsized":"2024-02-10T07:32:47.694985+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:14.454667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597842+0000","last_change":"2024-02-10T07:31:57.316162+0000","last_active":"2024-02-10T07:32:47.597842+0000","last_peered":"2024-02-10T07:32:47.597842+0000","last_clean":"2024-02-10T07:32:47.597842+0000","last_became_active":"2024-02-10T07:31:57.315781+0000","last_became_peered":"2024-02-10T07:31:57.315781+0000","last_unstale":"2024-02-10T07:32:47.597842+0000","last_undegraded":"2024-02-10T07:32:47.597842+0000","last_fullsized":"2024-02-10T07:32:47.597842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:08.947342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694902+0000","last_change":"2024-02-10T07:31:55.301556+0000","last_active":"2024-02-10T07:32:47.694902+0000","last_peered":"2024-02-10T07:32:47.694902+0000","last_clean":"2024-02-10T07:32:47.694902+0000","last_became_active":"2024-02-10T07:31:55.301260+0000","last_became_peered":"2024-02-10T07:31:55.301260+0000","last_unstale":"2024-02-10T07:32:47.694902+0000","last_undegraded":"2024-02-10T07:32:47.694902+0000","last_fullsized":"2024-02-10T07:32:47.694902+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:17.634914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641706+0000","last_change":"2024-02-10T07:31:51.286248+0000","last_active":"2024-02-10T07:32:46.641706+0000","last_peered":"2024-02-10T07:32:46.641706+0000","last_clean":"2024-02-10T07:32:46.641706+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:32:46.641706+0000","last_undegraded":"2024-02-10T07:32:46.641706+0000","last_fullsized":"2024-02-10T07:32:46.641706+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:06.511452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":96,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:00.143959+0000","last_change":"2024-02-10T07:33:00.144015+0000","last_active":"2024-02-10T07:33:00.143959+0000","last_peered":"2024-02-10T07:33:00.143959+0000","last_clean":"2024-02-10T07:33:00.143959+0000","last_became_active":"2024-02-10T07:32:18.389564+0000","last_became_peered":"2024-02-10T07:32:18.389564+0000","last_unstale":"2024-02-10T07:33:00.143959+0000","last_undegraded":"2024-02-10T07:33:00.143959+0000","last_fullsized":"2024-02-10T07:33:00.143959+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:33:00.143859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:00.143859+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:28.135685+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101249+0000","last_change":"2024-02-10T07:31:57.318330+0000","last_active":"2024-02-10T07:32:47.101249+0000","last_peered":"2024-02-10T07:32:47.101249+0000","last_clean":"2024-02-10T07:32:47.101249+0000","last_became_active":"2024-02-10T07:31:57.318009+0000","last_became_peered":"2024-02-10T07:31:57.318009+0000","last_unstale":"2024-02-10T07:32:47.101249+0000","last_undegraded":"2024-02-10T07:32:47.101249+0000","last_fullsized":"2024-02-10T07:32:47.101249+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:09.866798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":70,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.371818+0000","last_change":"2024-02-10T07:31:55.298387+0000","last_active":"2024-02-10T07:33:01.371818+0000","last_peered":"2024-02-10T07:33:01.371818+0000","last_clean":"2024-02-10T07:33:01.371818+0000","last_became_active":"2024-02-10T07:31:55.297670+0000","last_became_peered":"2024-02-10T07:31:55.297670+0000","last_unstale":"2024-02-10T07:33:01.371818+0000","last_undegraded":"2024-02-10T07:33:01.371818+0000","last_fullsized":"2024-02-10T07:33:01.371818+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.348418+0000","last_change":"2024-02-10T07:33:06.348460+0000","last_active":"2024-02-10T07:33:06.348418+0000","last_peered":"2024-02-10T07:33:06.348418+0000","last_clean":"2024-02-10T07:33:06.348418+0000","last_became_active":"2024-02-10T07:31:51.320644+0000","last_became_peered":"2024-02-10T07:31:51.320644+0000","last_unstale":"2024-02-10T07:33:06.348418+0000","last_undegraded":"2024-02-10T07:33:06.348418+0000","last_fullsized":"2024-02-10T07:33:06.348418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.348322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:15.227477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598312+0000","last_change":"2024-02-10T07:31:53.296048+0000","last_active":"2024-02-10T07:32:47.598312+0000","last_peered":"2024-02-10T07:32:47.598312+0000","last_clean":"2024-02-10T07:32:47.598312+0000","last_became_active":"2024-02-10T07:31:53.295641+0000","last_became_peered":"2024-02-10T07:31:53.295641+0000","last_unstale":"2024-02-10T07:32:47.598312+0000","last_undegraded":"2024-02-10T07:32:47.598312+0000","last_fullsized":"2024-02-10T07:32:47.598312+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:17.035054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642477+0000","last_change":"2024-02-10T07:31:57.307866+0000","last_active":"2024-02-10T07:32:46.642477+0000","last_peered":"2024-02-10T07:32:46.642477+0000","last_clean":"2024-02-10T07:32:46.642477+0000","last_became_active":"2024-02-10T07:31:57.307583+0000","last_became_peered":"2024-02-10T07:31:57.307583+0000","last_unstale":"2024-02-10T07:32:46.642477+0000","last_undegraded":"2024-02-10T07:32:46.642477+0000","last_fullsized":"2024-02-10T07:32:46.642477+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:25.642236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.371669+0000","last_change":"2024-02-10T07:31:55.294941+0000","last_active":"2024-02-10T07:33:01.371669+0000","last_peered":"2024-02-10T07:33:01.371669+0000","last_clean":"2024-02-10T07:33:01.371669+0000","last_became_active":"2024-02-10T07:31:55.294594+0000","last_became_peered":"2024-02-10T07:31:55.294594+0000","last_unstale":"2024-02-10T07:33:01.371669+0000","last_undegraded":"2024-02-10T07:33:01.371669+0000","last_fullsized":"2024-02-10T07:33:01.371669+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:08.937883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.496713+0000","last_change":"2024-02-10T07:33:01.496748+0000","last_active":"2024-02-10T07:33:01.496713+0000","last_peered":"2024-02-10T07:33:01.496713+0000","last_clean":"2024-02-10T07:33:01.496713+0000","last_became_active":"2024-02-10T07:31:51.282353+0000","last_became_peered":"2024-02-10T07:31:51.282353+0000","last_unstale":"2024-02-10T07:33:01.496713+0000","last_undegraded":"2024-02-10T07:33:01.496713+0000","last_fullsized":"2024-02-10T07:33:01.496713+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:01.496610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:01.496610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:02.323527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640928+0000","last_change":"2024-02-10T07:31:53.286183+0000","last_active":"2024-02-10T07:32:46.640928+0000","last_peered":"2024-02-10T07:32:46.640928+0000","last_clean":"2024-02-10T07:32:46.640928+0000","last_became_active":"2024-02-10T07:31:53.285777+0000","last_became_peered":"2024-02-10T07:31:53.285777+0000","last_unstale":"2024-02-10T07:32:46.640928+0000","last_undegraded":"2024-02-10T07:32:46.640928+0000","last_fullsized":"2024-02-10T07:32:46.640928+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:32:55.553428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.195301+0000","last_change":"2024-02-10T07:31:57.308913+0000","last_active":"2024-02-10T07:32:47.195301+0000","last_peered":"2024-02-10T07:32:47.195301+0000","last_clean":"2024-02-10T07:32:47.195301+0000","last_became_active":"2024-02-10T07:31:57.308615+0000","last_became_peered":"2024-02-10T07:31:57.308615+0000","last_unstale":"2024-02-10T07:32:47.195301+0000","last_undegraded":"2024-02-10T07:32:47.195301+0000","last_fullsized":"2024-02-10T07:32:47.195301+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:07.427216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598712+0000","last_change":"2024-02-10T07:31:55.297740+0000","last_active":"2024-02-10T07:32:47.598712+0000","last_peered":"2024-02-10T07:32:47.598712+0000","last_clean":"2024-02-10T07:32:47.598712+0000","last_became_active":"2024-02-10T07:31:55.297431+0000","last_became_peered":"2024-02-10T07:31:55.297431+0000","last_unstale":"2024-02-10T07:32:47.598712+0000","last_undegraded":"2024-02-10T07:32:47.598712+0000","last_fullsized":"2024-02-10T07:32:47.598712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:02.665928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639342+0000","last_change":"2024-02-10T07:31:51.284304+0000","last_active":"2024-02-10T07:32:46.639342+0000","last_peered":"2024-02-10T07:32:46.639342+0000","last_clean":"2024-02-10T07:32:46.639342+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:32:46.639342+0000","last_undegraded":"2024-02-10T07:32:46.639342+0000","last_fullsized":"2024-02-10T07:32:46.639342+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:02.877266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"89'13","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:56.585009+0000","last_change":"2024-02-10T07:32:56.585037+0000","last_active":"2024-02-10T07:32:56.585009+0000","last_peered":"2024-02-10T07:32:56.585009+0000","last_clean":"2024-02-10T07:32:56.585009+0000","last_became_active":"2024-02-10T07:31:53.295353+0000","last_became_peered":"2024-02-10T07:31:53.295353+0000","last_unstale":"2024-02-10T07:32:56.585009+0000","last_undegraded":"2024-02-10T07:32:56.585009+0000","last_fullsized":"2024-02-10T07:32:56.585009+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:32:56.584957+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:04.077539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640268+0000","last_change":"2024-02-10T07:31:57.305113+0000","last_active":"2024-02-10T07:32:46.640268+0000","last_peered":"2024-02-10T07:32:46.640268+0000","last_clean":"2024-02-10T07:32:46.640268+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:32:46.640268+0000","last_undegraded":"2024-02-10T07:32:46.640268+0000","last_fullsized":"2024-02-10T07:32:46.640268+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:11.938578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641217+0000","last_change":"2024-02-10T07:31:55.296699+0000","last_active":"2024-02-10T07:32:46.641217+0000","last_peered":"2024-02-10T07:32:46.641217+0000","last_clean":"2024-02-10T07:32:46.641217+0000","last_became_active":"2024-02-10T07:31:55.296369+0000","last_became_peered":"2024-02-10T07:31:55.296369+0000","last_unstale":"2024-02-10T07:32:46.641217+0000","last_undegraded":"2024-02-10T07:32:46.641217+0000","last_fullsized":"2024-02-10T07:32:46.641217+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.519413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694565+0000","last_change":"2024-02-10T07:31:51.285405+0000","last_active":"2024-02-10T07:32:47.694565+0000","last_peered":"2024-02-10T07:32:47.694565+0000","last_clean":"2024-02-10T07:32:47.694565+0000","last_became_active":"2024-02-10T07:31:51.285159+0000","last_became_peered":"2024-02-10T07:31:51.285159+0000","last_unstale":"2024-02-10T07:32:47.694565+0000","last_undegraded":"2024-02-10T07:32:47.694565+0000","last_fullsized":"2024-02-10T07:32:47.694565+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:03.320486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598658+0000","last_change":"2024-02-10T07:31:53.297115+0000","last_active":"2024-02-10T07:32:47.598658+0000","last_peered":"2024-02-10T07:32:47.598658+0000","last_clean":"2024-02-10T07:32:47.598658+0000","last_became_active":"2024-02-10T07:31:53.296831+0000","last_became_peered":"2024-02-10T07:31:53.296831+0000","last_unstale":"2024-02-10T07:32:47.598658+0000","last_undegraded":"2024-02-10T07:32:47.598658+0000","last_fullsized":"2024-02-10T07:32:47.598658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:21.861465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639876+0000","last_change":"2024-02-10T07:31:57.311522+0000","last_active":"2024-02-10T07:32:46.639876+0000","last_peered":"2024-02-10T07:32:46.639876+0000","last_clean":"2024-02-10T07:32:46.639876+0000","last_became_active":"2024-02-10T07:31:57.311110+0000","last_became_peered":"2024-02-10T07:31:57.311110+0000","last_unstale":"2024-02-10T07:32:46.639876+0000","last_undegraded":"2024-02-10T07:32:46.639876+0000","last_fullsized":"2024-02-10T07:32:46.639876+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:25.982197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640304+0000","last_change":"2024-02-10T07:31:51.286209+0000","last_active":"2024-02-10T07:32:46.640304+0000","last_peered":"2024-02-10T07:32:46.640304+0000","last_clean":"2024-02-10T07:32:46.640304+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:32:46.640304+0000","last_undegraded":"2024-02-10T07:32:46.640304+0000","last_fullsized":"2024-02-10T07:32:46.640304+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:08.757933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:59.385507+0000","last_change":"2024-02-10T07:32:59.385539+0000","last_active":"2024-02-10T07:32:59.385507+0000","last_peered":"2024-02-10T07:32:59.385507+0000","last_clean":"2024-02-10T07:32:59.385507+0000","last_became_active":"2024-02-10T07:31:53.284817+0000","last_became_peered":"2024-02-10T07:31:53.284817+0000","last_unstale":"2024-02-10T07:32:59.385507+0000","last_undegraded":"2024-02-10T07:32:59.385507+0000","last_fullsized":"2024-02-10T07:32:59.385507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:32:59.385448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:59.385448+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:59.831143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.371819+0000","last_change":"2024-02-10T07:31:55.301606+0000","last_active":"2024-02-10T07:33:01.371819+0000","last_peered":"2024-02-10T07:33:01.371819+0000","last_clean":"2024-02-10T07:33:01.371819+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:33:01.371819+0000","last_undegraded":"2024-02-10T07:33:01.371819+0000","last_fullsized":"2024-02-10T07:33:01.371819+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598067+0000","last_change":"2024-02-10T07:31:57.315186+0000","last_active":"2024-02-10T07:32:47.598067+0000","last_peered":"2024-02-10T07:32:47.598067+0000","last_clean":"2024-02-10T07:32:47.598067+0000","last_became_active":"2024-02-10T07:31:57.314872+0000","last_became_peered":"2024-02-10T07:31:57.314872+0000","last_unstale":"2024-02-10T07:32:47.598067+0000","last_undegraded":"2024-02-10T07:32:47.598067+0000","last_fullsized":"2024-02-10T07:32:47.598067+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:10.772524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640466+0000","last_change":"2024-02-10T07:31:51.287017+0000","last_active":"2024-02-10T07:32:46.640466+0000","last_peered":"2024-02-10T07:32:46.640466+0000","last_clean":"2024-02-10T07:32:46.640466+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:32:46.640466+0000","last_undegraded":"2024-02-10T07:32:46.640466+0000","last_fullsized":"2024-02-10T07:32:46.640466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:05.325144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":92,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640700+0000","last_change":"2024-02-10T07:32:18.396325+0000","last_active":"2024-02-10T07:32:46.640700+0000","last_peered":"2024-02-10T07:32:46.640700+0000","last_clean":"2024-02-10T07:32:46.640700+0000","last_became_active":"2024-02-10T07:32:18.389409+0000","last_became_peered":"2024-02-10T07:32:18.389409+0000","last_unstale":"2024-02-10T07:32:46.640700+0000","last_undegraded":"2024-02-10T07:32:46.640700+0000","last_fullsized":"2024-02-10T07:32:46.640700+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:07.708054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:59.218979+0000","last_change":"2024-02-10T07:32:59.218999+0000","last_active":"2024-02-10T07:32:59.218979+0000","last_peered":"2024-02-10T07:32:59.218979+0000","last_clean":"2024-02-10T07:32:59.218979+0000","last_became_active":"2024-02-10T07:31:55.303059+0000","last_became_peered":"2024-02-10T07:31:55.303059+0000","last_unstale":"2024-02-10T07:32:59.218979+0000","last_undegraded":"2024-02-10T07:32:59.218979+0000","last_fullsized":"2024-02-10T07:32:59.218979+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:59.218908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:59.218908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:06.437654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639057+0000","last_change":"2024-02-10T07:31:57.305254+0000","last_active":"2024-02-10T07:32:46.639057+0000","last_peered":"2024-02-10T07:32:46.639057+0000","last_clean":"2024-02-10T07:32:46.639057+0000","last_became_active":"2024-02-10T07:31:57.304971+0000","last_became_peered":"2024-02-10T07:31:57.304971+0000","last_unstale":"2024-02-10T07:32:46.639057+0000","last_undegraded":"2024-02-10T07:32:46.639057+0000","last_fullsized":"2024-02-10T07:32:46.639057+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:01.467119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.194617+0000","last_change":"2024-02-10T07:31:55.298725+0000","last_active":"2024-02-10T07:32:47.194617+0000","last_peered":"2024-02-10T07:32:47.194617+0000","last_clean":"2024-02-10T07:32:47.194617+0000","last_became_active":"2024-02-10T07:31:55.298322+0000","last_became_peered":"2024-02-10T07:31:55.298322+0000","last_unstale":"2024-02-10T07:32:47.194617+0000","last_undegraded":"2024-02-10T07:32:47.194617+0000","last_fullsized":"2024-02-10T07:32:47.194617+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.947435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641102+0000","last_change":"2024-02-10T07:31:51.283777+0000","last_active":"2024-02-10T07:32:46.641102+0000","last_peered":"2024-02-10T07:32:46.641102+0000","last_clean":"2024-02-10T07:32:46.641102+0000","last_became_active":"2024-02-10T07:31:51.283433+0000","last_became_peered":"2024-02-10T07:31:51.283433+0000","last_unstale":"2024-02-10T07:32:46.641102+0000","last_undegraded":"2024-02-10T07:32:46.641102+0000","last_fullsized":"2024-02-10T07:32:46.641102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:02.188930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.695470+0000","last_change":"2024-02-10T07:31:53.301616+0000","last_active":"2024-02-10T07:32:47.695470+0000","last_peered":"2024-02-10T07:32:47.695470+0000","last_clean":"2024-02-10T07:32:47.695470+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:32:47.695470+0000","last_undegraded":"2024-02-10T07:32:47.695470+0000","last_fullsized":"2024-02-10T07:32:47.695470+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:10.820328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.100519+0000","last_change":"2024-02-10T07:31:57.314885+0000","last_active":"2024-02-10T07:32:47.100519+0000","last_peered":"2024-02-10T07:32:47.100519+0000","last_clean":"2024-02-10T07:32:47.100519+0000","last_became_active":"2024-02-10T07:31:57.313394+0000","last_became_peered":"2024-02-10T07:31:57.313394+0000","last_unstale":"2024-02-10T07:32:47.100519+0000","last_undegraded":"2024-02-10T07:32:47.100519+0000","last_fullsized":"2024-02-10T07:32:47.100519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:04.536324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.194336+0000","last_change":"2024-02-10T07:31:55.298984+0000","last_active":"2024-02-10T07:32:47.194336+0000","last_peered":"2024-02-10T07:32:47.194336+0000","last_clean":"2024-02-10T07:32:47.194336+0000","last_became_active":"2024-02-10T07:31:55.298622+0000","last_became_peered":"2024-02-10T07:31:55.298622+0000","last_unstale":"2024-02-10T07:32:47.194336+0000","last_undegraded":"2024-02-10T07:32:47.194336+0000","last_fullsized":"2024-02-10T07:32:47.194336+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:11.544202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639978+0000","last_change":"2024-02-10T07:31:51.283676+0000","last_active":"2024-02-10T07:32:46.639978+0000","last_peered":"2024-02-10T07:32:46.639978+0000","last_clean":"2024-02-10T07:32:46.639978+0000","last_became_active":"2024-02-10T07:31:51.283208+0000","last_became_peered":"2024-02-10T07:31:51.283208+0000","last_unstale":"2024-02-10T07:32:46.639978+0000","last_undegraded":"2024-02-10T07:32:46.639978+0000","last_fullsized":"2024-02-10T07:32:46.639978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:11.381433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101117+0000","last_change":"2024-02-10T07:31:53.292486+0000","last_active":"2024-02-10T07:32:47.101117+0000","last_peered":"2024-02-10T07:32:47.101117+0000","last_clean":"2024-02-10T07:32:47.101117+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:32:47.101117+0000","last_undegraded":"2024-02-10T07:32:47.101117+0000","last_fullsized":"2024-02-10T07:32:47.101117+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:07.434520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598514+0000","last_change":"2024-02-10T07:31:57.314251+0000","last_active":"2024-02-10T07:32:47.598514+0000","last_peered":"2024-02-10T07:32:47.598514+0000","last_clean":"2024-02-10T07:32:47.598514+0000","last_became_active":"2024-02-10T07:31:57.313854+0000","last_became_peered":"2024-02-10T07:31:57.313854+0000","last_unstale":"2024-02-10T07:32:47.598514+0000","last_undegraded":"2024-02-10T07:32:47.598514+0000","last_fullsized":"2024-02-10T07:32:47.598514+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.959268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639330+0000","last_change":"2024-02-10T07:31:54.288414+0000","last_active":"2024-02-10T07:32:46.639330+0000","last_peered":"2024-02-10T07:32:46.639330+0000","last_clean":"2024-02-10T07:32:46.639330+0000","last_became_active":"2024-02-10T07:31:50.278548+0000","last_became_peered":"2024-02-10T07:31:50.278548+0000","last_unstale":"2024-02-10T07:32:46.639330+0000","last_undegraded":"2024-02-10T07:32:46.639330+0000","last_fullsized":"2024-02-10T07:32:46.639330+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:31:49.265852+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:32:55.653852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001361225,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641171+0000","last_change":"2024-02-10T07:31:51.285951+0000","last_active":"2024-02-10T07:32:46.641171+0000","last_peered":"2024-02-10T07:32:46.641171+0000","last_clean":"2024-02-10T07:32:46.641171+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:32:46.641171+0000","last_undegraded":"2024-02-10T07:32:46.641171+0000","last_fullsized":"2024-02-10T07:32:46.641171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:13.565080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":113,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.371614+0000","last_change":"2024-02-10T07:31:53.302218+0000","last_active":"2024-02-10T07:33:01.371614+0000","last_peered":"2024-02-10T07:33:01.371614+0000","last_clean":"2024-02-10T07:33:01.371614+0000","last_became_active":"2024-02-10T07:31:53.301863+0000","last_became_peered":"2024-02-10T07:31:53.301863+0000","last_unstale":"2024-02-10T07:33:01.371614+0000","last_undegraded":"2024-02-10T07:33:01.371614+0000","last_fullsized":"2024-02-10T07:33:01.371614+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101146+0000","last_change":"2024-02-10T07:31:55.296643+0000","last_active":"2024-02-10T07:32:47.101146+0000","last_peered":"2024-02-10T07:32:47.101146+0000","last_clean":"2024-02-10T07:32:47.101146+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:32:47.101146+0000","last_undegraded":"2024-02-10T07:32:47.101146+0000","last_fullsized":"2024-02-10T07:32:47.101146+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:05.656831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:58.354107+0000","last_change":"2024-02-10T07:32:58.354147+0000","last_active":"2024-02-10T07:32:58.354107+0000","last_peered":"2024-02-10T07:32:58.354107+0000","last_clean":"2024-02-10T07:32:58.354107+0000","last_became_active":"2024-02-10T07:31:57.299958+0000","last_became_peered":"2024-02-10T07:31:57.299958+0000","last_unstale":"2024-02-10T07:32:58.354107+0000","last_undegraded":"2024-02-10T07:32:58.354107+0000","last_fullsized":"2024-02-10T07:32:58.354107+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:58.354022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.354022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:17.659362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:03.198766+0000","last_change":"2024-02-10T07:33:03.198787+0000","last_active":"2024-02-10T07:33:03.198766+0000","last_peered":"2024-02-10T07:33:03.198766+0000","last_clean":"2024-02-10T07:33:03.198766+0000","last_became_active":"2024-02-10T07:31:51.320864+0000","last_became_peered":"2024-02-10T07:31:51.320864+0000","last_unstale":"2024-02-10T07:33:03.198766+0000","last_undegraded":"2024-02-10T07:33:03.198766+0000","last_fullsized":"2024-02-10T07:33:03.198766+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:03.198699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:03.198699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:13.704299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:58.209366+0000","last_change":"2024-02-10T07:32:58.209386+0000","last_active":"2024-02-10T07:32:58.209366+0000","last_peered":"2024-02-10T07:32:58.209366+0000","last_clean":"2024-02-10T07:32:58.209366+0000","last_became_active":"2024-02-10T07:31:50.281517+0000","last_became_peered":"2024-02-10T07:31:50.281517+0000","last_unstale":"2024-02-10T07:32:58.209366+0000","last_undegraded":"2024-02-10T07:32:58.209366+0000","last_fullsized":"2024-02-10T07:32:58.209366+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:58.209303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.209303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:28.009863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.194782+0000","last_change":"2024-02-10T07:32:18.398935+0000","last_active":"2024-02-10T07:32:47.194782+0000","last_peered":"2024-02-10T07:32:47.194782+0000","last_clean":"2024-02-10T07:32:47.194782+0000","last_became_active":"2024-02-10T07:32:18.391217+0000","last_became_peered":"2024-02-10T07:32:18.391217+0000","last_unstale":"2024-02-10T07:32:47.194782+0000","last_undegraded":"2024-02-10T07:32:47.194782+0000","last_fullsized":"2024-02-10T07:32:47.194782+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:16.649956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.183383+0000","last_change":"2024-02-10T07:33:06.183420+0000","last_active":"2024-02-10T07:33:06.183383+0000","last_peered":"2024-02-10T07:33:06.183383+0000","last_clean":"2024-02-10T07:33:06.183383+0000","last_became_active":"2024-02-10T07:31:55.298933+0000","last_became_peered":"2024-02-10T07:31:55.298933+0000","last_unstale":"2024-02-10T07:33:06.183383+0000","last_undegraded":"2024-02-10T07:33:06.183383+0000","last_fullsized":"2024-02-10T07:33:06.183383+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.183292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.183292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:12.631994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"74'1","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639813+0000","last_change":"2024-02-10T07:31:57.310237+0000","last_active":"2024-02-10T07:32:46.639813+0000","last_peered":"2024-02-10T07:32:46.639813+0000","last_clean":"2024-02-10T07:32:46.639813+0000","last_became_active":"2024-02-10T07:31:57.309929+0000","last_became_peered":"2024-02-10T07:31:57.309929+0000","last_unstale":"2024-02-10T07:32:46.639813+0000","last_undegraded":"2024-02-10T07:32:46.639813+0000","last_fullsized":"2024-02-10T07:32:46.639813+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:14.496768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":308,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641970+0000","last_change":"2024-02-10T07:32:25.175725+0000","last_active":"2024-02-10T07:32:46.641970+0000","last_peered":"2024-02-10T07:32:46.641970+0000","last_clean":"2024-02-10T07:32:46.641970+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:32:46.641970+0000","last_undegraded":"2024-02-10T07:32:46.641970+0000","last_fullsized":"2024-02-10T07:32:46.641970+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-02-10T07:32:25.175641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:32:25.175641+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:35.761800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597518+0000","last_change":"2024-02-10T07:31:51.283607+0000","last_active":"2024-02-10T07:32:47.597518+0000","last_peered":"2024-02-10T07:32:47.597518+0000","last_clean":"2024-02-10T07:32:47.597518+0000","last_became_active":"2024-02-10T07:31:51.283332+0000","last_became_peered":"2024-02-10T07:31:51.283332+0000","last_unstale":"2024-02-10T07:32:47.597518+0000","last_undegraded":"2024-02-10T07:32:47.597518+0000","last_fullsized":"2024-02-10T07:32:47.597518+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:06.299140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":88,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.371910+0000","last_change":"2024-02-10T07:31:53.290959+0000","last_active":"2024-02-10T07:33:01.371910+0000","last_peered":"2024-02-10T07:33:01.371910+0000","last_clean":"2024-02-10T07:33:01.371910+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:33:01.371910+0000","last_undegraded":"2024-02-10T07:33:01.371910+0000","last_fullsized":"2024-02-10T07:33:01.371910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640726+0000","last_change":"2024-02-10T07:31:55.300257+0000","last_active":"2024-02-10T07:32:46.640726+0000","last_peered":"2024-02-10T07:32:46.640726+0000","last_clean":"2024-02-10T07:32:46.640726+0000","last_became_active":"2024-02-10T07:31:55.299928+0000","last_became_peered":"2024-02-10T07:31:55.299928+0000","last_unstale":"2024-02-10T07:32:46.640726+0000","last_undegraded":"2024-02-10T07:32:46.640726+0000","last_fullsized":"2024-02-10T07:32:46.640726+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:15.486048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694890+0000","last_change":"2024-02-10T07:31:57.316758+0000","last_active":"2024-02-10T07:32:47.694890+0000","last_peered":"2024-02-10T07:32:47.694890+0000","last_clean":"2024-02-10T07:32:47.694890+0000","last_became_active":"2024-02-10T07:31:57.316432+0000","last_became_peered":"2024-02-10T07:31:57.316432+0000","last_unstale":"2024-02-10T07:32:47.694890+0000","last_undegraded":"2024-02-10T07:32:47.694890+0000","last_fullsized":"2024-02-10T07:32:47.694890+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:23.313144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.100688+0000","last_change":"2024-02-10T07:31:51.286361+0000","last_active":"2024-02-10T07:32:47.100688+0000","last_peered":"2024-02-10T07:32:47.100688+0000","last_clean":"2024-02-10T07:32:47.100688+0000","last_became_active":"2024-02-10T07:31:51.286076+0000","last_became_peered":"2024-02-10T07:31:51.286076+0000","last_unstale":"2024-02-10T07:32:47.100688+0000","last_undegraded":"2024-02-10T07:32:47.100688+0000","last_fullsized":"2024-02-10T07:32:47.100688+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:17.577559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":138,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.298755+0000","last_change":"2024-02-10T07:31:54.281755+0000","last_active":"2024-02-10T07:33:06.298755+0000","last_peered":"2024-02-10T07:33:06.298755+0000","last_clean":"2024-02-10T07:33:06.298755+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:33:06.298755+0000","last_undegraded":"2024-02-10T07:33:06.298755+0000","last_fullsized":"2024-02-10T07:33:06.298755+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:31:49.265852+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:11.035681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101395+0000","last_change":"2024-02-10T07:31:53.301026+0000","last_active":"2024-02-10T07:32:47.101395+0000","last_peered":"2024-02-10T07:32:47.101395+0000","last_clean":"2024-02-10T07:32:47.101395+0000","last_became_active":"2024-02-10T07:31:53.300687+0000","last_became_peered":"2024-02-10T07:31:53.300687+0000","last_unstale":"2024-02-10T07:32:47.101395+0000","last_undegraded":"2024-02-10T07:32:47.101395+0000","last_fullsized":"2024-02-10T07:32:47.101395+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:16.203387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640509+0000","last_change":"2024-02-10T07:31:55.297440+0000","last_active":"2024-02-10T07:32:46.640509+0000","last_peered":"2024-02-10T07:32:46.640509+0000","last_clean":"2024-02-10T07:32:46.640509+0000","last_became_active":"2024-02-10T07:31:55.297178+0000","last_became_peered":"2024-02-10T07:31:55.297178+0000","last_unstale":"2024-02-10T07:32:46.640509+0000","last_undegraded":"2024-02-10T07:32:46.640509+0000","last_fullsized":"2024-02-10T07:32:46.640509+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.962436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694624+0000","last_change":"2024-02-10T07:31:57.307961+0000","last_active":"2024-02-10T07:32:47.694624+0000","last_peered":"2024-02-10T07:32:47.694624+0000","last_clean":"2024-02-10T07:32:47.694624+0000","last_became_active":"2024-02-10T07:31:57.307615+0000","last_became_peered":"2024-02-10T07:31:57.307615+0000","last_unstale":"2024-02-10T07:32:47.694624+0000","last_undegraded":"2024-02-10T07:32:47.694624+0000","last_fullsized":"2024-02-10T07:32:47.694624+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:24.112392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.270015+0000","last_change":"2024-02-10T07:33:06.270053+0000","last_active":"2024-02-10T07:33:06.270015+0000","last_peered":"2024-02-10T07:33:06.270015+0000","last_clean":"2024-02-10T07:33:06.270015+0000","last_became_active":"2024-02-10T07:31:51.320752+0000","last_became_peered":"2024-02-10T07:31:51.320752+0000","last_unstale":"2024-02-10T07:33:06.270015+0000","last_undegraded":"2024-02-10T07:33:06.270015+0000","last_fullsized":"2024-02-10T07:33:06.270015+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.269938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.269938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:34.431697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.100974+0000","last_change":"2024-02-10T07:31:53.289754+0000","last_active":"2024-02-10T07:32:47.100974+0000","last_peered":"2024-02-10T07:32:47.100974+0000","last_clean":"2024-02-10T07:32:47.100974+0000","last_became_active":"2024-02-10T07:31:53.289381+0000","last_became_peered":"2024-02-10T07:31:53.289381+0000","last_unstale":"2024-02-10T07:32:47.100974+0000","last_undegraded":"2024-02-10T07:32:47.100974+0000","last_fullsized":"2024-02-10T07:32:47.100974+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:13.156780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.695474+0000","last_change":"2024-02-10T07:31:55.290623+0000","last_active":"2024-02-10T07:32:47.695474+0000","last_peered":"2024-02-10T07:32:47.695474+0000","last_clean":"2024-02-10T07:32:47.695474+0000","last_became_active":"2024-02-10T07:31:55.290319+0000","last_became_peered":"2024-02-10T07:31:55.290319+0000","last_unstale":"2024-02-10T07:32:47.695474+0000","last_undegraded":"2024-02-10T07:32:47.695474+0000","last_fullsized":"2024-02-10T07:32:47.695474+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:23.955277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:05.431132+0000","last_change":"2024-02-10T07:33:05.431169+0000","last_active":"2024-02-10T07:33:05.431132+0000","last_peered":"2024-02-10T07:33:05.431132+0000","last_clean":"2024-02-10T07:33:05.431132+0000","last_became_active":"2024-02-10T07:31:57.317976+0000","last_became_peered":"2024-02-10T07:31:57.317976+0000","last_unstale":"2024-02-10T07:33:05.431132+0000","last_undegraded":"2024-02-10T07:33:05.431132+0000","last_fullsized":"2024-02-10T07:33:05.431132+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:05.431037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:05.431037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:31.470574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640631+0000","last_change":"2024-02-10T07:31:51.320993+0000","last_active":"2024-02-10T07:32:46.640631+0000","last_peered":"2024-02-10T07:32:46.640631+0000","last_clean":"2024-02-10T07:32:46.640631+0000","last_became_active":"2024-02-10T07:31:51.320673+0000","last_became_peered":"2024-02-10T07:31:51.320673+0000","last_unstale":"2024-02-10T07:32:46.640631+0000","last_undegraded":"2024-02-10T07:32:46.640631+0000","last_fullsized":"2024-02-10T07:32:46.640631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:15.248068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:53.288892+0000","last_change":"2024-02-10T07:32:53.288963+0000","last_active":"2024-02-10T07:32:53.288892+0000","last_peered":"2024-02-10T07:32:53.288892+0000","last_clean":"2024-02-10T07:32:53.288892+0000","last_became_active":"2024-02-10T07:31:53.297150+0000","last_became_peered":"2024-02-10T07:31:53.297150+0000","last_unstale":"2024-02-10T07:32:53.288892+0000","last_undegraded":"2024-02-10T07:32:53.288892+0000","last_fullsized":"2024-02-10T07:32:53.288892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:32:53.288798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:53.288798+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:06.843116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":70,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.371605+0000","last_change":"2024-02-10T07:31:55.296952+0000","last_active":"2024-02-10T07:33:01.371605+0000","last_peered":"2024-02-10T07:33:01.371605+0000","last_clean":"2024-02-10T07:33:01.371605+0000","last_became_active":"2024-02-10T07:31:55.295732+0000","last_became_peered":"2024-02-10T07:31:55.295732+0000","last_unstale":"2024-02-10T07:33:01.371605+0000","last_undegraded":"2024-02-10T07:33:01.371605+0000","last_fullsized":"2024-02-10T07:33:01.371605+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.515830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639824+0000","last_change":"2024-02-10T07:31:57.305942+0000","last_active":"2024-02-10T07:32:46.639824+0000","last_peered":"2024-02-10T07:32:46.639824+0000","last_clean":"2024-02-10T07:32:46.639824+0000","last_became_active":"2024-02-10T07:31:57.305620+0000","last_became_peered":"2024-02-10T07:31:57.305620+0000","last_unstale":"2024-02-10T07:32:46.639824+0000","last_undegraded":"2024-02-10T07:32:46.639824+0000","last_fullsized":"2024-02-10T07:32:46.639824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:09.830636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101458+0000","last_change":"2024-02-10T07:31:51.285921+0000","last_active":"2024-02-10T07:32:47.101458+0000","last_peered":"2024-02-10T07:32:47.101458+0000","last_clean":"2024-02-10T07:32:47.101458+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:32:47.101458+0000","last_undegraded":"2024-02-10T07:32:47.101458+0000","last_fullsized":"2024-02-10T07:32:47.101458+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:05.329043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640503+0000","last_change":"2024-02-10T07:31:53.287199+0000","last_active":"2024-02-10T07:32:46.640503+0000","last_peered":"2024-02-10T07:32:46.640503+0000","last_clean":"2024-02-10T07:32:46.640503+0000","last_became_active":"2024-02-10T07:31:53.286884+0000","last_became_peered":"2024-02-10T07:31:53.286884+0000","last_unstale":"2024-02-10T07:32:46.640503+0000","last_undegraded":"2024-02-10T07:32:46.640503+0000","last_fullsized":"2024-02-10T07:32:46.640503+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.033998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.371608+0000","last_change":"2024-02-10T07:32:55.469184+0000","last_active":"2024-02-10T07:33:01.371608+0000","last_peered":"2024-02-10T07:33:01.371608+0000","last_clean":"2024-02-10T07:33:01.371608+0000","last_became_active":"2024-02-10T07:31:55.302863+0000","last_became_peered":"2024-02-10T07:31:55.302863+0000","last_unstale":"2024-02-10T07:33:01.371608+0000","last_undegraded":"2024-02-10T07:33:01.371608+0000","last_fullsized":"2024-02-10T07:33:01.371608+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:32:55.469099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:55.469099+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:13.566389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641909+0000","last_change":"2024-02-10T07:31:57.316332+0000","last_active":"2024-02-10T07:32:46.641909+0000","last_peered":"2024-02-10T07:32:46.641909+0000","last_clean":"2024-02-10T07:32:46.641909+0000","last_became_active":"2024-02-10T07:31:57.316047+0000","last_became_peered":"2024-02-10T07:31:57.316047+0000","last_unstale":"2024-02-10T07:32:46.641909+0000","last_undegraded":"2024-02-10T07:32:46.641909+0000","last_fullsized":"2024-02-10T07:32:46.641909+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:25.756992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597656+0000","last_change":"2024-02-10T07:31:51.285635+0000","last_active":"2024-02-10T07:32:47.597656+0000","last_peered":"2024-02-10T07:32:47.597656+0000","last_clean":"2024-02-10T07:32:47.597656+0000","last_became_active":"2024-02-10T07:31:51.285333+0000","last_became_peered":"2024-02-10T07:31:51.285333+0000","last_unstale":"2024-02-10T07:32:47.597656+0000","last_undegraded":"2024-02-10T07:32:47.597656+0000","last_fullsized":"2024-02-10T07:32:47.597656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:05.926246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598312+0000","last_change":"2024-02-10T07:31:53.290461+0000","last_active":"2024-02-10T07:32:47.598312+0000","last_peered":"2024-02-10T07:32:47.598312+0000","last_clean":"2024-02-10T07:32:47.598312+0000","last_became_active":"2024-02-10T07:31:53.290157+0000","last_became_peered":"2024-02-10T07:31:53.290157+0000","last_unstale":"2024-02-10T07:32:47.598312+0000","last_undegraded":"2024-02-10T07:32:47.598312+0000","last_fullsized":"2024-02-10T07:32:47.598312+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:32:58.963046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.184144+0000","last_change":"2024-02-10T07:33:01.184183+0000","last_active":"2024-02-10T07:33:01.184144+0000","last_peered":"2024-02-10T07:33:01.184144+0000","last_clean":"2024-02-10T07:33:01.184144+0000","last_became_active":"2024-02-10T07:31:55.295900+0000","last_became_peered":"2024-02-10T07:31:55.295900+0000","last_unstale":"2024-02-10T07:33:01.184144+0000","last_undegraded":"2024-02-10T07:33:01.184144+0000","last_fullsized":"2024-02-10T07:33:01.184144+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:01.184048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:01.184048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:11.992366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640531+0000","last_change":"2024-02-10T07:31:57.311731+0000","last_active":"2024-02-10T07:32:46.640531+0000","last_peered":"2024-02-10T07:32:46.640531+0000","last_clean":"2024-02-10T07:32:46.640531+0000","last_became_active":"2024-02-10T07:31:57.311054+0000","last_became_peered":"2024-02-10T07:31:57.311054+0000","last_unstale":"2024-02-10T07:32:46.640531+0000","last_undegraded":"2024-02-10T07:32:46.640531+0000","last_fullsized":"2024-02-10T07:32:46.640531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.402912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.371849+0000","last_change":"2024-02-10T07:31:55.295389+0000","last_active":"2024-02-10T07:33:01.371849+0000","last_peered":"2024-02-10T07:33:01.371849+0000","last_clean":"2024-02-10T07:33:01.371849+0000","last_became_active":"2024-02-10T07:31:55.295067+0000","last_became_peered":"2024-02-10T07:31:55.295067+0000","last_unstale":"2024-02-10T07:33:01.371849+0000","last_undegraded":"2024-02-10T07:33:01.371849+0000","last_fullsized":"2024-02-10T07:33:01.371849+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:24.085032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597500+0000","last_change":"2024-02-10T07:31:51.282545+0000","last_active":"2024-02-10T07:32:47.597500+0000","last_peered":"2024-02-10T07:32:47.597500+0000","last_clean":"2024-02-10T07:32:47.597500+0000","last_became_active":"2024-02-10T07:31:51.282137+0000","last_became_peered":"2024-02-10T07:31:51.282137+0000","last_unstale":"2024-02-10T07:32:47.597500+0000","last_undegraded":"2024-02-10T07:32:47.597500+0000","last_fullsized":"2024-02-10T07:32:47.597500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.188800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":82,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:57.341196+0000","last_change":"2024-02-10T07:32:57.341249+0000","last_active":"2024-02-10T07:32:57.341196+0000","last_peered":"2024-02-10T07:32:57.341196+0000","last_clean":"2024-02-10T07:32:57.341196+0000","last_became_active":"2024-02-10T07:31:53.294218+0000","last_became_peered":"2024-02-10T07:31:53.294218+0000","last_unstale":"2024-02-10T07:32:57.341196+0000","last_undegraded":"2024-02-10T07:32:57.341196+0000","last_fullsized":"2024-02-10T07:32:57.341196+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:32:57.341115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:57.341115+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:05.137575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:04.582259+0000","last_change":"2024-02-10T07:33:04.582281+0000","last_active":"2024-02-10T07:33:04.582259+0000","last_peered":"2024-02-10T07:33:04.582259+0000","last_clean":"2024-02-10T07:33:04.582259+0000","last_became_active":"2024-02-10T07:31:57.303235+0000","last_became_peered":"2024-02-10T07:31:57.303235+0000","last_unstale":"2024-02-10T07:33:04.582259+0000","last_undegraded":"2024-02-10T07:33:04.582259+0000","last_fullsized":"2024-02-10T07:33:04.582259+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:04.582188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:04.582188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:27.553405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.237029+0000","last_change":"2024-02-10T07:33:06.237068+0000","last_active":"2024-02-10T07:33:06.237029+0000","last_peered":"2024-02-10T07:33:06.237029+0000","last_clean":"2024-02-10T07:33:06.237029+0000","last_became_active":"2024-02-10T07:31:55.294376+0000","last_became_peered":"2024-02-10T07:31:55.294376+0000","last_unstale":"2024-02-10T07:33:06.237029+0000","last_undegraded":"2024-02-10T07:33:06.237029+0000","last_fullsized":"2024-02-10T07:33:06.237029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.236946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.236946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:10.623394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640348+0000","last_change":"2024-02-10T07:31:51.321090+0000","last_active":"2024-02-10T07:32:46.640348+0000","last_peered":"2024-02-10T07:32:46.640348+0000","last_clean":"2024-02-10T07:32:46.640348+0000","last_became_active":"2024-02-10T07:31:51.320715+0000","last_became_peered":"2024-02-10T07:31:51.320715+0000","last_unstale":"2024-02-10T07:32:46.640348+0000","last_undegraded":"2024-02-10T07:32:46.640348+0000","last_fullsized":"2024-02-10T07:32:46.640348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:04.387923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":70,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642166+0000","last_change":"2024-02-10T07:31:53.292945+0000","last_active":"2024-02-10T07:32:46.642166+0000","last_peered":"2024-02-10T07:32:46.642166+0000","last_clean":"2024-02-10T07:32:46.642166+0000","last_became_active":"2024-02-10T07:31:53.292510+0000","last_became_peered":"2024-02-10T07:31:53.292510+0000","last_unstale":"2024-02-10T07:32:46.642166+0000","last_undegraded":"2024-02-10T07:32:46.642166+0000","last_fullsized":"2024-02-10T07:32:46.642166+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:04.860425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:05.197704+0000","last_change":"2024-02-10T07:33:05.197724+0000","last_active":"2024-02-10T07:33:05.197704+0000","last_peered":"2024-02-10T07:33:05.197704+0000","last_clean":"2024-02-10T07:33:05.197704+0000","last_became_active":"2024-02-10T07:31:57.311427+0000","last_became_peered":"2024-02-10T07:31:57.311427+0000","last_unstale":"2024-02-10T07:33:05.197704+0000","last_undegraded":"2024-02-10T07:33:05.197704+0000","last_fullsized":"2024-02-10T07:33:05.197704+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:33:05.197642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:20.323214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.695093+0000","last_change":"2024-02-10T07:31:55.299573+0000","last_active":"2024-02-10T07:32:47.695093+0000","last_peered":"2024-02-10T07:32:47.695093+0000","last_clean":"2024-02-10T07:32:47.695093+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:32:47.695093+0000","last_undegraded":"2024-02-10T07:32:47.695093+0000","last_fullsized":"2024-02-10T07:32:47.695093+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:07.859918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:56.900575+0000","last_change":"2024-02-10T07:32:56.900599+0000","last_active":"2024-02-10T07:32:56.900575+0000","last_peered":"2024-02-10T07:32:56.900575+0000","last_clean":"2024-02-10T07:32:56.900575+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:32:56.900575+0000","last_undegraded":"2024-02-10T07:32:56.900575+0000","last_fullsized":"2024-02-10T07:32:56.900575+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:56.900501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:56.900501+0000","last_clean_scrub_stamp":"2024-02-10T07:32:56.900501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:16.803474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642534+0000","last_change":"2024-02-10T07:31:53.292581+0000","last_active":"2024-02-10T07:32:46.642534+0000","last_peered":"2024-02-10T07:32:46.642534+0000","last_clean":"2024-02-10T07:32:46.642534+0000","last_became_active":"2024-02-10T07:31:53.292277+0000","last_became_peered":"2024-02-10T07:31:53.292277+0000","last_unstale":"2024-02-10T07:32:46.642534+0000","last_undegraded":"2024-02-10T07:32:46.642534+0000","last_fullsized":"2024-02-10T07:32:46.642534+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:04.812915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642496+0000","last_change":"2024-02-10T07:31:57.308864+0000","last_active":"2024-02-10T07:32:46.642496+0000","last_peered":"2024-02-10T07:32:46.642496+0000","last_clean":"2024-02-10T07:32:46.642496+0000","last_became_active":"2024-02-10T07:31:57.308608+0000","last_became_peered":"2024-02-10T07:31:57.308608+0000","last_unstale":"2024-02-10T07:32:46.642496+0000","last_undegraded":"2024-02-10T07:32:46.642496+0000","last_fullsized":"2024-02-10T07:32:46.642496+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:09.099379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.371615+0000","last_change":"2024-02-10T07:31:55.298852+0000","last_active":"2024-02-10T07:33:01.371615+0000","last_peered":"2024-02-10T07:33:01.371615+0000","last_clean":"2024-02-10T07:33:01.371615+0000","last_became_active":"2024-02-10T07:31:55.298389+0000","last_became_peered":"2024-02-10T07:31:55.298389+0000","last_unstale":"2024-02-10T07:33:01.371615+0000","last_undegraded":"2024-02-10T07:33:01.371615+0000","last_fullsized":"2024-02-10T07:33:01.371615+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.686032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597506+0000","last_change":"2024-02-10T07:31:51.288638+0000","last_active":"2024-02-10T07:32:47.597506+0000","last_peered":"2024-02-10T07:32:47.597506+0000","last_clean":"2024-02-10T07:32:47.597506+0000","last_became_active":"2024-02-10T07:31:51.288197+0000","last_became_peered":"2024-02-10T07:31:51.288197+0000","last_unstale":"2024-02-10T07:32:47.597506+0000","last_undegraded":"2024-02-10T07:32:47.597506+0000","last_fullsized":"2024-02-10T07:32:47.597506+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:04.642730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641567+0000","last_change":"2024-02-10T07:31:53.288289+0000","last_active":"2024-02-10T07:32:46.641567+0000","last_peered":"2024-02-10T07:32:46.641567+0000","last_clean":"2024-02-10T07:32:46.641567+0000","last_became_active":"2024-02-10T07:31:53.287984+0000","last_became_peered":"2024-02-10T07:31:53.287984+0000","last_unstale":"2024-02-10T07:32:46.641567+0000","last_undegraded":"2024-02-10T07:32:46.641567+0000","last_fullsized":"2024-02-10T07:32:46.641567+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.716780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":397,"ondisk_log_size":397,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037780,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1495040,"data_stored":1013417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.48399999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.625}]},{"osd":3,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.53200000000000003}]},{"osd":4,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.60199999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.75900000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.73599999999999999}]}]},{"osd":6,"up_from":50,"seq":214748364825,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28460,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716692,"statfs":{"total":95995035648,"available":95965892608,"internally_reserved":0,"allocated":1478656,"data_stored":1012764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.438}]},{"osd":1,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.45800000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":19.664999999999999,"5min":19.664999999999999,"15min":19.664999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.47499999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.46500000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.378},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.34200000000000003}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.44700000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36236,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708916,"statfs":{"total":95995035648,"available":95957929984,"internally_reserved":0,"allocated":921600,"data_stored":423123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.36599999999999999,"15min":0.34399999999999997},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.39000000000000001,"15min":0.35999999999999999},"min":{"1min":0.29099999999999998,"5min":0.22,"15min":0.22},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.32600000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41299999999999998,"15min":0.40000000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42899999999999999,"15min":0.41799999999999998},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.28399999999999997}]},{"osd":3,"last update":"Sat Feb 10 07:32:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.433,"15min":0.42699999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.504},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51600000000000001,"15min":0.51000000000000001},"min":{"1min":0.376,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.44900000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.98799999999999999,"15min":1.0469999999999999},"min":{"1min":0.435,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80700000000000005,"5min":10.292,"15min":10.292},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58399999999999996,"15min":0.58299999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.71599999999999997,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.45900000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:32:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.627,"15min":0.63},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80500000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.47099999999999997}]},{"osd":6,"last update":"Sat Feb 10 07:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.55800000000000005}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.436},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.42499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32644,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712508,"statfs":{"total":95995035648,"available":95961608192,"internally_reserved":0,"allocated":1503232,"data_stored":1014034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:32:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35099999999999998,"15min":0.34599999999999997},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.56200000000000006,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.377,"15min":0.36199999999999999},"min":{"1min":0.26000000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61199999999999999,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.38500000000000001,"15min":0.38800000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.51200000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.254},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42899999999999999,"15min":0.42499999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57899999999999996,"5min":0.629,"15min":0.629},"last":0.28000000000000003}]},{"osd":3,"last update":"Sat Feb 10 07:33:05 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.432,"15min":0.443},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53800000000000003,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47499999999999998,"15min":0.47199999999999998},"min":{"1min":0.32400000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64700000000000002,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.32400000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.74099999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.433,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.59799999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:32:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59699999999999998,"15min":0.59399999999999997},"min":{"1min":0.438,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.67000000000000004}]},{"osd":6,"last update":"Sat Feb 10 07:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.65100000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":19.178000000000001,"5min":19.178000000000001,"15min":19.178000000000001},"last":0.624},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.58399999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":905216,"data_stored":423646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.379,"15min":0.38},"min":{"1min":0.255,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56499999999999995,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.42399999999999999,"15min":0.41599999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.38400000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.44,"15min":0.42399999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.435},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.44,"15min":0.42199999999999999},"min":{"1min":0.27200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.42099999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51700000000000002,"15min":0.53400000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.53100000000000003,"15min":0.55300000000000005},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.34699999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63100000000000001,"15min":0.626},"min":{"1min":0.50700000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.68899999999999995,"15min":0.65300000000000002},"min":{"1min":0.50700000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":5.343,"5min":5.343,"15min":5.343},"last":0.60399999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.64900000000000002,"15min":0.64400000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.628,"15min":0.623},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.61799999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":5.3230000000000004,"5min":5.3230000000000004,"15min":5.3230000000000004},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.66300000000000003}]},{"osd":7,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.75700000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":921600,"data_stored":422902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.372,"15min":0.378},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55500000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.41299999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.42099999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.376},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.442,"15min":0.437},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.317}]},{"osd":2,"last update":"Sat Feb 10 07:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.31,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42799999999999999,"15min":0.434},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.33100000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.89700000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.623}]},{"osd":5,"last update":"Sat Feb 10 07:32:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.64400000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.71399999999999997}]},{"osd":7,"last update":"Sat Feb 10 07:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.68100000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953507,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":423014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.58099999999999996,"15min":0.58499999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.69899999999999995,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39200000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.70599999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.35399999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:32:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.44,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69499999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72899999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:32:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.67300000000000004,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78800000000000003,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.46000000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:32:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57999999999999996,"15min":0.58299999999999996},"min":{"1min":0.44900000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.76000000000000001,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.376,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70499999999999996,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.51000000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.378,"15min":0.373},"min":{"1min":0.30099999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60799999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.435},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.40300000000000002,"15min":0.39300000000000002},"min":{"1min":0.32300000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.57099999999999995}]},{"osd":6,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.62},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Feb 10 07:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.54400000000000004}]}]},{"osd":5,"up_from":40,"seq":171798691870,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32052,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713100,"statfs":{"total":95995035648,"available":95962214400,"internally_reserved":0,"allocated":897024,"data_stored":423608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77400000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.378},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.57699999999999996,"15min":0.56100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.2029999999999998,"5min":3.2029999999999998,"15min":3.2029999999999998},"last":0.66500000000000004}]},{"osd":1,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.75,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.61299999999999999,"15min":0.59699999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.238,"5min":3.238,"15min":3.238},"last":0.55500000000000005}]},{"osd":2,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.438,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76200000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.748,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.60299999999999998},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80800000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.76500000000000001,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.48199999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.41099999999999998,"15min":0.40100000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.39700000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.52600000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.64300000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.71099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:33:14.124 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:33:14.212 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[29816]: 5.c scrub starts 2024-02-10T07:33:14.213 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[29816]: 5.c scrub ok 2024-02-10T07:33:14.213 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[29816]: 6.1b scrub starts 2024-02-10T07:33:14.213 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[29816]: 6.1b scrub ok 2024-02-10T07:33:14.213 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:14.213 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[29816]: 5.a scrub starts 2024-02-10T07:33:14.213 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[29816]: 5.a scrub ok 2024-02-10T07:33:14.213 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[29816]: 5.11 scrub starts 2024-02-10T07:33:14.213 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[29816]: 5.11 scrub ok 2024-02-10T07:33:14.213 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[40743]: 5.c scrub starts 2024-02-10T07:33:14.213 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[40743]: 5.c scrub ok 2024-02-10T07:33:14.213 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[40743]: 6.1b scrub starts 2024-02-10T07:33:14.213 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[40743]: 6.1b scrub ok 2024-02-10T07:33:14.214 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:14.214 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[40743]: 5.a scrub starts 2024-02-10T07:33:14.214 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[40743]: 5.a scrub ok 2024-02-10T07:33:14.214 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[40743]: 5.11 scrub starts 2024-02-10T07:33:14.214 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-mon[40743]: 5.11 scrub ok 2024-02-10T07:33:14.559 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:14 smithi008 podman[75733]: 2024-02-10 07:33:14.451296536 +0000 UTC m=+0.120195371 volume create e097244bafa46188b9bcc3334d1545736e2ef8f40385c3601cc293e1121de9fc 2024-02-10T07:33:14.560 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:14 smithi008 podman[75733]: 2024-02-10 07:33:14.559860904 +0000 UTC m=+0.228759780 container create 583cb0039d939852601fa18a520037a41bc64c3f7830f21129acbfbae668cbc2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:14 smithi170 ceph-mon[33394]: 5.c scrub starts 2024-02-10T07:33:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:14 smithi170 ceph-mon[33394]: 5.c scrub ok 2024-02-10T07:33:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:14 smithi170 ceph-mon[33394]: 6.1b scrub starts 2024-02-10T07:33:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:14 smithi170 ceph-mon[33394]: 6.1b scrub ok 2024-02-10T07:33:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:14 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:14 smithi170 ceph-mon[33394]: 5.a scrub starts 2024-02-10T07:33:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:14 smithi170 ceph-mon[33394]: 5.a scrub ok 2024-02-10T07:33:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:14 smithi170 ceph-mon[33394]: 5.11 scrub starts 2024-02-10T07:33:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:14 smithi170 ceph-mon[33394]: 5.11 scrub ok 2024-02-10T07:33:15.075 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:33:15] ENGINE Bus STOPPING 2024-02-10T07:33:15.075 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:33:15] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:33:15.075 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:33:15] ENGINE Bus STOPPED 2024-02-10T07:33:15.075 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:14 smithi008 podman[75733]: 2024-02-10 07:33:14.820657998 +0000 UTC m=+0.489556833 container init 583cb0039d939852601fa18a520037a41bc64c3f7830f21129acbfbae668cbc2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:15.076 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:14 smithi008 podman[75733]: 2024-02-10 07:33:14.823457914 +0000 UTC m=+0.492356760 container start 583cb0039d939852601fa18a520037a41bc64c3f7830f21129acbfbae668cbc2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:15.076 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[75760]: ts=2024-02-10T07:33:14.835Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-10T07:33:15.076 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[75760]: ts=2024-02-10T07:33:14.835Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-10T07:33:15.076 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[75760]: ts=2024-02-10T07:33:14.836Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.8 port=9094 2024-02-10T07:33:15.076 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[75760]: ts=2024-02-10T07:33:14.837Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T07:33:15.076 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[75760]: ts=2024-02-10T07:33:14.859Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T07:33:15.076 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[75760]: ts=2024-02-10T07:33:14.859Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T07:33:15.076 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[75760]: ts=2024-02-10T07:33:14.862Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-10T07:33:15.076 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:14 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[75760]: ts=2024-02-10T07:33:14.862Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-10T07:33:15.076 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:14 smithi008 bash[75733]: 583cb0039d939852601fa18a520037a41bc64c3f7830f21129acbfbae668cbc2 2024-02-10T07:33:15.076 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:14 smithi008 systemd[1]: Started Ceph alertmanager.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:33:15.086 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:15.331 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:33:15] ENGINE Bus STARTING 2024-02-10T07:33:15.331 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:33:15] ENGINE Serving on http://:::9283 2024-02-10T07:33:15.332 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:33:15] ENGINE Bus STARTED 2024-02-10T07:33:15.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: 4.1d scrub starts 2024-02-10T07:33:15.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: 4.1d scrub ok 2024-02-10T07:33:15.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: 4.1c deep-scrub starts 2024-02-10T07:33:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: 4.1c deep-scrub ok 2024-02-10T07:33:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: 6.9 scrub starts 2024-02-10T07:33:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: 6.9 scrub ok 2024-02-10T07:33:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:33:15.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: 4.1d scrub starts 2024-02-10T07:33:15.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: 4.1d scrub ok 2024-02-10T07:33:15.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:15.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: 4.1c deep-scrub starts 2024-02-10T07:33:15.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: 4.1c deep-scrub ok 2024-02-10T07:33:15.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: 6.9 scrub starts 2024-02-10T07:33:15.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: 6.9 scrub ok 2024-02-10T07:33:15.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:33:15.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:15 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: 4.1d scrub starts 2024-02-10T07:33:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: 4.1d scrub ok 2024-02-10T07:33:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:15.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: 4.1c deep-scrub starts 2024-02-10T07:33:15.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: 4.1c deep-scrub ok 2024-02-10T07:33:15.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: 6.9 scrub starts 2024-02-10T07:33:15.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: 6.9 scrub ok 2024-02-10T07:33:15.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:15.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:33:15.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:15 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:16.287 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[29816]: 5.19 scrub starts 2024-02-10T07:33:16.287 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[29816]: 5.19 scrub ok 2024-02-10T07:33:16.287 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[29816]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T07:33:16.287 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[29816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:33:16.287 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[29816]: Deploying daemon grafana.a on smithi170 2024-02-10T07:33:16.287 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[29816]: 5.15 scrub starts 2024-02-10T07:33:16.287 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[29816]: 5.15 scrub ok 2024-02-10T07:33:16.287 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[29816]: 3.6 scrub starts 2024-02-10T07:33:16.288 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[29816]: 3.6 scrub ok 2024-02-10T07:33:16.288 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[40743]: 5.19 scrub starts 2024-02-10T07:33:16.288 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[40743]: 5.19 scrub ok 2024-02-10T07:33:16.288 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[40743]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T07:33:16.288 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[40743]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:33:16.288 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[40743]: Deploying daemon grafana.a on smithi170 2024-02-10T07:33:16.288 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[40743]: 5.15 scrub starts 2024-02-10T07:33:16.288 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[40743]: 5.15 scrub ok 2024-02-10T07:33:16.288 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[40743]: 3.6 scrub starts 2024-02-10T07:33:16.288 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-mon[40743]: 3.6 scrub ok 2024-02-10T07:33:16.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:16 smithi170 ceph-mon[33394]: 5.19 scrub starts 2024-02-10T07:33:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:16 smithi170 ceph-mon[33394]: 5.19 scrub ok 2024-02-10T07:33:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:16 smithi170 ceph-mon[33394]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T07:33:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:16 smithi170 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T07:33:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:16 smithi170 ceph-mon[33394]: Deploying daemon grafana.a on smithi170 2024-02-10T07:33:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:16 smithi170 ceph-mon[33394]: 5.15 scrub starts 2024-02-10T07:33:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:16 smithi170 ceph-mon[33394]: 5.15 scrub ok 2024-02-10T07:33:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:16 smithi170 ceph-mon[33394]: 3.6 scrub starts 2024-02-10T07:33:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:16 smithi170 ceph-mon[33394]: 3.6 scrub ok 2024-02-10T07:33:17.028 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:33:17.030 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:33:17.168 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:16 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[75760]: ts=2024-02-10T07:33:16.837Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000495546s 2024-02-10T07:33:17.497 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:17 smithi008 ceph-mon[29816]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:17.497 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:17 smithi008 ceph-mon[29816]: 5.b scrub starts 2024-02-10T07:33:17.497 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:17 smithi008 ceph-mon[29816]: 5.b scrub ok 2024-02-10T07:33:17.497 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:17 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:17.497 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:17 smithi008 ceph-mon[29816]: 4.a scrub starts 2024-02-10T07:33:17.497 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:17 smithi008 ceph-mon[29816]: 4.a scrub ok 2024-02-10T07:33:17.497 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:17 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:17.497 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:17 smithi008 ceph-mon[40743]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:17.498 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:17 smithi008 ceph-mon[40743]: 5.b scrub starts 2024-02-10T07:33:17.498 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:17 smithi008 ceph-mon[40743]: 5.b scrub ok 2024-02-10T07:33:17.498 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:17 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:17.498 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:17 smithi008 ceph-mon[40743]: 4.a scrub starts 2024-02-10T07:33:17.498 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:17 smithi008 ceph-mon[40743]: 4.a scrub ok 2024-02-10T07:33:17.498 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:17 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:17 smithi170 ceph-mon[33394]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:17 smithi170 ceph-mon[33394]: 5.b scrub starts 2024-02-10T07:33:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:17 smithi170 ceph-mon[33394]: 5.b scrub ok 2024-02-10T07:33:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:17 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:17 smithi170 ceph-mon[33394]: 4.a scrub starts 2024-02-10T07:33:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:17 smithi170 ceph-mon[33394]: 4.a scrub ok 2024-02-10T07:33:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:17 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:17.756 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-02-10T07:33:15.676121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1009,"num_read_kb":855,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":612,"ondisk_log_size":612,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":254076,"kb_used_data":8828,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749707140,"statfs":{"total":767960285184,"available":767700111360,"internally_reserved":0,"allocated":9039872,"data_stored":5156508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003197"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640262+0000","last_change":"2024-02-10T07:31:57.311638+0000","last_active":"2024-02-10T07:32:46.640262+0000","last_peered":"2024-02-10T07:32:46.640262+0000","last_clean":"2024-02-10T07:32:46.640262+0000","last_became_active":"2024-02-10T07:31:57.311360+0000","last_became_peered":"2024-02-10T07:31:57.311360+0000","last_unstale":"2024-02-10T07:32:46.640262+0000","last_undegraded":"2024-02-10T07:32:46.640262+0000","last_fullsized":"2024-02-10T07:32:46.640262+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.653308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694646+0000","last_change":"2024-02-10T07:31:55.300335+0000","last_active":"2024-02-10T07:32:47.694646+0000","last_peered":"2024-02-10T07:32:47.694646+0000","last_clean":"2024-02-10T07:32:47.694646+0000","last_became_active":"2024-02-10T07:31:55.299946+0000","last_became_peered":"2024-02-10T07:31:55.299946+0000","last_unstale":"2024-02-10T07:32:47.694646+0000","last_undegraded":"2024-02-10T07:32:47.694646+0000","last_fullsized":"2024-02-10T07:32:47.694646+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.036416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101374+0000","last_change":"2024-02-10T07:31:51.321177+0000","last_active":"2024-02-10T07:32:47.101374+0000","last_peered":"2024-02-10T07:32:47.101374+0000","last_clean":"2024-02-10T07:32:47.101374+0000","last_became_active":"2024-02-10T07:31:51.320873+0000","last_became_peered":"2024-02-10T07:31:51.320873+0000","last_unstale":"2024-02-10T07:32:47.101374+0000","last_undegraded":"2024-02-10T07:32:47.101374+0000","last_fullsized":"2024-02-10T07:32:47.101374+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:09.885451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640003+0000","last_change":"2024-02-10T07:31:53.301175+0000","last_active":"2024-02-10T07:32:46.640003+0000","last_peered":"2024-02-10T07:32:46.640003+0000","last_clean":"2024-02-10T07:32:46.640003+0000","last_became_active":"2024-02-10T07:31:53.300846+0000","last_became_peered":"2024-02-10T07:31:53.300846+0000","last_unstale":"2024-02-10T07:32:46.640003+0000","last_undegraded":"2024-02-10T07:32:46.640003+0000","last_fullsized":"2024-02-10T07:32:46.640003+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:15.839827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:12.587926+0000","last_change":"2024-02-10T07:33:12.587948+0000","last_active":"2024-02-10T07:33:12.587926+0000","last_peered":"2024-02-10T07:33:12.587926+0000","last_clean":"2024-02-10T07:33:12.587926+0000","last_became_active":"2024-02-10T07:31:57.300056+0000","last_became_peered":"2024-02-10T07:31:57.300056+0000","last_unstale":"2024-02-10T07:33:12.587926+0000","last_undegraded":"2024-02-10T07:33:12.587926+0000","last_fullsized":"2024-02-10T07:33:12.587926+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:12.587860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:12.587860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:17.246465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:58.168423+0000","last_change":"2024-02-10T07:32:58.168446+0000","last_active":"2024-02-10T07:32:58.168423+0000","last_peered":"2024-02-10T07:32:58.168423+0000","last_clean":"2024-02-10T07:32:58.168423+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:32:58.168423+0000","last_undegraded":"2024-02-10T07:32:58.168423+0000","last_fullsized":"2024-02-10T07:32:58.168423+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:58.168353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.168353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:11.296018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597637+0000","last_change":"2024-02-10T07:31:51.323657+0000","last_active":"2024-02-10T07:32:47.597637+0000","last_peered":"2024-02-10T07:32:47.597637+0000","last_clean":"2024-02-10T07:32:47.597637+0000","last_became_active":"2024-02-10T07:31:51.323331+0000","last_became_peered":"2024-02-10T07:31:51.323331+0000","last_unstale":"2024-02-10T07:32:47.597637+0000","last_undegraded":"2024-02-10T07:32:47.597637+0000","last_fullsized":"2024-02-10T07:32:47.597637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:07.093553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":84,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:00.565446+0000","last_change":"2024-02-10T07:33:00.565478+0000","last_active":"2024-02-10T07:33:00.565446+0000","last_peered":"2024-02-10T07:33:00.565446+0000","last_clean":"2024-02-10T07:33:00.565446+0000","last_became_active":"2024-02-10T07:31:53.286957+0000","last_became_peered":"2024-02-10T07:31:53.286957+0000","last_unstale":"2024-02-10T07:33:00.565446+0000","last_undegraded":"2024-02-10T07:33:00.565446+0000","last_fullsized":"2024-02-10T07:33:00.565446+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:33:00.565384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:00.565384+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:05.401431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.100632+0000","last_change":"2024-02-10T07:31:57.313731+0000","last_active":"2024-02-10T07:32:47.100632+0000","last_peered":"2024-02-10T07:32:47.100632+0000","last_clean":"2024-02-10T07:32:47.100632+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:32:47.100632+0000","last_undegraded":"2024-02-10T07:32:47.100632+0000","last_fullsized":"2024-02-10T07:32:47.100632+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:16.885483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639717+0000","last_change":"2024-02-10T07:31:55.298018+0000","last_active":"2024-02-10T07:32:46.639717+0000","last_peered":"2024-02-10T07:32:46.639717+0000","last_clean":"2024-02-10T07:32:46.639717+0000","last_became_active":"2024-02-10T07:31:55.297715+0000","last_became_peered":"2024-02-10T07:31:55.297715+0000","last_unstale":"2024-02-10T07:32:46.639717+0000","last_undegraded":"2024-02-10T07:32:46.639717+0000","last_fullsized":"2024-02-10T07:32:46.639717+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.150692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:51.545567+0000","last_change":"2024-02-10T07:32:51.545597+0000","last_active":"2024-02-10T07:32:51.545567+0000","last_peered":"2024-02-10T07:32:51.545567+0000","last_clean":"2024-02-10T07:32:51.545567+0000","last_became_active":"2024-02-10T07:31:51.283229+0000","last_became_peered":"2024-02-10T07:31:51.283229+0000","last_unstale":"2024-02-10T07:32:51.545567+0000","last_undegraded":"2024-02-10T07:32:51.545567+0000","last_fullsized":"2024-02-10T07:32:51.545567+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:32:51.545493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:32:51.545493+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:17.023144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642023+0000","last_change":"2024-02-10T07:31:53.288142+0000","last_active":"2024-02-10T07:32:46.642023+0000","last_peered":"2024-02-10T07:32:46.642023+0000","last_clean":"2024-02-10T07:32:46.642023+0000","last_became_active":"2024-02-10T07:31:53.287858+0000","last_became_peered":"2024-02-10T07:31:53.287858+0000","last_unstale":"2024-02-10T07:32:46.642023+0000","last_undegraded":"2024-02-10T07:32:46.642023+0000","last_fullsized":"2024-02-10T07:32:46.642023+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:20.421670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639021+0000","last_change":"2024-02-10T07:31:57.310358+0000","last_active":"2024-02-10T07:32:46.639021+0000","last_peered":"2024-02-10T07:32:46.639021+0000","last_clean":"2024-02-10T07:32:46.639021+0000","last_became_active":"2024-02-10T07:31:57.310088+0000","last_became_peered":"2024-02-10T07:31:57.310088+0000","last_unstale":"2024-02-10T07:32:46.639021+0000","last_undegraded":"2024-02-10T07:32:46.639021+0000","last_fullsized":"2024-02-10T07:32:46.639021+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:23.164310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639188+0000","last_change":"2024-02-10T07:31:55.303127+0000","last_active":"2024-02-10T07:32:46.639188+0000","last_peered":"2024-02-10T07:32:46.639188+0000","last_clean":"2024-02-10T07:32:46.639188+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:32:46.639188+0000","last_undegraded":"2024-02-10T07:32:46.639188+0000","last_fullsized":"2024-02-10T07:32:46.639188+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:08.495702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641300+0000","last_change":"2024-02-10T07:31:51.276682+0000","last_active":"2024-02-10T07:32:46.641300+0000","last_peered":"2024-02-10T07:32:46.641300+0000","last_clean":"2024-02-10T07:32:46.641300+0000","last_became_active":"2024-02-10T07:31:51.276350+0000","last_became_peered":"2024-02-10T07:31:51.276350+0000","last_unstale":"2024-02-10T07:32:46.641300+0000","last_undegraded":"2024-02-10T07:32:46.641300+0000","last_fullsized":"2024-02-10T07:32:46.641300+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:11.602006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641535+0000","last_change":"2024-02-10T07:31:53.292840+0000","last_active":"2024-02-10T07:32:46.641535+0000","last_peered":"2024-02-10T07:32:46.641535+0000","last_clean":"2024-02-10T07:32:46.641535+0000","last_became_active":"2024-02-10T07:31:53.292417+0000","last_became_peered":"2024-02-10T07:31:53.292417+0000","last_unstale":"2024-02-10T07:32:46.641535+0000","last_undegraded":"2024-02-10T07:32:46.641535+0000","last_fullsized":"2024-02-10T07:32:46.641535+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:22.210604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642273+0000","last_change":"2024-02-10T07:31:57.306629+0000","last_active":"2024-02-10T07:32:46.642273+0000","last_peered":"2024-02-10T07:32:46.642273+0000","last_clean":"2024-02-10T07:32:46.642273+0000","last_became_active":"2024-02-10T07:31:57.305613+0000","last_became_peered":"2024-02-10T07:31:57.305613+0000","last_unstale":"2024-02-10T07:32:46.642273+0000","last_undegraded":"2024-02-10T07:32:46.642273+0000","last_fullsized":"2024-02-10T07:32:46.642273+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:00.617793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.695149+0000","last_change":"2024-02-10T07:31:55.299041+0000","last_active":"2024-02-10T07:32:47.695149+0000","last_peered":"2024-02-10T07:32:47.695149+0000","last_clean":"2024-02-10T07:32:47.695149+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:32:47.695149+0000","last_undegraded":"2024-02-10T07:32:47.695149+0000","last_fullsized":"2024-02-10T07:32:47.695149+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:14.438305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101011+0000","last_change":"2024-02-10T07:31:51.287400+0000","last_active":"2024-02-10T07:32:47.101011+0000","last_peered":"2024-02-10T07:32:47.101011+0000","last_clean":"2024-02-10T07:32:47.101011+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:32:47.101011+0000","last_undegraded":"2024-02-10T07:32:47.101011+0000","last_fullsized":"2024-02-10T07:32:47.101011+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:17.607773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":75,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.195006+0000","last_change":"2024-02-10T07:31:53.295830+0000","last_active":"2024-02-10T07:32:47.195006+0000","last_peered":"2024-02-10T07:32:47.195006+0000","last_clean":"2024-02-10T07:32:47.195006+0000","last_became_active":"2024-02-10T07:31:53.295527+0000","last_became_peered":"2024-02-10T07:31:53.295527+0000","last_unstale":"2024-02-10T07:32:47.195006+0000","last_undegraded":"2024-02-10T07:32:47.195006+0000","last_fullsized":"2024-02-10T07:32:47.195006+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.171389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598595+0000","last_change":"2024-02-10T07:31:57.306606+0000","last_active":"2024-02-10T07:32:47.598595+0000","last_peered":"2024-02-10T07:32:47.598595+0000","last_clean":"2024-02-10T07:32:47.598595+0000","last_became_active":"2024-02-10T07:31:57.306315+0000","last_became_peered":"2024-02-10T07:31:57.306315+0000","last_unstale":"2024-02-10T07:32:47.598595+0000","last_undegraded":"2024-02-10T07:32:47.598595+0000","last_fullsized":"2024-02-10T07:32:47.598595+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:22.626690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641457+0000","last_change":"2024-02-10T07:31:55.298789+0000","last_active":"2024-02-10T07:32:46.641457+0000","last_peered":"2024-02-10T07:32:46.641457+0000","last_clean":"2024-02-10T07:32:46.641457+0000","last_became_active":"2024-02-10T07:31:55.298474+0000","last_became_peered":"2024-02-10T07:31:55.298474+0000","last_unstale":"2024-02-10T07:32:46.641457+0000","last_undegraded":"2024-02-10T07:32:46.641457+0000","last_fullsized":"2024-02-10T07:32:46.641457+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.404110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:02.198071+0000","last_change":"2024-02-10T07:33:02.198096+0000","last_active":"2024-02-10T07:33:02.198071+0000","last_peered":"2024-02-10T07:33:02.198071+0000","last_clean":"2024-02-10T07:33:02.198071+0000","last_became_active":"2024-02-10T07:31:51.321237+0000","last_became_peered":"2024-02-10T07:31:51.321237+0000","last_unstale":"2024-02-10T07:33:02.198071+0000","last_undegraded":"2024-02-10T07:33:02.198071+0000","last_fullsized":"2024-02-10T07:33:02.198071+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:02.197996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:02.197996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:08.678151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598463+0000","last_change":"2024-02-10T07:31:53.295233+0000","last_active":"2024-02-10T07:32:47.598463+0000","last_peered":"2024-02-10T07:32:47.598463+0000","last_clean":"2024-02-10T07:32:47.598463+0000","last_became_active":"2024-02-10T07:31:53.294914+0000","last_became_peered":"2024-02-10T07:31:53.294914+0000","last_unstale":"2024-02-10T07:32:47.598463+0000","last_undegraded":"2024-02-10T07:32:47.598463+0000","last_fullsized":"2024-02-10T07:32:47.598463+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:00.458485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640318+0000","last_change":"2024-02-10T07:31:57.307172+0000","last_active":"2024-02-10T07:32:46.640318+0000","last_peered":"2024-02-10T07:32:46.640318+0000","last_clean":"2024-02-10T07:32:46.640318+0000","last_became_active":"2024-02-10T07:31:57.306892+0000","last_became_peered":"2024-02-10T07:31:57.306892+0000","last_unstale":"2024-02-10T07:32:46.640318+0000","last_undegraded":"2024-02-10T07:32:46.640318+0000","last_fullsized":"2024-02-10T07:32:46.640318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:15.856701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.372896+0000","last_change":"2024-02-10T07:31:55.295914+0000","last_active":"2024-02-10T07:33:06.372896+0000","last_peered":"2024-02-10T07:33:06.372896+0000","last_clean":"2024-02-10T07:33:06.372896+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:33:06.372896+0000","last_undegraded":"2024-02-10T07:33:06.372896+0000","last_fullsized":"2024-02-10T07:33:06.372896+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:17.555191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694224+0000","last_change":"2024-02-10T07:31:51.282350+0000","last_active":"2024-02-10T07:32:47.694224+0000","last_peered":"2024-02-10T07:32:47.694224+0000","last_clean":"2024-02-10T07:32:47.694224+0000","last_became_active":"2024-02-10T07:31:51.282077+0000","last_became_peered":"2024-02-10T07:31:51.282077+0000","last_unstale":"2024-02-10T07:32:47.694224+0000","last_undegraded":"2024-02-10T07:32:47.694224+0000","last_fullsized":"2024-02-10T07:32:47.694224+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:13.165010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:03.462205+0000","last_change":"2024-02-10T07:33:03.462253+0000","last_active":"2024-02-10T07:33:03.462205+0000","last_peered":"2024-02-10T07:33:03.462205+0000","last_clean":"2024-02-10T07:33:03.462205+0000","last_became_active":"2024-02-10T07:31:53.292936+0000","last_became_peered":"2024-02-10T07:31:53.292936+0000","last_unstale":"2024-02-10T07:33:03.462205+0000","last_undegraded":"2024-02-10T07:33:03.462205+0000","last_fullsized":"2024-02-10T07:33:03.462205+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:33:03.462109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:03.462109+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:31.967818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.695230+0000","last_change":"2024-02-10T07:31:57.309137+0000","last_active":"2024-02-10T07:32:47.695230+0000","last_peered":"2024-02-10T07:32:47.695230+0000","last_clean":"2024-02-10T07:32:47.695230+0000","last_became_active":"2024-02-10T07:31:57.308820+0000","last_became_peered":"2024-02-10T07:31:57.308820+0000","last_unstale":"2024-02-10T07:32:47.695230+0000","last_undegraded":"2024-02-10T07:32:47.695230+0000","last_fullsized":"2024-02-10T07:32:47.695230+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:13.939044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101652+0000","last_change":"2024-02-10T07:31:55.297952+0000","last_active":"2024-02-10T07:32:47.101652+0000","last_peered":"2024-02-10T07:32:47.101652+0000","last_clean":"2024-02-10T07:32:47.101652+0000","last_became_active":"2024-02-10T07:31:55.297595+0000","last_became_peered":"2024-02-10T07:31:55.297595+0000","last_unstale":"2024-02-10T07:32:47.101652+0000","last_undegraded":"2024-02-10T07:32:47.101652+0000","last_fullsized":"2024-02-10T07:32:47.101652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.915423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597500+0000","last_change":"2024-02-10T07:31:51.285841+0000","last_active":"2024-02-10T07:32:47.597500+0000","last_peered":"2024-02-10T07:32:47.597500+0000","last_clean":"2024-02-10T07:32:47.597500+0000","last_became_active":"2024-02-10T07:31:51.285405+0000","last_became_peered":"2024-02-10T07:31:51.285405+0000","last_unstale":"2024-02-10T07:32:47.597500+0000","last_undegraded":"2024-02-10T07:32:47.597500+0000","last_fullsized":"2024-02-10T07:32:47.597500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:00.794907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":77,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:00.861580+0000","last_change":"2024-02-10T07:33:00.861634+0000","last_active":"2024-02-10T07:33:00.861580+0000","last_peered":"2024-02-10T07:33:00.861580+0000","last_clean":"2024-02-10T07:33:00.861580+0000","last_became_active":"2024-02-10T07:31:53.300871+0000","last_became_peered":"2024-02-10T07:31:53.300871+0000","last_unstale":"2024-02-10T07:33:00.861580+0000","last_undegraded":"2024-02-10T07:33:00.861580+0000","last_fullsized":"2024-02-10T07:33:00.861580+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:33:00.861482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:00.861482+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:09.147490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639149+0000","last_change":"2024-02-10T07:31:57.307069+0000","last_active":"2024-02-10T07:32:46.639149+0000","last_peered":"2024-02-10T07:32:46.639149+0000","last_clean":"2024-02-10T07:32:46.639149+0000","last_became_active":"2024-02-10T07:31:57.306776+0000","last_became_peered":"2024-02-10T07:31:57.306776+0000","last_unstale":"2024-02-10T07:32:46.639149+0000","last_undegraded":"2024-02-10T07:32:46.639149+0000","last_fullsized":"2024-02-10T07:32:46.639149+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:09.159262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.695257+0000","last_change":"2024-02-10T07:31:55.291538+0000","last_active":"2024-02-10T07:32:47.695257+0000","last_peered":"2024-02-10T07:32:47.695257+0000","last_clean":"2024-02-10T07:32:47.695257+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:32:47.695257+0000","last_undegraded":"2024-02-10T07:32:47.695257+0000","last_fullsized":"2024-02-10T07:32:47.695257+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:04.691663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:11.581128+0000","last_change":"2024-02-10T07:33:11.581151+0000","last_active":"2024-02-10T07:33:11.581128+0000","last_peered":"2024-02-10T07:33:11.581128+0000","last_clean":"2024-02-10T07:33:11.581128+0000","last_became_active":"2024-02-10T07:31:51.284022+0000","last_became_peered":"2024-02-10T07:31:51.284022+0000","last_unstale":"2024-02-10T07:33:11.581128+0000","last_undegraded":"2024-02-10T07:33:11.581128+0000","last_fullsized":"2024-02-10T07:33:11.581128+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:11.581063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:11.581063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:32.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597927+0000","last_change":"2024-02-10T07:31:53.295829+0000","last_active":"2024-02-10T07:32:47.597927+0000","last_peered":"2024-02-10T07:32:47.597927+0000","last_clean":"2024-02-10T07:32:47.597927+0000","last_became_active":"2024-02-10T07:31:53.295520+0000","last_became_peered":"2024-02-10T07:31:53.295520+0000","last_unstale":"2024-02-10T07:32:47.597927+0000","last_undegraded":"2024-02-10T07:32:47.597927+0000","last_fullsized":"2024-02-10T07:32:47.597927+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.493903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598100+0000","last_change":"2024-02-10T07:31:57.315994+0000","last_active":"2024-02-10T07:32:47.598100+0000","last_peered":"2024-02-10T07:32:47.598100+0000","last_clean":"2024-02-10T07:32:47.598100+0000","last_became_active":"2024-02-10T07:31:57.315738+0000","last_became_peered":"2024-02-10T07:31:57.315738+0000","last_unstale":"2024-02-10T07:32:47.598100+0000","last_undegraded":"2024-02-10T07:32:47.598100+0000","last_fullsized":"2024-02-10T07:32:47.598100+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:07.663297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642118+0000","last_change":"2024-02-10T07:31:55.302310+0000","last_active":"2024-02-10T07:32:46.642118+0000","last_peered":"2024-02-10T07:32:46.642118+0000","last_clean":"2024-02-10T07:32:46.642118+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:32:46.642118+0000","last_undegraded":"2024-02-10T07:32:46.642118+0000","last_fullsized":"2024-02-10T07:32:46.642118+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:05.175419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:53.470063+0000","last_change":"2024-02-10T07:32:53.470087+0000","last_active":"2024-02-10T07:32:53.470063+0000","last_peered":"2024-02-10T07:32:53.470063+0000","last_clean":"2024-02-10T07:32:53.470063+0000","last_became_active":"2024-02-10T07:31:51.279438+0000","last_became_peered":"2024-02-10T07:31:51.279438+0000","last_unstale":"2024-02-10T07:32:53.470063+0000","last_undegraded":"2024-02-10T07:32:53.470063+0000","last_fullsized":"2024-02-10T07:32:53.470063+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:32:53.469958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:00.926321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:58.537802+0000","last_change":"2024-02-10T07:32:58.537831+0000","last_active":"2024-02-10T07:32:58.537802+0000","last_peered":"2024-02-10T07:32:58.537802+0000","last_clean":"2024-02-10T07:32:58.537802+0000","last_became_active":"2024-02-10T07:31:53.287823+0000","last_became_peered":"2024-02-10T07:31:53.287823+0000","last_unstale":"2024-02-10T07:32:58.537802+0000","last_undegraded":"2024-02-10T07:32:58.537802+0000","last_fullsized":"2024-02-10T07:32:58.537802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:32:58.537746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.537746+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:07.841413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.100440+0000","last_change":"2024-02-10T07:31:57.313822+0000","last_active":"2024-02-10T07:32:47.100440+0000","last_peered":"2024-02-10T07:32:47.100440+0000","last_clean":"2024-02-10T07:32:47.100440+0000","last_became_active":"2024-02-10T07:31:57.313529+0000","last_became_peered":"2024-02-10T07:31:57.313529+0000","last_unstale":"2024-02-10T07:32:47.100440+0000","last_undegraded":"2024-02-10T07:32:47.100440+0000","last_fullsized":"2024-02-10T07:32:47.100440+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:23.816714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:59.584337+0000","last_change":"2024-02-10T07:32:59.584373+0000","last_active":"2024-02-10T07:32:59.584337+0000","last_peered":"2024-02-10T07:32:59.584337+0000","last_clean":"2024-02-10T07:32:59.584337+0000","last_became_active":"2024-02-10T07:31:55.296670+0000","last_became_peered":"2024-02-10T07:31:55.296670+0000","last_unstale":"2024-02-10T07:32:59.584337+0000","last_undegraded":"2024-02-10T07:32:59.584337+0000","last_fullsized":"2024-02-10T07:32:59.584337+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:59.584219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:59.584219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:22.310522+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640519+0000","last_change":"2024-02-10T07:31:51.288447+0000","last_active":"2024-02-10T07:32:46.640519+0000","last_peered":"2024-02-10T07:32:46.640519+0000","last_clean":"2024-02-10T07:32:46.640519+0000","last_became_active":"2024-02-10T07:31:51.288165+0000","last_became_peered":"2024-02-10T07:31:51.288165+0000","last_unstale":"2024-02-10T07:32:46.640519+0000","last_undegraded":"2024-02-10T07:32:46.640519+0000","last_fullsized":"2024-02-10T07:32:46.640519+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:32:59.646262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694985+0000","last_change":"2024-02-10T07:31:53.292496+0000","last_active":"2024-02-10T07:32:47.694985+0000","last_peered":"2024-02-10T07:32:47.694985+0000","last_clean":"2024-02-10T07:32:47.694985+0000","last_became_active":"2024-02-10T07:31:53.292182+0000","last_became_peered":"2024-02-10T07:31:53.292182+0000","last_unstale":"2024-02-10T07:32:47.694985+0000","last_undegraded":"2024-02-10T07:32:47.694985+0000","last_fullsized":"2024-02-10T07:32:47.694985+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:14.454667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597842+0000","last_change":"2024-02-10T07:31:57.316162+0000","last_active":"2024-02-10T07:32:47.597842+0000","last_peered":"2024-02-10T07:32:47.597842+0000","last_clean":"2024-02-10T07:32:47.597842+0000","last_became_active":"2024-02-10T07:31:57.315781+0000","last_became_peered":"2024-02-10T07:31:57.315781+0000","last_unstale":"2024-02-10T07:32:47.597842+0000","last_undegraded":"2024-02-10T07:32:47.597842+0000","last_fullsized":"2024-02-10T07:32:47.597842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:08.947342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694902+0000","last_change":"2024-02-10T07:31:55.301556+0000","last_active":"2024-02-10T07:32:47.694902+0000","last_peered":"2024-02-10T07:32:47.694902+0000","last_clean":"2024-02-10T07:32:47.694902+0000","last_became_active":"2024-02-10T07:31:55.301260+0000","last_became_peered":"2024-02-10T07:31:55.301260+0000","last_unstale":"2024-02-10T07:32:47.694902+0000","last_undegraded":"2024-02-10T07:32:47.694902+0000","last_fullsized":"2024-02-10T07:32:47.694902+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:17.634914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641706+0000","last_change":"2024-02-10T07:31:51.286248+0000","last_active":"2024-02-10T07:32:46.641706+0000","last_peered":"2024-02-10T07:32:46.641706+0000","last_clean":"2024-02-10T07:32:46.641706+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:32:46.641706+0000","last_undegraded":"2024-02-10T07:32:46.641706+0000","last_fullsized":"2024-02-10T07:32:46.641706+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:06.511452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":96,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:00.143959+0000","last_change":"2024-02-10T07:33:00.144015+0000","last_active":"2024-02-10T07:33:00.143959+0000","last_peered":"2024-02-10T07:33:00.143959+0000","last_clean":"2024-02-10T07:33:00.143959+0000","last_became_active":"2024-02-10T07:32:18.389564+0000","last_became_peered":"2024-02-10T07:32:18.389564+0000","last_unstale":"2024-02-10T07:33:00.143959+0000","last_undegraded":"2024-02-10T07:33:00.143959+0000","last_fullsized":"2024-02-10T07:33:00.143959+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:33:00.143859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:00.143859+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:28.135685+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101249+0000","last_change":"2024-02-10T07:31:57.318330+0000","last_active":"2024-02-10T07:32:47.101249+0000","last_peered":"2024-02-10T07:32:47.101249+0000","last_clean":"2024-02-10T07:32:47.101249+0000","last_became_active":"2024-02-10T07:31:57.318009+0000","last_became_peered":"2024-02-10T07:31:57.318009+0000","last_unstale":"2024-02-10T07:32:47.101249+0000","last_undegraded":"2024-02-10T07:32:47.101249+0000","last_fullsized":"2024-02-10T07:32:47.101249+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:09.866798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.372897+0000","last_change":"2024-02-10T07:31:55.298387+0000","last_active":"2024-02-10T07:33:06.372897+0000","last_peered":"2024-02-10T07:33:06.372897+0000","last_clean":"2024-02-10T07:33:06.372897+0000","last_became_active":"2024-02-10T07:31:55.297670+0000","last_became_peered":"2024-02-10T07:31:55.297670+0000","last_unstale":"2024-02-10T07:33:06.372897+0000","last_undegraded":"2024-02-10T07:33:06.372897+0000","last_fullsized":"2024-02-10T07:33:06.372897+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.348418+0000","last_change":"2024-02-10T07:33:06.348460+0000","last_active":"2024-02-10T07:33:06.348418+0000","last_peered":"2024-02-10T07:33:06.348418+0000","last_clean":"2024-02-10T07:33:06.348418+0000","last_became_active":"2024-02-10T07:31:51.320644+0000","last_became_peered":"2024-02-10T07:31:51.320644+0000","last_unstale":"2024-02-10T07:33:06.348418+0000","last_undegraded":"2024-02-10T07:33:06.348418+0000","last_fullsized":"2024-02-10T07:33:06.348418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.348322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:15.227477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598312+0000","last_change":"2024-02-10T07:31:53.296048+0000","last_active":"2024-02-10T07:32:47.598312+0000","last_peered":"2024-02-10T07:32:47.598312+0000","last_clean":"2024-02-10T07:32:47.598312+0000","last_became_active":"2024-02-10T07:31:53.295641+0000","last_became_peered":"2024-02-10T07:31:53.295641+0000","last_unstale":"2024-02-10T07:32:47.598312+0000","last_undegraded":"2024-02-10T07:32:47.598312+0000","last_fullsized":"2024-02-10T07:32:47.598312+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:17.035054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642477+0000","last_change":"2024-02-10T07:31:57.307866+0000","last_active":"2024-02-10T07:32:46.642477+0000","last_peered":"2024-02-10T07:32:46.642477+0000","last_clean":"2024-02-10T07:32:46.642477+0000","last_became_active":"2024-02-10T07:31:57.307583+0000","last_became_peered":"2024-02-10T07:31:57.307583+0000","last_unstale":"2024-02-10T07:32:46.642477+0000","last_undegraded":"2024-02-10T07:32:46.642477+0000","last_fullsized":"2024-02-10T07:32:46.642477+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:25.642236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.372563+0000","last_change":"2024-02-10T07:31:55.294941+0000","last_active":"2024-02-10T07:33:06.372563+0000","last_peered":"2024-02-10T07:33:06.372563+0000","last_clean":"2024-02-10T07:33:06.372563+0000","last_became_active":"2024-02-10T07:31:55.294594+0000","last_became_peered":"2024-02-10T07:31:55.294594+0000","last_unstale":"2024-02-10T07:33:06.372563+0000","last_undegraded":"2024-02-10T07:33:06.372563+0000","last_fullsized":"2024-02-10T07:33:06.372563+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.496713+0000","last_change":"2024-02-10T07:33:01.496748+0000","last_active":"2024-02-10T07:33:01.496713+0000","last_peered":"2024-02-10T07:33:01.496713+0000","last_clean":"2024-02-10T07:33:01.496713+0000","last_became_active":"2024-02-10T07:31:51.282353+0000","last_became_peered":"2024-02-10T07:31:51.282353+0000","last_unstale":"2024-02-10T07:33:01.496713+0000","last_undegraded":"2024-02-10T07:33:01.496713+0000","last_fullsized":"2024-02-10T07:33:01.496713+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:01.496610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:01.496610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:02.323527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:09.243572+0000","last_change":"2024-02-10T07:33:09.243608+0000","last_active":"2024-02-10T07:33:09.243572+0000","last_peered":"2024-02-10T07:33:09.243572+0000","last_clean":"2024-02-10T07:33:09.243572+0000","last_became_active":"2024-02-10T07:31:53.285777+0000","last_became_peered":"2024-02-10T07:31:53.285777+0000","last_unstale":"2024-02-10T07:33:09.243572+0000","last_undegraded":"2024-02-10T07:33:09.243572+0000","last_fullsized":"2024-02-10T07:33:09.243572+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:33:09.243500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:09.243500+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:37.334914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:10.120521+0000","last_change":"2024-02-10T07:33:10.120541+0000","last_active":"2024-02-10T07:33:10.120521+0000","last_peered":"2024-02-10T07:33:10.120521+0000","last_clean":"2024-02-10T07:33:10.120521+0000","last_became_active":"2024-02-10T07:31:57.308615+0000","last_became_peered":"2024-02-10T07:31:57.308615+0000","last_unstale":"2024-02-10T07:33:10.120521+0000","last_undegraded":"2024-02-10T07:33:10.120521+0000","last_fullsized":"2024-02-10T07:33:10.120521+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:33:10.120462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:13.165236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598712+0000","last_change":"2024-02-10T07:31:55.297740+0000","last_active":"2024-02-10T07:32:47.598712+0000","last_peered":"2024-02-10T07:32:47.598712+0000","last_clean":"2024-02-10T07:32:47.598712+0000","last_became_active":"2024-02-10T07:31:55.297431+0000","last_became_peered":"2024-02-10T07:31:55.297431+0000","last_unstale":"2024-02-10T07:32:47.598712+0000","last_undegraded":"2024-02-10T07:32:47.598712+0000","last_fullsized":"2024-02-10T07:32:47.598712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:02.665928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639342+0000","last_change":"2024-02-10T07:31:51.284304+0000","last_active":"2024-02-10T07:32:46.639342+0000","last_peered":"2024-02-10T07:32:46.639342+0000","last_clean":"2024-02-10T07:32:46.639342+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:32:46.639342+0000","last_undegraded":"2024-02-10T07:32:46.639342+0000","last_fullsized":"2024-02-10T07:32:46.639342+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:02.877266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"89'13","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:56.585009+0000","last_change":"2024-02-10T07:32:56.585037+0000","last_active":"2024-02-10T07:32:56.585009+0000","last_peered":"2024-02-10T07:32:56.585009+0000","last_clean":"2024-02-10T07:32:56.585009+0000","last_became_active":"2024-02-10T07:31:53.295353+0000","last_became_peered":"2024-02-10T07:31:53.295353+0000","last_unstale":"2024-02-10T07:32:56.585009+0000","last_undegraded":"2024-02-10T07:32:56.585009+0000","last_fullsized":"2024-02-10T07:32:56.585009+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:32:56.584957+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:04.077539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640268+0000","last_change":"2024-02-10T07:31:57.305113+0000","last_active":"2024-02-10T07:32:46.640268+0000","last_peered":"2024-02-10T07:32:46.640268+0000","last_clean":"2024-02-10T07:32:46.640268+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:32:46.640268+0000","last_undegraded":"2024-02-10T07:32:46.640268+0000","last_fullsized":"2024-02-10T07:32:46.640268+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:11.938578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641217+0000","last_change":"2024-02-10T07:31:55.296699+0000","last_active":"2024-02-10T07:32:46.641217+0000","last_peered":"2024-02-10T07:32:46.641217+0000","last_clean":"2024-02-10T07:32:46.641217+0000","last_became_active":"2024-02-10T07:31:55.296369+0000","last_became_peered":"2024-02-10T07:31:55.296369+0000","last_unstale":"2024-02-10T07:32:46.641217+0000","last_undegraded":"2024-02-10T07:32:46.641217+0000","last_fullsized":"2024-02-10T07:32:46.641217+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.519413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:09.871264+0000","last_change":"2024-02-10T07:33:09.871286+0000","last_active":"2024-02-10T07:33:09.871264+0000","last_peered":"2024-02-10T07:33:09.871264+0000","last_clean":"2024-02-10T07:33:09.871264+0000","last_became_active":"2024-02-10T07:31:51.285159+0000","last_became_peered":"2024-02-10T07:31:51.285159+0000","last_unstale":"2024-02-10T07:33:09.871264+0000","last_undegraded":"2024-02-10T07:33:09.871264+0000","last_fullsized":"2024-02-10T07:33:09.871264+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:09.871194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:09.871194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:38.560272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598658+0000","last_change":"2024-02-10T07:31:53.297115+0000","last_active":"2024-02-10T07:32:47.598658+0000","last_peered":"2024-02-10T07:32:47.598658+0000","last_clean":"2024-02-10T07:32:47.598658+0000","last_became_active":"2024-02-10T07:31:53.296831+0000","last_became_peered":"2024-02-10T07:31:53.296831+0000","last_unstale":"2024-02-10T07:32:47.598658+0000","last_undegraded":"2024-02-10T07:32:47.598658+0000","last_fullsized":"2024-02-10T07:32:47.598658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:21.861465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639876+0000","last_change":"2024-02-10T07:31:57.311522+0000","last_active":"2024-02-10T07:32:46.639876+0000","last_peered":"2024-02-10T07:32:46.639876+0000","last_clean":"2024-02-10T07:32:46.639876+0000","last_became_active":"2024-02-10T07:31:57.311110+0000","last_became_peered":"2024-02-10T07:31:57.311110+0000","last_unstale":"2024-02-10T07:32:46.639876+0000","last_undegraded":"2024-02-10T07:32:46.639876+0000","last_fullsized":"2024-02-10T07:32:46.639876+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:25.982197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640304+0000","last_change":"2024-02-10T07:31:51.286209+0000","last_active":"2024-02-10T07:32:46.640304+0000","last_peered":"2024-02-10T07:32:46.640304+0000","last_clean":"2024-02-10T07:32:46.640304+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:32:46.640304+0000","last_undegraded":"2024-02-10T07:32:46.640304+0000","last_fullsized":"2024-02-10T07:32:46.640304+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:08.757933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:59.385507+0000","last_change":"2024-02-10T07:32:59.385539+0000","last_active":"2024-02-10T07:32:59.385507+0000","last_peered":"2024-02-10T07:32:59.385507+0000","last_clean":"2024-02-10T07:32:59.385507+0000","last_became_active":"2024-02-10T07:31:53.284817+0000","last_became_peered":"2024-02-10T07:31:53.284817+0000","last_unstale":"2024-02-10T07:32:59.385507+0000","last_undegraded":"2024-02-10T07:32:59.385507+0000","last_fullsized":"2024-02-10T07:32:59.385507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:32:59.385448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:59.385448+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:59.831143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.372889+0000","last_change":"2024-02-10T07:31:55.301606+0000","last_active":"2024-02-10T07:33:06.372889+0000","last_peered":"2024-02-10T07:33:06.372889+0000","last_clean":"2024-02-10T07:33:06.372889+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:33:06.372889+0000","last_undegraded":"2024-02-10T07:33:06.372889+0000","last_fullsized":"2024-02-10T07:33:06.372889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598067+0000","last_change":"2024-02-10T07:31:57.315186+0000","last_active":"2024-02-10T07:32:47.598067+0000","last_peered":"2024-02-10T07:32:47.598067+0000","last_clean":"2024-02-10T07:32:47.598067+0000","last_became_active":"2024-02-10T07:31:57.314872+0000","last_became_peered":"2024-02-10T07:31:57.314872+0000","last_unstale":"2024-02-10T07:32:47.598067+0000","last_undegraded":"2024-02-10T07:32:47.598067+0000","last_fullsized":"2024-02-10T07:32:47.598067+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:10.772524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640466+0000","last_change":"2024-02-10T07:31:51.287017+0000","last_active":"2024-02-10T07:32:46.640466+0000","last_peered":"2024-02-10T07:32:46.640466+0000","last_clean":"2024-02-10T07:32:46.640466+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:32:46.640466+0000","last_undegraded":"2024-02-10T07:32:46.640466+0000","last_fullsized":"2024-02-10T07:32:46.640466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:05.325144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":92,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640700+0000","last_change":"2024-02-10T07:32:18.396325+0000","last_active":"2024-02-10T07:32:46.640700+0000","last_peered":"2024-02-10T07:32:46.640700+0000","last_clean":"2024-02-10T07:32:46.640700+0000","last_became_active":"2024-02-10T07:32:18.389409+0000","last_became_peered":"2024-02-10T07:32:18.389409+0000","last_unstale":"2024-02-10T07:32:46.640700+0000","last_undegraded":"2024-02-10T07:32:46.640700+0000","last_fullsized":"2024-02-10T07:32:46.640700+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:07.708054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:59.218979+0000","last_change":"2024-02-10T07:32:59.218999+0000","last_active":"2024-02-10T07:32:59.218979+0000","last_peered":"2024-02-10T07:32:59.218979+0000","last_clean":"2024-02-10T07:32:59.218979+0000","last_became_active":"2024-02-10T07:31:55.303059+0000","last_became_peered":"2024-02-10T07:31:55.303059+0000","last_unstale":"2024-02-10T07:32:59.218979+0000","last_undegraded":"2024-02-10T07:32:59.218979+0000","last_fullsized":"2024-02-10T07:32:59.218979+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:59.218908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:59.218908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:06.437654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639057+0000","last_change":"2024-02-10T07:31:57.305254+0000","last_active":"2024-02-10T07:32:46.639057+0000","last_peered":"2024-02-10T07:32:46.639057+0000","last_clean":"2024-02-10T07:32:46.639057+0000","last_became_active":"2024-02-10T07:31:57.304971+0000","last_became_peered":"2024-02-10T07:31:57.304971+0000","last_unstale":"2024-02-10T07:32:46.639057+0000","last_undegraded":"2024-02-10T07:32:46.639057+0000","last_fullsized":"2024-02-10T07:32:46.639057+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:01.467119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.194617+0000","last_change":"2024-02-10T07:31:55.298725+0000","last_active":"2024-02-10T07:32:47.194617+0000","last_peered":"2024-02-10T07:32:47.194617+0000","last_clean":"2024-02-10T07:32:47.194617+0000","last_became_active":"2024-02-10T07:31:55.298322+0000","last_became_peered":"2024-02-10T07:31:55.298322+0000","last_unstale":"2024-02-10T07:32:47.194617+0000","last_undegraded":"2024-02-10T07:32:47.194617+0000","last_fullsized":"2024-02-10T07:32:47.194617+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.947435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641102+0000","last_change":"2024-02-10T07:31:51.283777+0000","last_active":"2024-02-10T07:32:46.641102+0000","last_peered":"2024-02-10T07:32:46.641102+0000","last_clean":"2024-02-10T07:32:46.641102+0000","last_became_active":"2024-02-10T07:31:51.283433+0000","last_became_peered":"2024-02-10T07:31:51.283433+0000","last_unstale":"2024-02-10T07:32:46.641102+0000","last_undegraded":"2024-02-10T07:32:46.641102+0000","last_fullsized":"2024-02-10T07:32:46.641102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:02.188930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.695470+0000","last_change":"2024-02-10T07:31:53.301616+0000","last_active":"2024-02-10T07:32:47.695470+0000","last_peered":"2024-02-10T07:32:47.695470+0000","last_clean":"2024-02-10T07:32:47.695470+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:32:47.695470+0000","last_undegraded":"2024-02-10T07:32:47.695470+0000","last_fullsized":"2024-02-10T07:32:47.695470+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:10.820328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.100519+0000","last_change":"2024-02-10T07:31:57.314885+0000","last_active":"2024-02-10T07:32:47.100519+0000","last_peered":"2024-02-10T07:32:47.100519+0000","last_clean":"2024-02-10T07:32:47.100519+0000","last_became_active":"2024-02-10T07:31:57.313394+0000","last_became_peered":"2024-02-10T07:31:57.313394+0000","last_unstale":"2024-02-10T07:32:47.100519+0000","last_undegraded":"2024-02-10T07:32:47.100519+0000","last_fullsized":"2024-02-10T07:32:47.100519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:04.536324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.194336+0000","last_change":"2024-02-10T07:31:55.298984+0000","last_active":"2024-02-10T07:32:47.194336+0000","last_peered":"2024-02-10T07:32:47.194336+0000","last_clean":"2024-02-10T07:32:47.194336+0000","last_became_active":"2024-02-10T07:31:55.298622+0000","last_became_peered":"2024-02-10T07:31:55.298622+0000","last_unstale":"2024-02-10T07:32:47.194336+0000","last_undegraded":"2024-02-10T07:32:47.194336+0000","last_fullsized":"2024-02-10T07:32:47.194336+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:11.544202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639978+0000","last_change":"2024-02-10T07:31:51.283676+0000","last_active":"2024-02-10T07:32:46.639978+0000","last_peered":"2024-02-10T07:32:46.639978+0000","last_clean":"2024-02-10T07:32:46.639978+0000","last_became_active":"2024-02-10T07:31:51.283208+0000","last_became_peered":"2024-02-10T07:31:51.283208+0000","last_unstale":"2024-02-10T07:32:46.639978+0000","last_undegraded":"2024-02-10T07:32:46.639978+0000","last_fullsized":"2024-02-10T07:32:46.639978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:11.381433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101117+0000","last_change":"2024-02-10T07:31:53.292486+0000","last_active":"2024-02-10T07:32:47.101117+0000","last_peered":"2024-02-10T07:32:47.101117+0000","last_clean":"2024-02-10T07:32:47.101117+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:32:47.101117+0000","last_undegraded":"2024-02-10T07:32:47.101117+0000","last_fullsized":"2024-02-10T07:32:47.101117+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:07.434520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598514+0000","last_change":"2024-02-10T07:31:57.314251+0000","last_active":"2024-02-10T07:32:47.598514+0000","last_peered":"2024-02-10T07:32:47.598514+0000","last_clean":"2024-02-10T07:32:47.598514+0000","last_became_active":"2024-02-10T07:31:57.313854+0000","last_became_peered":"2024-02-10T07:31:57.313854+0000","last_unstale":"2024-02-10T07:32:47.598514+0000","last_undegraded":"2024-02-10T07:32:47.598514+0000","last_fullsized":"2024-02-10T07:32:47.598514+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.959268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:12.268090+0000","last_change":"2024-02-10T07:33:12.268121+0000","last_active":"2024-02-10T07:33:12.268090+0000","last_peered":"2024-02-10T07:33:12.268090+0000","last_clean":"2024-02-10T07:33:12.268090+0000","last_became_active":"2024-02-10T07:31:50.278548+0000","last_became_peered":"2024-02-10T07:31:50.278548+0000","last_unstale":"2024-02-10T07:33:12.268090+0000","last_undegraded":"2024-02-10T07:33:12.268090+0000","last_fullsized":"2024-02-10T07:33:12.268090+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:33:12.268020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:33:12.268020+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:13.552959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001361225,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641171+0000","last_change":"2024-02-10T07:31:51.285951+0000","last_active":"2024-02-10T07:32:46.641171+0000","last_peered":"2024-02-10T07:32:46.641171+0000","last_clean":"2024-02-10T07:32:46.641171+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:32:46.641171+0000","last_undegraded":"2024-02-10T07:32:46.641171+0000","last_fullsized":"2024-02-10T07:32:46.641171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:13.565080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":122,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:09.271524+0000","last_change":"2024-02-10T07:33:09.271524+0000","last_active":"2024-02-10T07:33:09.271524+0000","last_peered":"2024-02-10T07:33:09.271524+0000","last_clean":"2024-02-10T07:33:09.271524+0000","last_became_active":"2024-02-10T07:31:53.301863+0000","last_became_peered":"2024-02-10T07:31:53.301863+0000","last_unstale":"2024-02-10T07:33:09.271524+0000","last_undegraded":"2024-02-10T07:33:09.271524+0000","last_fullsized":"2024-02-10T07:33:09.271524+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-02-10T07:33:09.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:09.271402+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:33.250857+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101146+0000","last_change":"2024-02-10T07:31:55.296643+0000","last_active":"2024-02-10T07:32:47.101146+0000","last_peered":"2024-02-10T07:32:47.101146+0000","last_clean":"2024-02-10T07:32:47.101146+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:32:47.101146+0000","last_undegraded":"2024-02-10T07:32:47.101146+0000","last_fullsized":"2024-02-10T07:32:47.101146+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:05.656831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:58.354107+0000","last_change":"2024-02-10T07:32:58.354147+0000","last_active":"2024-02-10T07:32:58.354107+0000","last_peered":"2024-02-10T07:32:58.354107+0000","last_clean":"2024-02-10T07:32:58.354107+0000","last_became_active":"2024-02-10T07:31:57.299958+0000","last_became_peered":"2024-02-10T07:31:57.299958+0000","last_unstale":"2024-02-10T07:32:58.354107+0000","last_undegraded":"2024-02-10T07:32:58.354107+0000","last_fullsized":"2024-02-10T07:32:58.354107+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:58.354022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.354022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:17.659362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:03.198766+0000","last_change":"2024-02-10T07:33:03.198787+0000","last_active":"2024-02-10T07:33:03.198766+0000","last_peered":"2024-02-10T07:33:03.198766+0000","last_clean":"2024-02-10T07:33:03.198766+0000","last_became_active":"2024-02-10T07:31:51.320864+0000","last_became_peered":"2024-02-10T07:31:51.320864+0000","last_unstale":"2024-02-10T07:33:03.198766+0000","last_undegraded":"2024-02-10T07:33:03.198766+0000","last_fullsized":"2024-02-10T07:33:03.198766+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:03.198699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:03.198699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:13.704299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:58.209366+0000","last_change":"2024-02-10T07:32:58.209386+0000","last_active":"2024-02-10T07:32:58.209366+0000","last_peered":"2024-02-10T07:32:58.209366+0000","last_clean":"2024-02-10T07:32:58.209366+0000","last_became_active":"2024-02-10T07:31:50.281517+0000","last_became_peered":"2024-02-10T07:31:50.281517+0000","last_unstale":"2024-02-10T07:32:58.209366+0000","last_undegraded":"2024-02-10T07:32:58.209366+0000","last_fullsized":"2024-02-10T07:32:58.209366+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:58.209303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.209303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:28.009863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.194782+0000","last_change":"2024-02-10T07:32:18.398935+0000","last_active":"2024-02-10T07:32:47.194782+0000","last_peered":"2024-02-10T07:32:47.194782+0000","last_clean":"2024-02-10T07:32:47.194782+0000","last_became_active":"2024-02-10T07:32:18.391217+0000","last_became_peered":"2024-02-10T07:32:18.391217+0000","last_unstale":"2024-02-10T07:32:47.194782+0000","last_undegraded":"2024-02-10T07:32:47.194782+0000","last_fullsized":"2024-02-10T07:32:47.194782+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:16.649956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.183383+0000","last_change":"2024-02-10T07:33:06.183420+0000","last_active":"2024-02-10T07:33:06.183383+0000","last_peered":"2024-02-10T07:33:06.183383+0000","last_clean":"2024-02-10T07:33:06.183383+0000","last_became_active":"2024-02-10T07:31:55.298933+0000","last_became_peered":"2024-02-10T07:31:55.298933+0000","last_unstale":"2024-02-10T07:33:06.183383+0000","last_undegraded":"2024-02-10T07:33:06.183383+0000","last_fullsized":"2024-02-10T07:33:06.183383+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.183292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.183292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:12.631994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"74'1","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639813+0000","last_change":"2024-02-10T07:31:57.310237+0000","last_active":"2024-02-10T07:32:46.639813+0000","last_peered":"2024-02-10T07:32:46.639813+0000","last_clean":"2024-02-10T07:32:46.639813+0000","last_became_active":"2024-02-10T07:31:57.309929+0000","last_became_peered":"2024-02-10T07:31:57.309929+0000","last_unstale":"2024-02-10T07:32:46.639813+0000","last_undegraded":"2024-02-10T07:32:46.639813+0000","last_fullsized":"2024-02-10T07:32:46.639813+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:14.496768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":308,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641970+0000","last_change":"2024-02-10T07:32:25.175725+0000","last_active":"2024-02-10T07:32:46.641970+0000","last_peered":"2024-02-10T07:32:46.641970+0000","last_clean":"2024-02-10T07:32:46.641970+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:32:46.641970+0000","last_undegraded":"2024-02-10T07:32:46.641970+0000","last_fullsized":"2024-02-10T07:32:46.641970+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-02-10T07:32:25.175641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:32:25.175641+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:35.761800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597518+0000","last_change":"2024-02-10T07:31:51.283607+0000","last_active":"2024-02-10T07:32:47.597518+0000","last_peered":"2024-02-10T07:32:47.597518+0000","last_clean":"2024-02-10T07:32:47.597518+0000","last_became_active":"2024-02-10T07:31:51.283332+0000","last_became_peered":"2024-02-10T07:31:51.283332+0000","last_unstale":"2024-02-10T07:32:47.597518+0000","last_undegraded":"2024-02-10T07:32:47.597518+0000","last_fullsized":"2024-02-10T07:32:47.597518+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:06.299140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"89'24","reported_seq":105,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:11.841273+0000","last_change":"2024-02-10T07:33:11.841322+0000","last_active":"2024-02-10T07:33:11.841273+0000","last_peered":"2024-02-10T07:33:11.841273+0000","last_clean":"2024-02-10T07:33:11.841273+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:33:11.841273+0000","last_undegraded":"2024-02-10T07:33:11.841273+0000","last_fullsized":"2024-02-10T07:33:11.841273+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'24","last_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:33:11.841200+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:33.809552+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640726+0000","last_change":"2024-02-10T07:31:55.300257+0000","last_active":"2024-02-10T07:32:46.640726+0000","last_peered":"2024-02-10T07:32:46.640726+0000","last_clean":"2024-02-10T07:32:46.640726+0000","last_became_active":"2024-02-10T07:31:55.299928+0000","last_became_peered":"2024-02-10T07:31:55.299928+0000","last_unstale":"2024-02-10T07:32:46.640726+0000","last_undegraded":"2024-02-10T07:32:46.640726+0000","last_fullsized":"2024-02-10T07:32:46.640726+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:15.486048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694890+0000","last_change":"2024-02-10T07:31:57.316758+0000","last_active":"2024-02-10T07:32:47.694890+0000","last_peered":"2024-02-10T07:32:47.694890+0000","last_clean":"2024-02-10T07:32:47.694890+0000","last_became_active":"2024-02-10T07:31:57.316432+0000","last_became_peered":"2024-02-10T07:31:57.316432+0000","last_unstale":"2024-02-10T07:32:47.694890+0000","last_undegraded":"2024-02-10T07:32:47.694890+0000","last_fullsized":"2024-02-10T07:32:47.694890+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:23.313144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.100688+0000","last_change":"2024-02-10T07:31:51.286361+0000","last_active":"2024-02-10T07:32:47.100688+0000","last_peered":"2024-02-10T07:32:47.100688+0000","last_clean":"2024-02-10T07:32:47.100688+0000","last_became_active":"2024-02-10T07:31:51.286076+0000","last_became_peered":"2024-02-10T07:31:51.286076+0000","last_unstale":"2024-02-10T07:32:47.100688+0000","last_undegraded":"2024-02-10T07:32:47.100688+0000","last_fullsized":"2024-02-10T07:32:47.100688+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:17.577559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":143,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:11.306526+0000","last_change":"2024-02-10T07:31:54.281755+0000","last_active":"2024-02-10T07:33:11.306526+0000","last_peered":"2024-02-10T07:33:11.306526+0000","last_clean":"2024-02-10T07:33:11.306526+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:33:11.306526+0000","last_undegraded":"2024-02-10T07:33:11.306526+0000","last_fullsized":"2024-02-10T07:33:11.306526+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:31:49.265852+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101395+0000","last_change":"2024-02-10T07:31:53.301026+0000","last_active":"2024-02-10T07:32:47.101395+0000","last_peered":"2024-02-10T07:32:47.101395+0000","last_clean":"2024-02-10T07:32:47.101395+0000","last_became_active":"2024-02-10T07:31:53.300687+0000","last_became_peered":"2024-02-10T07:31:53.300687+0000","last_unstale":"2024-02-10T07:32:47.101395+0000","last_undegraded":"2024-02-10T07:32:47.101395+0000","last_fullsized":"2024-02-10T07:32:47.101395+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:16.203387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640509+0000","last_change":"2024-02-10T07:31:55.297440+0000","last_active":"2024-02-10T07:32:46.640509+0000","last_peered":"2024-02-10T07:32:46.640509+0000","last_clean":"2024-02-10T07:32:46.640509+0000","last_became_active":"2024-02-10T07:31:55.297178+0000","last_became_peered":"2024-02-10T07:31:55.297178+0000","last_unstale":"2024-02-10T07:32:46.640509+0000","last_undegraded":"2024-02-10T07:32:46.640509+0000","last_fullsized":"2024-02-10T07:32:46.640509+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.962436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694624+0000","last_change":"2024-02-10T07:31:57.307961+0000","last_active":"2024-02-10T07:32:47.694624+0000","last_peered":"2024-02-10T07:32:47.694624+0000","last_clean":"2024-02-10T07:32:47.694624+0000","last_became_active":"2024-02-10T07:31:57.307615+0000","last_became_peered":"2024-02-10T07:31:57.307615+0000","last_unstale":"2024-02-10T07:32:47.694624+0000","last_undegraded":"2024-02-10T07:32:47.694624+0000","last_fullsized":"2024-02-10T07:32:47.694624+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:24.112392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.270015+0000","last_change":"2024-02-10T07:33:06.270053+0000","last_active":"2024-02-10T07:33:06.270015+0000","last_peered":"2024-02-10T07:33:06.270015+0000","last_clean":"2024-02-10T07:33:06.270015+0000","last_became_active":"2024-02-10T07:31:51.320752+0000","last_became_peered":"2024-02-10T07:31:51.320752+0000","last_unstale":"2024-02-10T07:33:06.270015+0000","last_undegraded":"2024-02-10T07:33:06.270015+0000","last_fullsized":"2024-02-10T07:33:06.270015+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.269938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.269938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:34.431697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.100974+0000","last_change":"2024-02-10T07:31:53.289754+0000","last_active":"2024-02-10T07:32:47.100974+0000","last_peered":"2024-02-10T07:32:47.100974+0000","last_clean":"2024-02-10T07:32:47.100974+0000","last_became_active":"2024-02-10T07:31:53.289381+0000","last_became_peered":"2024-02-10T07:31:53.289381+0000","last_unstale":"2024-02-10T07:32:47.100974+0000","last_undegraded":"2024-02-10T07:32:47.100974+0000","last_fullsized":"2024-02-10T07:32:47.100974+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:13.156780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.695474+0000","last_change":"2024-02-10T07:31:55.290623+0000","last_active":"2024-02-10T07:32:47.695474+0000","last_peered":"2024-02-10T07:32:47.695474+0000","last_clean":"2024-02-10T07:32:47.695474+0000","last_became_active":"2024-02-10T07:31:55.290319+0000","last_became_peered":"2024-02-10T07:31:55.290319+0000","last_unstale":"2024-02-10T07:32:47.695474+0000","last_undegraded":"2024-02-10T07:32:47.695474+0000","last_fullsized":"2024-02-10T07:32:47.695474+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:23.955277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:05.431132+0000","last_change":"2024-02-10T07:33:05.431169+0000","last_active":"2024-02-10T07:33:05.431132+0000","last_peered":"2024-02-10T07:33:05.431132+0000","last_clean":"2024-02-10T07:33:05.431132+0000","last_became_active":"2024-02-10T07:31:57.317976+0000","last_became_peered":"2024-02-10T07:31:57.317976+0000","last_unstale":"2024-02-10T07:33:05.431132+0000","last_undegraded":"2024-02-10T07:33:05.431132+0000","last_fullsized":"2024-02-10T07:33:05.431132+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:05.431037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:05.431037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:31.470574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640631+0000","last_change":"2024-02-10T07:31:51.320993+0000","last_active":"2024-02-10T07:32:46.640631+0000","last_peered":"2024-02-10T07:32:46.640631+0000","last_clean":"2024-02-10T07:32:46.640631+0000","last_became_active":"2024-02-10T07:31:51.320673+0000","last_became_peered":"2024-02-10T07:31:51.320673+0000","last_unstale":"2024-02-10T07:32:46.640631+0000","last_undegraded":"2024-02-10T07:32:46.640631+0000","last_fullsized":"2024-02-10T07:32:46.640631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:15.248068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:53.288892+0000","last_change":"2024-02-10T07:32:53.288963+0000","last_active":"2024-02-10T07:32:53.288892+0000","last_peered":"2024-02-10T07:32:53.288892+0000","last_clean":"2024-02-10T07:32:53.288892+0000","last_became_active":"2024-02-10T07:31:53.297150+0000","last_became_peered":"2024-02-10T07:31:53.297150+0000","last_unstale":"2024-02-10T07:32:53.288892+0000","last_undegraded":"2024-02-10T07:32:53.288892+0000","last_fullsized":"2024-02-10T07:32:53.288892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:32:53.288798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:53.288798+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:06.843116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.372502+0000","last_change":"2024-02-10T07:31:55.296952+0000","last_active":"2024-02-10T07:33:06.372502+0000","last_peered":"2024-02-10T07:33:06.372502+0000","last_clean":"2024-02-10T07:33:06.372502+0000","last_became_active":"2024-02-10T07:31:55.295732+0000","last_became_peered":"2024-02-10T07:31:55.295732+0000","last_unstale":"2024-02-10T07:33:06.372502+0000","last_undegraded":"2024-02-10T07:33:06.372502+0000","last_fullsized":"2024-02-10T07:33:06.372502+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.515830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639824+0000","last_change":"2024-02-10T07:31:57.305942+0000","last_active":"2024-02-10T07:32:46.639824+0000","last_peered":"2024-02-10T07:32:46.639824+0000","last_clean":"2024-02-10T07:32:46.639824+0000","last_became_active":"2024-02-10T07:31:57.305620+0000","last_became_peered":"2024-02-10T07:31:57.305620+0000","last_unstale":"2024-02-10T07:32:46.639824+0000","last_undegraded":"2024-02-10T07:32:46.639824+0000","last_fullsized":"2024-02-10T07:32:46.639824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:09.830636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101458+0000","last_change":"2024-02-10T07:31:51.285921+0000","last_active":"2024-02-10T07:32:47.101458+0000","last_peered":"2024-02-10T07:32:47.101458+0000","last_clean":"2024-02-10T07:32:47.101458+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:32:47.101458+0000","last_undegraded":"2024-02-10T07:32:47.101458+0000","last_fullsized":"2024-02-10T07:32:47.101458+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:05.329043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640503+0000","last_change":"2024-02-10T07:31:53.287199+0000","last_active":"2024-02-10T07:32:46.640503+0000","last_peered":"2024-02-10T07:32:46.640503+0000","last_clean":"2024-02-10T07:32:46.640503+0000","last_became_active":"2024-02-10T07:31:53.286884+0000","last_became_peered":"2024-02-10T07:31:53.286884+0000","last_unstale":"2024-02-10T07:32:46.640503+0000","last_undegraded":"2024-02-10T07:32:46.640503+0000","last_fullsized":"2024-02-10T07:32:46.640503+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.033998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.372539+0000","last_change":"2024-02-10T07:32:55.469184+0000","last_active":"2024-02-10T07:33:06.372539+0000","last_peered":"2024-02-10T07:33:06.372539+0000","last_clean":"2024-02-10T07:33:06.372539+0000","last_became_active":"2024-02-10T07:31:55.302863+0000","last_became_peered":"2024-02-10T07:31:55.302863+0000","last_unstale":"2024-02-10T07:33:06.372539+0000","last_undegraded":"2024-02-10T07:33:06.372539+0000","last_fullsized":"2024-02-10T07:33:06.372539+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:32:55.469099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:55.469099+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:13.566389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641909+0000","last_change":"2024-02-10T07:31:57.316332+0000","last_active":"2024-02-10T07:32:46.641909+0000","last_peered":"2024-02-10T07:32:46.641909+0000","last_clean":"2024-02-10T07:32:46.641909+0000","last_became_active":"2024-02-10T07:31:57.316047+0000","last_became_peered":"2024-02-10T07:31:57.316047+0000","last_unstale":"2024-02-10T07:32:46.641909+0000","last_undegraded":"2024-02-10T07:32:46.641909+0000","last_fullsized":"2024-02-10T07:32:46.641909+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:25.756992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597656+0000","last_change":"2024-02-10T07:31:51.285635+0000","last_active":"2024-02-10T07:32:47.597656+0000","last_peered":"2024-02-10T07:32:47.597656+0000","last_clean":"2024-02-10T07:32:47.597656+0000","last_became_active":"2024-02-10T07:31:51.285333+0000","last_became_peered":"2024-02-10T07:31:51.285333+0000","last_unstale":"2024-02-10T07:32:47.597656+0000","last_undegraded":"2024-02-10T07:32:47.597656+0000","last_fullsized":"2024-02-10T07:32:47.597656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:05.926246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:08.541222+0000","last_change":"2024-02-10T07:33:08.541272+0000","last_active":"2024-02-10T07:33:08.541222+0000","last_peered":"2024-02-10T07:33:08.541222+0000","last_clean":"2024-02-10T07:33:08.541222+0000","last_became_active":"2024-02-10T07:31:53.290157+0000","last_became_peered":"2024-02-10T07:31:53.290157+0000","last_unstale":"2024-02-10T07:33:08.541222+0000","last_undegraded":"2024-02-10T07:33:08.541222+0000","last_fullsized":"2024-02-10T07:33:08.541222+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-10T07:33:08.541155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:08.541155+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:33.263033+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.184144+0000","last_change":"2024-02-10T07:33:01.184183+0000","last_active":"2024-02-10T07:33:01.184144+0000","last_peered":"2024-02-10T07:33:01.184144+0000","last_clean":"2024-02-10T07:33:01.184144+0000","last_became_active":"2024-02-10T07:31:55.295900+0000","last_became_peered":"2024-02-10T07:31:55.295900+0000","last_unstale":"2024-02-10T07:33:01.184144+0000","last_undegraded":"2024-02-10T07:33:01.184144+0000","last_fullsized":"2024-02-10T07:33:01.184144+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:01.184048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:01.184048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:11.992366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640531+0000","last_change":"2024-02-10T07:31:57.311731+0000","last_active":"2024-02-10T07:32:46.640531+0000","last_peered":"2024-02-10T07:32:46.640531+0000","last_clean":"2024-02-10T07:32:46.640531+0000","last_became_active":"2024-02-10T07:31:57.311054+0000","last_became_peered":"2024-02-10T07:31:57.311054+0000","last_unstale":"2024-02-10T07:32:46.640531+0000","last_undegraded":"2024-02-10T07:32:46.640531+0000","last_fullsized":"2024-02-10T07:32:46.640531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.402912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.372913+0000","last_change":"2024-02-10T07:31:55.295389+0000","last_active":"2024-02-10T07:33:06.372913+0000","last_peered":"2024-02-10T07:33:06.372913+0000","last_clean":"2024-02-10T07:33:06.372913+0000","last_became_active":"2024-02-10T07:31:55.295067+0000","last_became_peered":"2024-02-10T07:31:55.295067+0000","last_unstale":"2024-02-10T07:33:06.372913+0000","last_undegraded":"2024-02-10T07:33:06.372913+0000","last_fullsized":"2024-02-10T07:33:06.372913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:24.085032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597500+0000","last_change":"2024-02-10T07:31:51.282545+0000","last_active":"2024-02-10T07:32:47.597500+0000","last_peered":"2024-02-10T07:32:47.597500+0000","last_clean":"2024-02-10T07:32:47.597500+0000","last_became_active":"2024-02-10T07:31:51.282137+0000","last_became_peered":"2024-02-10T07:31:51.282137+0000","last_unstale":"2024-02-10T07:32:47.597500+0000","last_undegraded":"2024-02-10T07:32:47.597500+0000","last_fullsized":"2024-02-10T07:32:47.597500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.188800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":82,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:57.341196+0000","last_change":"2024-02-10T07:32:57.341249+0000","last_active":"2024-02-10T07:32:57.341196+0000","last_peered":"2024-02-10T07:32:57.341196+0000","last_clean":"2024-02-10T07:32:57.341196+0000","last_became_active":"2024-02-10T07:31:53.294218+0000","last_became_peered":"2024-02-10T07:31:53.294218+0000","last_unstale":"2024-02-10T07:32:57.341196+0000","last_undegraded":"2024-02-10T07:32:57.341196+0000","last_fullsized":"2024-02-10T07:32:57.341196+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:32:57.341115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:57.341115+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:05.137575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:04.582259+0000","last_change":"2024-02-10T07:33:04.582281+0000","last_active":"2024-02-10T07:33:04.582259+0000","last_peered":"2024-02-10T07:33:04.582259+0000","last_clean":"2024-02-10T07:33:04.582259+0000","last_became_active":"2024-02-10T07:31:57.303235+0000","last_became_peered":"2024-02-10T07:31:57.303235+0000","last_unstale":"2024-02-10T07:33:04.582259+0000","last_undegraded":"2024-02-10T07:33:04.582259+0000","last_fullsized":"2024-02-10T07:33:04.582259+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:04.582188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:04.582188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:27.553405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.237029+0000","last_change":"2024-02-10T07:33:06.237068+0000","last_active":"2024-02-10T07:33:06.237029+0000","last_peered":"2024-02-10T07:33:06.237029+0000","last_clean":"2024-02-10T07:33:06.237029+0000","last_became_active":"2024-02-10T07:31:55.294376+0000","last_became_peered":"2024-02-10T07:31:55.294376+0000","last_unstale":"2024-02-10T07:33:06.237029+0000","last_undegraded":"2024-02-10T07:33:06.237029+0000","last_fullsized":"2024-02-10T07:33:06.237029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.236946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.236946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:10.623394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640348+0000","last_change":"2024-02-10T07:31:51.321090+0000","last_active":"2024-02-10T07:32:46.640348+0000","last_peered":"2024-02-10T07:32:46.640348+0000","last_clean":"2024-02-10T07:32:46.640348+0000","last_became_active":"2024-02-10T07:31:51.320715+0000","last_became_peered":"2024-02-10T07:31:51.320715+0000","last_unstale":"2024-02-10T07:32:46.640348+0000","last_undegraded":"2024-02-10T07:32:46.640348+0000","last_fullsized":"2024-02-10T07:32:46.640348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:04.387923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":70,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642166+0000","last_change":"2024-02-10T07:31:53.292945+0000","last_active":"2024-02-10T07:32:46.642166+0000","last_peered":"2024-02-10T07:32:46.642166+0000","last_clean":"2024-02-10T07:32:46.642166+0000","last_became_active":"2024-02-10T07:31:53.292510+0000","last_became_peered":"2024-02-10T07:31:53.292510+0000","last_unstale":"2024-02-10T07:32:46.642166+0000","last_undegraded":"2024-02-10T07:32:46.642166+0000","last_fullsized":"2024-02-10T07:32:46.642166+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:04.860425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:05.197704+0000","last_change":"2024-02-10T07:33:05.197724+0000","last_active":"2024-02-10T07:33:05.197704+0000","last_peered":"2024-02-10T07:33:05.197704+0000","last_clean":"2024-02-10T07:33:05.197704+0000","last_became_active":"2024-02-10T07:31:57.311427+0000","last_became_peered":"2024-02-10T07:31:57.311427+0000","last_unstale":"2024-02-10T07:33:05.197704+0000","last_undegraded":"2024-02-10T07:33:05.197704+0000","last_fullsized":"2024-02-10T07:33:05.197704+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:33:05.197642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:20.323214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:12.259572+0000","last_change":"2024-02-10T07:33:12.259596+0000","last_active":"2024-02-10T07:33:12.259572+0000","last_peered":"2024-02-10T07:33:12.259572+0000","last_clean":"2024-02-10T07:33:12.259572+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:33:12.259572+0000","last_undegraded":"2024-02-10T07:33:12.259572+0000","last_fullsized":"2024-02-10T07:33:12.259572+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:12.259478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:12.259478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:41.862761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:56.900575+0000","last_change":"2024-02-10T07:32:56.900599+0000","last_active":"2024-02-10T07:32:56.900575+0000","last_peered":"2024-02-10T07:32:56.900575+0000","last_clean":"2024-02-10T07:32:56.900575+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:32:56.900575+0000","last_undegraded":"2024-02-10T07:32:56.900575+0000","last_fullsized":"2024-02-10T07:32:56.900575+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:56.900501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:56.900501+0000","last_clean_scrub_stamp":"2024-02-10T07:32:56.900501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:16.803474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642534+0000","last_change":"2024-02-10T07:31:53.292581+0000","last_active":"2024-02-10T07:32:46.642534+0000","last_peered":"2024-02-10T07:32:46.642534+0000","last_clean":"2024-02-10T07:32:46.642534+0000","last_became_active":"2024-02-10T07:31:53.292277+0000","last_became_peered":"2024-02-10T07:31:53.292277+0000","last_unstale":"2024-02-10T07:32:46.642534+0000","last_undegraded":"2024-02-10T07:32:46.642534+0000","last_fullsized":"2024-02-10T07:32:46.642534+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:04.812915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642496+0000","last_change":"2024-02-10T07:31:57.308864+0000","last_active":"2024-02-10T07:32:46.642496+0000","last_peered":"2024-02-10T07:32:46.642496+0000","last_clean":"2024-02-10T07:32:46.642496+0000","last_became_active":"2024-02-10T07:31:57.308608+0000","last_became_peered":"2024-02-10T07:31:57.308608+0000","last_unstale":"2024-02-10T07:32:46.642496+0000","last_undegraded":"2024-02-10T07:32:46.642496+0000","last_fullsized":"2024-02-10T07:32:46.642496+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:09.099379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.372502+0000","last_change":"2024-02-10T07:31:55.298852+0000","last_active":"2024-02-10T07:33:06.372502+0000","last_peered":"2024-02-10T07:33:06.372502+0000","last_clean":"2024-02-10T07:33:06.372502+0000","last_became_active":"2024-02-10T07:31:55.298389+0000","last_became_peered":"2024-02-10T07:31:55.298389+0000","last_unstale":"2024-02-10T07:33:06.372502+0000","last_undegraded":"2024-02-10T07:33:06.372502+0000","last_fullsized":"2024-02-10T07:33:06.372502+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.686032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597506+0000","last_change":"2024-02-10T07:31:51.288638+0000","last_active":"2024-02-10T07:32:47.597506+0000","last_peered":"2024-02-10T07:32:47.597506+0000","last_clean":"2024-02-10T07:32:47.597506+0000","last_became_active":"2024-02-10T07:31:51.288197+0000","last_became_peered":"2024-02-10T07:31:51.288197+0000","last_unstale":"2024-02-10T07:32:47.597506+0000","last_undegraded":"2024-02-10T07:32:47.597506+0000","last_fullsized":"2024-02-10T07:32:47.597506+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:04.642730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641567+0000","last_change":"2024-02-10T07:31:53.288289+0000","last_active":"2024-02-10T07:32:46.641567+0000","last_peered":"2024-02-10T07:32:46.641567+0000","last_clean":"2024-02-10T07:32:46.641567+0000","last_became_active":"2024-02-10T07:31:53.287984+0000","last_became_peered":"2024-02-10T07:31:53.287984+0000","last_unstale":"2024-02-10T07:32:46.641567+0000","last_undegraded":"2024-02-10T07:32:46.641567+0000","last_fullsized":"2024-02-10T07:32:46.641567+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.716780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":405,"ondisk_log_size":405,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037781,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1495040,"data_stored":1013417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.79900000000000004}]},{"osd":1,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.77200000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.73299999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.76000000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.61499999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.68500000000000005}]},{"osd":6,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.747}]}]},{"osd":6,"up_from":50,"seq":214748364826,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28460,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716692,"statfs":{"total":95995035648,"available":95965892608,"internally_reserved":0,"allocated":1478656,"data_stored":1012764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.497},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59399999999999997}]},{"osd":1,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.64700000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.749},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.72499999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":19.664999999999999,"5min":19.664999999999999,"15min":19.664999999999999},"last":0.623},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.71199999999999997}]},{"osd":4,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.67500000000000004}]},{"osd":5,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.70099999999999996}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.73599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36236,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708916,"statfs":{"total":95995035648,"available":95957929984,"internally_reserved":0,"allocated":921600,"data_stored":423123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.374,"15min":0.34599999999999997},"min":{"1min":0.249,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.40400000000000003,"15min":0.36399999999999999},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.47099999999999997}]},{"osd":2,"last update":"Sat Feb 10 07:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41299999999999998,"15min":0.40000000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42899999999999999,"15min":0.41799999999999998},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.52800000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:33:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.42599999999999999,"15min":0.42399999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.58499999999999996,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.51200000000000001,"15min":0.50900000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.497}]},{"osd":4,"last update":"Sat Feb 10 07:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.98799999999999999,"15min":1.0469999999999999},"min":{"1min":0.435,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80700000000000005,"5min":10.292,"15min":10.292},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58399999999999996,"15min":0.58299999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.71599999999999997,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.69999999999999996}]},{"osd":5,"last update":"Sat Feb 10 07:32:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.627,"15min":0.63},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80500000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.751}]},{"osd":6,"last update":"Sat Feb 10 07:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.623,"15min":0.628},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.79800000000000004,"5min":1.002,"15min":1.002},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.83699999999999997,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.71399999999999997}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.82099999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32644,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712508,"statfs":{"total":95995035648,"available":95961608192,"internally_reserved":0,"allocated":1503232,"data_stored":1014034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34599999999999997,"15min":0.34399999999999997},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.41099999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.36499999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.36699999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.38500000000000001,"15min":0.38800000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.51200000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42899999999999999,"15min":0.42499999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57899999999999996,"5min":0.629,"15min":0.629},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Feb 10 07:33:05 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.432,"15min":0.443},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53800000000000003,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47499999999999998,"15min":0.47199999999999998},"min":{"1min":0.32400000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64700000000000002,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.436}]},{"osd":4,"last update":"Sat Feb 10 07:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.74099999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.433,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.58899999999999997}]},{"osd":5,"last update":"Sat Feb 10 07:32:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59699999999999998,"15min":0.59399999999999997},"min":{"1min":0.438,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.55000000000000004}]},{"osd":6,"last update":"Sat Feb 10 07:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":0.83499999999999996,"15min":0.70599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":19.16,"5min":19.16,"15min":19.16},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.61899999999999999,"15min":0.63100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.67200000000000004,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.45100000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":19.178000000000001,"5min":19.178000000000001,"15min":19.178000000000001},"last":0.63},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.66900000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":905216,"data_stored":423646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.379,"15min":0.38},"min":{"1min":0.255,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56499999999999995,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.42399999999999999,"15min":0.41599999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.38100000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.44,"15min":0.42399999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.314},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.44,"15min":0.42199999999999999},"min":{"1min":0.27200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.32600000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51700000000000002,"15min":0.53400000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.53100000000000003,"15min":0.55300000000000005},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.30099999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63100000000000001,"15min":0.626},"min":{"1min":0.50700000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.68899999999999995,"15min":0.65300000000000002},"min":{"1min":0.50700000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":5.343,"5min":5.343,"15min":5.343},"last":0.51300000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.64900000000000002,"15min":0.64400000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.628,"15min":0.623},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.56399999999999995}]},{"osd":6,"last update":"Sat Feb 10 07:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":5.3230000000000004,"5min":5.3230000000000004,"15min":5.3230000000000004},"last":0.627},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.44}]},{"osd":7,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.60599999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":921600,"data_stored":422902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.372,"15min":0.378},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55500000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.316}]},{"osd":1,"last update":"Sat Feb 10 07:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.42099999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.442,"15min":0.437},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.37}]},{"osd":2,"last update":"Sat Feb 10 07:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.31,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42799999999999999,"15min":0.434},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.35299999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.89700000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:32:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.54600000000000004}]},{"osd":6,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.60799999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.59999999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953508,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":423014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.58099999999999996,"15min":0.58499999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.69899999999999995,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.39200000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.70599999999999996,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.34100000000000003}]},{"osd":1,"last update":"Sat Feb 10 07:32:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.44,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.69499999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72899999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:32:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.67300000000000004,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.59899999999999998,"15min":0.60499999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78800000000000003,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.53000000000000003}]},{"osd":3,"last update":"Sat Feb 10 07:32:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57999999999999996,"15min":0.58299999999999996},"min":{"1min":0.44900000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.76000000000000001,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.439},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.376,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70499999999999996,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.56999999999999995}]},{"osd":5,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.378,"15min":0.373},"min":{"1min":0.30099999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60799999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.307},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.40300000000000002,"15min":0.39300000000000002},"min":{"1min":0.32300000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.39000000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.46100000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.59299999999999997}]}]},{"osd":5,"up_from":40,"seq":171798691871,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32052,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713100,"statfs":{"total":95995035648,"available":95962214400,"internally_reserved":0,"allocated":897024,"data_stored":423608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77400000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.57699999999999996,"15min":0.56100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.2029999999999998,"5min":3.2029999999999998,"15min":3.2029999999999998},"last":0.48699999999999999}]},{"osd":1,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.75,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.61299999999999999,"15min":0.59699999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.238,"5min":3.238,"15min":3.238},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.438,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76200000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.748,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.46700000000000003}]},{"osd":3,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.60299999999999998},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80800000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.44},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.76500000000000001,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.41499999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.41099999999999998,"15min":0.40100000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.39700000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.51000000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.58999999999999997}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:33:17.763 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-10T07:33:17.763 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-10T07:33:17.764 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-10T07:33:17.764 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph health --format=json 2024-02-10T07:33:18.013 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:18.441 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[29816]: 3.19 scrub starts 2024-02-10T07:33:18.441 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[29816]: 3.19 scrub ok 2024-02-10T07:33:18.441 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[29816]: 3.18 scrub starts 2024-02-10T07:33:18.441 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[29816]: 3.18 scrub ok 2024-02-10T07:33:18.442 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[29816]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:33:18.442 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[29816]: 4.7 scrub starts 2024-02-10T07:33:18.442 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[29816]: 4.7 scrub ok 2024-02-10T07:33:18.442 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[29816]: 5.5 scrub starts 2024-02-10T07:33:18.442 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[29816]: 5.5 scrub ok 2024-02-10T07:33:18.442 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[29816]: 5.1f scrub starts 2024-02-10T07:33:18.442 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[29816]: 5.1f scrub ok 2024-02-10T07:33:18.443 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[40743]: 3.19 scrub starts 2024-02-10T07:33:18.443 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[40743]: 3.19 scrub ok 2024-02-10T07:33:18.444 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[40743]: 3.18 scrub starts 2024-02-10T07:33:18.444 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[40743]: 3.18 scrub ok 2024-02-10T07:33:18.444 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[40743]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:33:18.444 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[40743]: 4.7 scrub starts 2024-02-10T07:33:18.444 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[40743]: 4.7 scrub ok 2024-02-10T07:33:18.444 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[40743]: 5.5 scrub starts 2024-02-10T07:33:18.444 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[40743]: 5.5 scrub ok 2024-02-10T07:33:18.444 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[40743]: 5.1f scrub starts 2024-02-10T07:33:18.444 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-mon[40743]: 5.1f scrub ok 2024-02-10T07:33:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:18 smithi170 ceph-mon[33394]: 3.19 scrub starts 2024-02-10T07:33:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:18 smithi170 ceph-mon[33394]: 3.19 scrub ok 2024-02-10T07:33:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:18 smithi170 ceph-mon[33394]: 3.18 scrub starts 2024-02-10T07:33:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:18 smithi170 ceph-mon[33394]: 3.18 scrub ok 2024-02-10T07:33:18.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:18 smithi170 ceph-mon[33394]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:33:18.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:18 smithi170 ceph-mon[33394]: 4.7 scrub starts 2024-02-10T07:33:18.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:18 smithi170 ceph-mon[33394]: 4.7 scrub ok 2024-02-10T07:33:18.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:18 smithi170 ceph-mon[33394]: 5.5 scrub starts 2024-02-10T07:33:18.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:18 smithi170 ceph-mon[33394]: 5.5 scrub ok 2024-02-10T07:33:18.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:18 smithi170 ceph-mon[33394]: 5.1f scrub starts 2024-02-10T07:33:18.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:18 smithi170 ceph-mon[33394]: 5.1f scrub ok 2024-02-10T07:33:19.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:33:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:33:19.380 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[29816]: 5.17 deep-scrub starts 2024-02-10T07:33:19.381 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[29816]: 5.17 deep-scrub ok 2024-02-10T07:33:19.381 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[29816]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:19.381 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[29816]: 6.1e scrub starts 2024-02-10T07:33:19.381 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[29816]: 6.1e scrub ok 2024-02-10T07:33:19.381 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[29816]: 6.1d scrub starts 2024-02-10T07:33:19.381 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[29816]: 6.1d scrub ok 2024-02-10T07:33:19.381 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[29816]: 4.b scrub starts 2024-02-10T07:33:19.381 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[29816]: 4.b scrub ok 2024-02-10T07:33:19.381 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[40743]: 5.17 deep-scrub starts 2024-02-10T07:33:19.381 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[40743]: 5.17 deep-scrub ok 2024-02-10T07:33:19.381 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[40743]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:19.382 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[40743]: 6.1e scrub starts 2024-02-10T07:33:19.382 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[40743]: 6.1e scrub ok 2024-02-10T07:33:19.382 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[40743]: 6.1d scrub starts 2024-02-10T07:33:19.382 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[40743]: 6.1d scrub ok 2024-02-10T07:33:19.382 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[40743]: 4.b scrub starts 2024-02-10T07:33:19.382 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:19 smithi008 ceph-mon[40743]: 4.b scrub ok 2024-02-10T07:33:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:19 smithi170 ceph-mon[33394]: 5.17 deep-scrub starts 2024-02-10T07:33:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:19 smithi170 ceph-mon[33394]: 5.17 deep-scrub ok 2024-02-10T07:33:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:19 smithi170 ceph-mon[33394]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:19 smithi170 ceph-mon[33394]: 6.1e scrub starts 2024-02-10T07:33:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:19 smithi170 ceph-mon[33394]: 6.1e scrub ok 2024-02-10T07:33:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:19 smithi170 ceph-mon[33394]: 6.1d scrub starts 2024-02-10T07:33:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:19 smithi170 ceph-mon[33394]: 6.1d scrub ok 2024-02-10T07:33:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:19 smithi170 ceph-mon[33394]: 4.b scrub starts 2024-02-10T07:33:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:19 smithi170 ceph-mon[33394]: 4.b scrub ok 2024-02-10T07:33:19.861 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:33:19.861 INFO:teuthology.orchestra.run.smithi008.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-10T07:33:20.553 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[29816]: 3.15 scrub starts 2024-02-10T07:33:20.553 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[29816]: 3.15 scrub ok 2024-02-10T07:33:20.553 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[29816]: 3.9 scrub starts 2024-02-10T07:33:20.553 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[29816]: 3.9 scrub ok 2024-02-10T07:33:20.553 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[29816]: 5.1d scrub starts 2024-02-10T07:33:20.553 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[29816]: 5.1d scrub ok 2024-02-10T07:33:20.554 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[29816]: 3.c scrub starts 2024-02-10T07:33:20.554 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[29816]: 3.c scrub ok 2024-02-10T07:33:20.554 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/680722090' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T07:33:20.554 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[40743]: 3.15 scrub starts 2024-02-10T07:33:20.554 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[40743]: 3.15 scrub ok 2024-02-10T07:33:20.554 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[40743]: 3.9 scrub starts 2024-02-10T07:33:20.554 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[40743]: 3.9 scrub ok 2024-02-10T07:33:20.554 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[40743]: 5.1d scrub starts 2024-02-10T07:33:20.554 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[40743]: 5.1d scrub ok 2024-02-10T07:33:20.554 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[40743]: 3.c scrub starts 2024-02-10T07:33:20.554 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[40743]: 3.c scrub ok 2024-02-10T07:33:20.555 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:20 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/680722090' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T07:33:20.581 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-10T07:33:20.582 INFO:tasks.cephadm:Setup complete, yielding 2024-02-10T07:33:20.582 INFO:teuthology.run_tasks:Running task thrashosds... 2024-02-10T07:33:20.593 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-10T07:33:20.593 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}} 2024-02-10T07:33:20.593 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-10T07:33:20.594 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-02-10T07:33:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:20 smithi170 ceph-mon[33394]: 3.15 scrub starts 2024-02-10T07:33:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:20 smithi170 ceph-mon[33394]: 3.15 scrub ok 2024-02-10T07:33:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:20 smithi170 ceph-mon[33394]: 3.9 scrub starts 2024-02-10T07:33:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:20 smithi170 ceph-mon[33394]: 3.9 scrub ok 2024-02-10T07:33:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:20 smithi170 ceph-mon[33394]: 5.1d scrub starts 2024-02-10T07:33:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:20 smithi170 ceph-mon[33394]: 5.1d scrub ok 2024-02-10T07:33:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:20 smithi170 ceph-mon[33394]: 3.c scrub starts 2024-02-10T07:33:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:20 smithi170 ceph-mon[33394]: 3.c scrub ok 2024-02-10T07:33:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:20 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/680722090' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T07:33:20.842 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:21.272 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:21 smithi008 ceph-mon[29816]: 3.b scrub starts 2024-02-10T07:33:21.272 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:21 smithi008 ceph-mon[29816]: 3.b scrub ok 2024-02-10T07:33:21.272 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:21 smithi008 ceph-mon[29816]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:21.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:21 smithi008 ceph-mon[29816]: 4.d deep-scrub starts 2024-02-10T07:33:21.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:21 smithi008 ceph-mon[29816]: 4.d deep-scrub ok 2024-02-10T07:33:21.273 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:21 smithi008 ceph-mon[40743]: 3.b scrub starts 2024-02-10T07:33:21.273 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:21 smithi008 ceph-mon[40743]: 3.b scrub ok 2024-02-10T07:33:21.273 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:21 smithi008 ceph-mon[40743]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:21.273 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:21 smithi008 ceph-mon[40743]: 4.d deep-scrub starts 2024-02-10T07:33:21.273 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:21 smithi008 ceph-mon[40743]: 4.d deep-scrub ok 2024-02-10T07:33:21.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:21 smithi170 ceph-mon[33394]: 3.b scrub starts 2024-02-10T07:33:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:21 smithi170 ceph-mon[33394]: 3.b scrub ok 2024-02-10T07:33:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:21 smithi170 ceph-mon[33394]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:21 smithi170 ceph-mon[33394]: 4.d deep-scrub starts 2024-02-10T07:33:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:21 smithi170 ceph-mon[33394]: 4.d deep-scrub ok 2024-02-10T07:33:22.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-mon[40743]: 3.7 scrub starts 2024-02-10T07:33:22.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-mon[40743]: 3.7 scrub ok 2024-02-10T07:33:22.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-mon[40743]: 4.12 scrub starts 2024-02-10T07:33:22.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-mon[40743]: 4.12 scrub ok 2024-02-10T07:33:22.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-mon[40743]: 4.e deep-scrub starts 2024-02-10T07:33:22.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-mon[40743]: 4.e deep-scrub ok 2024-02-10T07:33:22.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-mon[29816]: 3.7 scrub starts 2024-02-10T07:33:22.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-mon[29816]: 3.7 scrub ok 2024-02-10T07:33:22.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-mon[29816]: 4.12 scrub starts 2024-02-10T07:33:22.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-mon[29816]: 4.12 scrub ok 2024-02-10T07:33:22.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-mon[29816]: 4.e deep-scrub starts 2024-02-10T07:33:22.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-mon[29816]: 4.e deep-scrub ok 2024-02-10T07:33:22.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:22 smithi170 ceph-mon[33394]: 3.7 scrub starts 2024-02-10T07:33:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:22 smithi170 ceph-mon[33394]: 3.7 scrub ok 2024-02-10T07:33:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:22 smithi170 ceph-mon[33394]: 4.12 scrub starts 2024-02-10T07:33:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:22 smithi170 ceph-mon[33394]: 4.12 scrub ok 2024-02-10T07:33:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:22 smithi170 ceph-mon[33394]: 4.e deep-scrub starts 2024-02-10T07:33:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:22 smithi170 ceph-mon[33394]: 4.e deep-scrub ok 2024-02-10T07:33:23.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:33:22] ENGINE Bus STOPPING 2024-02-10T07:33:23.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:33:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:33:23.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:33:22] ENGINE Bus STOPPED 2024-02-10T07:33:23.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:33:22] ENGINE Bus STARTING 2024-02-10T07:33:23.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:33:22] ENGINE Serving on http://:::9283 2024-02-10T07:33:23.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:22 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:33:22] ENGINE Bus STARTED 2024-02-10T07:33:23.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:33:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:33:23.359 INFO:tasks.thrashosds:Beginning thrashosds... 2024-02-10T07:33:23.359 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-10T07:33:23.359 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:33:23.386 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:23 smithi008 ceph-mon[40743]: 6.8 scrub starts 2024-02-10T07:33:23.386 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:23 smithi008 ceph-mon[40743]: 6.8 scrub ok 2024-02-10T07:33:23.386 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:23 smithi008 ceph-mon[40743]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:23.386 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:23 smithi008 ceph-mon[40743]: 6.16 scrub starts 2024-02-10T07:33:23.386 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:23 smithi008 ceph-mon[40743]: 6.16 scrub ok 2024-02-10T07:33:23.386 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:23 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2772507422' entity='client.admin' 2024-02-10T07:33:23.386 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:23 smithi008 ceph-mon[29816]: 6.8 scrub starts 2024-02-10T07:33:23.387 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:23 smithi008 ceph-mon[29816]: 6.8 scrub ok 2024-02-10T07:33:23.387 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:23 smithi008 ceph-mon[29816]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:23.387 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:23 smithi008 ceph-mon[29816]: 6.16 scrub starts 2024-02-10T07:33:23.387 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:23 smithi008 ceph-mon[29816]: 6.16 scrub ok 2024-02-10T07:33:23.387 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:23 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2772507422' entity='client.admin' 2024-02-10T07:33:23.608 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:23.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:23 smithi170 ceph-mon[33394]: 6.8 scrub starts 2024-02-10T07:33:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:23 smithi170 ceph-mon[33394]: 6.8 scrub ok 2024-02-10T07:33:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:23 smithi170 ceph-mon[33394]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:23 smithi170 ceph-mon[33394]: 6.16 scrub starts 2024-02-10T07:33:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:23 smithi170 ceph-mon[33394]: 6.16 scrub ok 2024-02-10T07:33:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:23 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2772507422' entity='client.admin' 2024-02-10T07:33:24.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:24 smithi008 ceph-mon[29816]: 3.14 scrub starts 2024-02-10T07:33:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:24 smithi008 ceph-mon[29816]: 3.14 scrub ok 2024-02-10T07:33:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:24 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:24 smithi008 ceph-mon[29816]: 5.12 scrub starts 2024-02-10T07:33:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:24 smithi008 ceph-mon[29816]: 5.12 scrub ok 2024-02-10T07:33:24.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:24 smithi008 ceph-mon[40743]: 3.14 scrub starts 2024-02-10T07:33:24.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:24 smithi008 ceph-mon[40743]: 3.14 scrub ok 2024-02-10T07:33:24.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:24 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:24.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:24 smithi008 ceph-mon[40743]: 5.12 scrub starts 2024-02-10T07:33:24.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:24 smithi008 ceph-mon[40743]: 5.12 scrub ok 2024-02-10T07:33:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:24 smithi170 ceph-mon[33394]: 3.14 scrub starts 2024-02-10T07:33:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:24 smithi170 ceph-mon[33394]: 3.14 scrub ok 2024-02-10T07:33:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:24 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:24 smithi170 ceph-mon[33394]: 5.12 scrub starts 2024-02-10T07:33:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:24 smithi170 ceph-mon[33394]: 5.12 scrub ok 2024-02-10T07:33:24.939 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:33:24 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[75760]: ts=2024-02-10T07:33:24.841Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004111774s 2024-02-10T07:33:25.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:25 smithi008 ceph-mon[29816]: 3.2 scrub starts 2024-02-10T07:33:25.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:25 smithi008 ceph-mon[29816]: 3.2 scrub ok 2024-02-10T07:33:25.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:25 smithi008 ceph-mon[29816]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:25 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:33:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:25 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:33:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:25 smithi008 ceph-mon[40743]: 3.2 scrub starts 2024-02-10T07:33:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:25 smithi008 ceph-mon[40743]: 3.2 scrub ok 2024-02-10T07:33:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:25 smithi008 ceph-mon[40743]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:25 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:33:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:25 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:33:25.364 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:33:25.367 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:33:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:25 smithi170 ceph-mon[33394]: 3.2 scrub starts 2024-02-10T07:33:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:25 smithi170 ceph-mon[33394]: 3.2 scrub ok 2024-02-10T07:33:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:25 smithi170 ceph-mon[33394]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:25 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:33:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:25 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:33:26.142 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-02-10T07:33:25.192394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1019,"num_read_kb":865,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":632,"ondisk_log_size":632,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":254076,"kb_used_data":8828,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749707140,"statfs":{"total":767960285184,"available":767700111360,"internally_reserved":0,"allocated":9039872,"data_stored":5156508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.516924"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640262+0000","last_change":"2024-02-10T07:31:57.311638+0000","last_active":"2024-02-10T07:32:46.640262+0000","last_peered":"2024-02-10T07:32:46.640262+0000","last_clean":"2024-02-10T07:32:46.640262+0000","last_became_active":"2024-02-10T07:31:57.311360+0000","last_became_peered":"2024-02-10T07:31:57.311360+0000","last_unstale":"2024-02-10T07:32:46.640262+0000","last_undegraded":"2024-02-10T07:32:46.640262+0000","last_fullsized":"2024-02-10T07:32:46.640262+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.653308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:14.320656+0000","last_change":"2024-02-10T07:33:14.320677+0000","last_active":"2024-02-10T07:33:14.320656+0000","last_peered":"2024-02-10T07:33:14.320656+0000","last_clean":"2024-02-10T07:33:14.320656+0000","last_became_active":"2024-02-10T07:31:55.299946+0000","last_became_peered":"2024-02-10T07:31:55.299946+0000","last_unstale":"2024-02-10T07:33:14.320656+0000","last_undegraded":"2024-02-10T07:33:14.320656+0000","last_fullsized":"2024-02-10T07:33:14.320656+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:14.320588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:14.320588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:38.014309+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101374+0000","last_change":"2024-02-10T07:31:51.321177+0000","last_active":"2024-02-10T07:32:47.101374+0000","last_peered":"2024-02-10T07:32:47.101374+0000","last_clean":"2024-02-10T07:32:47.101374+0000","last_became_active":"2024-02-10T07:31:51.320873+0000","last_became_peered":"2024-02-10T07:31:51.320873+0000","last_unstale":"2024-02-10T07:32:47.101374+0000","last_undegraded":"2024-02-10T07:32:47.101374+0000","last_fullsized":"2024-02-10T07:32:47.101374+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:09.885451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640003+0000","last_change":"2024-02-10T07:31:53.301175+0000","last_active":"2024-02-10T07:32:46.640003+0000","last_peered":"2024-02-10T07:32:46.640003+0000","last_clean":"2024-02-10T07:32:46.640003+0000","last_became_active":"2024-02-10T07:31:53.300846+0000","last_became_peered":"2024-02-10T07:31:53.300846+0000","last_unstale":"2024-02-10T07:32:46.640003+0000","last_undegraded":"2024-02-10T07:32:46.640003+0000","last_fullsized":"2024-02-10T07:32:46.640003+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:15.839827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:12.587926+0000","last_change":"2024-02-10T07:33:12.587948+0000","last_active":"2024-02-10T07:33:12.587926+0000","last_peered":"2024-02-10T07:33:12.587926+0000","last_clean":"2024-02-10T07:33:12.587926+0000","last_became_active":"2024-02-10T07:31:57.300056+0000","last_became_peered":"2024-02-10T07:31:57.300056+0000","last_unstale":"2024-02-10T07:33:12.587926+0000","last_undegraded":"2024-02-10T07:33:12.587926+0000","last_fullsized":"2024-02-10T07:33:12.587926+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:12.587860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:12.587860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:17.246465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:58.168423+0000","last_change":"2024-02-10T07:32:58.168446+0000","last_active":"2024-02-10T07:32:58.168423+0000","last_peered":"2024-02-10T07:32:58.168423+0000","last_clean":"2024-02-10T07:32:58.168423+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:32:58.168423+0000","last_undegraded":"2024-02-10T07:32:58.168423+0000","last_fullsized":"2024-02-10T07:32:58.168423+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:58.168353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.168353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:11.296018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597637+0000","last_change":"2024-02-10T07:31:51.323657+0000","last_active":"2024-02-10T07:32:47.597637+0000","last_peered":"2024-02-10T07:32:47.597637+0000","last_clean":"2024-02-10T07:32:47.597637+0000","last_became_active":"2024-02-10T07:31:51.323331+0000","last_became_peered":"2024-02-10T07:31:51.323331+0000","last_unstale":"2024-02-10T07:32:47.597637+0000","last_undegraded":"2024-02-10T07:32:47.597637+0000","last_fullsized":"2024-02-10T07:32:47.597637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:07.093553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":84,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:00.565446+0000","last_change":"2024-02-10T07:33:00.565478+0000","last_active":"2024-02-10T07:33:00.565446+0000","last_peered":"2024-02-10T07:33:00.565446+0000","last_clean":"2024-02-10T07:33:00.565446+0000","last_became_active":"2024-02-10T07:31:53.286957+0000","last_became_peered":"2024-02-10T07:31:53.286957+0000","last_unstale":"2024-02-10T07:33:00.565446+0000","last_undegraded":"2024-02-10T07:33:00.565446+0000","last_fullsized":"2024-02-10T07:33:00.565446+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:33:00.565384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:00.565384+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:05.401431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.100632+0000","last_change":"2024-02-10T07:31:57.313731+0000","last_active":"2024-02-10T07:32:47.100632+0000","last_peered":"2024-02-10T07:32:47.100632+0000","last_clean":"2024-02-10T07:32:47.100632+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:32:47.100632+0000","last_undegraded":"2024-02-10T07:32:47.100632+0000","last_fullsized":"2024-02-10T07:32:47.100632+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:16.885483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639717+0000","last_change":"2024-02-10T07:31:55.298018+0000","last_active":"2024-02-10T07:32:46.639717+0000","last_peered":"2024-02-10T07:32:46.639717+0000","last_clean":"2024-02-10T07:32:46.639717+0000","last_became_active":"2024-02-10T07:31:55.297715+0000","last_became_peered":"2024-02-10T07:31:55.297715+0000","last_unstale":"2024-02-10T07:32:46.639717+0000","last_undegraded":"2024-02-10T07:32:46.639717+0000","last_fullsized":"2024-02-10T07:32:46.639717+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.150692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:51.545567+0000","last_change":"2024-02-10T07:32:51.545597+0000","last_active":"2024-02-10T07:32:51.545567+0000","last_peered":"2024-02-10T07:32:51.545567+0000","last_clean":"2024-02-10T07:32:51.545567+0000","last_became_active":"2024-02-10T07:31:51.283229+0000","last_became_peered":"2024-02-10T07:31:51.283229+0000","last_unstale":"2024-02-10T07:32:51.545567+0000","last_undegraded":"2024-02-10T07:32:51.545567+0000","last_fullsized":"2024-02-10T07:32:51.545567+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:32:51.545493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:32:51.545493+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:17.023144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642023+0000","last_change":"2024-02-10T07:31:53.288142+0000","last_active":"2024-02-10T07:32:46.642023+0000","last_peered":"2024-02-10T07:32:46.642023+0000","last_clean":"2024-02-10T07:32:46.642023+0000","last_became_active":"2024-02-10T07:31:53.287858+0000","last_became_peered":"2024-02-10T07:31:53.287858+0000","last_unstale":"2024-02-10T07:32:46.642023+0000","last_undegraded":"2024-02-10T07:32:46.642023+0000","last_fullsized":"2024-02-10T07:32:46.642023+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:20.421670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639021+0000","last_change":"2024-02-10T07:31:57.310358+0000","last_active":"2024-02-10T07:32:46.639021+0000","last_peered":"2024-02-10T07:32:46.639021+0000","last_clean":"2024-02-10T07:32:46.639021+0000","last_became_active":"2024-02-10T07:31:57.310088+0000","last_became_peered":"2024-02-10T07:31:57.310088+0000","last_unstale":"2024-02-10T07:32:46.639021+0000","last_undegraded":"2024-02-10T07:32:46.639021+0000","last_fullsized":"2024-02-10T07:32:46.639021+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:23.164310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639188+0000","last_change":"2024-02-10T07:31:55.303127+0000","last_active":"2024-02-10T07:32:46.639188+0000","last_peered":"2024-02-10T07:32:46.639188+0000","last_clean":"2024-02-10T07:32:46.639188+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:32:46.639188+0000","last_undegraded":"2024-02-10T07:32:46.639188+0000","last_fullsized":"2024-02-10T07:32:46.639188+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:08.495702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641300+0000","last_change":"2024-02-10T07:31:51.276682+0000","last_active":"2024-02-10T07:32:46.641300+0000","last_peered":"2024-02-10T07:32:46.641300+0000","last_clean":"2024-02-10T07:32:46.641300+0000","last_became_active":"2024-02-10T07:31:51.276350+0000","last_became_peered":"2024-02-10T07:31:51.276350+0000","last_unstale":"2024-02-10T07:32:46.641300+0000","last_undegraded":"2024-02-10T07:32:46.641300+0000","last_fullsized":"2024-02-10T07:32:46.641300+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:11.602006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641535+0000","last_change":"2024-02-10T07:31:53.292840+0000","last_active":"2024-02-10T07:32:46.641535+0000","last_peered":"2024-02-10T07:32:46.641535+0000","last_clean":"2024-02-10T07:32:46.641535+0000","last_became_active":"2024-02-10T07:31:53.292417+0000","last_became_peered":"2024-02-10T07:31:53.292417+0000","last_unstale":"2024-02-10T07:32:46.641535+0000","last_undegraded":"2024-02-10T07:32:46.641535+0000","last_fullsized":"2024-02-10T07:32:46.641535+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:22.210604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:18.207493+0000","last_change":"2024-02-10T07:33:18.207530+0000","last_active":"2024-02-10T07:33:18.207493+0000","last_peered":"2024-02-10T07:33:18.207493+0000","last_clean":"2024-02-10T07:33:18.207493+0000","last_became_active":"2024-02-10T07:31:57.305613+0000","last_became_peered":"2024-02-10T07:31:57.305613+0000","last_unstale":"2024-02-10T07:33:18.207493+0000","last_undegraded":"2024-02-10T07:33:18.207493+0000","last_fullsized":"2024-02-10T07:33:18.207493+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:18.207406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:18.207406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:27.440541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:19.252448+0000","last_change":"2024-02-10T07:33:19.252470+0000","last_active":"2024-02-10T07:33:19.252448+0000","last_peered":"2024-02-10T07:33:19.252448+0000","last_clean":"2024-02-10T07:33:19.252448+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:33:19.252448+0000","last_undegraded":"2024-02-10T07:33:19.252448+0000","last_fullsized":"2024-02-10T07:33:19.252448+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:19.252386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:19.252386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:47.255925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101011+0000","last_change":"2024-02-10T07:31:51.287400+0000","last_active":"2024-02-10T07:32:47.101011+0000","last_peered":"2024-02-10T07:32:47.101011+0000","last_clean":"2024-02-10T07:32:47.101011+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:32:47.101011+0000","last_undegraded":"2024-02-10T07:32:47.101011+0000","last_fullsized":"2024-02-10T07:32:47.101011+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:17.607773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"89'22","reported_seq":90,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:14.170176+0000","last_change":"2024-02-10T07:33:14.170253+0000","last_active":"2024-02-10T07:33:14.170176+0000","last_peered":"2024-02-10T07:33:14.170176+0000","last_clean":"2024-02-10T07:33:14.170176+0000","last_became_active":"2024-02-10T07:31:53.295527+0000","last_became_peered":"2024-02-10T07:31:53.295527+0000","last_unstale":"2024-02-10T07:33:14.170176+0000","last_undegraded":"2024-02-10T07:33:14.170176+0000","last_fullsized":"2024-02-10T07:33:14.170176+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:33:14.170072+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:27.215444+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598595+0000","last_change":"2024-02-10T07:31:57.306606+0000","last_active":"2024-02-10T07:32:47.598595+0000","last_peered":"2024-02-10T07:32:47.598595+0000","last_clean":"2024-02-10T07:32:47.598595+0000","last_became_active":"2024-02-10T07:31:57.306315+0000","last_became_peered":"2024-02-10T07:31:57.306315+0000","last_unstale":"2024-02-10T07:32:47.598595+0000","last_undegraded":"2024-02-10T07:32:47.598595+0000","last_fullsized":"2024-02-10T07:32:47.598595+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:22.626690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641457+0000","last_change":"2024-02-10T07:31:55.298789+0000","last_active":"2024-02-10T07:32:46.641457+0000","last_peered":"2024-02-10T07:32:46.641457+0000","last_clean":"2024-02-10T07:32:46.641457+0000","last_became_active":"2024-02-10T07:31:55.298474+0000","last_became_peered":"2024-02-10T07:31:55.298474+0000","last_unstale":"2024-02-10T07:32:46.641457+0000","last_undegraded":"2024-02-10T07:32:46.641457+0000","last_fullsized":"2024-02-10T07:32:46.641457+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.404110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:02.198071+0000","last_change":"2024-02-10T07:33:02.198096+0000","last_active":"2024-02-10T07:33:02.198071+0000","last_peered":"2024-02-10T07:33:02.198071+0000","last_clean":"2024-02-10T07:33:02.198071+0000","last_became_active":"2024-02-10T07:31:51.321237+0000","last_became_peered":"2024-02-10T07:31:51.321237+0000","last_unstale":"2024-02-10T07:33:02.198071+0000","last_undegraded":"2024-02-10T07:33:02.198071+0000","last_fullsized":"2024-02-10T07:33:02.198071+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:02.197996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:02.197996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:08.678151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":82,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:13.556340+0000","last_change":"2024-02-10T07:33:13.556374+0000","last_active":"2024-02-10T07:33:13.556340+0000","last_peered":"2024-02-10T07:33:13.556340+0000","last_clean":"2024-02-10T07:33:13.556340+0000","last_became_active":"2024-02-10T07:31:53.294914+0000","last_became_peered":"2024-02-10T07:31:53.294914+0000","last_unstale":"2024-02-10T07:33:13.556340+0000","last_undegraded":"2024-02-10T07:33:13.556340+0000","last_fullsized":"2024-02-10T07:33:13.556340+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:33:13.556271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:13.556271+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:22.295459+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640318+0000","last_change":"2024-02-10T07:31:57.307172+0000","last_active":"2024-02-10T07:32:46.640318+0000","last_peered":"2024-02-10T07:32:46.640318+0000","last_clean":"2024-02-10T07:32:46.640318+0000","last_became_active":"2024-02-10T07:31:57.306892+0000","last_became_peered":"2024-02-10T07:31:57.306892+0000","last_unstale":"2024-02-10T07:32:46.640318+0000","last_undegraded":"2024-02-10T07:32:46.640318+0000","last_fullsized":"2024-02-10T07:32:46.640318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:15.856701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":84,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:17.787432+0000","last_change":"2024-02-10T07:33:17.787432+0000","last_active":"2024-02-10T07:33:17.787432+0000","last_peered":"2024-02-10T07:33:17.787432+0000","last_clean":"2024-02-10T07:33:17.787432+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:33:17.787432+0000","last_undegraded":"2024-02-10T07:33:17.787432+0000","last_fullsized":"2024-02-10T07:33:17.787432+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:33:17.787314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:17.787314+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:24.569254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:16.291248+0000","last_change":"2024-02-10T07:33:16.291275+0000","last_active":"2024-02-10T07:33:16.291248+0000","last_peered":"2024-02-10T07:33:16.291248+0000","last_clean":"2024-02-10T07:33:16.291248+0000","last_became_active":"2024-02-10T07:31:51.282077+0000","last_became_peered":"2024-02-10T07:31:51.282077+0000","last_unstale":"2024-02-10T07:33:16.291248+0000","last_undegraded":"2024-02-10T07:33:16.291248+0000","last_fullsized":"2024-02-10T07:33:16.291248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:33:16.291171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:16.291171+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:24.579644+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:03.462205+0000","last_change":"2024-02-10T07:33:03.462253+0000","last_active":"2024-02-10T07:33:03.462205+0000","last_peered":"2024-02-10T07:33:03.462205+0000","last_clean":"2024-02-10T07:33:03.462205+0000","last_became_active":"2024-02-10T07:31:53.292936+0000","last_became_peered":"2024-02-10T07:31:53.292936+0000","last_unstale":"2024-02-10T07:33:03.462205+0000","last_undegraded":"2024-02-10T07:33:03.462205+0000","last_fullsized":"2024-02-10T07:33:03.462205+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:33:03.462109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:03.462109+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:31.967818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:18.242863+0000","last_change":"2024-02-10T07:33:18.242898+0000","last_active":"2024-02-10T07:33:18.242863+0000","last_peered":"2024-02-10T07:33:18.242863+0000","last_clean":"2024-02-10T07:33:18.242863+0000","last_became_active":"2024-02-10T07:31:57.308820+0000","last_became_peered":"2024-02-10T07:31:57.308820+0000","last_unstale":"2024-02-10T07:33:18.242863+0000","last_undegraded":"2024-02-10T07:33:18.242863+0000","last_fullsized":"2024-02-10T07:33:18.242863+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:18.242759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:18.242759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:31.207474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101652+0000","last_change":"2024-02-10T07:31:55.297952+0000","last_active":"2024-02-10T07:32:47.101652+0000","last_peered":"2024-02-10T07:32:47.101652+0000","last_clean":"2024-02-10T07:32:47.101652+0000","last_became_active":"2024-02-10T07:31:55.297595+0000","last_became_peered":"2024-02-10T07:31:55.297595+0000","last_unstale":"2024-02-10T07:32:47.101652+0000","last_undegraded":"2024-02-10T07:32:47.101652+0000","last_fullsized":"2024-02-10T07:32:47.101652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.915423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:16.558680+0000","last_change":"2024-02-10T07:33:16.558719+0000","last_active":"2024-02-10T07:33:16.558680+0000","last_peered":"2024-02-10T07:33:16.558680+0000","last_clean":"2024-02-10T07:33:16.558680+0000","last_became_active":"2024-02-10T07:31:51.285405+0000","last_became_peered":"2024-02-10T07:31:51.285405+0000","last_unstale":"2024-02-10T07:33:16.558680+0000","last_undegraded":"2024-02-10T07:33:16.558680+0000","last_fullsized":"2024-02-10T07:33:16.558680+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:16.558593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:16.558593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:21.969603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":77,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:00.861580+0000","last_change":"2024-02-10T07:33:00.861634+0000","last_active":"2024-02-10T07:33:00.861580+0000","last_peered":"2024-02-10T07:33:00.861580+0000","last_clean":"2024-02-10T07:33:00.861580+0000","last_became_active":"2024-02-10T07:31:53.300871+0000","last_became_peered":"2024-02-10T07:31:53.300871+0000","last_unstale":"2024-02-10T07:33:00.861580+0000","last_undegraded":"2024-02-10T07:33:00.861580+0000","last_fullsized":"2024-02-10T07:33:00.861580+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:33:00.861482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:00.861482+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:09.147490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639149+0000","last_change":"2024-02-10T07:31:57.307069+0000","last_active":"2024-02-10T07:32:46.639149+0000","last_peered":"2024-02-10T07:32:46.639149+0000","last_clean":"2024-02-10T07:32:46.639149+0000","last_became_active":"2024-02-10T07:31:57.306776+0000","last_became_peered":"2024-02-10T07:31:57.306776+0000","last_unstale":"2024-02-10T07:32:46.639149+0000","last_undegraded":"2024-02-10T07:32:46.639149+0000","last_fullsized":"2024-02-10T07:32:46.639149+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:09.159262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:13.822410+0000","last_change":"2024-02-10T07:33:13.822432+0000","last_active":"2024-02-10T07:33:13.822410+0000","last_peered":"2024-02-10T07:33:13.822410+0000","last_clean":"2024-02-10T07:33:13.822410+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:33:13.822410+0000","last_undegraded":"2024-02-10T07:33:13.822410+0000","last_fullsized":"2024-02-10T07:33:13.822410+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:13.822335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:13.822335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:35.258891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:11.581128+0000","last_change":"2024-02-10T07:33:11.581151+0000","last_active":"2024-02-10T07:33:11.581128+0000","last_peered":"2024-02-10T07:33:11.581128+0000","last_clean":"2024-02-10T07:33:11.581128+0000","last_became_active":"2024-02-10T07:31:51.284022+0000","last_became_peered":"2024-02-10T07:31:51.284022+0000","last_unstale":"2024-02-10T07:33:11.581128+0000","last_undegraded":"2024-02-10T07:33:11.581128+0000","last_fullsized":"2024-02-10T07:33:11.581128+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:11.581063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:11.581063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:32.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597927+0000","last_change":"2024-02-10T07:31:53.295829+0000","last_active":"2024-02-10T07:32:47.597927+0000","last_peered":"2024-02-10T07:32:47.597927+0000","last_clean":"2024-02-10T07:32:47.597927+0000","last_became_active":"2024-02-10T07:31:53.295520+0000","last_became_peered":"2024-02-10T07:31:53.295520+0000","last_unstale":"2024-02-10T07:32:47.597927+0000","last_undegraded":"2024-02-10T07:32:47.597927+0000","last_fullsized":"2024-02-10T07:32:47.597927+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.493903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598100+0000","last_change":"2024-02-10T07:31:57.315994+0000","last_active":"2024-02-10T07:32:47.598100+0000","last_peered":"2024-02-10T07:32:47.598100+0000","last_clean":"2024-02-10T07:32:47.598100+0000","last_became_active":"2024-02-10T07:31:57.315738+0000","last_became_peered":"2024-02-10T07:31:57.315738+0000","last_unstale":"2024-02-10T07:32:47.598100+0000","last_undegraded":"2024-02-10T07:32:47.598100+0000","last_fullsized":"2024-02-10T07:32:47.598100+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:07.663297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642118+0000","last_change":"2024-02-10T07:31:55.302310+0000","last_active":"2024-02-10T07:32:46.642118+0000","last_peered":"2024-02-10T07:32:46.642118+0000","last_clean":"2024-02-10T07:32:46.642118+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:32:46.642118+0000","last_undegraded":"2024-02-10T07:32:46.642118+0000","last_fullsized":"2024-02-10T07:32:46.642118+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:05.175419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:53.470063+0000","last_change":"2024-02-10T07:32:53.470087+0000","last_active":"2024-02-10T07:32:53.470063+0000","last_peered":"2024-02-10T07:32:53.470063+0000","last_clean":"2024-02-10T07:32:53.470063+0000","last_became_active":"2024-02-10T07:31:51.279438+0000","last_became_peered":"2024-02-10T07:31:51.279438+0000","last_unstale":"2024-02-10T07:32:53.470063+0000","last_undegraded":"2024-02-10T07:32:53.470063+0000","last_fullsized":"2024-02-10T07:32:53.470063+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:32:53.469958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:00.926321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:58.537802+0000","last_change":"2024-02-10T07:32:58.537831+0000","last_active":"2024-02-10T07:32:58.537802+0000","last_peered":"2024-02-10T07:32:58.537802+0000","last_clean":"2024-02-10T07:32:58.537802+0000","last_became_active":"2024-02-10T07:31:53.287823+0000","last_became_peered":"2024-02-10T07:31:53.287823+0000","last_unstale":"2024-02-10T07:32:58.537802+0000","last_undegraded":"2024-02-10T07:32:58.537802+0000","last_fullsized":"2024-02-10T07:32:58.537802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:32:58.537746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.537746+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:07.841413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.100440+0000","last_change":"2024-02-10T07:31:57.313822+0000","last_active":"2024-02-10T07:32:47.100440+0000","last_peered":"2024-02-10T07:32:47.100440+0000","last_clean":"2024-02-10T07:32:47.100440+0000","last_became_active":"2024-02-10T07:31:57.313529+0000","last_became_peered":"2024-02-10T07:31:57.313529+0000","last_unstale":"2024-02-10T07:32:47.100440+0000","last_undegraded":"2024-02-10T07:32:47.100440+0000","last_fullsized":"2024-02-10T07:32:47.100440+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:23.816714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:59.584337+0000","last_change":"2024-02-10T07:32:59.584373+0000","last_active":"2024-02-10T07:32:59.584337+0000","last_peered":"2024-02-10T07:32:59.584337+0000","last_clean":"2024-02-10T07:32:59.584337+0000","last_became_active":"2024-02-10T07:31:55.296670+0000","last_became_peered":"2024-02-10T07:31:55.296670+0000","last_unstale":"2024-02-10T07:32:59.584337+0000","last_undegraded":"2024-02-10T07:32:59.584337+0000","last_fullsized":"2024-02-10T07:32:59.584337+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:59.584219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:59.584219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:22.310522+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:18.329943+0000","last_change":"2024-02-10T07:33:18.329963+0000","last_active":"2024-02-10T07:33:18.329943+0000","last_peered":"2024-02-10T07:33:18.329943+0000","last_clean":"2024-02-10T07:33:18.329943+0000","last_became_active":"2024-02-10T07:31:51.288165+0000","last_became_peered":"2024-02-10T07:31:51.288165+0000","last_unstale":"2024-02-10T07:33:18.329943+0000","last_undegraded":"2024-02-10T07:33:18.329943+0000","last_fullsized":"2024-02-10T07:33:18.329943+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:18.329878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:18.329878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:37.623492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:21.268099+0000","last_change":"2024-02-10T07:33:21.268130+0000","last_active":"2024-02-10T07:33:21.268099+0000","last_peered":"2024-02-10T07:33:21.268099+0000","last_clean":"2024-02-10T07:33:21.268099+0000","last_became_active":"2024-02-10T07:31:53.292182+0000","last_became_peered":"2024-02-10T07:31:53.292182+0000","last_unstale":"2024-02-10T07:33:21.268099+0000","last_undegraded":"2024-02-10T07:33:21.268099+0000","last_fullsized":"2024-02-10T07:33:21.268099+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:33:21.268030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:21.268030+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:45.065389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597842+0000","last_change":"2024-02-10T07:31:57.316162+0000","last_active":"2024-02-10T07:32:47.597842+0000","last_peered":"2024-02-10T07:32:47.597842+0000","last_clean":"2024-02-10T07:32:47.597842+0000","last_became_active":"2024-02-10T07:31:57.315781+0000","last_became_peered":"2024-02-10T07:31:57.315781+0000","last_unstale":"2024-02-10T07:32:47.597842+0000","last_undegraded":"2024-02-10T07:32:47.597842+0000","last_fullsized":"2024-02-10T07:32:47.597842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:08.947342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694902+0000","last_change":"2024-02-10T07:31:55.301556+0000","last_active":"2024-02-10T07:32:47.694902+0000","last_peered":"2024-02-10T07:32:47.694902+0000","last_clean":"2024-02-10T07:32:47.694902+0000","last_became_active":"2024-02-10T07:31:55.301260+0000","last_became_peered":"2024-02-10T07:31:55.301260+0000","last_unstale":"2024-02-10T07:32:47.694902+0000","last_undegraded":"2024-02-10T07:32:47.694902+0000","last_fullsized":"2024-02-10T07:32:47.694902+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:17.634914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:22.333901+0000","last_change":"2024-02-10T07:33:22.333921+0000","last_active":"2024-02-10T07:33:22.333901+0000","last_peered":"2024-02-10T07:33:22.333901+0000","last_clean":"2024-02-10T07:33:22.333901+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:33:22.333901+0000","last_undegraded":"2024-02-10T07:33:22.333901+0000","last_fullsized":"2024-02-10T07:33:22.333901+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:22.333839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:22.333839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:28.687307+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":96,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:00.143959+0000","last_change":"2024-02-10T07:33:00.144015+0000","last_active":"2024-02-10T07:33:00.143959+0000","last_peered":"2024-02-10T07:33:00.143959+0000","last_clean":"2024-02-10T07:33:00.143959+0000","last_became_active":"2024-02-10T07:32:18.389564+0000","last_became_peered":"2024-02-10T07:32:18.389564+0000","last_unstale":"2024-02-10T07:33:00.143959+0000","last_undegraded":"2024-02-10T07:33:00.143959+0000","last_fullsized":"2024-02-10T07:33:00.143959+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:33:00.143859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:00.143859+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:28.135685+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:22.616384+0000","last_change":"2024-02-10T07:33:22.616403+0000","last_active":"2024-02-10T07:33:22.616384+0000","last_peered":"2024-02-10T07:33:22.616384+0000","last_clean":"2024-02-10T07:33:22.616384+0000","last_became_active":"2024-02-10T07:31:57.318009+0000","last_became_peered":"2024-02-10T07:31:57.318009+0000","last_unstale":"2024-02-10T07:33:22.616384+0000","last_undegraded":"2024-02-10T07:33:22.616384+0000","last_fullsized":"2024-02-10T07:33:22.616384+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:22.616327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:22.616327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:36.984762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:16.374847+0000","last_change":"2024-02-10T07:33:15.216416+0000","last_active":"2024-02-10T07:33:16.374847+0000","last_peered":"2024-02-10T07:33:16.374847+0000","last_clean":"2024-02-10T07:33:16.374847+0000","last_became_active":"2024-02-10T07:31:55.297670+0000","last_became_peered":"2024-02-10T07:31:55.297670+0000","last_unstale":"2024-02-10T07:33:16.374847+0000","last_undegraded":"2024-02-10T07:33:16.374847+0000","last_fullsized":"2024-02-10T07:33:16.374847+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:33:15.216295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:15.216295+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:35.214128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.348418+0000","last_change":"2024-02-10T07:33:06.348460+0000","last_active":"2024-02-10T07:33:06.348418+0000","last_peered":"2024-02-10T07:33:06.348418+0000","last_clean":"2024-02-10T07:33:06.348418+0000","last_became_active":"2024-02-10T07:31:51.320644+0000","last_became_peered":"2024-02-10T07:31:51.320644+0000","last_unstale":"2024-02-10T07:33:06.348418+0000","last_undegraded":"2024-02-10T07:33:06.348418+0000","last_fullsized":"2024-02-10T07:33:06.348418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.348322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:15.227477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598312+0000","last_change":"2024-02-10T07:31:53.296048+0000","last_active":"2024-02-10T07:32:47.598312+0000","last_peered":"2024-02-10T07:32:47.598312+0000","last_clean":"2024-02-10T07:32:47.598312+0000","last_became_active":"2024-02-10T07:31:53.295641+0000","last_became_peered":"2024-02-10T07:31:53.295641+0000","last_unstale":"2024-02-10T07:32:47.598312+0000","last_undegraded":"2024-02-10T07:32:47.598312+0000","last_fullsized":"2024-02-10T07:32:47.598312+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:17.035054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642477+0000","last_change":"2024-02-10T07:31:57.307866+0000","last_active":"2024-02-10T07:32:46.642477+0000","last_peered":"2024-02-10T07:32:46.642477+0000","last_clean":"2024-02-10T07:32:46.642477+0000","last_became_active":"2024-02-10T07:31:57.307583+0000","last_became_peered":"2024-02-10T07:31:57.307583+0000","last_unstale":"2024-02-10T07:32:46.642477+0000","last_undegraded":"2024-02-10T07:32:46.642477+0000","last_fullsized":"2024-02-10T07:32:46.642477+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:25.642236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:16.374657+0000","last_change":"2024-02-10T07:31:55.294941+0000","last_active":"2024-02-10T07:33:16.374657+0000","last_peered":"2024-02-10T07:33:16.374657+0000","last_clean":"2024-02-10T07:33:16.374657+0000","last_became_active":"2024-02-10T07:31:55.294594+0000","last_became_peered":"2024-02-10T07:31:55.294594+0000","last_unstale":"2024-02-10T07:33:16.374657+0000","last_undegraded":"2024-02-10T07:33:16.374657+0000","last_fullsized":"2024-02-10T07:33:16.374657+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.496713+0000","last_change":"2024-02-10T07:33:01.496748+0000","last_active":"2024-02-10T07:33:01.496713+0000","last_peered":"2024-02-10T07:33:01.496713+0000","last_clean":"2024-02-10T07:33:01.496713+0000","last_became_active":"2024-02-10T07:31:51.282353+0000","last_became_peered":"2024-02-10T07:31:51.282353+0000","last_unstale":"2024-02-10T07:33:01.496713+0000","last_undegraded":"2024-02-10T07:33:01.496713+0000","last_fullsized":"2024-02-10T07:33:01.496713+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:01.496610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:01.496610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:02.323527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:09.243572+0000","last_change":"2024-02-10T07:33:09.243608+0000","last_active":"2024-02-10T07:33:09.243572+0000","last_peered":"2024-02-10T07:33:09.243572+0000","last_clean":"2024-02-10T07:33:09.243572+0000","last_became_active":"2024-02-10T07:31:53.285777+0000","last_became_peered":"2024-02-10T07:31:53.285777+0000","last_unstale":"2024-02-10T07:33:09.243572+0000","last_undegraded":"2024-02-10T07:33:09.243572+0000","last_fullsized":"2024-02-10T07:33:09.243572+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:33:09.243500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:09.243500+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:37.334914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:10.120521+0000","last_change":"2024-02-10T07:33:10.120541+0000","last_active":"2024-02-10T07:33:10.120521+0000","last_peered":"2024-02-10T07:33:10.120521+0000","last_clean":"2024-02-10T07:33:10.120521+0000","last_became_active":"2024-02-10T07:31:57.308615+0000","last_became_peered":"2024-02-10T07:31:57.308615+0000","last_unstale":"2024-02-10T07:33:10.120521+0000","last_undegraded":"2024-02-10T07:33:10.120521+0000","last_fullsized":"2024-02-10T07:33:10.120521+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:33:10.120462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:13.165236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:17.587517+0000","last_change":"2024-02-10T07:33:17.587557+0000","last_active":"2024-02-10T07:33:17.587517+0000","last_peered":"2024-02-10T07:33:17.587517+0000","last_clean":"2024-02-10T07:33:17.587517+0000","last_became_active":"2024-02-10T07:31:55.297431+0000","last_became_peered":"2024-02-10T07:31:55.297431+0000","last_unstale":"2024-02-10T07:33:17.587517+0000","last_undegraded":"2024-02-10T07:33:17.587517+0000","last_fullsized":"2024-02-10T07:33:17.587517+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:33:17.587433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:40.484002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639342+0000","last_change":"2024-02-10T07:31:51.284304+0000","last_active":"2024-02-10T07:32:46.639342+0000","last_peered":"2024-02-10T07:32:46.639342+0000","last_clean":"2024-02-10T07:32:46.639342+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:32:46.639342+0000","last_undegraded":"2024-02-10T07:32:46.639342+0000","last_fullsized":"2024-02-10T07:32:46.639342+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:02.877266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"89'13","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:56.585009+0000","last_change":"2024-02-10T07:32:56.585037+0000","last_active":"2024-02-10T07:32:56.585009+0000","last_peered":"2024-02-10T07:32:56.585009+0000","last_clean":"2024-02-10T07:32:56.585009+0000","last_became_active":"2024-02-10T07:31:53.295353+0000","last_became_peered":"2024-02-10T07:31:53.295353+0000","last_unstale":"2024-02-10T07:32:56.585009+0000","last_undegraded":"2024-02-10T07:32:56.585009+0000","last_fullsized":"2024-02-10T07:32:56.585009+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:32:56.584957+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:04.077539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640268+0000","last_change":"2024-02-10T07:31:57.305113+0000","last_active":"2024-02-10T07:32:46.640268+0000","last_peered":"2024-02-10T07:32:46.640268+0000","last_clean":"2024-02-10T07:32:46.640268+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:32:46.640268+0000","last_undegraded":"2024-02-10T07:32:46.640268+0000","last_fullsized":"2024-02-10T07:32:46.640268+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:11.938578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641217+0000","last_change":"2024-02-10T07:31:55.296699+0000","last_active":"2024-02-10T07:32:46.641217+0000","last_peered":"2024-02-10T07:32:46.641217+0000","last_clean":"2024-02-10T07:32:46.641217+0000","last_became_active":"2024-02-10T07:31:55.296369+0000","last_became_peered":"2024-02-10T07:31:55.296369+0000","last_unstale":"2024-02-10T07:32:46.641217+0000","last_undegraded":"2024-02-10T07:32:46.641217+0000","last_fullsized":"2024-02-10T07:32:46.641217+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.519413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:09.871264+0000","last_change":"2024-02-10T07:33:09.871286+0000","last_active":"2024-02-10T07:33:09.871264+0000","last_peered":"2024-02-10T07:33:09.871264+0000","last_clean":"2024-02-10T07:33:09.871264+0000","last_became_active":"2024-02-10T07:31:51.285159+0000","last_became_peered":"2024-02-10T07:31:51.285159+0000","last_unstale":"2024-02-10T07:33:09.871264+0000","last_undegraded":"2024-02-10T07:33:09.871264+0000","last_fullsized":"2024-02-10T07:33:09.871264+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:09.871194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:09.871194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:38.560272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598658+0000","last_change":"2024-02-10T07:31:53.297115+0000","last_active":"2024-02-10T07:32:47.598658+0000","last_peered":"2024-02-10T07:32:47.598658+0000","last_clean":"2024-02-10T07:32:47.598658+0000","last_became_active":"2024-02-10T07:31:53.296831+0000","last_became_peered":"2024-02-10T07:31:53.296831+0000","last_unstale":"2024-02-10T07:32:47.598658+0000","last_undegraded":"2024-02-10T07:32:47.598658+0000","last_fullsized":"2024-02-10T07:32:47.598658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:21.861465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639876+0000","last_change":"2024-02-10T07:31:57.311522+0000","last_active":"2024-02-10T07:32:46.639876+0000","last_peered":"2024-02-10T07:32:46.639876+0000","last_clean":"2024-02-10T07:32:46.639876+0000","last_became_active":"2024-02-10T07:31:57.311110+0000","last_became_peered":"2024-02-10T07:31:57.311110+0000","last_unstale":"2024-02-10T07:32:46.639876+0000","last_undegraded":"2024-02-10T07:32:46.639876+0000","last_fullsized":"2024-02-10T07:32:46.639876+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:25.982197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640304+0000","last_change":"2024-02-10T07:31:51.286209+0000","last_active":"2024-02-10T07:32:46.640304+0000","last_peered":"2024-02-10T07:32:46.640304+0000","last_clean":"2024-02-10T07:32:46.640304+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:32:46.640304+0000","last_undegraded":"2024-02-10T07:32:46.640304+0000","last_fullsized":"2024-02-10T07:32:46.640304+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:08.757933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:59.385507+0000","last_change":"2024-02-10T07:32:59.385539+0000","last_active":"2024-02-10T07:32:59.385507+0000","last_peered":"2024-02-10T07:32:59.385507+0000","last_clean":"2024-02-10T07:32:59.385507+0000","last_became_active":"2024-02-10T07:31:53.284817+0000","last_became_peered":"2024-02-10T07:31:53.284817+0000","last_unstale":"2024-02-10T07:32:59.385507+0000","last_undegraded":"2024-02-10T07:32:59.385507+0000","last_fullsized":"2024-02-10T07:32:59.385507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:32:59.385448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:59.385448+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:59.831143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:16.374862+0000","last_change":"2024-02-10T07:31:55.301606+0000","last_active":"2024-02-10T07:33:16.374862+0000","last_peered":"2024-02-10T07:33:16.374862+0000","last_clean":"2024-02-10T07:33:16.374862+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:33:16.374862+0000","last_undegraded":"2024-02-10T07:33:16.374862+0000","last_fullsized":"2024-02-10T07:33:16.374862+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598067+0000","last_change":"2024-02-10T07:31:57.315186+0000","last_active":"2024-02-10T07:32:47.598067+0000","last_peered":"2024-02-10T07:32:47.598067+0000","last_clean":"2024-02-10T07:32:47.598067+0000","last_became_active":"2024-02-10T07:31:57.314872+0000","last_became_peered":"2024-02-10T07:31:57.314872+0000","last_unstale":"2024-02-10T07:32:47.598067+0000","last_undegraded":"2024-02-10T07:32:47.598067+0000","last_fullsized":"2024-02-10T07:32:47.598067+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:10.772524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640466+0000","last_change":"2024-02-10T07:31:51.287017+0000","last_active":"2024-02-10T07:32:46.640466+0000","last_peered":"2024-02-10T07:32:46.640466+0000","last_clean":"2024-02-10T07:32:46.640466+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:32:46.640466+0000","last_undegraded":"2024-02-10T07:32:46.640466+0000","last_fullsized":"2024-02-10T07:32:46.640466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:05.325144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":92,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640700+0000","last_change":"2024-02-10T07:32:18.396325+0000","last_active":"2024-02-10T07:32:46.640700+0000","last_peered":"2024-02-10T07:32:46.640700+0000","last_clean":"2024-02-10T07:32:46.640700+0000","last_became_active":"2024-02-10T07:32:18.389409+0000","last_became_peered":"2024-02-10T07:32:18.389409+0000","last_unstale":"2024-02-10T07:32:46.640700+0000","last_undegraded":"2024-02-10T07:32:46.640700+0000","last_fullsized":"2024-02-10T07:32:46.640700+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:07.708054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:59.218979+0000","last_change":"2024-02-10T07:32:59.218999+0000","last_active":"2024-02-10T07:32:59.218979+0000","last_peered":"2024-02-10T07:32:59.218979+0000","last_clean":"2024-02-10T07:32:59.218979+0000","last_became_active":"2024-02-10T07:31:55.303059+0000","last_became_peered":"2024-02-10T07:31:55.303059+0000","last_unstale":"2024-02-10T07:32:59.218979+0000","last_undegraded":"2024-02-10T07:32:59.218979+0000","last_fullsized":"2024-02-10T07:32:59.218979+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:59.218908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:59.218908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:06.437654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:21.361055+0000","last_change":"2024-02-10T07:33:21.361093+0000","last_active":"2024-02-10T07:33:21.361055+0000","last_peered":"2024-02-10T07:33:21.361055+0000","last_clean":"2024-02-10T07:33:21.361055+0000","last_became_active":"2024-02-10T07:31:57.304971+0000","last_became_peered":"2024-02-10T07:31:57.304971+0000","last_unstale":"2024-02-10T07:33:21.361055+0000","last_undegraded":"2024-02-10T07:33:21.361055+0000","last_fullsized":"2024-02-10T07:33:21.361055+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:21.360834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:21.360834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:29.470777+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:16.199527+0000","last_change":"2024-02-10T07:33:16.199553+0000","last_active":"2024-02-10T07:33:16.199527+0000","last_peered":"2024-02-10T07:33:16.199527+0000","last_clean":"2024-02-10T07:33:16.199527+0000","last_became_active":"2024-02-10T07:31:55.298322+0000","last_became_peered":"2024-02-10T07:31:55.298322+0000","last_unstale":"2024-02-10T07:33:16.199527+0000","last_undegraded":"2024-02-10T07:33:16.199527+0000","last_fullsized":"2024-02-10T07:33:16.199527+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:16.199447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:16.199447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:27.934282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641102+0000","last_change":"2024-02-10T07:31:51.283777+0000","last_active":"2024-02-10T07:32:46.641102+0000","last_peered":"2024-02-10T07:32:46.641102+0000","last_clean":"2024-02-10T07:32:46.641102+0000","last_became_active":"2024-02-10T07:31:51.283433+0000","last_became_peered":"2024-02-10T07:31:51.283433+0000","last_unstale":"2024-02-10T07:32:46.641102+0000","last_undegraded":"2024-02-10T07:32:46.641102+0000","last_fullsized":"2024-02-10T07:32:46.641102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:02.188930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:16.799167+0000","last_change":"2024-02-10T07:33:16.799197+0000","last_active":"2024-02-10T07:33:16.799167+0000","last_peered":"2024-02-10T07:33:16.799167+0000","last_clean":"2024-02-10T07:33:16.799167+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:33:16.799167+0000","last_undegraded":"2024-02-10T07:33:16.799167+0000","last_fullsized":"2024-02-10T07:33:16.799167+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-10T07:33:16.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:16.799093+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:16.832132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:14.610191+0000","last_change":"2024-02-10T07:33:14.610211+0000","last_active":"2024-02-10T07:33:14.610191+0000","last_peered":"2024-02-10T07:33:14.610191+0000","last_clean":"2024-02-10T07:33:14.610191+0000","last_became_active":"2024-02-10T07:31:57.313394+0000","last_became_peered":"2024-02-10T07:31:57.313394+0000","last_unstale":"2024-02-10T07:33:14.610191+0000","last_undegraded":"2024-02-10T07:33:14.610191+0000","last_fullsized":"2024-02-10T07:33:14.610191+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:14.610134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:14.610134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:20.009681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:13.158121+0000","last_change":"2024-02-10T07:33:13.158162+0000","last_active":"2024-02-10T07:33:13.158121+0000","last_peered":"2024-02-10T07:33:13.158121+0000","last_clean":"2024-02-10T07:33:13.158121+0000","last_became_active":"2024-02-10T07:31:55.298622+0000","last_became_peered":"2024-02-10T07:31:55.298622+0000","last_unstale":"2024-02-10T07:33:13.158121+0000","last_undegraded":"2024-02-10T07:33:13.158121+0000","last_fullsized":"2024-02-10T07:33:13.158121+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:13.157988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:13.157988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:36.826105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:19.291618+0000","last_change":"2024-02-10T07:33:19.291637+0000","last_active":"2024-02-10T07:33:19.291618+0000","last_peered":"2024-02-10T07:33:19.291618+0000","last_clean":"2024-02-10T07:33:19.291618+0000","last_became_active":"2024-02-10T07:31:51.283208+0000","last_became_peered":"2024-02-10T07:31:51.283208+0000","last_unstale":"2024-02-10T07:33:19.291618+0000","last_undegraded":"2024-02-10T07:33:19.291618+0000","last_fullsized":"2024-02-10T07:33:19.291618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:19.291556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:19.291556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:45.842544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:18.612915+0000","last_change":"2024-02-10T07:33:18.612946+0000","last_active":"2024-02-10T07:33:18.612915+0000","last_peered":"2024-02-10T07:33:18.612915+0000","last_clean":"2024-02-10T07:33:18.612915+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:33:18.612915+0000","last_undegraded":"2024-02-10T07:33:18.612915+0000","last_fullsized":"2024-02-10T07:33:18.612915+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:33:18.612847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:18.612847+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:39.762053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.598514+0000","last_change":"2024-02-10T07:31:57.314251+0000","last_active":"2024-02-10T07:32:47.598514+0000","last_peered":"2024-02-10T07:32:47.598514+0000","last_clean":"2024-02-10T07:32:47.598514+0000","last_became_active":"2024-02-10T07:31:57.313854+0000","last_became_peered":"2024-02-10T07:31:57.313854+0000","last_unstale":"2024-02-10T07:32:47.598514+0000","last_undegraded":"2024-02-10T07:32:47.598514+0000","last_fullsized":"2024-02-10T07:32:47.598514+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.959268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:12.268090+0000","last_change":"2024-02-10T07:33:12.268121+0000","last_active":"2024-02-10T07:33:12.268090+0000","last_peered":"2024-02-10T07:33:12.268090+0000","last_clean":"2024-02-10T07:33:12.268090+0000","last_became_active":"2024-02-10T07:31:50.278548+0000","last_became_peered":"2024-02-10T07:31:50.278548+0000","last_unstale":"2024-02-10T07:33:12.268090+0000","last_undegraded":"2024-02-10T07:33:12.268090+0000","last_fullsized":"2024-02-10T07:33:12.268090+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:33:12.268020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:33:12.268020+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:13.552959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001361225,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641171+0000","last_change":"2024-02-10T07:31:51.285951+0000","last_active":"2024-02-10T07:32:46.641171+0000","last_peered":"2024-02-10T07:32:46.641171+0000","last_clean":"2024-02-10T07:32:46.641171+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:32:46.641171+0000","last_undegraded":"2024-02-10T07:32:46.641171+0000","last_fullsized":"2024-02-10T07:32:46.641171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:13.565080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":124,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:16.374609+0000","last_change":"2024-02-10T07:33:09.271524+0000","last_active":"2024-02-10T07:33:16.374609+0000","last_peered":"2024-02-10T07:33:16.374609+0000","last_clean":"2024-02-10T07:33:16.374609+0000","last_became_active":"2024-02-10T07:31:53.301863+0000","last_became_peered":"2024-02-10T07:31:53.301863+0000","last_unstale":"2024-02-10T07:33:16.374609+0000","last_undegraded":"2024-02-10T07:33:16.374609+0000","last_fullsized":"2024-02-10T07:33:16.374609+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-02-10T07:33:09.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:09.271402+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:33.250857+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:17.587727+0000","last_change":"2024-02-10T07:33:17.587762+0000","last_active":"2024-02-10T07:33:17.587727+0000","last_peered":"2024-02-10T07:33:17.587727+0000","last_clean":"2024-02-10T07:33:17.587727+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:33:17.587727+0000","last_undegraded":"2024-02-10T07:33:17.587727+0000","last_fullsized":"2024-02-10T07:33:17.587727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:17.587649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:17.587649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:30.335639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:58.354107+0000","last_change":"2024-02-10T07:32:58.354147+0000","last_active":"2024-02-10T07:32:58.354107+0000","last_peered":"2024-02-10T07:32:58.354107+0000","last_clean":"2024-02-10T07:32:58.354107+0000","last_became_active":"2024-02-10T07:31:57.299958+0000","last_became_peered":"2024-02-10T07:31:57.299958+0000","last_unstale":"2024-02-10T07:32:58.354107+0000","last_undegraded":"2024-02-10T07:32:58.354107+0000","last_fullsized":"2024-02-10T07:32:58.354107+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:58.354022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.354022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:17.659362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:03.198766+0000","last_change":"2024-02-10T07:33:03.198787+0000","last_active":"2024-02-10T07:33:03.198766+0000","last_peered":"2024-02-10T07:33:03.198766+0000","last_clean":"2024-02-10T07:33:03.198766+0000","last_became_active":"2024-02-10T07:31:51.320864+0000","last_became_peered":"2024-02-10T07:31:51.320864+0000","last_unstale":"2024-02-10T07:33:03.198766+0000","last_undegraded":"2024-02-10T07:33:03.198766+0000","last_fullsized":"2024-02-10T07:33:03.198766+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:03.198699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:03.198699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:13.704299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:58.209366+0000","last_change":"2024-02-10T07:32:58.209386+0000","last_active":"2024-02-10T07:32:58.209366+0000","last_peered":"2024-02-10T07:32:58.209366+0000","last_clean":"2024-02-10T07:32:58.209366+0000","last_became_active":"2024-02-10T07:31:50.281517+0000","last_became_peered":"2024-02-10T07:31:50.281517+0000","last_unstale":"2024-02-10T07:32:58.209366+0000","last_undegraded":"2024-02-10T07:32:58.209366+0000","last_fullsized":"2024-02-10T07:32:58.209366+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:58.209303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.209303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:28.009863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":52,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:17.180201+0000","last_change":"2024-02-10T07:33:17.180228+0000","last_active":"2024-02-10T07:33:17.180201+0000","last_peered":"2024-02-10T07:33:17.180201+0000","last_clean":"2024-02-10T07:33:17.180201+0000","last_became_active":"2024-02-10T07:32:18.391217+0000","last_became_peered":"2024-02-10T07:32:18.391217+0000","last_unstale":"2024-02-10T07:33:17.180201+0000","last_undegraded":"2024-02-10T07:33:17.180201+0000","last_fullsized":"2024-02-10T07:33:17.180201+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-10T07:33:17.180142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:17.180142+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:40.010786+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.183383+0000","last_change":"2024-02-10T07:33:06.183420+0000","last_active":"2024-02-10T07:33:06.183383+0000","last_peered":"2024-02-10T07:33:06.183383+0000","last_clean":"2024-02-10T07:33:06.183383+0000","last_became_active":"2024-02-10T07:31:55.298933+0000","last_became_peered":"2024-02-10T07:31:55.298933+0000","last_unstale":"2024-02-10T07:33:06.183383+0000","last_undegraded":"2024-02-10T07:33:06.183383+0000","last_fullsized":"2024-02-10T07:33:06.183383+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.183292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.183292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:12.631994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"74'1","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639813+0000","last_change":"2024-02-10T07:31:57.310237+0000","last_active":"2024-02-10T07:32:46.639813+0000","last_peered":"2024-02-10T07:32:46.639813+0000","last_clean":"2024-02-10T07:32:46.639813+0000","last_became_active":"2024-02-10T07:31:57.309929+0000","last_became_peered":"2024-02-10T07:31:57.309929+0000","last_unstale":"2024-02-10T07:32:46.639813+0000","last_undegraded":"2024-02-10T07:32:46.639813+0000","last_fullsized":"2024-02-10T07:32:46.639813+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:14.496768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":308,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641970+0000","last_change":"2024-02-10T07:32:25.175725+0000","last_active":"2024-02-10T07:32:46.641970+0000","last_peered":"2024-02-10T07:32:46.641970+0000","last_clean":"2024-02-10T07:32:46.641970+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:32:46.641970+0000","last_undegraded":"2024-02-10T07:32:46.641970+0000","last_fullsized":"2024-02-10T07:32:46.641970+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-02-10T07:32:25.175641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:32:25.175641+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:35.761800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597518+0000","last_change":"2024-02-10T07:31:51.283607+0000","last_active":"2024-02-10T07:32:47.597518+0000","last_peered":"2024-02-10T07:32:47.597518+0000","last_clean":"2024-02-10T07:32:47.597518+0000","last_became_active":"2024-02-10T07:31:51.283332+0000","last_became_peered":"2024-02-10T07:31:51.283332+0000","last_unstale":"2024-02-10T07:32:47.597518+0000","last_undegraded":"2024-02-10T07:32:47.597518+0000","last_fullsized":"2024-02-10T07:32:47.597518+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:06.299140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"89'24","reported_seq":107,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:16.374859+0000","last_change":"2024-02-10T07:33:11.841322+0000","last_active":"2024-02-10T07:33:16.374859+0000","last_peered":"2024-02-10T07:33:16.374859+0000","last_clean":"2024-02-10T07:33:16.374859+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:33:16.374859+0000","last_undegraded":"2024-02-10T07:33:16.374859+0000","last_fullsized":"2024-02-10T07:33:16.374859+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'24","last_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:33:11.841200+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:33.809552+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":47,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640726+0000","last_change":"2024-02-10T07:31:55.300257+0000","last_active":"2024-02-10T07:32:46.640726+0000","last_peered":"2024-02-10T07:32:46.640726+0000","last_clean":"2024-02-10T07:32:46.640726+0000","last_became_active":"2024-02-10T07:31:55.299928+0000","last_became_peered":"2024-02-10T07:31:55.299928+0000","last_unstale":"2024-02-10T07:32:46.640726+0000","last_undegraded":"2024-02-10T07:32:46.640726+0000","last_fullsized":"2024-02-10T07:32:46.640726+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:15.486048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694890+0000","last_change":"2024-02-10T07:31:57.316758+0000","last_active":"2024-02-10T07:32:47.694890+0000","last_peered":"2024-02-10T07:32:47.694890+0000","last_clean":"2024-02-10T07:32:47.694890+0000","last_became_active":"2024-02-10T07:31:57.316432+0000","last_became_peered":"2024-02-10T07:31:57.316432+0000","last_unstale":"2024-02-10T07:32:47.694890+0000","last_undegraded":"2024-02-10T07:32:47.694890+0000","last_fullsized":"2024-02-10T07:32:47.694890+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:23.313144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.100688+0000","last_change":"2024-02-10T07:31:51.286361+0000","last_active":"2024-02-10T07:32:47.100688+0000","last_peered":"2024-02-10T07:32:47.100688+0000","last_clean":"2024-02-10T07:32:47.100688+0000","last_became_active":"2024-02-10T07:31:51.286076+0000","last_became_peered":"2024-02-10T07:31:51.286076+0000","last_unstale":"2024-02-10T07:32:47.100688+0000","last_undegraded":"2024-02-10T07:32:47.100688+0000","last_fullsized":"2024-02-10T07:32:47.100688+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:17.577559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":153,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:21.319228+0000","last_change":"2024-02-10T07:31:54.281755+0000","last_active":"2024-02-10T07:33:21.319228+0000","last_peered":"2024-02-10T07:33:21.319228+0000","last_clean":"2024-02-10T07:33:21.319228+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:33:21.319228+0000","last_undegraded":"2024-02-10T07:33:21.319228+0000","last_fullsized":"2024-02-10T07:33:21.319228+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:31:49.265852+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.101395+0000","last_change":"2024-02-10T07:31:53.301026+0000","last_active":"2024-02-10T07:32:47.101395+0000","last_peered":"2024-02-10T07:32:47.101395+0000","last_clean":"2024-02-10T07:32:47.101395+0000","last_became_active":"2024-02-10T07:31:53.300687+0000","last_became_peered":"2024-02-10T07:31:53.300687+0000","last_unstale":"2024-02-10T07:32:47.101395+0000","last_undegraded":"2024-02-10T07:32:47.101395+0000","last_fullsized":"2024-02-10T07:32:47.101395+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:16.203387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640509+0000","last_change":"2024-02-10T07:31:55.297440+0000","last_active":"2024-02-10T07:32:46.640509+0000","last_peered":"2024-02-10T07:32:46.640509+0000","last_clean":"2024-02-10T07:32:46.640509+0000","last_became_active":"2024-02-10T07:31:55.297178+0000","last_became_peered":"2024-02-10T07:31:55.297178+0000","last_unstale":"2024-02-10T07:32:46.640509+0000","last_undegraded":"2024-02-10T07:32:46.640509+0000","last_fullsized":"2024-02-10T07:32:46.640509+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.962436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":41,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.694624+0000","last_change":"2024-02-10T07:31:57.307961+0000","last_active":"2024-02-10T07:32:47.694624+0000","last_peered":"2024-02-10T07:32:47.694624+0000","last_clean":"2024-02-10T07:32:47.694624+0000","last_became_active":"2024-02-10T07:31:57.307615+0000","last_became_peered":"2024-02-10T07:31:57.307615+0000","last_unstale":"2024-02-10T07:32:47.694624+0000","last_undegraded":"2024-02-10T07:32:47.694624+0000","last_fullsized":"2024-02-10T07:32:47.694624+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:24.112392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":61,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.270015+0000","last_change":"2024-02-10T07:33:06.270053+0000","last_active":"2024-02-10T07:33:06.270015+0000","last_peered":"2024-02-10T07:33:06.270015+0000","last_clean":"2024-02-10T07:33:06.270015+0000","last_became_active":"2024-02-10T07:31:51.320752+0000","last_became_peered":"2024-02-10T07:31:51.320752+0000","last_unstale":"2024-02-10T07:33:06.270015+0000","last_undegraded":"2024-02-10T07:33:06.270015+0000","last_fullsized":"2024-02-10T07:33:06.270015+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.269938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.269938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:34.431697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.100974+0000","last_change":"2024-02-10T07:31:53.289754+0000","last_active":"2024-02-10T07:32:47.100974+0000","last_peered":"2024-02-10T07:32:47.100974+0000","last_clean":"2024-02-10T07:32:47.100974+0000","last_became_active":"2024-02-10T07:31:53.289381+0000","last_became_peered":"2024-02-10T07:31:53.289381+0000","last_unstale":"2024-02-10T07:32:47.100974+0000","last_undegraded":"2024-02-10T07:32:47.100974+0000","last_fullsized":"2024-02-10T07:32:47.100974+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:13.156780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":46,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.695474+0000","last_change":"2024-02-10T07:31:55.290623+0000","last_active":"2024-02-10T07:32:47.695474+0000","last_peered":"2024-02-10T07:32:47.695474+0000","last_clean":"2024-02-10T07:32:47.695474+0000","last_became_active":"2024-02-10T07:31:55.290319+0000","last_became_peered":"2024-02-10T07:31:55.290319+0000","last_unstale":"2024-02-10T07:32:47.695474+0000","last_undegraded":"2024-02-10T07:32:47.695474+0000","last_fullsized":"2024-02-10T07:32:47.695474+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:23.955277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:05.431132+0000","last_change":"2024-02-10T07:33:05.431169+0000","last_active":"2024-02-10T07:33:05.431132+0000","last_peered":"2024-02-10T07:33:05.431132+0000","last_clean":"2024-02-10T07:33:05.431132+0000","last_became_active":"2024-02-10T07:31:57.317976+0000","last_became_peered":"2024-02-10T07:31:57.317976+0000","last_unstale":"2024-02-10T07:33:05.431132+0000","last_undegraded":"2024-02-10T07:33:05.431132+0000","last_fullsized":"2024-02-10T07:33:05.431132+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:05.431037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:05.431037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:31.470574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640631+0000","last_change":"2024-02-10T07:31:51.320993+0000","last_active":"2024-02-10T07:32:46.640631+0000","last_peered":"2024-02-10T07:32:46.640631+0000","last_clean":"2024-02-10T07:32:46.640631+0000","last_became_active":"2024-02-10T07:31:51.320673+0000","last_became_peered":"2024-02-10T07:31:51.320673+0000","last_unstale":"2024-02-10T07:32:46.640631+0000","last_undegraded":"2024-02-10T07:32:46.640631+0000","last_fullsized":"2024-02-10T07:32:46.640631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:15.248068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:53.288892+0000","last_change":"2024-02-10T07:32:53.288963+0000","last_active":"2024-02-10T07:32:53.288892+0000","last_peered":"2024-02-10T07:32:53.288892+0000","last_clean":"2024-02-10T07:32:53.288892+0000","last_became_active":"2024-02-10T07:31:53.297150+0000","last_became_peered":"2024-02-10T07:31:53.297150+0000","last_unstale":"2024-02-10T07:32:53.288892+0000","last_undegraded":"2024-02-10T07:32:53.288892+0000","last_fullsized":"2024-02-10T07:32:53.288892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:32:53.288798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:53.288798+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:06.843116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":73,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:16.374631+0000","last_change":"2024-02-10T07:31:55.296952+0000","last_active":"2024-02-10T07:33:16.374631+0000","last_peered":"2024-02-10T07:33:16.374631+0000","last_clean":"2024-02-10T07:33:16.374631+0000","last_became_active":"2024-02-10T07:31:55.295732+0000","last_became_peered":"2024-02-10T07:31:55.295732+0000","last_unstale":"2024-02-10T07:33:16.374631+0000","last_undegraded":"2024-02-10T07:33:16.374631+0000","last_fullsized":"2024-02-10T07:33:16.374631+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.639824+0000","last_change":"2024-02-10T07:31:57.305942+0000","last_active":"2024-02-10T07:32:46.639824+0000","last_peered":"2024-02-10T07:32:46.639824+0000","last_clean":"2024-02-10T07:32:46.639824+0000","last_became_active":"2024-02-10T07:31:57.305620+0000","last_became_peered":"2024-02-10T07:31:57.305620+0000","last_unstale":"2024-02-10T07:32:46.639824+0000","last_undegraded":"2024-02-10T07:32:46.639824+0000","last_fullsized":"2024-02-10T07:32:46.639824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:09.830636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:15.563990+0000","last_change":"2024-02-10T07:33:15.564016+0000","last_active":"2024-02-10T07:33:15.563990+0000","last_peered":"2024-02-10T07:33:15.563990+0000","last_clean":"2024-02-10T07:33:15.563990+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:33:15.563990+0000","last_undegraded":"2024-02-10T07:33:15.563990+0000","last_fullsized":"2024-02-10T07:33:15.563990+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:33:15.563932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:15.563932+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:23.107286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640503+0000","last_change":"2024-02-10T07:31:53.287199+0000","last_active":"2024-02-10T07:32:46.640503+0000","last_peered":"2024-02-10T07:32:46.640503+0000","last_clean":"2024-02-10T07:32:46.640503+0000","last_became_active":"2024-02-10T07:31:53.286884+0000","last_became_peered":"2024-02-10T07:31:53.286884+0000","last_unstale":"2024-02-10T07:32:46.640503+0000","last_undegraded":"2024-02-10T07:32:46.640503+0000","last_fullsized":"2024-02-10T07:32:46.640503+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.033998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":82,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:16.374609+0000","last_change":"2024-02-10T07:32:55.469184+0000","last_active":"2024-02-10T07:33:16.374609+0000","last_peered":"2024-02-10T07:33:16.374609+0000","last_clean":"2024-02-10T07:33:16.374609+0000","last_became_active":"2024-02-10T07:31:55.302863+0000","last_became_peered":"2024-02-10T07:31:55.302863+0000","last_unstale":"2024-02-10T07:33:16.374609+0000","last_undegraded":"2024-02-10T07:33:16.374609+0000","last_fullsized":"2024-02-10T07:33:16.374609+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:32:55.469099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:55.469099+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:13.566389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641909+0000","last_change":"2024-02-10T07:31:57.316332+0000","last_active":"2024-02-10T07:32:46.641909+0000","last_peered":"2024-02-10T07:32:46.641909+0000","last_clean":"2024-02-10T07:32:46.641909+0000","last_became_active":"2024-02-10T07:31:57.316047+0000","last_became_peered":"2024-02-10T07:31:57.316047+0000","last_unstale":"2024-02-10T07:32:46.641909+0000","last_undegraded":"2024-02-10T07:32:46.641909+0000","last_fullsized":"2024-02-10T07:32:46.641909+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:25.756992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:20.491429+0000","last_change":"2024-02-10T07:33:20.491471+0000","last_active":"2024-02-10T07:33:20.491429+0000","last_peered":"2024-02-10T07:33:20.491429+0000","last_clean":"2024-02-10T07:33:20.491429+0000","last_became_active":"2024-02-10T07:31:51.285333+0000","last_became_peered":"2024-02-10T07:31:51.285333+0000","last_unstale":"2024-02-10T07:33:20.491429+0000","last_undegraded":"2024-02-10T07:33:20.491429+0000","last_fullsized":"2024-02-10T07:33:20.491429+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:20.491354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:20.491354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:29.702384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:08.541222+0000","last_change":"2024-02-10T07:33:08.541272+0000","last_active":"2024-02-10T07:33:08.541222+0000","last_peered":"2024-02-10T07:33:08.541222+0000","last_clean":"2024-02-10T07:33:08.541222+0000","last_became_active":"2024-02-10T07:31:53.290157+0000","last_became_peered":"2024-02-10T07:31:53.290157+0000","last_unstale":"2024-02-10T07:33:08.541222+0000","last_undegraded":"2024-02-10T07:33:08.541222+0000","last_fullsized":"2024-02-10T07:33:08.541222+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-10T07:33:08.541155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:08.541155+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:33.263033+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:01.184144+0000","last_change":"2024-02-10T07:33:01.184183+0000","last_active":"2024-02-10T07:33:01.184144+0000","last_peered":"2024-02-10T07:33:01.184144+0000","last_clean":"2024-02-10T07:33:01.184144+0000","last_became_active":"2024-02-10T07:31:55.295900+0000","last_became_peered":"2024-02-10T07:31:55.295900+0000","last_unstale":"2024-02-10T07:33:01.184144+0000","last_undegraded":"2024-02-10T07:33:01.184144+0000","last_fullsized":"2024-02-10T07:33:01.184144+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:01.184048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:01.184048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:11.992366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":42,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.640531+0000","last_change":"2024-02-10T07:31:57.311731+0000","last_active":"2024-02-10T07:32:46.640531+0000","last_peered":"2024-02-10T07:32:46.640531+0000","last_clean":"2024-02-10T07:32:46.640531+0000","last_became_active":"2024-02-10T07:31:57.311054+0000","last_became_peered":"2024-02-10T07:31:57.311054+0000","last_unstale":"2024-02-10T07:32:46.640531+0000","last_undegraded":"2024-02-10T07:32:46.640531+0000","last_fullsized":"2024-02-10T07:32:46.640531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.402912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:16.374849+0000","last_change":"2024-02-10T07:31:55.295389+0000","last_active":"2024-02-10T07:33:16.374849+0000","last_peered":"2024-02-10T07:33:16.374849+0000","last_clean":"2024-02-10T07:33:16.374849+0000","last_became_active":"2024-02-10T07:31:55.295067+0000","last_became_peered":"2024-02-10T07:31:55.295067+0000","last_unstale":"2024-02-10T07:33:16.374849+0000","last_undegraded":"2024-02-10T07:33:16.374849+0000","last_fullsized":"2024-02-10T07:33:16.374849+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:24.085032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:47.597500+0000","last_change":"2024-02-10T07:31:51.282545+0000","last_active":"2024-02-10T07:32:47.597500+0000","last_peered":"2024-02-10T07:32:47.597500+0000","last_clean":"2024-02-10T07:32:47.597500+0000","last_became_active":"2024-02-10T07:31:51.282137+0000","last_became_peered":"2024-02-10T07:31:51.282137+0000","last_unstale":"2024-02-10T07:32:47.597500+0000","last_undegraded":"2024-02-10T07:32:47.597500+0000","last_fullsized":"2024-02-10T07:32:47.597500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:12.188800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":82,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:57.341196+0000","last_change":"2024-02-10T07:32:57.341249+0000","last_active":"2024-02-10T07:32:57.341196+0000","last_peered":"2024-02-10T07:32:57.341196+0000","last_clean":"2024-02-10T07:32:57.341196+0000","last_became_active":"2024-02-10T07:31:53.294218+0000","last_became_peered":"2024-02-10T07:31:53.294218+0000","last_unstale":"2024-02-10T07:32:57.341196+0000","last_undegraded":"2024-02-10T07:32:57.341196+0000","last_fullsized":"2024-02-10T07:32:57.341196+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:32:57.341115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:57.341115+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:05.137575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":51,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:04.582259+0000","last_change":"2024-02-10T07:33:04.582281+0000","last_active":"2024-02-10T07:33:04.582259+0000","last_peered":"2024-02-10T07:33:04.582259+0000","last_clean":"2024-02-10T07:33:04.582259+0000","last_became_active":"2024-02-10T07:31:57.303235+0000","last_became_peered":"2024-02-10T07:31:57.303235+0000","last_unstale":"2024-02-10T07:33:04.582259+0000","last_undegraded":"2024-02-10T07:33:04.582259+0000","last_fullsized":"2024-02-10T07:33:04.582259+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:04.582188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:04.582188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:27.553405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":54,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:06.237029+0000","last_change":"2024-02-10T07:33:06.237068+0000","last_active":"2024-02-10T07:33:06.237029+0000","last_peered":"2024-02-10T07:33:06.237029+0000","last_clean":"2024-02-10T07:33:06.237029+0000","last_became_active":"2024-02-10T07:31:55.294376+0000","last_became_peered":"2024-02-10T07:31:55.294376+0000","last_unstale":"2024-02-10T07:33:06.237029+0000","last_undegraded":"2024-02-10T07:33:06.237029+0000","last_fullsized":"2024-02-10T07:33:06.237029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.236946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.236946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:10.623394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:19.231683+0000","last_change":"2024-02-10T07:33:19.231720+0000","last_active":"2024-02-10T07:33:19.231683+0000","last_peered":"2024-02-10T07:33:19.231683+0000","last_clean":"2024-02-10T07:33:19.231683+0000","last_became_active":"2024-02-10T07:31:51.320715+0000","last_became_peered":"2024-02-10T07:31:51.320715+0000","last_unstale":"2024-02-10T07:33:19.231683+0000","last_undegraded":"2024-02-10T07:33:19.231683+0000","last_fullsized":"2024-02-10T07:33:19.231683+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:19.231613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:19.231613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:34.092643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"89'16","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:21.287044+0000","last_change":"2024-02-10T07:33:21.287093+0000","last_active":"2024-02-10T07:33:21.287044+0000","last_peered":"2024-02-10T07:33:21.287044+0000","last_clean":"2024-02-10T07:33:21.287044+0000","last_became_active":"2024-02-10T07:31:53.292510+0000","last_became_peered":"2024-02-10T07:31:53.292510+0000","last_unstale":"2024-02-10T07:33:21.287044+0000","last_undegraded":"2024-02-10T07:33:21.287044+0000","last_fullsized":"2024-02-10T07:33:21.287044+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'16","last_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:33:21.286955+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:43.692133+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":50,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:05.197704+0000","last_change":"2024-02-10T07:33:05.197724+0000","last_active":"2024-02-10T07:33:05.197704+0000","last_peered":"2024-02-10T07:33:05.197704+0000","last_clean":"2024-02-10T07:33:05.197704+0000","last_became_active":"2024-02-10T07:31:57.311427+0000","last_became_peered":"2024-02-10T07:31:57.311427+0000","last_unstale":"2024-02-10T07:33:05.197704+0000","last_undegraded":"2024-02-10T07:33:05.197704+0000","last_fullsized":"2024-02-10T07:33:05.197704+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:33:05.197642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:20.323214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:12.259572+0000","last_change":"2024-02-10T07:33:12.259596+0000","last_active":"2024-02-10T07:33:12.259572+0000","last_peered":"2024-02-10T07:33:12.259572+0000","last_clean":"2024-02-10T07:33:12.259572+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:33:12.259572+0000","last_undegraded":"2024-02-10T07:33:12.259572+0000","last_fullsized":"2024-02-10T07:33:12.259572+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:12.259478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:12.259478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:41.862761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:56.900575+0000","last_change":"2024-02-10T07:32:56.900599+0000","last_active":"2024-02-10T07:32:56.900575+0000","last_peered":"2024-02-10T07:32:56.900575+0000","last_clean":"2024-02-10T07:32:56.900575+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:32:56.900575+0000","last_undegraded":"2024-02-10T07:32:56.900575+0000","last_fullsized":"2024-02-10T07:32:56.900575+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:56.900501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:56.900501+0000","last_clean_scrub_stamp":"2024-02-10T07:32:56.900501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:16.803474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"89'25","reported_seq":95,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:20.252075+0000","last_change":"2024-02-10T07:33:20.252130+0000","last_active":"2024-02-10T07:33:20.252075+0000","last_peered":"2024-02-10T07:33:20.252075+0000","last_clean":"2024-02-10T07:33:20.252075+0000","last_became_active":"2024-02-10T07:31:53.292277+0000","last_became_peered":"2024-02-10T07:31:53.292277+0000","last_unstale":"2024-02-10T07:33:20.252075+0000","last_undegraded":"2024-02-10T07:33:20.252075+0000","last_fullsized":"2024-02-10T07:33:20.252075+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'25","last_scrub_stamp":"2024-02-10T07:33:20.251980+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:33:20.251980+0000","last_clean_scrub_stamp":"2024-02-10T07:33:20.251980+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:50.016769+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":43,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.642496+0000","last_change":"2024-02-10T07:31:57.308864+0000","last_active":"2024-02-10T07:32:46.642496+0000","last_peered":"2024-02-10T07:32:46.642496+0000","last_clean":"2024-02-10T07:32:46.642496+0000","last_became_active":"2024-02-10T07:31:57.308608+0000","last_became_peered":"2024-02-10T07:31:57.308608+0000","last_unstale":"2024-02-10T07:32:46.642496+0000","last_undegraded":"2024-02-10T07:32:46.642496+0000","last_fullsized":"2024-02-10T07:32:46.642496+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:09.099379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":76,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:16.374618+0000","last_change":"2024-02-10T07:31:55.298852+0000","last_active":"2024-02-10T07:33:16.374618+0000","last_peered":"2024-02-10T07:33:16.374618+0000","last_clean":"2024-02-10T07:33:16.374618+0000","last_became_active":"2024-02-10T07:31:55.298389+0000","last_became_peered":"2024-02-10T07:31:55.298389+0000","last_unstale":"2024-02-10T07:33:16.374618+0000","last_undegraded":"2024-02-10T07:33:16.374618+0000","last_fullsized":"2024-02-10T07:33:16.374618+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":63,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:33:19.515519+0000","last_change":"2024-02-10T07:33:19.515540+0000","last_active":"2024-02-10T07:33:19.515519+0000","last_peered":"2024-02-10T07:33:19.515519+0000","last_clean":"2024-02-10T07:33:19.515519+0000","last_became_active":"2024-02-10T07:31:51.288197+0000","last_became_peered":"2024-02-10T07:31:51.288197+0000","last_unstale":"2024-02-10T07:33:19.515519+0000","last_undegraded":"2024-02-10T07:33:19.515519+0000","last_fullsized":"2024-02-10T07:33:19.515519+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:19.515464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:19.515464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:46.814548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-10T07:32:46.641567+0000","last_change":"2024-02-10T07:31:53.288289+0000","last_active":"2024-02-10T07:32:46.641567+0000","last_peered":"2024-02-10T07:32:46.641567+0000","last_clean":"2024-02-10T07:32:46.641567+0000","last_became_active":"2024-02-10T07:31:53.287984+0000","last_became_peered":"2024-02-10T07:31:53.287984+0000","last_unstale":"2024-02-10T07:32:46.641567+0000","last_undegraded":"2024-02-10T07:32:46.641567+0000","last_fullsized":"2024-02-10T07:32:46.641567+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:19.716780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037783,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1495040,"data_stored":1013417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.495},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.376}]},{"osd":1,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.436},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.48299999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.443},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51400000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.314}]},{"osd":5,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.45500000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.58199999999999996}]}]},{"osd":6,"up_from":50,"seq":214748364828,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28460,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716692,"statfs":{"total":95995035648,"available":95965892608,"internally_reserved":0,"allocated":1478656,"data_stored":1012764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.36099999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77000000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.51500000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.503},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66600000000000004,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.63200000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.77400000000000002,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73499999999999999,"5min":0.872,"15min":0.872},"last":0.496}]},{"osd":3,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":1.544,"15min":1.7070000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72399999999999998,"5min":19.664999999999999,"15min":19.664999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.61299999999999999,"15min":0.61499999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83599999999999997,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.65800000000000003}]},{"osd":4,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.35999999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.38},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.48799999999999999,"15min":0.47899999999999998},"min":{"1min":0.34399999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.39600000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48999999999999999,"15min":0.48799999999999999},"min":{"1min":0.33700000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.51400000000000001,"15min":0.51000000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.315}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.35299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36236,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708916,"statfs":{"total":95995035648,"available":95957929984,"internally_reserved":0,"allocated":921600,"data_stored":423123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.374,"15min":0.34599999999999997},"min":{"1min":0.249,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.40400000000000003,"15min":0.36399999999999999},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.54700000000000004}]},{"osd":2,"last update":"Sat Feb 10 07:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41299999999999998,"15min":0.40000000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42899999999999999,"15min":0.41799999999999998},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.48699999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:33:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.42599999999999999,"15min":0.42399999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.58499999999999996,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.51200000000000001,"15min":0.50900000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.69399999999999995}]},{"osd":4,"last update":"Sat Feb 10 07:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.98799999999999999,"15min":1.0469999999999999},"min":{"1min":0.435,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80700000000000005,"5min":10.292,"15min":10.292},"last":0.749},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58399999999999996,"15min":0.58299999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.71599999999999997,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.70899999999999996}]},{"osd":5,"last update":"Sat Feb 10 07:32:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.627,"15min":0.63},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80500000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.66400000000000003}]},{"osd":6,"last update":"Sat Feb 10 07:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.623,"15min":0.628},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.79800000000000004,"5min":1.002,"15min":1.002},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.83699999999999997,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.78100000000000003}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.876}]}]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32644,"kb_used_data":1468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712508,"statfs":{"total":95995035648,"available":95961608192,"internally_reserved":0,"allocated":1503232,"data_stored":1014034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34599999999999997,"15min":0.34399999999999997},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.41099999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.36499999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.33300000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.38500000000000001,"15min":0.38800000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.51200000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42899999999999999,"15min":0.42499999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57899999999999996,"5min":0.629,"15min":0.629},"last":0.379}]},{"osd":3,"last update":"Sat Feb 10 07:33:05 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.432,"15min":0.443},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53800000000000003,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47499999999999998,"15min":0.47199999999999998},"min":{"1min":0.32400000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64700000000000002,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.53000000000000003}]},{"osd":4,"last update":"Sat Feb 10 07:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.74099999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.433,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.36699999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:32:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59699999999999998,"15min":0.59399999999999997},"min":{"1min":0.438,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.48699999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":0.83499999999999996,"15min":0.70599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":19.16,"5min":19.16,"15min":19.16},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.61899999999999999,"15min":0.63100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.67200000000000004,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.51800000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":19.178000000000001,"5min":19.178000000000001,"15min":19.178000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.61699999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":905216,"data_stored":423646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.379,"15min":0.38},"min":{"1min":0.255,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56499999999999995,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.42399999999999999,"15min":0.41599999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Feb 10 07:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.44,"15min":0.42399999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.44,"15min":0.42199999999999999},"min":{"1min":0.27200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.34300000000000003}]},{"osd":3,"last update":"Sat Feb 10 07:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51700000000000002,"15min":0.53400000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.53100000000000003,"15min":0.55300000000000005},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.30399999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63100000000000001,"15min":0.626},"min":{"1min":0.50700000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.68899999999999995,"15min":0.65300000000000002},"min":{"1min":0.50700000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":5.343,"5min":5.343,"15min":5.343},"last":0.41699999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.64900000000000002,"15min":0.64400000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.628,"15min":0.623},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.53300000000000003}]},{"osd":6,"last update":"Sat Feb 10 07:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.85099999999999998,"15min":0.879},"min":{"1min":0.45900000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.321,"5min":5.3230000000000004,"15min":5.3230000000000004},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64500000000000002,"15min":0.64300000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.57799999999999996}]},{"osd":7,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.45600000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":921600,"data_stored":422902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.372,"15min":0.378},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55500000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.42099999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.442,"15min":0.437},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.32000000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.31,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.38},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42799999999999999,"15min":0.434},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.441}]},{"osd":4,"last update":"Sat Feb 10 07:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.89700000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.46800000000000003}]},{"osd":5,"last update":"Sat Feb 10 07:32:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.52200000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67000000000000004,"15min":0.67400000000000004},"min":{"1min":0.41999999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.65300000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.54300000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953510,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":423014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.64500000000000002,"15min":0.60599999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":7.5700000000000003,"5min":7.5700000000000003,"15min":7.5700000000000003},"last":7.5700000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72899999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.46999999999999997}]},{"osd":1,"last update":"Sat Feb 10 07:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54100000000000004,"15min":0.54800000000000004},"min":{"1min":0.34799999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.63,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.56699999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.40799999999999997}]},{"osd":2,"last update":"Sat Feb 10 07:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54100000000000004,"15min":0.53800000000000003},"min":{"1min":0.40999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69699999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66500000000000004,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.51500000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57899999999999996,"15min":0.58299999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.76900000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.434},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.626,"15min":0.57699999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":7.601,"5min":7.601,"15min":7.601},"last":7.601}]},{"osd":5,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.378,"15min":0.373},"min":{"1min":0.30099999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60799999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.40300000000000002,"15min":0.39300000000000002},"min":{"1min":0.32300000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.33800000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.44400000000000001,"15min":0.42999999999999999},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.44,"15min":0.42999999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.47899999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.42199999999999999}]}]},{"osd":5,"up_from":40,"seq":171798691873,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32052,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713100,"statfs":{"total":95995035648,"available":95962214400,"internally_reserved":0,"allocated":897024,"data_stored":423608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77400000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.57699999999999996,"15min":0.56100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.2029999999999998,"5min":3.2029999999999998,"15min":3.2029999999999998},"last":0.37}]},{"osd":1,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.75,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.61299999999999999,"15min":0.59699999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.238,"5min":3.238,"15min":3.238},"last":0.51200000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.438,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76200000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.748,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.60299999999999998},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80800000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.76500000000000001,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.35099999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.41099999999999998,"15min":0.40100000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.39700000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.33100000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.437,"15min":0.42499999999999999},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.442},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.47299999999999998,"15min":0.45600000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.29899999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.501},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.41499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:33:26.149 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:33:26.396 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:26.517 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[29816]: 3.d scrub starts 2024-02-10T07:33:26.517 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[29816]: 3.d scrub ok 2024-02-10T07:33:26.517 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[29816]: 4.9 scrub starts 2024-02-10T07:33:26.517 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[29816]: 4.9 scrub ok 2024-02-10T07:33:26.517 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[29816]: 3.1e deep-scrub starts 2024-02-10T07:33:26.517 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[29816]: 3.1e deep-scrub ok 2024-02-10T07:33:26.517 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[29816]: osdmap e90: 8 total, 8 up, 8 in 2024-02-10T07:33:26.517 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[29816]: 6.e scrub starts 2024-02-10T07:33:26.517 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[29816]: 6.e scrub ok 2024-02-10T07:33:26.518 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[40743]: 3.d scrub starts 2024-02-10T07:33:26.518 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[40743]: 3.d scrub ok 2024-02-10T07:33:26.518 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[40743]: 4.9 scrub starts 2024-02-10T07:33:26.518 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[40743]: 4.9 scrub ok 2024-02-10T07:33:26.518 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[40743]: 3.1e deep-scrub starts 2024-02-10T07:33:26.518 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[40743]: 3.1e deep-scrub ok 2024-02-10T07:33:26.518 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[40743]: osdmap e90: 8 total, 8 up, 8 in 2024-02-10T07:33:26.518 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[40743]: 6.e scrub starts 2024-02-10T07:33:26.518 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:26 smithi008 ceph-mon[40743]: 6.e scrub ok 2024-02-10T07:33:26.625 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:26 smithi170 ceph-mon[33394]: 3.d scrub starts 2024-02-10T07:33:26.625 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:26 smithi170 ceph-mon[33394]: 3.d scrub ok 2024-02-10T07:33:26.625 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:26 smithi170 ceph-mon[33394]: 4.9 scrub starts 2024-02-10T07:33:26.625 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:26 smithi170 ceph-mon[33394]: 4.9 scrub ok 2024-02-10T07:33:26.625 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:26 smithi170 ceph-mon[33394]: 3.1e deep-scrub starts 2024-02-10T07:33:26.625 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:26 smithi170 ceph-mon[33394]: 3.1e deep-scrub ok 2024-02-10T07:33:26.625 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:26 smithi170 ceph-mon[33394]: osdmap e90: 8 total, 8 up, 8 in 2024-02-10T07:33:26.625 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:26 smithi170 ceph-mon[33394]: 6.e scrub starts 2024-02-10T07:33:26.625 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:26 smithi170 ceph-mon[33394]: 6.e scrub ok 2024-02-10T07:33:27.480 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:27 smithi008 ceph-mon[29816]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:33:27.480 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:27 smithi008 ceph-mon[29816]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:27.480 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:27 smithi008 ceph-mon[29816]: 5.d scrub starts 2024-02-10T07:33:27.480 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:27 smithi008 ceph-mon[29816]: 5.d scrub ok 2024-02-10T07:33:27.480 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:27 smithi008 ceph-mon[29816]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T07:33:27.480 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:27 smithi008 ceph-mon[40743]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:33:27.481 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:27 smithi008 ceph-mon[40743]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:27.481 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:27 smithi008 ceph-mon[40743]: 5.d scrub starts 2024-02-10T07:33:27.481 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:27 smithi008 ceph-mon[40743]: 5.d scrub ok 2024-02-10T07:33:27.481 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:27 smithi008 ceph-mon[40743]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T07:33:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:27 smithi170 ceph-mon[33394]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:33:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:27 smithi170 ceph-mon[33394]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:27 smithi170 ceph-mon[33394]: 5.d scrub starts 2024-02-10T07:33:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:27 smithi170 ceph-mon[33394]: 5.d scrub ok 2024-02-10T07:33:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:27 smithi170 ceph-mon[33394]: osdmap e91: 8 total, 8 up, 8 in 2024-02-10T07:33:28.217 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:33:28.219 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:33:28.339 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:28 smithi170 ceph-mon[33394]: 6.13 scrub starts 2024-02-10T07:33:28.340 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:28 smithi170 ceph-mon[33394]: 6.13 scrub ok 2024-02-10T07:33:28.340 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:28 smithi170 ceph-mon[33394]: 3.11 scrub starts 2024-02-10T07:33:28.340 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:28 smithi170 ceph-mon[33394]: 3.11 scrub ok 2024-02-10T07:33:28.340 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:28 smithi170 ceph-mon[33394]: 6.4 deep-scrub starts 2024-02-10T07:33:28.340 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:28 smithi170 ceph-mon[33394]: 6.4 deep-scrub ok 2024-02-10T07:33:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:28 smithi008 ceph-mon[40743]: 6.13 scrub starts 2024-02-10T07:33:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:28 smithi008 ceph-mon[40743]: 6.13 scrub ok 2024-02-10T07:33:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:28 smithi008 ceph-mon[40743]: 3.11 scrub starts 2024-02-10T07:33:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:28 smithi008 ceph-mon[40743]: 3.11 scrub ok 2024-02-10T07:33:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:28 smithi008 ceph-mon[40743]: 6.4 deep-scrub starts 2024-02-10T07:33:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:28 smithi008 ceph-mon[40743]: 6.4 deep-scrub ok 2024-02-10T07:33:28.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:28 smithi008 ceph-mon[29816]: 6.13 scrub starts 2024-02-10T07:33:28.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:28 smithi008 ceph-mon[29816]: 6.13 scrub ok 2024-02-10T07:33:28.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:28 smithi008 ceph-mon[29816]: 3.11 scrub starts 2024-02-10T07:33:28.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:28 smithi008 ceph-mon[29816]: 3.11 scrub ok 2024-02-10T07:33:28.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:28 smithi008 ceph-mon[29816]: 6.4 deep-scrub starts 2024-02-10T07:33:28.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:28 smithi008 ceph-mon[29816]: 6.4 deep-scrub ok 2024-02-10T07:33:28.981 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:33:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:33:29.008 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-02-10T07:33:27.678764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1024,"num_read_kb":870,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":632,"ondisk_log_size":632,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":254284,"kb_used_data":8984,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749706932,"statfs":{"total":767960285184,"available":767699898368,"internally_reserved":0,"allocated":9199616,"data_stored":5257112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001854"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657946+0000","last_change":"2024-02-10T07:31:57.311638+0000","last_active":"2024-02-10T07:33:26.657946+0000","last_peered":"2024-02-10T07:33:26.657946+0000","last_clean":"2024-02-10T07:33:26.657946+0000","last_became_active":"2024-02-10T07:31:57.311360+0000","last_became_peered":"2024-02-10T07:31:57.311360+0000","last_unstale":"2024-02-10T07:33:26.657946+0000","last_undegraded":"2024-02-10T07:33:26.657946+0000","last_fullsized":"2024-02-10T07:33:26.657946+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.295989+0000","last_change":"2024-02-10T07:33:14.320677+0000","last_active":"2024-02-10T07:33:27.295989+0000","last_peered":"2024-02-10T07:33:27.295989+0000","last_clean":"2024-02-10T07:33:27.295989+0000","last_became_active":"2024-02-10T07:31:55.299946+0000","last_became_peered":"2024-02-10T07:31:55.299946+0000","last_unstale":"2024-02-10T07:33:27.295989+0000","last_undegraded":"2024-02-10T07:33:27.295989+0000","last_fullsized":"2024-02-10T07:33:27.295989+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:14.320588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:14.320588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:38.014309+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.592352+0000","last_change":"2024-02-10T07:31:51.321177+0000","last_active":"2024-02-10T07:33:27.592352+0000","last_peered":"2024-02-10T07:33:27.592352+0000","last_clean":"2024-02-10T07:33:27.592352+0000","last_became_active":"2024-02-10T07:31:51.320873+0000","last_became_peered":"2024-02-10T07:31:51.320873+0000","last_unstale":"2024-02-10T07:33:27.592352+0000","last_undegraded":"2024-02-10T07:33:27.592352+0000","last_fullsized":"2024-02-10T07:33:27.592352+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657663+0000","last_change":"2024-02-10T07:31:53.301175+0000","last_active":"2024-02-10T07:33:26.657663+0000","last_peered":"2024-02-10T07:33:26.657663+0000","last_clean":"2024-02-10T07:33:26.657663+0000","last_became_active":"2024-02-10T07:31:53.300846+0000","last_became_peered":"2024-02-10T07:31:53.300846+0000","last_unstale":"2024-02-10T07:33:26.657663+0000","last_undegraded":"2024-02-10T07:33:26.657663+0000","last_fullsized":"2024-02-10T07:33:26.657663+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656691+0000","last_change":"2024-02-10T07:33:12.587948+0000","last_active":"2024-02-10T07:33:26.656691+0000","last_peered":"2024-02-10T07:33:26.656691+0000","last_clean":"2024-02-10T07:33:26.656691+0000","last_became_active":"2024-02-10T07:31:57.300056+0000","last_became_peered":"2024-02-10T07:31:57.300056+0000","last_unstale":"2024-02-10T07:33:26.656691+0000","last_undegraded":"2024-02-10T07:33:26.656691+0000","last_fullsized":"2024-02-10T07:33:26.656691+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:12.587860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:12.587860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:17.246465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657767+0000","last_change":"2024-02-10T07:32:58.168446+0000","last_active":"2024-02-10T07:33:26.657767+0000","last_peered":"2024-02-10T07:33:26.657767+0000","last_clean":"2024-02-10T07:33:26.657767+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:33:26.657767+0000","last_undegraded":"2024-02-10T07:33:26.657767+0000","last_fullsized":"2024-02-10T07:33:26.657767+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:58.168353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.168353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:11.296018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657292+0000","last_change":"2024-02-10T07:33:24.458907+0000","last_active":"2024-02-10T07:33:26.657292+0000","last_peered":"2024-02-10T07:33:26.657292+0000","last_clean":"2024-02-10T07:33:26.657292+0000","last_became_active":"2024-02-10T07:31:51.323331+0000","last_became_peered":"2024-02-10T07:31:51.323331+0000","last_unstale":"2024-02-10T07:33:26.657292+0000","last_undegraded":"2024-02-10T07:33:26.657292+0000","last_fullsized":"2024-02-10T07:33:26.657292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:33:24.458825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:24.622317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656590+0000","last_change":"2024-02-10T07:33:00.565478+0000","last_active":"2024-02-10T07:33:26.656590+0000","last_peered":"2024-02-10T07:33:26.656590+0000","last_clean":"2024-02-10T07:33:26.656590+0000","last_became_active":"2024-02-10T07:31:53.286957+0000","last_became_peered":"2024-02-10T07:31:53.286957+0000","last_unstale":"2024-02-10T07:33:26.656590+0000","last_undegraded":"2024-02-10T07:33:26.656590+0000","last_fullsized":"2024-02-10T07:33:26.656590+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:33:00.565384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:00.565384+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:05.401431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.592577+0000","last_change":"2024-02-10T07:31:57.313731+0000","last_active":"2024-02-10T07:33:27.592577+0000","last_peered":"2024-02-10T07:33:27.592577+0000","last_clean":"2024-02-10T07:33:27.592577+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:33:27.592577+0000","last_undegraded":"2024-02-10T07:33:27.592577+0000","last_fullsized":"2024-02-10T07:33:27.592577+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.215034+0000","last_change":"2024-02-10T07:31:55.298018+0000","last_active":"2024-02-10T07:33:27.215034+0000","last_peered":"2024-02-10T07:33:27.215034+0000","last_clean":"2024-02-10T07:33:27.215034+0000","last_became_active":"2024-02-10T07:31:55.297715+0000","last_became_peered":"2024-02-10T07:31:55.297715+0000","last_unstale":"2024-02-10T07:33:27.215034+0000","last_undegraded":"2024-02-10T07:33:27.215034+0000","last_fullsized":"2024-02-10T07:33:27.215034+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"67'1","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.215241+0000","last_change":"2024-02-10T07:32:51.545597+0000","last_active":"2024-02-10T07:33:27.215241+0000","last_peered":"2024-02-10T07:33:27.215241+0000","last_clean":"2024-02-10T07:33:27.215241+0000","last_became_active":"2024-02-10T07:31:51.283229+0000","last_became_peered":"2024-02-10T07:31:51.283229+0000","last_unstale":"2024-02-10T07:33:27.215241+0000","last_undegraded":"2024-02-10T07:33:27.215241+0000","last_fullsized":"2024-02-10T07:33:27.215241+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:32:51.545493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:32:51.545493+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:17.023144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.658198+0000","last_change":"2024-02-10T07:31:53.288142+0000","last_active":"2024-02-10T07:33:26.658198+0000","last_peered":"2024-02-10T07:33:26.658198+0000","last_clean":"2024-02-10T07:33:26.658198+0000","last_became_active":"2024-02-10T07:31:53.287858+0000","last_became_peered":"2024-02-10T07:31:53.287858+0000","last_unstale":"2024-02-10T07:33:26.658198+0000","last_undegraded":"2024-02-10T07:33:26.658198+0000","last_fullsized":"2024-02-10T07:33:26.658198+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.214744+0000","last_change":"2024-02-10T07:31:57.310358+0000","last_active":"2024-02-10T07:33:27.214744+0000","last_peered":"2024-02-10T07:33:27.214744+0000","last_clean":"2024-02-10T07:33:27.214744+0000","last_became_active":"2024-02-10T07:31:57.310088+0000","last_became_peered":"2024-02-10T07:31:57.310088+0000","last_unstale":"2024-02-10T07:33:27.214744+0000","last_undegraded":"2024-02-10T07:33:27.214744+0000","last_fullsized":"2024-02-10T07:33:27.214744+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656962+0000","last_change":"2024-02-10T07:31:55.303127+0000","last_active":"2024-02-10T07:33:26.656962+0000","last_peered":"2024-02-10T07:33:26.656962+0000","last_clean":"2024-02-10T07:33:26.656962+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:33:26.656962+0000","last_undegraded":"2024-02-10T07:33:26.656962+0000","last_fullsized":"2024-02-10T07:33:26.656962+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.215162+0000","last_change":"2024-02-10T07:31:51.276682+0000","last_active":"2024-02-10T07:33:27.215162+0000","last_peered":"2024-02-10T07:33:27.215162+0000","last_clean":"2024-02-10T07:33:27.215162+0000","last_became_active":"2024-02-10T07:31:51.276350+0000","last_became_peered":"2024-02-10T07:31:51.276350+0000","last_unstale":"2024-02-10T07:33:27.215162+0000","last_undegraded":"2024-02-10T07:33:27.215162+0000","last_fullsized":"2024-02-10T07:33:27.215162+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657495+0000","last_change":"2024-02-10T07:31:53.292840+0000","last_active":"2024-02-10T07:33:26.657495+0000","last_peered":"2024-02-10T07:33:26.657495+0000","last_clean":"2024-02-10T07:33:26.657495+0000","last_became_active":"2024-02-10T07:31:53.292417+0000","last_became_peered":"2024-02-10T07:31:53.292417+0000","last_unstale":"2024-02-10T07:33:26.657495+0000","last_undegraded":"2024-02-10T07:33:26.657495+0000","last_fullsized":"2024-02-10T07:33:26.657495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.659064+0000","last_change":"2024-02-10T07:33:18.207530+0000","last_active":"2024-02-10T07:33:26.659064+0000","last_peered":"2024-02-10T07:33:26.659064+0000","last_clean":"2024-02-10T07:33:26.659064+0000","last_became_active":"2024-02-10T07:31:57.305613+0000","last_became_peered":"2024-02-10T07:31:57.305613+0000","last_unstale":"2024-02-10T07:33:26.659064+0000","last_undegraded":"2024-02-10T07:33:26.659064+0000","last_fullsized":"2024-02-10T07:33:26.659064+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:18.207406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:18.207406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:27.440541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.296620+0000","last_change":"2024-02-10T07:33:19.252470+0000","last_active":"2024-02-10T07:33:27.296620+0000","last_peered":"2024-02-10T07:33:27.296620+0000","last_clean":"2024-02-10T07:33:27.296620+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:33:27.296620+0000","last_undegraded":"2024-02-10T07:33:27.296620+0000","last_fullsized":"2024-02-10T07:33:27.296620+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:19.252386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:19.252386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:47.255925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.592244+0000","last_change":"2024-02-10T07:31:51.287400+0000","last_active":"2024-02-10T07:33:27.592244+0000","last_peered":"2024-02-10T07:33:27.592244+0000","last_clean":"2024-02-10T07:33:27.592244+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:33:27.592244+0000","last_undegraded":"2024-02-10T07:33:27.592244+0000","last_fullsized":"2024-02-10T07:33:27.592244+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"89'22","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.699654+0000","last_change":"2024-02-10T07:33:14.170253+0000","last_active":"2024-02-10T07:33:26.699654+0000","last_peered":"2024-02-10T07:33:26.699654+0000","last_clean":"2024-02-10T07:33:26.699654+0000","last_became_active":"2024-02-10T07:31:53.295527+0000","last_became_peered":"2024-02-10T07:31:53.295527+0000","last_unstale":"2024-02-10T07:33:26.699654+0000","last_undegraded":"2024-02-10T07:33:26.699654+0000","last_fullsized":"2024-02-10T07:33:26.699654+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:33:14.170072+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:27.215444+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657227+0000","last_change":"2024-02-10T07:31:57.306606+0000","last_active":"2024-02-10T07:33:26.657227+0000","last_peered":"2024-02-10T07:33:26.657227+0000","last_clean":"2024-02-10T07:33:26.657227+0000","last_became_active":"2024-02-10T07:31:57.306315+0000","last_became_peered":"2024-02-10T07:31:57.306315+0000","last_unstale":"2024-02-10T07:33:26.657227+0000","last_undegraded":"2024-02-10T07:33:26.657227+0000","last_fullsized":"2024-02-10T07:33:26.657227+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.368871+0000","last_change":"2024-02-10T07:33:27.368898+0000","last_active":"2024-02-10T07:33:27.368871+0000","last_peered":"2024-02-10T07:33:27.368871+0000","last_clean":"2024-02-10T07:33:27.368871+0000","last_became_active":"2024-02-10T07:31:55.298474+0000","last_became_peered":"2024-02-10T07:31:55.298474+0000","last_unstale":"2024-02-10T07:33:27.368871+0000","last_undegraded":"2024-02-10T07:33:27.368871+0000","last_fullsized":"2024-02-10T07:33:27.368871+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:27.368799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:27.368799+0000","last_clean_scrub_stamp":"2024-02-10T07:33:27.368799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:33.627384+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.658310+0000","last_change":"2024-02-10T07:33:02.198096+0000","last_active":"2024-02-10T07:33:26.658310+0000","last_peered":"2024-02-10T07:33:26.658310+0000","last_clean":"2024-02-10T07:33:26.658310+0000","last_became_active":"2024-02-10T07:31:51.321237+0000","last_became_peered":"2024-02-10T07:31:51.321237+0000","last_unstale":"2024-02-10T07:33:26.658310+0000","last_undegraded":"2024-02-10T07:33:26.658310+0000","last_fullsized":"2024-02-10T07:33:26.658310+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:02.197996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:02.197996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:08.678151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657001+0000","last_change":"2024-02-10T07:33:13.556374+0000","last_active":"2024-02-10T07:33:26.657001+0000","last_peered":"2024-02-10T07:33:26.657001+0000","last_clean":"2024-02-10T07:33:26.657001+0000","last_became_active":"2024-02-10T07:31:53.294914+0000","last_became_peered":"2024-02-10T07:31:53.294914+0000","last_unstale":"2024-02-10T07:33:26.657001+0000","last_undegraded":"2024-02-10T07:33:26.657001+0000","last_fullsized":"2024-02-10T07:33:26.657001+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:33:13.556271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:13.556271+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:22.295459+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":48,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657223+0000","last_change":"2024-02-10T07:31:57.307172+0000","last_active":"2024-02-10T07:33:26.657223+0000","last_peered":"2024-02-10T07:33:26.657223+0000","last_clean":"2024-02-10T07:33:26.657223+0000","last_became_active":"2024-02-10T07:31:57.306892+0000","last_became_peered":"2024-02-10T07:31:57.306892+0000","last_unstale":"2024-02-10T07:33:26.657223+0000","last_undegraded":"2024-02-10T07:33:26.657223+0000","last_fullsized":"2024-02-10T07:33:26.657223+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":89,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.278646+0000","last_change":"2024-02-10T07:33:17.787432+0000","last_active":"2024-02-10T07:33:27.278646+0000","last_peered":"2024-02-10T07:33:27.278646+0000","last_clean":"2024-02-10T07:33:27.278646+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:33:27.278646+0000","last_undegraded":"2024-02-10T07:33:27.278646+0000","last_fullsized":"2024-02-10T07:33:27.278646+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:33:17.787314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:17.787314+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:24.569254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.295417+0000","last_change":"2024-02-10T07:33:16.291275+0000","last_active":"2024-02-10T07:33:27.295417+0000","last_peered":"2024-02-10T07:33:27.295417+0000","last_clean":"2024-02-10T07:33:27.295417+0000","last_became_active":"2024-02-10T07:31:51.282077+0000","last_became_peered":"2024-02-10T07:31:51.282077+0000","last_unstale":"2024-02-10T07:33:27.295417+0000","last_undegraded":"2024-02-10T07:33:27.295417+0000","last_fullsized":"2024-02-10T07:33:27.295417+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:33:16.291171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:16.291171+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:24.579644+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.593078+0000","last_change":"2024-02-10T07:33:03.462253+0000","last_active":"2024-02-10T07:33:27.593078+0000","last_peered":"2024-02-10T07:33:27.593078+0000","last_clean":"2024-02-10T07:33:27.593078+0000","last_became_active":"2024-02-10T07:31:53.292936+0000","last_became_peered":"2024-02-10T07:31:53.292936+0000","last_unstale":"2024-02-10T07:33:27.593078+0000","last_undegraded":"2024-02-10T07:33:27.593078+0000","last_fullsized":"2024-02-10T07:33:27.593078+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:33:03.462109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:03.462109+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:31.967818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.296612+0000","last_change":"2024-02-10T07:33:18.242898+0000","last_active":"2024-02-10T07:33:27.296612+0000","last_peered":"2024-02-10T07:33:27.296612+0000","last_clean":"2024-02-10T07:33:27.296612+0000","last_became_active":"2024-02-10T07:31:57.308820+0000","last_became_peered":"2024-02-10T07:31:57.308820+0000","last_unstale":"2024-02-10T07:33:27.296612+0000","last_undegraded":"2024-02-10T07:33:27.296612+0000","last_fullsized":"2024-02-10T07:33:27.296612+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:18.242759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:18.242759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:31.207474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.593212+0000","last_change":"2024-02-10T07:31:55.297952+0000","last_active":"2024-02-10T07:33:27.593212+0000","last_peered":"2024-02-10T07:33:27.593212+0000","last_clean":"2024-02-10T07:33:27.593212+0000","last_became_active":"2024-02-10T07:31:55.297595+0000","last_became_peered":"2024-02-10T07:31:55.297595+0000","last_unstale":"2024-02-10T07:33:27.593212+0000","last_undegraded":"2024-02-10T07:33:27.593212+0000","last_fullsized":"2024-02-10T07:33:27.593212+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656328+0000","last_change":"2024-02-10T07:33:16.558719+0000","last_active":"2024-02-10T07:33:26.656328+0000","last_peered":"2024-02-10T07:33:26.656328+0000","last_clean":"2024-02-10T07:33:26.656328+0000","last_became_active":"2024-02-10T07:31:51.285405+0000","last_became_peered":"2024-02-10T07:31:51.285405+0000","last_unstale":"2024-02-10T07:33:26.656328+0000","last_undegraded":"2024-02-10T07:33:26.656328+0000","last_fullsized":"2024-02-10T07:33:26.656328+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:16.558593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:16.558593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:21.969603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.278622+0000","last_change":"2024-02-10T07:33:00.861634+0000","last_active":"2024-02-10T07:33:27.278622+0000","last_peered":"2024-02-10T07:33:27.278622+0000","last_clean":"2024-02-10T07:33:27.278622+0000","last_became_active":"2024-02-10T07:31:53.300871+0000","last_became_peered":"2024-02-10T07:31:53.300871+0000","last_unstale":"2024-02-10T07:33:27.278622+0000","last_undegraded":"2024-02-10T07:33:27.278622+0000","last_fullsized":"2024-02-10T07:33:27.278622+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:33:00.861482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:00.861482+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:09.147490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656837+0000","last_change":"2024-02-10T07:31:57.307069+0000","last_active":"2024-02-10T07:33:26.656837+0000","last_peered":"2024-02-10T07:33:26.656837+0000","last_clean":"2024-02-10T07:33:26.656837+0000","last_became_active":"2024-02-10T07:31:57.306776+0000","last_became_peered":"2024-02-10T07:31:57.306776+0000","last_unstale":"2024-02-10T07:33:26.656837+0000","last_undegraded":"2024-02-10T07:33:26.656837+0000","last_fullsized":"2024-02-10T07:33:26.656837+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.277842+0000","last_change":"2024-02-10T07:33:13.822432+0000","last_active":"2024-02-10T07:33:27.277842+0000","last_peered":"2024-02-10T07:33:27.277842+0000","last_clean":"2024-02-10T07:33:27.277842+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:33:27.277842+0000","last_undegraded":"2024-02-10T07:33:27.277842+0000","last_fullsized":"2024-02-10T07:33:27.277842+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:13.822335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:13.822335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:35.258891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.592498+0000","last_change":"2024-02-10T07:33:11.581151+0000","last_active":"2024-02-10T07:33:27.592498+0000","last_peered":"2024-02-10T07:33:27.592498+0000","last_clean":"2024-02-10T07:33:27.592498+0000","last_became_active":"2024-02-10T07:31:51.284022+0000","last_became_peered":"2024-02-10T07:31:51.284022+0000","last_unstale":"2024-02-10T07:33:27.592498+0000","last_undegraded":"2024-02-10T07:33:27.592498+0000","last_fullsized":"2024-02-10T07:33:27.592498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:11.581063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:11.581063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:32.492652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":61,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656340+0000","last_change":"2024-02-10T07:31:53.295829+0000","last_active":"2024-02-10T07:33:26.656340+0000","last_peered":"2024-02-10T07:33:26.656340+0000","last_clean":"2024-02-10T07:33:26.656340+0000","last_became_active":"2024-02-10T07:31:53.295520+0000","last_became_peered":"2024-02-10T07:31:53.295520+0000","last_unstale":"2024-02-10T07:33:26.656340+0000","last_undegraded":"2024-02-10T07:33:26.656340+0000","last_fullsized":"2024-02-10T07:33:26.656340+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656585+0000","last_change":"2024-02-10T07:33:26.457321+0000","last_active":"2024-02-10T07:33:26.656585+0000","last_peered":"2024-02-10T07:33:26.656585+0000","last_clean":"2024-02-10T07:33:26.656585+0000","last_became_active":"2024-02-10T07:31:57.315738+0000","last_became_peered":"2024-02-10T07:31:57.315738+0000","last_unstale":"2024-02-10T07:33:26.656585+0000","last_undegraded":"2024-02-10T07:33:26.656585+0000","last_fullsized":"2024-02-10T07:33:26.656585+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:26.457244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:26.457244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:31.708097+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656978+0000","last_change":"2024-02-10T07:31:55.302310+0000","last_active":"2024-02-10T07:33:26.656978+0000","last_peered":"2024-02-10T07:33:26.656978+0000","last_clean":"2024-02-10T07:33:26.656978+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:33:26.656978+0000","last_undegraded":"2024-02-10T07:33:26.656978+0000","last_fullsized":"2024-02-10T07:33:26.656978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.215338+0000","last_change":"2024-02-10T07:32:53.470087+0000","last_active":"2024-02-10T07:33:27.215338+0000","last_peered":"2024-02-10T07:33:27.215338+0000","last_clean":"2024-02-10T07:33:27.215338+0000","last_became_active":"2024-02-10T07:31:51.279438+0000","last_became_peered":"2024-02-10T07:31:51.279438+0000","last_unstale":"2024-02-10T07:33:27.215338+0000","last_undegraded":"2024-02-10T07:33:27.215338+0000","last_fullsized":"2024-02-10T07:33:27.215338+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:32:53.469958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:00.926321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656585+0000","last_change":"2024-02-10T07:32:58.537831+0000","last_active":"2024-02-10T07:33:26.656585+0000","last_peered":"2024-02-10T07:33:26.656585+0000","last_clean":"2024-02-10T07:33:26.656585+0000","last_became_active":"2024-02-10T07:31:53.287823+0000","last_became_peered":"2024-02-10T07:31:53.287823+0000","last_unstale":"2024-02-10T07:33:26.656585+0000","last_undegraded":"2024-02-10T07:33:26.656585+0000","last_fullsized":"2024-02-10T07:33:26.656585+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:32:58.537746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.537746+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:07.841413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.592405+0000","last_change":"2024-02-10T07:31:57.313822+0000","last_active":"2024-02-10T07:33:27.592405+0000","last_peered":"2024-02-10T07:33:27.592405+0000","last_clean":"2024-02-10T07:33:27.592405+0000","last_became_active":"2024-02-10T07:31:57.313529+0000","last_became_peered":"2024-02-10T07:31:57.313529+0000","last_unstale":"2024-02-10T07:33:27.592405+0000","last_undegraded":"2024-02-10T07:33:27.592405+0000","last_fullsized":"2024-02-10T07:33:27.592405+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656796+0000","last_change":"2024-02-10T07:32:59.584373+0000","last_active":"2024-02-10T07:33:26.656796+0000","last_peered":"2024-02-10T07:33:26.656796+0000","last_clean":"2024-02-10T07:33:26.656796+0000","last_became_active":"2024-02-10T07:31:55.296670+0000","last_became_peered":"2024-02-10T07:31:55.296670+0000","last_unstale":"2024-02-10T07:33:26.656796+0000","last_undegraded":"2024-02-10T07:33:26.656796+0000","last_fullsized":"2024-02-10T07:33:26.656796+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:59.584219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:59.584219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:22.310522+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.658323+0000","last_change":"2024-02-10T07:33:18.329963+0000","last_active":"2024-02-10T07:33:26.658323+0000","last_peered":"2024-02-10T07:33:26.658323+0000","last_clean":"2024-02-10T07:33:26.658323+0000","last_became_active":"2024-02-10T07:31:51.288165+0000","last_became_peered":"2024-02-10T07:31:51.288165+0000","last_unstale":"2024-02-10T07:33:26.658323+0000","last_undegraded":"2024-02-10T07:33:26.658323+0000","last_fullsized":"2024-02-10T07:33:26.658323+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:18.329878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:18.329878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:37.623492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.296151+0000","last_change":"2024-02-10T07:33:21.268130+0000","last_active":"2024-02-10T07:33:27.296151+0000","last_peered":"2024-02-10T07:33:27.296151+0000","last_clean":"2024-02-10T07:33:27.296151+0000","last_became_active":"2024-02-10T07:31:53.292182+0000","last_became_peered":"2024-02-10T07:31:53.292182+0000","last_unstale":"2024-02-10T07:33:27.296151+0000","last_undegraded":"2024-02-10T07:33:27.296151+0000","last_fullsized":"2024-02-10T07:33:27.296151+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:33:21.268030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:21.268030+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:45.065389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656344+0000","last_change":"2024-02-10T07:31:57.316162+0000","last_active":"2024-02-10T07:33:26.656344+0000","last_peered":"2024-02-10T07:33:26.656344+0000","last_clean":"2024-02-10T07:33:26.656344+0000","last_became_active":"2024-02-10T07:31:57.315781+0000","last_became_peered":"2024-02-10T07:31:57.315781+0000","last_unstale":"2024-02-10T07:33:26.656344+0000","last_undegraded":"2024-02-10T07:33:26.656344+0000","last_fullsized":"2024-02-10T07:33:26.656344+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.295964+0000","last_change":"2024-02-10T07:33:23.288263+0000","last_active":"2024-02-10T07:33:27.295964+0000","last_peered":"2024-02-10T07:33:27.295964+0000","last_clean":"2024-02-10T07:33:27.295964+0000","last_became_active":"2024-02-10T07:31:55.301260+0000","last_became_peered":"2024-02-10T07:31:55.301260+0000","last_unstale":"2024-02-10T07:33:27.295964+0000","last_undegraded":"2024-02-10T07:33:27.295964+0000","last_fullsized":"2024-02-10T07:33:27.295964+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:23.288135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:23.288135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:30.639812+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.658518+0000","last_change":"2024-02-10T07:33:22.333921+0000","last_active":"2024-02-10T07:33:26.658518+0000","last_peered":"2024-02-10T07:33:26.658518+0000","last_clean":"2024-02-10T07:33:26.658518+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:33:26.658518+0000","last_undegraded":"2024-02-10T07:33:26.658518+0000","last_fullsized":"2024-02-10T07:33:26.658518+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:22.333839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:22.333839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:28.687307+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657282+0000","last_change":"2024-02-10T07:33:00.144015+0000","last_active":"2024-02-10T07:33:26.657282+0000","last_peered":"2024-02-10T07:33:26.657282+0000","last_clean":"2024-02-10T07:33:26.657282+0000","last_became_active":"2024-02-10T07:32:18.389564+0000","last_became_peered":"2024-02-10T07:32:18.389564+0000","last_unstale":"2024-02-10T07:33:26.657282+0000","last_undegraded":"2024-02-10T07:33:26.657282+0000","last_fullsized":"2024-02-10T07:33:26.657282+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:33:00.143859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:00.143859+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:28.135685+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.592880+0000","last_change":"2024-02-10T07:33:22.616403+0000","last_active":"2024-02-10T07:33:27.592880+0000","last_peered":"2024-02-10T07:33:27.592880+0000","last_clean":"2024-02-10T07:33:27.592880+0000","last_became_active":"2024-02-10T07:31:57.318009+0000","last_became_peered":"2024-02-10T07:31:57.318009+0000","last_unstale":"2024-02-10T07:33:27.592880+0000","last_undegraded":"2024-02-10T07:33:27.592880+0000","last_fullsized":"2024-02-10T07:33:27.592880+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:22.616327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:22.616327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:36.984762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.215358+0000","last_change":"2024-02-10T07:33:15.216416+0000","last_active":"2024-02-10T07:33:27.215358+0000","last_peered":"2024-02-10T07:33:27.215358+0000","last_clean":"2024-02-10T07:33:27.215358+0000","last_became_active":"2024-02-10T07:31:55.297670+0000","last_became_peered":"2024-02-10T07:31:55.297670+0000","last_unstale":"2024-02-10T07:33:27.215358+0000","last_undegraded":"2024-02-10T07:33:27.215358+0000","last_fullsized":"2024-02-10T07:33:27.215358+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:33:15.216295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:15.216295+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:35.214128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.658197+0000","last_change":"2024-02-10T07:33:06.348460+0000","last_active":"2024-02-10T07:33:26.658197+0000","last_peered":"2024-02-10T07:33:26.658197+0000","last_clean":"2024-02-10T07:33:26.658197+0000","last_became_active":"2024-02-10T07:31:51.320644+0000","last_became_peered":"2024-02-10T07:31:51.320644+0000","last_unstale":"2024-02-10T07:33:26.658197+0000","last_undegraded":"2024-02-10T07:33:26.658197+0000","last_fullsized":"2024-02-10T07:33:26.658197+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.348322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:15.227477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656870+0000","last_change":"2024-02-10T07:31:53.296048+0000","last_active":"2024-02-10T07:33:26.656870+0000","last_peered":"2024-02-10T07:33:26.656870+0000","last_clean":"2024-02-10T07:33:26.656870+0000","last_became_active":"2024-02-10T07:31:53.295641+0000","last_became_peered":"2024-02-10T07:31:53.295641+0000","last_unstale":"2024-02-10T07:33:26.656870+0000","last_undegraded":"2024-02-10T07:33:26.656870+0000","last_fullsized":"2024-02-10T07:33:26.656870+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.659316+0000","last_change":"2024-02-10T07:31:57.307866+0000","last_active":"2024-02-10T07:33:26.659316+0000","last_peered":"2024-02-10T07:33:26.659316+0000","last_clean":"2024-02-10T07:33:26.659316+0000","last_became_active":"2024-02-10T07:31:57.307583+0000","last_became_peered":"2024-02-10T07:31:57.307583+0000","last_unstale":"2024-02-10T07:33:26.659316+0000","last_undegraded":"2024-02-10T07:33:26.659316+0000","last_fullsized":"2024-02-10T07:33:26.659316+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656985+0000","last_change":"2024-02-10T07:31:55.294941+0000","last_active":"2024-02-10T07:33:26.656985+0000","last_peered":"2024-02-10T07:33:26.656985+0000","last_clean":"2024-02-10T07:33:26.656985+0000","last_became_active":"2024-02-10T07:31:55.294594+0000","last_became_peered":"2024-02-10T07:31:55.294594+0000","last_unstale":"2024-02-10T07:33:26.656985+0000","last_undegraded":"2024-02-10T07:33:26.656985+0000","last_fullsized":"2024-02-10T07:33:26.656985+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.592264+0000","last_change":"2024-02-10T07:33:01.496748+0000","last_active":"2024-02-10T07:33:27.592264+0000","last_peered":"2024-02-10T07:33:27.592264+0000","last_clean":"2024-02-10T07:33:27.592264+0000","last_became_active":"2024-02-10T07:31:51.282353+0000","last_became_peered":"2024-02-10T07:31:51.282353+0000","last_unstale":"2024-02-10T07:33:27.592264+0000","last_undegraded":"2024-02-10T07:33:27.592264+0000","last_fullsized":"2024-02-10T07:33:27.592264+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:01.496610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:01.496610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:02.323527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.215491+0000","last_change":"2024-02-10T07:33:09.243608+0000","last_active":"2024-02-10T07:33:27.215491+0000","last_peered":"2024-02-10T07:33:27.215491+0000","last_clean":"2024-02-10T07:33:27.215491+0000","last_became_active":"2024-02-10T07:31:53.285777+0000","last_became_peered":"2024-02-10T07:31:53.285777+0000","last_unstale":"2024-02-10T07:33:27.215491+0000","last_undegraded":"2024-02-10T07:33:27.215491+0000","last_fullsized":"2024-02-10T07:33:27.215491+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:33:09.243500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:09.243500+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:37.334914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.699905+0000","last_change":"2024-02-10T07:33:10.120541+0000","last_active":"2024-02-10T07:33:26.699905+0000","last_peered":"2024-02-10T07:33:26.699905+0000","last_clean":"2024-02-10T07:33:26.699905+0000","last_became_active":"2024-02-10T07:31:57.308615+0000","last_became_peered":"2024-02-10T07:31:57.308615+0000","last_unstale":"2024-02-10T07:33:26.699905+0000","last_undegraded":"2024-02-10T07:33:26.699905+0000","last_fullsized":"2024-02-10T07:33:26.699905+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:33:10.120462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:13.165236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657437+0000","last_change":"2024-02-10T07:33:17.587557+0000","last_active":"2024-02-10T07:33:26.657437+0000","last_peered":"2024-02-10T07:33:26.657437+0000","last_clean":"2024-02-10T07:33:26.657437+0000","last_became_active":"2024-02-10T07:31:55.297431+0000","last_became_peered":"2024-02-10T07:31:55.297431+0000","last_unstale":"2024-02-10T07:33:26.657437+0000","last_undegraded":"2024-02-10T07:33:26.657437+0000","last_fullsized":"2024-02-10T07:33:26.657437+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:33:17.587433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:40.484002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.279932+0000","last_change":"2024-02-10T07:33:27.279956+0000","last_active":"2024-02-10T07:33:27.279932+0000","last_peered":"2024-02-10T07:33:27.279932+0000","last_clean":"2024-02-10T07:33:27.279932+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:33:27.279932+0000","last_undegraded":"2024-02-10T07:33:27.279932+0000","last_fullsized":"2024-02-10T07:33:27.279932+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:27.279843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:27.279843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:36.220022+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"89'13","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.593136+0000","last_change":"2024-02-10T07:32:56.585037+0000","last_active":"2024-02-10T07:33:27.593136+0000","last_peered":"2024-02-10T07:33:27.593136+0000","last_clean":"2024-02-10T07:33:27.593136+0000","last_became_active":"2024-02-10T07:31:53.295353+0000","last_became_peered":"2024-02-10T07:31:53.295353+0000","last_unstale":"2024-02-10T07:33:27.593136+0000","last_undegraded":"2024-02-10T07:33:27.593136+0000","last_fullsized":"2024-02-10T07:33:27.593136+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:32:56.584957+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:04.077539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.658218+0000","last_change":"2024-02-10T07:31:57.305113+0000","last_active":"2024-02-10T07:33:26.658218+0000","last_peered":"2024-02-10T07:33:26.658218+0000","last_clean":"2024-02-10T07:33:26.658218+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:33:26.658218+0000","last_undegraded":"2024-02-10T07:33:26.658218+0000","last_fullsized":"2024-02-10T07:33:26.658218+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.215411+0000","last_change":"2024-02-10T07:31:55.296699+0000","last_active":"2024-02-10T07:33:27.215411+0000","last_peered":"2024-02-10T07:33:27.215411+0000","last_clean":"2024-02-10T07:33:27.215411+0000","last_became_active":"2024-02-10T07:31:55.296369+0000","last_became_peered":"2024-02-10T07:31:55.296369+0000","last_unstale":"2024-02-10T07:33:27.215411+0000","last_undegraded":"2024-02-10T07:33:27.215411+0000","last_fullsized":"2024-02-10T07:33:27.215411+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.277767+0000","last_change":"2024-02-10T07:33:09.871286+0000","last_active":"2024-02-10T07:33:27.277767+0000","last_peered":"2024-02-10T07:33:27.277767+0000","last_clean":"2024-02-10T07:33:27.277767+0000","last_became_active":"2024-02-10T07:31:51.285159+0000","last_became_peered":"2024-02-10T07:31:51.285159+0000","last_unstale":"2024-02-10T07:33:27.277767+0000","last_undegraded":"2024-02-10T07:33:27.277767+0000","last_fullsized":"2024-02-10T07:33:27.277767+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:09.871194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:09.871194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:38.560272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657393+0000","last_change":"2024-02-10T07:31:53.297115+0000","last_active":"2024-02-10T07:33:26.657393+0000","last_peered":"2024-02-10T07:33:26.657393+0000","last_clean":"2024-02-10T07:33:26.657393+0000","last_became_active":"2024-02-10T07:31:53.296831+0000","last_became_peered":"2024-02-10T07:31:53.296831+0000","last_unstale":"2024-02-10T07:33:26.657393+0000","last_undegraded":"2024-02-10T07:33:26.657393+0000","last_fullsized":"2024-02-10T07:33:26.657393+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.214771+0000","last_change":"2024-02-10T07:31:57.311522+0000","last_active":"2024-02-10T07:33:27.214771+0000","last_peered":"2024-02-10T07:33:27.214771+0000","last_clean":"2024-02-10T07:33:27.214771+0000","last_became_active":"2024-02-10T07:31:57.311110+0000","last_became_peered":"2024-02-10T07:31:57.311110+0000","last_unstale":"2024-02-10T07:33:27.214771+0000","last_undegraded":"2024-02-10T07:33:27.214771+0000","last_fullsized":"2024-02-10T07:33:27.214771+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657270+0000","last_change":"2024-02-10T07:31:51.286209+0000","last_active":"2024-02-10T07:33:26.657270+0000","last_peered":"2024-02-10T07:33:26.657270+0000","last_clean":"2024-02-10T07:33:26.657270+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:33:26.657270+0000","last_undegraded":"2024-02-10T07:33:26.657270+0000","last_fullsized":"2024-02-10T07:33:26.657270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.214760+0000","last_change":"2024-02-10T07:32:59.385539+0000","last_active":"2024-02-10T07:33:27.214760+0000","last_peered":"2024-02-10T07:33:27.214760+0000","last_clean":"2024-02-10T07:33:27.214760+0000","last_became_active":"2024-02-10T07:31:53.284817+0000","last_became_peered":"2024-02-10T07:31:53.284817+0000","last_unstale":"2024-02-10T07:33:27.214760+0000","last_undegraded":"2024-02-10T07:33:27.214760+0000","last_fullsized":"2024-02-10T07:33:27.214760+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:32:59.385448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:59.385448+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:59.831143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.658095+0000","last_change":"2024-02-10T07:31:55.301606+0000","last_active":"2024-02-10T07:33:26.658095+0000","last_peered":"2024-02-10T07:33:26.658095+0000","last_clean":"2024-02-10T07:33:26.658095+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:33:26.658095+0000","last_undegraded":"2024-02-10T07:33:26.658095+0000","last_fullsized":"2024-02-10T07:33:26.658095+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656689+0000","last_change":"2024-02-10T07:31:57.315186+0000","last_active":"2024-02-10T07:33:26.656689+0000","last_peered":"2024-02-10T07:33:26.656689+0000","last_clean":"2024-02-10T07:33:26.656689+0000","last_became_active":"2024-02-10T07:31:57.314872+0000","last_became_peered":"2024-02-10T07:31:57.314872+0000","last_unstale":"2024-02-10T07:33:26.656689+0000","last_undegraded":"2024-02-10T07:33:26.656689+0000","last_fullsized":"2024-02-10T07:33:26.656689+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657795+0000","last_change":"2024-02-10T07:31:51.287017+0000","last_active":"2024-02-10T07:33:26.657795+0000","last_peered":"2024-02-10T07:33:26.657795+0000","last_clean":"2024-02-10T07:33:26.657795+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:33:26.657795+0000","last_undegraded":"2024-02-10T07:33:26.657795+0000","last_fullsized":"2024-02-10T07:33:26.657795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657832+0000","last_change":"2024-02-10T07:33:24.357572+0000","last_active":"2024-02-10T07:33:26.657832+0000","last_peered":"2024-02-10T07:33:26.657832+0000","last_clean":"2024-02-10T07:33:26.657832+0000","last_became_active":"2024-02-10T07:32:18.389409+0000","last_became_peered":"2024-02-10T07:32:18.389409+0000","last_unstale":"2024-02-10T07:33:26.657832+0000","last_undegraded":"2024-02-10T07:33:26.657832+0000","last_fullsized":"2024-02-10T07:33:26.657832+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:33:24.357485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:24.357485+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:44.468360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.699412+0000","last_change":"2024-02-10T07:32:59.218999+0000","last_active":"2024-02-10T07:33:26.699412+0000","last_peered":"2024-02-10T07:33:26.699412+0000","last_clean":"2024-02-10T07:33:26.699412+0000","last_became_active":"2024-02-10T07:31:55.303059+0000","last_became_peered":"2024-02-10T07:31:55.303059+0000","last_unstale":"2024-02-10T07:33:26.699412+0000","last_undegraded":"2024-02-10T07:33:26.699412+0000","last_fullsized":"2024-02-10T07:33:26.699412+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:59.218908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:59.218908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:06.437654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656624+0000","last_change":"2024-02-10T07:33:21.361093+0000","last_active":"2024-02-10T07:33:26.656624+0000","last_peered":"2024-02-10T07:33:26.656624+0000","last_clean":"2024-02-10T07:33:26.656624+0000","last_became_active":"2024-02-10T07:31:57.304971+0000","last_became_peered":"2024-02-10T07:31:57.304971+0000","last_unstale":"2024-02-10T07:33:26.656624+0000","last_undegraded":"2024-02-10T07:33:26.656624+0000","last_fullsized":"2024-02-10T07:33:26.656624+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:21.360834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:21.360834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:29.470777+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.699642+0000","last_change":"2024-02-10T07:33:16.199553+0000","last_active":"2024-02-10T07:33:26.699642+0000","last_peered":"2024-02-10T07:33:26.699642+0000","last_clean":"2024-02-10T07:33:26.699642+0000","last_became_active":"2024-02-10T07:31:55.298322+0000","last_became_peered":"2024-02-10T07:31:55.298322+0000","last_unstale":"2024-02-10T07:33:26.699642+0000","last_undegraded":"2024-02-10T07:33:26.699642+0000","last_fullsized":"2024-02-10T07:33:26.699642+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:16.199447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:16.199447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:27.934282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.658547+0000","last_change":"2024-02-10T07:33:24.347354+0000","last_active":"2024-02-10T07:33:26.658547+0000","last_peered":"2024-02-10T07:33:26.658547+0000","last_clean":"2024-02-10T07:33:26.658547+0000","last_became_active":"2024-02-10T07:31:51.283433+0000","last_became_peered":"2024-02-10T07:31:51.283433+0000","last_unstale":"2024-02-10T07:33:26.658547+0000","last_undegraded":"2024-02-10T07:33:26.658547+0000","last_fullsized":"2024-02-10T07:33:26.658547+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:24.347275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:24.347275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:39.523904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":93,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.278034+0000","last_change":"2024-02-10T07:33:16.799197+0000","last_active":"2024-02-10T07:33:27.278034+0000","last_peered":"2024-02-10T07:33:27.278034+0000","last_clean":"2024-02-10T07:33:27.278034+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:33:27.278034+0000","last_undegraded":"2024-02-10T07:33:27.278034+0000","last_fullsized":"2024-02-10T07:33:27.278034+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-10T07:33:16.799093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:16.799093+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:16.832132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.592480+0000","last_change":"2024-02-10T07:33:14.610211+0000","last_active":"2024-02-10T07:33:27.592480+0000","last_peered":"2024-02-10T07:33:27.592480+0000","last_clean":"2024-02-10T07:33:27.592480+0000","last_became_active":"2024-02-10T07:31:57.313394+0000","last_became_peered":"2024-02-10T07:31:57.313394+0000","last_unstale":"2024-02-10T07:33:27.592480+0000","last_undegraded":"2024-02-10T07:33:27.592480+0000","last_fullsized":"2024-02-10T07:33:27.592480+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:14.610134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:14.610134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:20.009681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.699579+0000","last_change":"2024-02-10T07:33:13.158162+0000","last_active":"2024-02-10T07:33:26.699579+0000","last_peered":"2024-02-10T07:33:26.699579+0000","last_clean":"2024-02-10T07:33:26.699579+0000","last_became_active":"2024-02-10T07:31:55.298622+0000","last_became_peered":"2024-02-10T07:31:55.298622+0000","last_unstale":"2024-02-10T07:33:26.699579+0000","last_undegraded":"2024-02-10T07:33:26.699579+0000","last_fullsized":"2024-02-10T07:33:26.699579+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:13.157988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:13.157988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:36.826105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.215014+0000","last_change":"2024-02-10T07:33:19.291637+0000","last_active":"2024-02-10T07:33:27.215014+0000","last_peered":"2024-02-10T07:33:27.215014+0000","last_clean":"2024-02-10T07:33:27.215014+0000","last_became_active":"2024-02-10T07:31:51.283208+0000","last_became_peered":"2024-02-10T07:31:51.283208+0000","last_unstale":"2024-02-10T07:33:27.215014+0000","last_undegraded":"2024-02-10T07:33:27.215014+0000","last_fullsized":"2024-02-10T07:33:27.215014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:19.291556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:19.291556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:45.842544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.592976+0000","last_change":"2024-02-10T07:33:18.612946+0000","last_active":"2024-02-10T07:33:27.592976+0000","last_peered":"2024-02-10T07:33:27.592976+0000","last_clean":"2024-02-10T07:33:27.592976+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:33:27.592976+0000","last_undegraded":"2024-02-10T07:33:27.592976+0000","last_fullsized":"2024-02-10T07:33:27.592976+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:33:18.612847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:18.612847+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:39.762053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657155+0000","last_change":"2024-02-10T07:31:57.314251+0000","last_active":"2024-02-10T07:33:26.657155+0000","last_peered":"2024-02-10T07:33:26.657155+0000","last_clean":"2024-02-10T07:33:26.657155+0000","last_became_active":"2024-02-10T07:31:57.313854+0000","last_became_peered":"2024-02-10T07:31:57.313854+0000","last_unstale":"2024-02-10T07:33:26.657155+0000","last_undegraded":"2024-02-10T07:33:26.657155+0000","last_fullsized":"2024-02-10T07:33:26.657155+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.214946+0000","last_change":"2024-02-10T07:33:12.268121+0000","last_active":"2024-02-10T07:33:27.214946+0000","last_peered":"2024-02-10T07:33:27.214946+0000","last_clean":"2024-02-10T07:33:27.214946+0000","last_became_active":"2024-02-10T07:31:50.278548+0000","last_became_peered":"2024-02-10T07:31:50.278548+0000","last_unstale":"2024-02-10T07:33:27.214946+0000","last_undegraded":"2024-02-10T07:33:27.214946+0000","last_fullsized":"2024-02-10T07:33:27.214946+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:33:12.268020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:33:12.268020+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:13.552959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001361225,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657122+0000","last_change":"2024-02-10T07:31:51.285951+0000","last_active":"2024-02-10T07:33:26.657122+0000","last_peered":"2024-02-10T07:33:26.657122+0000","last_clean":"2024-02-10T07:33:26.657122+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:33:26.657122+0000","last_undegraded":"2024-02-10T07:33:26.657122+0000","last_fullsized":"2024-02-10T07:33:26.657122+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":129,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.296579+0000","last_change":"2024-02-10T07:33:09.271524+0000","last_active":"2024-02-10T07:33:27.296579+0000","last_peered":"2024-02-10T07:33:27.296579+0000","last_clean":"2024-02-10T07:33:27.296579+0000","last_became_active":"2024-02-10T07:31:53.301863+0000","last_became_peered":"2024-02-10T07:31:53.301863+0000","last_unstale":"2024-02-10T07:33:27.296579+0000","last_undegraded":"2024-02-10T07:33:27.296579+0000","last_fullsized":"2024-02-10T07:33:27.296579+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-02-10T07:33:09.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:09.271402+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:33.250857+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.593114+0000","last_change":"2024-02-10T07:33:17.587762+0000","last_active":"2024-02-10T07:33:27.593114+0000","last_peered":"2024-02-10T07:33:27.593114+0000","last_clean":"2024-02-10T07:33:27.593114+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:33:27.593114+0000","last_undegraded":"2024-02-10T07:33:27.593114+0000","last_fullsized":"2024-02-10T07:33:27.593114+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:17.587649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:17.587649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:30.335639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.658615+0000","last_change":"2024-02-10T07:32:58.354147+0000","last_active":"2024-02-10T07:33:26.658615+0000","last_peered":"2024-02-10T07:33:26.658615+0000","last_clean":"2024-02-10T07:33:26.658615+0000","last_became_active":"2024-02-10T07:31:57.299958+0000","last_became_peered":"2024-02-10T07:31:57.299958+0000","last_unstale":"2024-02-10T07:33:26.658615+0000","last_undegraded":"2024-02-10T07:33:26.658615+0000","last_fullsized":"2024-02-10T07:33:26.658615+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:58.354022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.354022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:17.659362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.295819+0000","last_change":"2024-02-10T07:33:03.198787+0000","last_active":"2024-02-10T07:33:27.295819+0000","last_peered":"2024-02-10T07:33:27.295819+0000","last_clean":"2024-02-10T07:33:27.295819+0000","last_became_active":"2024-02-10T07:31:51.320864+0000","last_became_peered":"2024-02-10T07:31:51.320864+0000","last_unstale":"2024-02-10T07:33:27.295819+0000","last_undegraded":"2024-02-10T07:33:27.295819+0000","last_fullsized":"2024-02-10T07:33:27.295819+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:03.198699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:03.198699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:13.704299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.699561+0000","last_change":"2024-02-10T07:32:58.209386+0000","last_active":"2024-02-10T07:33:26.699561+0000","last_peered":"2024-02-10T07:33:26.699561+0000","last_clean":"2024-02-10T07:33:26.699561+0000","last_became_active":"2024-02-10T07:31:50.281517+0000","last_became_peered":"2024-02-10T07:31:50.281517+0000","last_unstale":"2024-02-10T07:33:26.699561+0000","last_undegraded":"2024-02-10T07:33:26.699561+0000","last_fullsized":"2024-02-10T07:33:26.699561+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:58.209303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:32:58.209303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:28.009863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.699808+0000","last_change":"2024-02-10T07:33:17.180228+0000","last_active":"2024-02-10T07:33:26.699808+0000","last_peered":"2024-02-10T07:33:26.699808+0000","last_clean":"2024-02-10T07:33:26.699808+0000","last_became_active":"2024-02-10T07:32:18.391217+0000","last_became_peered":"2024-02-10T07:32:18.391217+0000","last_unstale":"2024-02-10T07:33:26.699808+0000","last_undegraded":"2024-02-10T07:33:26.699808+0000","last_fullsized":"2024-02-10T07:33:26.699808+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-10T07:33:17.180142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:17.180142+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:40.010786+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.699770+0000","last_change":"2024-02-10T07:33:06.183420+0000","last_active":"2024-02-10T07:33:26.699770+0000","last_peered":"2024-02-10T07:33:26.699770+0000","last_clean":"2024-02-10T07:33:26.699770+0000","last_became_active":"2024-02-10T07:31:55.298933+0000","last_became_peered":"2024-02-10T07:31:55.298933+0000","last_unstale":"2024-02-10T07:33:26.699770+0000","last_undegraded":"2024-02-10T07:33:26.699770+0000","last_fullsized":"2024-02-10T07:33:26.699770+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.183292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.183292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:12.631994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"74'1","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.215420+0000","last_change":"2024-02-10T07:31:57.310237+0000","last_active":"2024-02-10T07:33:27.215420+0000","last_peered":"2024-02-10T07:33:27.215420+0000","last_clean":"2024-02-10T07:33:27.215420+0000","last_became_active":"2024-02-10T07:31:57.309929+0000","last_became_peered":"2024-02-10T07:31:57.309929+0000","last_unstale":"2024-02-10T07:33:27.215420+0000","last_undegraded":"2024-02-10T07:33:27.215420+0000","last_fullsized":"2024-02-10T07:33:27.215420+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":312,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656759+0000","last_change":"2024-02-10T07:32:25.175725+0000","last_active":"2024-02-10T07:33:26.656759+0000","last_peered":"2024-02-10T07:33:26.656759+0000","last_clean":"2024-02-10T07:33:26.656759+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:33:26.656759+0000","last_undegraded":"2024-02-10T07:33:26.656759+0000","last_fullsized":"2024-02-10T07:33:26.656759+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-02-10T07:32:25.175641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:32:25.175641+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:33:35.761800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656584+0000","last_change":"2024-02-10T07:33:23.493888+0000","last_active":"2024-02-10T07:33:26.656584+0000","last_peered":"2024-02-10T07:33:26.656584+0000","last_clean":"2024-02-10T07:33:26.656584+0000","last_became_active":"2024-02-10T07:31:51.283332+0000","last_became_peered":"2024-02-10T07:31:51.283332+0000","last_unstale":"2024-02-10T07:33:26.656584+0000","last_undegraded":"2024-02-10T07:33:26.656584+0000","last_fullsized":"2024-02-10T07:33:26.656584+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:23.493813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:23.493813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:48.821258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"89'24","reported_seq":112,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.278465+0000","last_change":"2024-02-10T07:33:11.841322+0000","last_active":"2024-02-10T07:33:27.278465+0000","last_peered":"2024-02-10T07:33:27.278465+0000","last_clean":"2024-02-10T07:33:27.278465+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:33:27.278465+0000","last_undegraded":"2024-02-10T07:33:27.278465+0000","last_fullsized":"2024-02-10T07:33:27.278465+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'24","last_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:33:11.841200+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:33.809552+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657386+0000","last_change":"2024-02-10T07:31:55.300257+0000","last_active":"2024-02-10T07:33:26.657386+0000","last_peered":"2024-02-10T07:33:26.657386+0000","last_clean":"2024-02-10T07:33:26.657386+0000","last_became_active":"2024-02-10T07:31:55.299928+0000","last_became_peered":"2024-02-10T07:31:55.299928+0000","last_unstale":"2024-02-10T07:33:26.657386+0000","last_undegraded":"2024-02-10T07:33:26.657386+0000","last_fullsized":"2024-02-10T07:33:26.657386+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":53,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.297140+0000","last_change":"2024-02-10T07:33:27.297160+0000","last_active":"2024-02-10T07:33:27.297140+0000","last_peered":"2024-02-10T07:33:27.297140+0000","last_clean":"2024-02-10T07:33:27.297140+0000","last_became_active":"2024-02-10T07:31:57.316432+0000","last_became_peered":"2024-02-10T07:31:57.316432+0000","last_unstale":"2024-02-10T07:33:27.297140+0000","last_undegraded":"2024-02-10T07:33:27.297140+0000","last_fullsized":"2024-02-10T07:33:27.297140+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:27.297064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:27.297064+0000","last_clean_scrub_stamp":"2024-02-10T07:33:27.297064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:31.787062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.592205+0000","last_change":"2024-02-10T07:31:51.286361+0000","last_active":"2024-02-10T07:33:27.592205+0000","last_peered":"2024-02-10T07:33:27.592205+0000","last_clean":"2024-02-10T07:33:27.592205+0000","last_became_active":"2024-02-10T07:31:51.286076+0000","last_became_peered":"2024-02-10T07:31:51.286076+0000","last_unstale":"2024-02-10T07:33:27.592205+0000","last_undegraded":"2024-02-10T07:33:27.592205+0000","last_fullsized":"2024-02-10T07:33:27.592205+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":162,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656773+0000","last_change":"2024-02-10T07:31:54.281755+0000","last_active":"2024-02-10T07:33:26.656773+0000","last_peered":"2024-02-10T07:33:26.656773+0000","last_clean":"2024-02-10T07:33:26.656773+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:33:26.656773+0000","last_undegraded":"2024-02-10T07:33:26.656773+0000","last_fullsized":"2024-02-10T07:33:26.656773+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:31:49.265852+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.592999+0000","last_change":"2024-02-10T07:31:53.301026+0000","last_active":"2024-02-10T07:33:27.592999+0000","last_peered":"2024-02-10T07:33:27.592999+0000","last_clean":"2024-02-10T07:33:27.592999+0000","last_became_active":"2024-02-10T07:31:53.300687+0000","last_became_peered":"2024-02-10T07:31:53.300687+0000","last_unstale":"2024-02-10T07:33:27.592999+0000","last_undegraded":"2024-02-10T07:33:27.592999+0000","last_fullsized":"2024-02-10T07:33:27.592999+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.215608+0000","last_change":"2024-02-10T07:31:55.297440+0000","last_active":"2024-02-10T07:33:27.215608+0000","last_peered":"2024-02-10T07:33:27.215608+0000","last_clean":"2024-02-10T07:33:27.215608+0000","last_became_active":"2024-02-10T07:31:55.297178+0000","last_became_peered":"2024-02-10T07:31:55.297178+0000","last_unstale":"2024-02-10T07:33:27.215608+0000","last_undegraded":"2024-02-10T07:33:27.215608+0000","last_fullsized":"2024-02-10T07:33:27.215608+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.295791+0000","last_change":"2024-02-10T07:31:57.307961+0000","last_active":"2024-02-10T07:33:27.295791+0000","last_peered":"2024-02-10T07:33:27.295791+0000","last_clean":"2024-02-10T07:33:27.295791+0000","last_became_active":"2024-02-10T07:31:57.307615+0000","last_became_peered":"2024-02-10T07:31:57.307615+0000","last_unstale":"2024-02-10T07:33:27.295791+0000","last_undegraded":"2024-02-10T07:33:27.295791+0000","last_fullsized":"2024-02-10T07:33:27.295791+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.296477+0000","last_change":"2024-02-10T07:33:06.270053+0000","last_active":"2024-02-10T07:33:27.296477+0000","last_peered":"2024-02-10T07:33:27.296477+0000","last_clean":"2024-02-10T07:33:27.296477+0000","last_became_active":"2024-02-10T07:31:51.320752+0000","last_became_peered":"2024-02-10T07:31:51.320752+0000","last_unstale":"2024-02-10T07:33:27.296477+0000","last_undegraded":"2024-02-10T07:33:27.296477+0000","last_fullsized":"2024-02-10T07:33:27.296477+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.269938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.269938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:34.431697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":90,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.592919+0000","last_change":"2024-02-10T07:31:53.289754+0000","last_active":"2024-02-10T07:33:27.592919+0000","last_peered":"2024-02-10T07:33:27.592919+0000","last_clean":"2024-02-10T07:33:27.592919+0000","last_became_active":"2024-02-10T07:31:53.289381+0000","last_became_peered":"2024-02-10T07:31:53.289381+0000","last_unstale":"2024-02-10T07:33:27.592919+0000","last_undegraded":"2024-02-10T07:33:27.592919+0000","last_fullsized":"2024-02-10T07:33:27.592919+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.278055+0000","last_change":"2024-02-10T07:31:55.290623+0000","last_active":"2024-02-10T07:33:27.278055+0000","last_peered":"2024-02-10T07:33:27.278055+0000","last_clean":"2024-02-10T07:33:27.278055+0000","last_became_active":"2024-02-10T07:31:55.290319+0000","last_became_peered":"2024-02-10T07:31:55.290319+0000","last_unstale":"2024-02-10T07:33:27.278055+0000","last_undegraded":"2024-02-10T07:33:27.278055+0000","last_fullsized":"2024-02-10T07:33:27.278055+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.592430+0000","last_change":"2024-02-10T07:33:05.431169+0000","last_active":"2024-02-10T07:33:27.592430+0000","last_peered":"2024-02-10T07:33:27.592430+0000","last_clean":"2024-02-10T07:33:27.592430+0000","last_became_active":"2024-02-10T07:31:57.317976+0000","last_became_peered":"2024-02-10T07:31:57.317976+0000","last_unstale":"2024-02-10T07:33:27.592430+0000","last_undegraded":"2024-02-10T07:33:27.592430+0000","last_fullsized":"2024-02-10T07:33:27.592430+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:05.431037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:05.431037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:31.470574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.215182+0000","last_change":"2024-02-10T07:31:51.320993+0000","last_active":"2024-02-10T07:33:27.215182+0000","last_peered":"2024-02-10T07:33:27.215182+0000","last_clean":"2024-02-10T07:33:27.215182+0000","last_became_active":"2024-02-10T07:31:51.320673+0000","last_became_peered":"2024-02-10T07:31:51.320673+0000","last_unstale":"2024-02-10T07:33:27.215182+0000","last_undegraded":"2024-02-10T07:33:27.215182+0000","last_fullsized":"2024-02-10T07:33:27.215182+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.296120+0000","last_change":"2024-02-10T07:32:53.288963+0000","last_active":"2024-02-10T07:33:27.296120+0000","last_peered":"2024-02-10T07:33:27.296120+0000","last_clean":"2024-02-10T07:33:27.296120+0000","last_became_active":"2024-02-10T07:31:53.297150+0000","last_became_peered":"2024-02-10T07:31:53.297150+0000","last_unstale":"2024-02-10T07:33:27.296120+0000","last_undegraded":"2024-02-10T07:33:27.296120+0000","last_fullsized":"2024-02-10T07:33:27.296120+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:32:53.288798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:53.288798+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:06.843116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.593077+0000","last_change":"2024-02-10T07:31:55.296952+0000","last_active":"2024-02-10T07:33:27.593077+0000","last_peered":"2024-02-10T07:33:27.593077+0000","last_clean":"2024-02-10T07:33:27.593077+0000","last_became_active":"2024-02-10T07:31:55.295732+0000","last_became_peered":"2024-02-10T07:31:55.295732+0000","last_unstale":"2024-02-10T07:33:27.593077+0000","last_undegraded":"2024-02-10T07:33:27.593077+0000","last_fullsized":"2024-02-10T07:33:27.593077+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.658295+0000","last_change":"2024-02-10T07:31:57.305942+0000","last_active":"2024-02-10T07:33:26.658295+0000","last_peered":"2024-02-10T07:33:26.658295+0000","last_clean":"2024-02-10T07:33:26.658295+0000","last_became_active":"2024-02-10T07:31:57.305620+0000","last_became_peered":"2024-02-10T07:31:57.305620+0000","last_unstale":"2024-02-10T07:33:26.658295+0000","last_undegraded":"2024-02-10T07:33:26.658295+0000","last_fullsized":"2024-02-10T07:33:26.658295+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.592273+0000","last_change":"2024-02-10T07:33:15.564016+0000","last_active":"2024-02-10T07:33:27.592273+0000","last_peered":"2024-02-10T07:33:27.592273+0000","last_clean":"2024-02-10T07:33:27.592273+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:33:27.592273+0000","last_undegraded":"2024-02-10T07:33:27.592273+0000","last_fullsized":"2024-02-10T07:33:27.592273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:33:15.563932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:15.563932+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:23.107286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657609+0000","last_change":"2024-02-10T07:31:53.287199+0000","last_active":"2024-02-10T07:33:26.657609+0000","last_peered":"2024-02-10T07:33:26.657609+0000","last_clean":"2024-02-10T07:33:26.657609+0000","last_became_active":"2024-02-10T07:31:53.286884+0000","last_became_peered":"2024-02-10T07:31:53.286884+0000","last_unstale":"2024-02-10T07:33:26.657609+0000","last_undegraded":"2024-02-10T07:33:26.657609+0000","last_fullsized":"2024-02-10T07:33:26.657609+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656504+0000","last_change":"2024-02-10T07:32:55.469184+0000","last_active":"2024-02-10T07:33:26.656504+0000","last_peered":"2024-02-10T07:33:26.656504+0000","last_clean":"2024-02-10T07:33:26.656504+0000","last_became_active":"2024-02-10T07:31:55.302863+0000","last_became_peered":"2024-02-10T07:31:55.302863+0000","last_unstale":"2024-02-10T07:33:26.656504+0000","last_undegraded":"2024-02-10T07:33:26.656504+0000","last_fullsized":"2024-02-10T07:33:26.656504+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:32:55.469099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:55.469099+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:13.566389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.658056+0000","last_change":"2024-02-10T07:31:57.316332+0000","last_active":"2024-02-10T07:33:26.658056+0000","last_peered":"2024-02-10T07:33:26.658056+0000","last_clean":"2024-02-10T07:33:26.658056+0000","last_became_active":"2024-02-10T07:31:57.316047+0000","last_became_peered":"2024-02-10T07:31:57.316047+0000","last_unstale":"2024-02-10T07:33:26.658056+0000","last_undegraded":"2024-02-10T07:33:26.658056+0000","last_fullsized":"2024-02-10T07:33:26.658056+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657329+0000","last_change":"2024-02-10T07:33:20.491471+0000","last_active":"2024-02-10T07:33:26.657329+0000","last_peered":"2024-02-10T07:33:26.657329+0000","last_clean":"2024-02-10T07:33:26.657329+0000","last_became_active":"2024-02-10T07:31:51.285333+0000","last_became_peered":"2024-02-10T07:31:51.285333+0000","last_unstale":"2024-02-10T07:33:26.657329+0000","last_undegraded":"2024-02-10T07:33:26.657329+0000","last_fullsized":"2024-02-10T07:33:26.657329+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:20.491354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:20.491354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:29.702384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656443+0000","last_change":"2024-02-10T07:33:08.541272+0000","last_active":"2024-02-10T07:33:26.656443+0000","last_peered":"2024-02-10T07:33:26.656443+0000","last_clean":"2024-02-10T07:33:26.656443+0000","last_became_active":"2024-02-10T07:31:53.290157+0000","last_became_peered":"2024-02-10T07:31:53.290157+0000","last_unstale":"2024-02-10T07:33:26.656443+0000","last_undegraded":"2024-02-10T07:33:26.656443+0000","last_fullsized":"2024-02-10T07:33:26.656443+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-10T07:33:08.541155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:08.541155+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:33.263033+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657358+0000","last_change":"2024-02-10T07:33:01.184183+0000","last_active":"2024-02-10T07:33:26.657358+0000","last_peered":"2024-02-10T07:33:26.657358+0000","last_clean":"2024-02-10T07:33:26.657358+0000","last_became_active":"2024-02-10T07:31:55.295900+0000","last_became_peered":"2024-02-10T07:31:55.295900+0000","last_unstale":"2024-02-10T07:33:26.657358+0000","last_undegraded":"2024-02-10T07:33:26.657358+0000","last_fullsized":"2024-02-10T07:33:26.657358+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:01.184048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:01.184048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:11.992366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.215100+0000","last_change":"2024-02-10T07:31:57.311731+0000","last_active":"2024-02-10T07:33:27.215100+0000","last_peered":"2024-02-10T07:33:27.215100+0000","last_clean":"2024-02-10T07:33:27.215100+0000","last_became_active":"2024-02-10T07:31:57.311054+0000","last_became_peered":"2024-02-10T07:31:57.311054+0000","last_unstale":"2024-02-10T07:33:27.215100+0000","last_undegraded":"2024-02-10T07:33:27.215100+0000","last_fullsized":"2024-02-10T07:33:27.215100+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:31:56.282886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.659114+0000","last_change":"2024-02-10T07:31:55.295389+0000","last_active":"2024-02-10T07:33:26.659114+0000","last_peered":"2024-02-10T07:33:26.659114+0000","last_clean":"2024-02-10T07:33:26.659114+0000","last_became_active":"2024-02-10T07:31:55.295067+0000","last_became_peered":"2024-02-10T07:31:55.295067+0000","last_unstale":"2024-02-10T07:33:26.659114+0000","last_undegraded":"2024-02-10T07:33:26.659114+0000","last_fullsized":"2024-02-10T07:33:26.659114+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:31:54.275917+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656418+0000","last_change":"2024-02-10T07:31:51.282545+0000","last_active":"2024-02-10T07:33:26.656418+0000","last_peered":"2024-02-10T07:33:26.656418+0000","last_clean":"2024-02-10T07:33:26.656418+0000","last_became_active":"2024-02-10T07:31:51.282137+0000","last_became_peered":"2024-02-10T07:31:51.282137+0000","last_unstale":"2024-02-10T07:33:26.656418+0000","last_undegraded":"2024-02-10T07:33:26.656418+0000","last_fullsized":"2024-02-10T07:33:26.656418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:31:50.269547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.296877+0000","last_change":"2024-02-10T07:32:57.341249+0000","last_active":"2024-02-10T07:33:27.296877+0000","last_peered":"2024-02-10T07:33:27.296877+0000","last_clean":"2024-02-10T07:33:27.296877+0000","last_became_active":"2024-02-10T07:31:53.294218+0000","last_became_peered":"2024-02-10T07:31:53.294218+0000","last_unstale":"2024-02-10T07:33:27.296877+0000","last_undegraded":"2024-02-10T07:33:27.296877+0000","last_fullsized":"2024-02-10T07:33:27.296877+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:32:57.341115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:32:57.341115+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:05.137575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656751+0000","last_change":"2024-02-10T07:33:04.582281+0000","last_active":"2024-02-10T07:33:26.656751+0000","last_peered":"2024-02-10T07:33:26.656751+0000","last_clean":"2024-02-10T07:33:26.656751+0000","last_became_active":"2024-02-10T07:31:57.303235+0000","last_became_peered":"2024-02-10T07:31:57.303235+0000","last_unstale":"2024-02-10T07:33:26.656751+0000","last_undegraded":"2024-02-10T07:33:26.656751+0000","last_fullsized":"2024-02-10T07:33:26.656751+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:04.582188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:04.582188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:27.553405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.215576+0000","last_change":"2024-02-10T07:33:06.237068+0000","last_active":"2024-02-10T07:33:27.215576+0000","last_peered":"2024-02-10T07:33:27.215576+0000","last_clean":"2024-02-10T07:33:27.215576+0000","last_became_active":"2024-02-10T07:31:55.294376+0000","last_became_peered":"2024-02-10T07:31:55.294376+0000","last_unstale":"2024-02-10T07:33:27.215576+0000","last_undegraded":"2024-02-10T07:33:27.215576+0000","last_fullsized":"2024-02-10T07:33:27.215576+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.236946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.236946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:10.623394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657450+0000","last_change":"2024-02-10T07:33:19.231720+0000","last_active":"2024-02-10T07:33:26.657450+0000","last_peered":"2024-02-10T07:33:26.657450+0000","last_clean":"2024-02-10T07:33:26.657450+0000","last_became_active":"2024-02-10T07:31:51.320715+0000","last_became_peered":"2024-02-10T07:31:51.320715+0000","last_unstale":"2024-02-10T07:33:26.657450+0000","last_undegraded":"2024-02-10T07:33:26.657450+0000","last_fullsized":"2024-02-10T07:33:26.657450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:19.231613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:19.231613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:34.092643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"89'16","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.658987+0000","last_change":"2024-02-10T07:33:21.287093+0000","last_active":"2024-02-10T07:33:26.658987+0000","last_peered":"2024-02-10T07:33:26.658987+0000","last_clean":"2024-02-10T07:33:26.658987+0000","last_became_active":"2024-02-10T07:31:53.292510+0000","last_became_peered":"2024-02-10T07:31:53.292510+0000","last_unstale":"2024-02-10T07:33:26.658987+0000","last_undegraded":"2024-02-10T07:33:26.658987+0000","last_fullsized":"2024-02-10T07:33:26.658987+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'16","last_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:33:21.286955+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:43.692133+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.699926+0000","last_change":"2024-02-10T07:33:05.197724+0000","last_active":"2024-02-10T07:33:26.699926+0000","last_peered":"2024-02-10T07:33:26.699926+0000","last_clean":"2024-02-10T07:33:26.699926+0000","last_became_active":"2024-02-10T07:31:57.311427+0000","last_became_peered":"2024-02-10T07:31:57.311427+0000","last_unstale":"2024-02-10T07:33:26.699926+0000","last_undegraded":"2024-02-10T07:33:26.699926+0000","last_fullsized":"2024-02-10T07:33:26.699926+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:33:05.197642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:20.323214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":57,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.296702+0000","last_change":"2024-02-10T07:33:12.259596+0000","last_active":"2024-02-10T07:33:27.296702+0000","last_peered":"2024-02-10T07:33:27.296702+0000","last_clean":"2024-02-10T07:33:27.296702+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:33:27.296702+0000","last_undegraded":"2024-02-10T07:33:27.296702+0000","last_fullsized":"2024-02-10T07:33:27.296702+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:12.259478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:12.259478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:41.862761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:27.278220+0000","last_change":"2024-02-10T07:32:56.900599+0000","last_active":"2024-02-10T07:33:27.278220+0000","last_peered":"2024-02-10T07:33:27.278220+0000","last_clean":"2024-02-10T07:33:27.278220+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:33:27.278220+0000","last_undegraded":"2024-02-10T07:33:27.278220+0000","last_fullsized":"2024-02-10T07:33:27.278220+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:56.900501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:56.900501+0000","last_clean_scrub_stamp":"2024-02-10T07:32:56.900501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:16.803474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"89'25","reported_seq":99,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.659037+0000","last_change":"2024-02-10T07:33:20.252130+0000","last_active":"2024-02-10T07:33:26.659037+0000","last_peered":"2024-02-10T07:33:26.659037+0000","last_clean":"2024-02-10T07:33:26.659037+0000","last_became_active":"2024-02-10T07:31:53.292277+0000","last_became_peered":"2024-02-10T07:31:53.292277+0000","last_unstale":"2024-02-10T07:33:26.659037+0000","last_undegraded":"2024-02-10T07:33:26.659037+0000","last_fullsized":"2024-02-10T07:33:26.659037+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'25","last_scrub_stamp":"2024-02-10T07:33:20.251980+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:33:20.251980+0000","last_clean_scrub_stamp":"2024-02-10T07:33:20.251980+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:50.016769+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.659202+0000","last_change":"2024-02-10T07:33:25.318627+0000","last_active":"2024-02-10T07:33:26.659202+0000","last_peered":"2024-02-10T07:33:26.659202+0000","last_clean":"2024-02-10T07:33:26.659202+0000","last_became_active":"2024-02-10T07:31:57.308608+0000","last_became_peered":"2024-02-10T07:31:57.308608+0000","last_unstale":"2024-02-10T07:33:26.659202+0000","last_undegraded":"2024-02-10T07:33:26.659202+0000","last_fullsized":"2024-02-10T07:33:26.659202+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:25.318521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:25.318521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:52.291449+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":89,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.699668+0000","last_change":"2024-02-10T07:33:26.105310+0000","last_active":"2024-02-10T07:33:26.699668+0000","last_peered":"2024-02-10T07:33:26.699668+0000","last_clean":"2024-02-10T07:33:26.699668+0000","last_became_active":"2024-02-10T07:31:55.298389+0000","last_became_peered":"2024-02-10T07:31:55.298389+0000","last_unstale":"2024-02-10T07:33:26.699668+0000","last_undegraded":"2024-02-10T07:33:26.699668+0000","last_fullsized":"2024-02-10T07:33:26.699668+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:33:26.105217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:26.105217+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:49.252325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.656799+0000","last_change":"2024-02-10T07:33:19.515540+0000","last_active":"2024-02-10T07:33:26.656799+0000","last_peered":"2024-02-10T07:33:26.656799+0000","last_clean":"2024-02-10T07:33:26.656799+0000","last_became_active":"2024-02-10T07:31:51.288197+0000","last_became_peered":"2024-02-10T07:31:51.288197+0000","last_unstale":"2024-02-10T07:33:26.656799+0000","last_undegraded":"2024-02-10T07:33:26.656799+0000","last_fullsized":"2024-02-10T07:33:26.656799+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:19.515464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:19.515464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:34:46.814548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-10T07:33:26.657247+0000","last_change":"2024-02-10T07:31:53.288289+0000","last_active":"2024-02-10T07:33:26.657247+0000","last_peered":"2024-02-10T07:33:26.657247+0000","last_clean":"2024-02-10T07:33:26.657247+0000","last_became_active":"2024-02-10T07:31:53.287984+0000","last_became_peered":"2024-02-10T07:31:53.287984+0000","last_unstale":"2024-02-10T07:33:26.657247+0000","last_undegraded":"2024-02-10T07:33:26.657247+0000","last_fullsized":"2024-02-10T07:33:26.657247+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:31:52.272420+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":425,"ondisk_log_size":425,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037784,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28508,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716644,"statfs":{"total":95995035648,"available":95965843456,"internally_reserved":0,"allocated":1519616,"data_stored":1028883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.748}]},{"osd":1,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.68400000000000005}]},{"osd":2,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.64100000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.72699999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.79100000000000004}]},{"osd":6,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.76300000000000001}]}]},{"osd":6,"up_from":50,"seq":214748364829,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1490944,"data_stored":1020522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.36099999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77000000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.64100000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.66600000000000004,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.67500000000000004}]},{"osd":2,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.77400000000000002,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.627},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73499999999999999,"5min":0.872,"15min":0.872},"last":0.68899999999999995}]},{"osd":3,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":1.544,"15min":1.7070000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.72399999999999998,"5min":19.664999999999999,"15min":19.664999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.61299999999999999,"15min":0.61499999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83599999999999997,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.58199999999999996}]},{"osd":4,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.47499999999999998,"15min":0.46999999999999997},"min":{"1min":0.35999999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.48799999999999999,"15min":0.47899999999999998},"min":{"1min":0.34399999999999997,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.36599999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48999999999999999,"15min":0.48799999999999999},"min":{"1min":0.33700000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.51400000000000001,"15min":0.51000000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.44900000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.47799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36252,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708900,"statfs":{"total":95995035648,"available":95957913600,"internally_reserved":0,"allocated":933888,"data_stored":430881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.374,"15min":0.34599999999999997},"min":{"1min":0.249,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.378},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.40400000000000003,"15min":0.36399999999999999},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.313}]},{"osd":2,"last update":"Sat Feb 10 07:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41299999999999998,"15min":0.40000000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42899999999999999,"15min":0.41799999999999998},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.35799999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:33:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.42599999999999999,"15min":0.42399999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.58499999999999996,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.51200000000000001,"15min":0.50900000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.504}]},{"osd":4,"last update":"Sat Feb 10 07:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.89200000000000002,"15min":1.0149999999999999},"min":{"1min":0.435,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.79200000000000004,"5min":10.292,"15min":10.292},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.41199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.70899999999999996,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.51500000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:32:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.627,"15min":0.63},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80500000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.44400000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.623,"15min":0.628},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.79800000000000004,"5min":1.002,"15min":1.002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.83699999999999997,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46300000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.47199999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712492,"statfs":{"total":95995035648,"available":95961591808,"internally_reserved":0,"allocated":1515520,"data_stored":1021792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34599999999999997,"15min":0.34399999999999997},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.41099999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.36499999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.55200000000000005}]},{"osd":2,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.38500000000000001,"15min":0.38800000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.51200000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42899999999999999,"15min":0.42499999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57899999999999996,"5min":0.629,"15min":0.629},"last":0.50900000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:33:05 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.432,"15min":0.443},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.53800000000000003,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47499999999999998,"15min":0.47199999999999998},"min":{"1min":0.32400000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64700000000000002,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.434}]},{"osd":4,"last update":"Sat Feb 10 07:32:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.74099999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.433,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56599999999999995}]},{"osd":5,"last update":"Sat Feb 10 07:32:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59699999999999998,"15min":0.59399999999999997},"min":{"1min":0.438,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.62}]},{"osd":6,"last update":"Sat Feb 10 07:33:12 2024","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":0.83499999999999996,"15min":0.70599999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":19.16,"5min":19.16,"15min":19.16},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.61899999999999999,"15min":0.63100000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.67200000000000004,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.66400000000000003}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":19.178000000000001,"5min":19.178000000000001,"15min":19.178000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.67500000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":929792,"data_stored":439112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.379,"15min":0.38},"min":{"1min":0.255,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56499999999999995,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.432,"5min":0.42399999999999999,"15min":0.41599999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.58499999999999996}]},{"osd":1,"last update":"Sat Feb 10 07:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.44,"15min":0.42399999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.44,"15min":0.42199999999999999},"min":{"1min":0.27200000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.54500000000000004}]},{"osd":3,"last update":"Sat Feb 10 07:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51700000000000002,"15min":0.53400000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.624},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.53100000000000003,"15min":0.55300000000000005},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.60899999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63100000000000001,"15min":0.626},"min":{"1min":0.50700000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.68899999999999995,"15min":0.65300000000000002},"min":{"1min":0.50700000000000001,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":5.343,"5min":5.343,"15min":5.343},"last":0.996}]},{"osd":5,"last update":"Sat Feb 10 07:32:51 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.64900000000000002,"15min":0.64400000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.628,"15min":0.623},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":1.0880000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.85099999999999998,"15min":0.879},"min":{"1min":0.45900000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.321,"5min":5.3230000000000004,"15min":5.3230000000000004},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64500000000000002,"15min":0.64300000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":1.0620000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":1.1040000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32108,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713044,"statfs":{"total":95995035648,"available":95962157056,"internally_reserved":0,"allocated":946176,"data_stored":438368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.372,"15min":0.378},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.55500000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.32300000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.42099999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.442,"15min":0.437},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:33:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.31,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.42799999999999999,"15min":0.434},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.30399999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.61499999999999999,"15min":0.61399999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.89700000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.45900000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:32:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.54000000000000004}]},{"osd":6,"last update":"Sat Feb 10 07:33:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67000000000000004,"15min":0.67400000000000004},"min":{"1min":0.41999999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.40999999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.42299999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953511,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":942080,"data_stored":438480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.64500000000000002,"15min":0.60599999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":7.5700000000000003,"5min":7.5700000000000003,"15min":7.5700000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72899999999999998,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.57499999999999996}]},{"osd":1,"last update":"Sat Feb 10 07:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54100000000000004,"15min":0.54800000000000004},"min":{"1min":0.34799999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.63,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55500000000000005,"15min":0.56699999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.46700000000000003}]},{"osd":2,"last update":"Sat Feb 10 07:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54100000000000004,"15min":0.53800000000000003},"min":{"1min":0.40999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69699999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.58399999999999996,"15min":0.59999999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66500000000000004,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.51700000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57899999999999996,"15min":0.58299999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.76900000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.626,"15min":0.57699999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":7.601,"5min":7.601,"15min":7.601},"last":0.501}]},{"osd":5,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.378,"15min":0.373},"min":{"1min":0.30099999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.60799999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.40300000000000002,"15min":0.39300000000000002},"min":{"1min":0.32300000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.432}]},{"osd":6,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.44400000000000001,"15min":0.42999999999999999},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.44,"15min":0.42999999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.41299999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.371},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.44900000000000001}]}]},{"osd":5,"up_from":40,"seq":171798691874,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":921600,"data_stored":439074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.77400000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.57699999999999996,"15min":0.56100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.2029999999999998,"5min":3.2029999999999998,"15min":3.2029999999999998},"last":0.39800000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.46200000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.75,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.61299999999999999,"15min":0.59699999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.238,"5min":3.238,"15min":3.238},"last":0.47299999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.438,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76200000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.748,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.76200000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60499999999999998,"15min":0.60299999999999998},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80800000000000005,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.76500000000000001,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.58599999999999997}]},{"osd":4,"last update":"Sat Feb 10 07:32:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.41099999999999998,"15min":0.40100000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.41099999999999998,"15min":0.39700000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.624}]},{"osd":6,"last update":"Sat Feb 10 07:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.437,"15min":0.42499999999999999},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.47299999999999998,"15min":0.45600000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41699999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:33:29.014 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-10T07:33:29.015 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd dump 2024-02-10T07:33:29.260 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[29816]: 5.1c deep-scrub starts 2024-02-10T07:33:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[29816]: 5.1c deep-scrub ok 2024-02-10T07:33:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[29816]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:33:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:33:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[29816]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:33:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[29816]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T07:33:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[29816]: 5.9 scrub starts 2024-02-10T07:33:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[29816]: 5.9 scrub ok 2024-02-10T07:33:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[29816]: 3.1c scrub starts 2024-02-10T07:33:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[29816]: 3.1c scrub ok 2024-02-10T07:33:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[40743]: 5.1c deep-scrub starts 2024-02-10T07:33:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[40743]: 5.1c deep-scrub ok 2024-02-10T07:33:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[40743]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:33:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:33:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[40743]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:33:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[40743]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T07:33:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[40743]: 5.9 scrub starts 2024-02-10T07:33:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[40743]: 5.9 scrub ok 2024-02-10T07:33:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[40743]: 3.1c scrub starts 2024-02-10T07:33:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:29 smithi008 ceph-mon[40743]: 3.1c scrub ok 2024-02-10T07:33:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-mon[33394]: 5.1c deep-scrub starts 2024-02-10T07:33:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-mon[33394]: 5.1c deep-scrub ok 2024-02-10T07:33:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-mon[33394]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:33:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:33:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:33:29.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-mon[33394]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:33:29.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-mon[33394]: osdmap e92: 8 total, 8 up, 8 in 2024-02-10T07:33:29.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-mon[33394]: 5.9 scrub starts 2024-02-10T07:33:29.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-mon[33394]: 5.9 scrub ok 2024-02-10T07:33:29.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-mon[33394]: 3.1c scrub starts 2024-02-10T07:33:29.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-mon[33394]: 3.1c scrub ok 2024-02-10T07:33:29.813 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 podman[64596]: 2024-02-10 07:33:29.732566249 +0000 UTC m=+0.346151999 container start a7f8d6fdc1d75e1189f6eba7fe3b975ad7ddab8443acde5a0941f36d293847c2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a, architecture=x86_64, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , release=239.1651231664, com.redhat.component=ubi8-container, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, description=Ceph Grafana Container, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., vcs-type=git, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8) 2024-02-10T07:33:29.813 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 bash[64596]: a7f8d6fdc1d75e1189f6eba7fe3b975ad7ddab8443acde5a0941f36d293847c2 2024-02-10T07:33:30.067 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 systemd[1]: Started Ceph grafana.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:33:30.067 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=settings t=2024-02-10T07:33:29.89068381Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-02-10T07:33:30.067 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=settings t=2024-02-10T07:33:29.890901073Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-02-10T07:33:30.067 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=settings t=2024-02-10T07:33:29.890921433Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-02-10T07:33:30.067 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=settings t=2024-02-10T07:33:29.890929672Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-02-10T07:33:30.067 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=settings t=2024-02-10T07:33:29.890935801Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-10T07:33:30.067 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=settings t=2024-02-10T07:33:29.890941845Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-10T07:33:30.067 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=settings t=2024-02-10T07:33:29.890947651Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-10T07:33:30.067 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=settings t=2024-02-10T07:33:29.890972114Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-10T07:33:30.067 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=settings t=2024-02-10T07:33:29.890978132Z level=info msg="Path Home" path=/usr/share/grafana 2024-02-10T07:33:30.067 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=settings t=2024-02-10T07:33:29.890983829Z level=info msg="Path Data" path=/var/lib/grafana 2024-02-10T07:33:30.067 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=settings t=2024-02-10T07:33:29.890989261Z level=info msg="Path Logs" path=/var/log/grafana 2024-02-10T07:33:30.068 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=settings t=2024-02-10T07:33:29.890995558Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-02-10T07:33:30.068 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=settings t=2024-02-10T07:33:29.891001184Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-02-10T07:33:30.068 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=settings t=2024-02-10T07:33:29.891007237Z level=info msg="App mode production" 2024-02-10T07:33:30.068 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=sqlstore t=2024-02-10T07:33:29.891053376Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-02-10T07:33:30.068 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=sqlstore t=2024-02-10T07:33:29.891070059Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-10T07:33:30.068 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.891918087Z level=info msg="Starting DB migrations" 2024-02-10T07:33:30.068 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.891997609Z level=info msg="Executing migration" id="create migration_log table" 2024-02-10T07:33:30.068 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.892526044Z level=info msg="Executing migration" id="create user table" 2024-02-10T07:33:30.068 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.892959244Z level=info msg="Executing migration" id="add unique index user.login" 2024-02-10T07:33:30.068 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.893394158Z level=info msg="Executing migration" id="add unique index user.email" 2024-02-10T07:33:30.068 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.893812691Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-02-10T07:33:30.068 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.894375639Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-02-10T07:33:30.068 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.89470834Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-02-10T07:33:30.068 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.895398705Z level=info msg="Executing migration" id="create user table v2" 2024-02-10T07:33:30.069 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.895750304Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-02-10T07:33:30.069 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.896119055Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-02-10T07:33:30.069 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.896499121Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-02-10T07:33:30.069 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.896789204Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-02-10T07:33:30.069 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.897114087Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-02-10T07:33:30.069 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.897469967Z level=info msg="Executing migration" id="Update user table charset" 2024-02-10T07:33:30.069 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.897710119Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-02-10T07:33:30.069 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.898100657Z level=info msg="Executing migration" id="Add missing user data" 2024-02-10T07:33:30.069 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.898388341Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-02-10T07:33:30.069 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.898756237Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-02-10T07:33:30.069 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.899154773Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-02-10T07:33:30.069 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.899507077Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-02-10T07:33:30.069 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.900606021Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-02-10T07:33:30.071 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.901004098Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-02-10T07:33:30.071 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.901422217Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-02-10T07:33:30.071 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.901821499Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-02-10T07:33:30.071 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.902236116Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-02-10T07:33:30.071 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.902635787Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-02-10T07:33:30.071 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.902906227Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-02-10T07:33:30.071 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.903281941Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-02-10T07:33:30.071 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.903598304Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-02-10T07:33:30.071 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.903953138Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.904323739Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.90495746Z level=info msg="Executing migration" id="create temp_user v2" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.90539778Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.905746178Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.906144067Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.906489488Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.906880374Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.907209274Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.90749018Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.907763484Z level=info msg="Executing migration" id="create star table" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.908121702Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.908555004Z level=info msg="Executing migration" id="create org table v1" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.908967022Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.909406282Z level=info msg="Executing migration" id="create org_user table v1" 2024-02-10T07:33:30.072 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.909807995Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.910180535Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.910542718Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.910898545Z level=info msg="Executing migration" id="Update org table charset" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.911146058Z level=info msg="Executing migration" id="Update org_user table charset" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.911364185Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.911618618Z level=info msg="Executing migration" id="create dashboard table" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.91198535Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.912359342Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.912776418Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.913121646Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.913482964Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.913801727Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.914654572Z level=info msg="Executing migration" id="create dashboard v2" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.91504444Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.915408905Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-02-10T07:33:30.073 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.915773094Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.916062717Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.916374017Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.916610912Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.917054908Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.917525635Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.917970993Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.918335768Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.918743809Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.919116694Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.919450372Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.91967969Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.91992732Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.92040294Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.920851093Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-02-10T07:33:30.074 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.921326195Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.921740141Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.922019929Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.922436017Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.922753562Z level=info msg="Executing migration" id="Update dashboard title length" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.923002297Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.923346119Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.923675492Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.924511082Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.924859627Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.925224283Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.925586135Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.925898741Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.926220353Z level=info msg="Executing migration" id="Add check_sum column" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.92665785Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.927031757Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-02-10T07:33:30.075 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.927292711Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.927523566Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.927896426Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.928381097Z level=info msg="Executing migration" id="create data_source table" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.928765056Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.929172772Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.929537323Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.929938453Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.930297133Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.931240574Z level=info msg="Executing migration" id="create data_source table v2" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.931611437Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.931980675Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.932348785Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.932630564Z level=info msg="Executing migration" id="Add column with_credentials" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.933107931Z level=info msg="Executing migration" id="Add secure json data column" 2024-02-10T07:33:30.076 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.933574124Z level=info msg="Executing migration" id="Update data_source table charset" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.933803823Z level=info msg="Executing migration" id="Update initial version to 1" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.934071617Z level=info msg="Executing migration" id="Add read_only data column" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.934509823Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.934783685Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.935044935Z level=info msg="Executing migration" id="Add uid column" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.935485377Z level=info msg="Executing migration" id="Update uid value" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.935750313Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.936127582Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.93649244Z level=info msg="Executing migration" id="create api_key table" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.936867668Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.937230701Z level=info msg="Executing migration" id="add index api_key.key" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.937588776Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.938001875Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.938346657Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-02-10T07:33:30.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.938663601Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.939044523Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.940024188Z level=info msg="Executing migration" id="create api_key table v2" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.940369629Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.940703232Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.941073658Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.941439705Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.941687194Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.942002847Z level=info msg="Executing migration" id="Update api_key table charset" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.94221421Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.942663181Z level=info msg="Executing migration" id="Add service account foreign key" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.94315342Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.94341959Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.943922909Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.944417885Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.944722215Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-02-10T07:33:30.078 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.94504979Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.945373363Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.945728664Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.946101093Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.946457756Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.946684288Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.946939413Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.947409666Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.947938965Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.948204135Z level=info msg="Executing migration" id="create quota table v1" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.948538501Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.948976981Z level=info msg="Executing migration" id="Update quota table charset" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.949230686Z level=info msg="Executing migration" id="create plugin_setting table" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.94957896Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.949961582Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-02-10T07:33:30.079 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.950479338Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.950698734Z level=info msg="Executing migration" id="create session table" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.951133975Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.951357587Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.951573931Z level=info msg="Executing migration" id="create playlist table v2" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.951923069Z level=info msg="Executing migration" id="create playlist item table v2" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.952260159Z level=info msg="Executing migration" id="Update playlist table charset" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.952472213Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.952679592Z level=info msg="Executing migration" id="drop preferences table v2" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.952941743Z level=info msg="Executing migration" id="drop preferences table v3" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.953192309Z level=info msg="Executing migration" id="create preferences table v3" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.953540872Z level=info msg="Executing migration" id="Update preferences table charset" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.953763985Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.954342814Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.954578986Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.955132702Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-02-10T07:33:30.080 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.95566436Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.955956027Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.956343115Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.956766019Z level=info msg="Executing migration" id="create alert table v1" 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.957170078Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.957572222Z level=info msg="Executing migration" id="add index alert state" 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.957948116Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.958318377Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.95865033Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.95903398Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.959387122Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.960780508Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.961124082Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.961502822Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.961754263Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-02-10T07:33:30.081 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.962095464Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.962403301Z level=info msg="Executing migration" id="Add column is_default" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.962992445Z level=info msg="Executing migration" id="Add column frequency" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.963564973Z level=info msg="Executing migration" id="Add column send_reminder" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.964195927Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.964776801Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.965152817Z level=info msg="Executing migration" id="Update alert table charset" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.965368767Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.965583901Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.965987228Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.966368094Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.966678592Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.967057623Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.967394826Z level=info msg="Executing migration" id="Add for to alert table" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.967991891Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-02-10T07:33:30.082 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.968573285Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.968857013Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.969239596Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.969560588Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.970155514Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.970415547Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.970794226Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.971195518Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.971436335Z level=info msg="Executing migration" id="create annotation table v5" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.97182915Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.972204187Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.972567677Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.973150048Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.973546406Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.97395589Z level=info msg="Executing migration" id="Update annotation table charset" 2024-02-10T07:33:30.083 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.974203972Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.974784613Z level=info msg="Executing migration" id="Drop category_id index" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.975152472Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.97572288Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.976135842Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.976511736Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.976878747Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.978521908Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.978882138Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.979266406Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.979511867Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.979791768Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.980059905Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.980638114Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.981253499Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-02-10T07:33:30.084 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.981633114Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-02-10T07:33:30.085 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.982012941Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-02-10T07:33:30.085 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.982289409Z level=info msg="Executing migration" id="Add epoch_end column" 2024-02-10T07:33:30.085 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.982918394Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-02-10T07:33:30.086 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.983299332Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-02-10T07:33:30.086 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.9835316Z level=info msg="Executing migration" id="Move region to single row" 2024-02-10T07:33:30.086 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.983882398Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-02-10T07:33:30.086 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.984240434Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-10T07:33:30.086 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.984572168Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-10T07:33:30.086 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.984974205Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-10T07:33:30.086 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.985384187Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.985707874Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.986091133Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.98632229Z level=info msg="Executing migration" id="create test_data table" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.986690618Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.987060187Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.98743375Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.987807424Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.988076535Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.988323049Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.988541314Z level=info msg="Executing migration" id="create team table" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.988898678Z level=info msg="Executing migration" id="add index team.org_id" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.989295272Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.989669505Z level=info msg="Executing migration" id="create team member table" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.990023042Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.990443322Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-02-10T07:33:30.087 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.990822244Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.991195857Z level=info msg="Executing migration" id="Add column email to team table" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.991848986Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.992533106Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.993222787Z level=info msg="Executing migration" id="create dashboard acl table" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.993577503Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.99401906Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.994406556Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.994784796Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.995218623Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.995591176Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.99597507Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.996347865Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.996682945Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.99697399Z level=info msg="Executing migration" id="create tag table" 2024-02-10T07:33:30.088 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.997311822Z level=info msg="Executing migration" id="add index tag.key_value" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.997684324Z level=info msg="Executing migration" id="create login attempt table" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.998072684Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.998444012Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:29.998782129Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.000581398Z level=info msg="Executing migration" id="create login_attempt v2" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.000939697Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.001318588Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.001565439Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.00189392Z level=info msg="Executing migration" id="create user auth table" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.002232576Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.002603479Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.002867673Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.003548143Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.004267797Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-02-10T07:33:30.089 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.004963093Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.00567709Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.006070251Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.006824704Z level=info msg="Executing migration" id="create server_lock table" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.007178158Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.00756146Z level=info msg="Executing migration" id="create user auth token table" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.00797124Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.008350532Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.008719289Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.009109202Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.009785512Z level=info msg="Executing migration" id="create cache_data table" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.010179912Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.010651891Z level=info msg="Executing migration" id="create short_url table v1" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.011100128Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.011548568Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.011839434Z level=info msg="Executing migration" id="delete alert_definition table" 2024-02-10T07:33:30.090 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.012125656Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.012543394Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.012972025Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.013378277Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.013629716Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.014033035Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.014433814Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.014854946Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.015286814Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.016109687Z level=info msg="Executing migration" id="drop alert_definition table" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.016553093Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.016871222Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.01728095Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.017687058Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.018086762Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-10T07:33:30.091 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.018312351Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.018667981Z level=info msg="Executing migration" id="create alert_instance table" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.019077677Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.019455303Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.019845173Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.020590974Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.020971853Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.021343084Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.024663384Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.027963599Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.028357716Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.028706028Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.029498742Z level=info msg="Executing migration" id="create alert_rule table" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.029883544Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.030271059Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-02-10T07:33:30.092 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.030649546Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.03103761Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.031295224Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.03206513Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.032869308Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.033638671Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.034023302Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.034410113Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.035176815Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.035941175Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.036332564Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.037114057Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.037916051Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.038193022Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.038592135Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-10T07:33:30.093 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.038991967Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.039383697Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.039608127Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.040411473Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.041226225Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.042013558Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.042827728Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.043622732Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.043907042Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.044282467Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.045100021Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.045392261Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.046257551Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.046665403Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.047489346Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.047889574Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-02-10T07:33:30.094 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.048334561Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.049180486Z level=info msg="Executing migration" id="create provenance_type table" 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.049558678Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.049983824Z level=info msg="Executing migration" id="create alert_image table" 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.050382403Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.050794975Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.051074105Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.05144864Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.051866417Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.051954425Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.052193445Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.0524848Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.052895258Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.053020675Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.053131601Z level=info msg="alerts found to migrate" alerts=0 2024-02-10T07:33:30.095 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.053489685Z level=info msg="Executing migration" id="create library_element table v1" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.053926222Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.054372791Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.054762323Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.055195745Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.055641706Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.0559481Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.056304402Z level=info msg="Executing migration" id="create data_keys table" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.056707301Z level=info msg="Executing migration" id="create secrets table" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.057130933Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.061549459Z level=info msg="Executing migration" id="add name column into data_keys" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.062446227Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.062744841Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.066922398Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.071196153Z level=info msg="Executing migration" id="create kv_store table v1" 2024-02-10T07:33:30.096 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.071622271Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-02-10T07:33:30.097 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.072076354Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-02-10T07:33:30.097 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.072409778Z level=info msg="Executing migration" id="create permission table" 2024-02-10T07:33:30.097 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.072791638Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-02-10T07:33:30.097 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.073225061Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-02-10T07:33:30.097 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.073664298Z level=info msg="Executing migration" id="create role table" 2024-02-10T07:33:30.097 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.074055686Z level=info msg="Executing migration" id="add column display_name" 2024-02-10T07:33:30.097 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.075013561Z level=info msg="Executing migration" id="add column group_name" 2024-02-10T07:33:30.097 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.075936428Z level=info msg="Executing migration" id="add index role.org_id" 2024-02-10T07:33:30.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[40743]: 5.14 scrub starts 2024-02-10T07:33:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[40743]: 5.14 scrub ok 2024-02-10T07:33:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[40743]: 5.10 scrub starts 2024-02-10T07:33:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[40743]: 5.10 scrub ok 2024-02-10T07:33:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[40743]: 6.1 scrub starts 2024-02-10T07:33:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[40743]: 6.1 scrub ok 2024-02-10T07:33:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[40743]: 3.1f scrub starts 2024-02-10T07:33:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[40743]: 3.1f scrub ok 2024-02-10T07:33:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[29816]: 5.14 scrub starts 2024-02-10T07:33:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[29816]: 5.14 scrub ok 2024-02-10T07:33:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[29816]: 5.10 scrub starts 2024-02-10T07:33:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[29816]: 5.10 scrub ok 2024-02-10T07:33:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[29816]: 6.1 scrub starts 2024-02-10T07:33:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[29816]: 6.1 scrub ok 2024-02-10T07:33:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[29816]: 3.1f scrub starts 2024-02-10T07:33:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[29816]: 3.1f scrub ok 2024-02-10T07:33:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:30 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-mon[33394]: 5.14 scrub starts 2024-02-10T07:33:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-mon[33394]: 5.14 scrub ok 2024-02-10T07:33:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-mon[33394]: 5.10 scrub starts 2024-02-10T07:33:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-mon[33394]: 5.10 scrub ok 2024-02-10T07:33:30.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-mon[33394]: 6.1 scrub starts 2024-02-10T07:33:30.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-mon[33394]: 6.1 scrub ok 2024-02-10T07:33:30.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-mon[33394]: 3.1f scrub starts 2024-02-10T07:33:30.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-mon[33394]: 3.1f scrub ok 2024-02-10T07:33:30.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:30.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:30.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:30.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:30.385 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.076353077Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-02-10T07:33:30.386 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.076776966Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-02-10T07:33:30.386 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.077168721Z level=info msg="Executing migration" id="create team role table" 2024-02-10T07:33:30.386 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.07753541Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-02-10T07:33:30.386 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.077929183Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-02-10T07:33:30.386 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.078359233Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-02-10T07:33:30.386 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.078746508Z level=info msg="Executing migration" id="create user role table" 2024-02-10T07:33:30.386 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.079096496Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-02-10T07:33:30.386 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.079474018Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-02-10T07:33:30.386 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.079864243Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-02-10T07:33:30.386 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.080251541Z level=info msg="Executing migration" id="create builtin role table" 2024-02-10T07:33:30.387 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.080677392Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-02-10T07:33:30.387 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.081085607Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-02-10T07:33:30.387 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.081497578Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-02-10T07:33:30.387 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.082452823Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-02-10T07:33:30.387 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.082914694Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-02-10T07:33:30.387 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.083321506Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-02-10T07:33:30.387 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.083676828Z level=info msg="Executing migration" id="add unique index role.uid" 2024-02-10T07:33:30.387 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.084159315Z level=info msg="Executing migration" id="create seed assignment table" 2024-02-10T07:33:30.387 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.084538638Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-02-10T07:33:30.388 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.084954091Z level=info msg="Executing migration" id="add column hidden to role table" 2024-02-10T07:33:30.388 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.085996266Z level=info msg="Executing migration" id="create query_history table v1" 2024-02-10T07:33:30.388 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.086364697Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-02-10T07:33:30.388 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.086784422Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-02-10T07:33:30.388 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.087055872Z level=info msg="Executing migration" id="teams permissions migration" 2024-02-10T07:33:30.388 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.087420927Z level=info msg="Executing migration" id="dashboard permissions" 2024-02-10T07:33:30.388 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.087854609Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-02-10T07:33:30.388 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.088201608Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-02-10T07:33:30.388 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.088454084Z level=info msg="Executing migration" id="alerting notification permissions" 2024-02-10T07:33:30.388 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.088757965Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-02-10T07:33:30.389 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.089113765Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-02-10T07:33:30.389 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.089531966Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-02-10T07:33:30.389 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.09045926Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-02-10T07:33:30.389 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.090758599Z level=info msg="Executing migration" id="create correlation table v1" 2024-02-10T07:33:30.389 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.091151495Z level=info msg="Executing migration" id="add index correlations.uid" 2024-02-10T07:33:30.389 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.091563197Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-02-10T07:33:30.389 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.091992932Z level=info msg="Executing migration" id="add correlation config column" 2024-02-10T07:33:30.389 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.092932207Z level=info msg="Executing migration" id="create entity_events table" 2024-02-10T07:33:30.389 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.093324844Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-02-10T07:33:30.389 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.093718816Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-10T07:33:30.389 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.093855261Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-10T07:33:30.389 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.094121947Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T07:33:30.390 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.094201455Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T07:33:30.390 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.094415908Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-02-10T07:33:30.390 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.094779403Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-02-10T07:33:30.390 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.09516842Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-02-10T07:33:30.390 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.095583746Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T07:33:30.390 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.096004729Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-02-10T07:33:30.390 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.096400049Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-10T07:33:30.390 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.096780125Z level=info msg="Executing migration" id="Drop public config table" 2024-02-10T07:33:30.390 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.097202729Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-02-10T07:33:30.390 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.097567208Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-02-10T07:33:30.390 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.09801826Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-10T07:33:30.390 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.098452625Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-02-10T07:33:30.390 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.098933114Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-02-10T07:33:30.390 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.101967016Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-02-10T07:33:30.391 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.102927615Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-02-10T07:33:30.391 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.103889582Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-02-10T07:33:30.391 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.104176452Z level=info msg="Executing migration" id="add share column" 2024-02-10T07:33:30.391 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.10519401Z level=info msg="Executing migration" id="create default alerting folders" 2024-02-10T07:33:30.391 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.10561784Z level=info msg="Executing migration" id="create file table" 2024-02-10T07:33:30.391 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.106073254Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-02-10T07:33:30.391 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.106488107Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-02-10T07:33:30.391 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.106940586Z level=info msg="Executing migration" id="create file_meta table" 2024-02-10T07:33:30.391 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.107306098Z level=info msg="Executing migration" id="file table idx: path key" 2024-02-10T07:33:30.391 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.107718964Z level=info msg="Executing migration" id="set path collation in file table" 2024-02-10T07:33:30.391 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.10799247Z level=info msg="Executing migration" id="managed permissions migration" 2024-02-10T07:33:30.391 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.108296636Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-02-10T07:33:30.391 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.108545095Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-02-10T07:33:30.391 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.10892829Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-02-10T07:33:30.392 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.109911148Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-02-10T07:33:30.392 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.110213005Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-02-10T07:33:30.392 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.110604461Z level=info msg="Executing migration" id="update group index for alert rules" 2024-02-10T07:33:30.392 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.110949958Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-02-10T07:33:30.392 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.111221329Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-02-10T07:33:30.392 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.111510182Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-02-10T07:33:30.392 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.112475599Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-02-10T07:33:30.393 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.113496457Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-02-10T07:33:30.393 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.11388528Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-02-10T07:33:30.393 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.123620438Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-02-10T07:33:30.393 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.124032039Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-02-10T07:33:30.393 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.12442298Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-02-10T07:33:30.393 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.127829126Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-02-10T07:33:30.393 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.128116174Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-02-10T07:33:30.393 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.128377091Z level=info msg="Executing migration" id="create folder table" 2024-02-10T07:33:30.393 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.128696474Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-02-10T07:33:30.393 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.129122507Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-02-10T07:33:30.393 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.129536558Z level=info msg="Executing migration" id="Update folder title length" 2024-02-10T07:33:30.393 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.129785697Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-02-10T07:33:30.394 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=migrator t=2024-02-10T07:33:30.13020787Z level=info msg="migrations completed" performed=485 skipped=0 duration=238.238762ms 2024-02-10T07:33:30.394 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=sqlstore t=2024-02-10T07:33:30.130591865Z level=info msg="Created default organization" 2024-02-10T07:33:30.394 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=plugin.loader t=2024-02-10T07:33:30.187567317Z level=info msg="Plugin registered" pluginID=input 2024-02-10T07:33:30.394 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=plugin.loader t=2024-02-10T07:33:30.215066202Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-02-10T07:33:30.394 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=plugin.loader t=2024-02-10T07:33:30.215090262Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-02-10T07:33:30.394 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=secrets t=2024-02-10T07:33:30.21512634Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-02-10T07:33:30.394 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=query_data t=2024-02-10T07:33:30.217293812Z level=info msg="Query Service initialization" 2024-02-10T07:33:30.394 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=live.push_http t=2024-02-10T07:33:30.219810193Z level=info msg="Live Push Gateway initialization" 2024-02-10T07:33:31.133 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=infra.usagestats.collector t=2024-02-10T07:33:30.821469163Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-02-10T07:33:31.133 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=provisioning.datasources t=2024-02-10T07:33:30.822432339Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-10T07:33:31.133 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=provisioning.datasources t=2024-02-10T07:33:30.833365716Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-02-10T07:33:31.133 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=provisioning.alerting t=2024-02-10T07:33:30.843947932Z level=info msg="starting to provision alerting" 2024-02-10T07:33:31.133 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=provisioning.alerting t=2024-02-10T07:33:30.843983577Z level=info msg="finished to provision alerting" 2024-02-10T07:33:31.133 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=ngalert.state.manager t=2024-02-10T07:33:30.844075679Z level=info msg="Warming state cache for startup" 2024-02-10T07:33:31.133 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=ngalert.state.manager t=2024-02-10T07:33:30.84427851Z level=info msg="State cache has been initialized" states=0 duration=202.97µs 2024-02-10T07:33:31.133 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=ticker t=2024-02-10T07:33:30.844347079Z level=info msg=starting first_tick=2024-02-10T07:33:40Z 2024-02-10T07:33:31.133 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=grafanaStorageLogger t=2024-02-10T07:33:30.844103314Z level=info msg="storage starting" 2024-02-10T07:33:31.134 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=ngalert.multiorg.alertmanager t=2024-02-10T07:33:30.844381034Z level=info msg="starting MultiOrg Alertmanager" 2024-02-10T07:33:31.134 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=http.server t=2024-02-10T07:33:30.846046521Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-02-10T07:33:31.134 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=sqlstore.transactions t=2024-02-10T07:33:30.860344916Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-10T07:33:31.134 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=sqlstore.transactions t=2024-02-10T07:33:30.862405613Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-10T07:33:31.134 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:33:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=sqlstore.transactions t=2024-02-10T07:33:30.870922726Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-02-10T07:33:31.276 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:31 smithi008 ceph-mon[29816]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:31.277 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:31 smithi008 ceph-mon[29816]: Deploying daemon node-exporter.a on smithi008 2024-02-10T07:33:31.277 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:31 smithi008 ceph-mon[29816]: 5.2 scrub starts 2024-02-10T07:33:31.277 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:31 smithi008 ceph-mon[29816]: 5.2 scrub ok 2024-02-10T07:33:31.277 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:31 smithi008 ceph-mon[40743]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:31.277 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:31 smithi008 ceph-mon[40743]: Deploying daemon node-exporter.a on smithi008 2024-02-10T07:33:31.277 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:31 smithi008 ceph-mon[40743]: 5.2 scrub starts 2024-02-10T07:33:31.277 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:31 smithi008 ceph-mon[40743]: 5.2 scrub ok 2024-02-10T07:33:31.311 INFO:teuthology.orchestra.run.smithi008.stdout:epoch 92 2024-02-10T07:33:31.312 INFO:teuthology.orchestra.run.smithi008.stdout:fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:33:31.312 INFO:teuthology.orchestra.run.smithi008.stdout:created 2024-02-10T07:26:21.316239+0000 2024-02-10T07:33:31.312 INFO:teuthology.orchestra.run.smithi008.stdout:modified 2024-02-10T07:33:28.198743+0000 2024-02-10T07:33:31.312 INFO:teuthology.orchestra.run.smithi008.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-10T07:33:31.312 INFO:teuthology.orchestra.run.smithi008.stdout:crush_version 18 2024-02-10T07:33:31.312 INFO:teuthology.orchestra.run.smithi008.stdout:full_ratio 0.95 2024-02-10T07:33:31.312 INFO:teuthology.orchestra.run.smithi008.stdout:backfillfull_ratio 0.9 2024-02-10T07:33:31.312 INFO:teuthology.orchestra.run.smithi008.stdout:nearfull_ratio 0.85 2024-02-10T07:33:31.312 INFO:teuthology.orchestra.run.smithi008.stdout:require_min_compat_client luminous 2024-02-10T07:33:31.312 INFO:teuthology.orchestra.run.smithi008.stdout:min_compat_client luminous 2024-02-10T07:33:31.312 INFO:teuthology.orchestra.run.smithi008.stdout:require_osd_release squid 2024-02-10T07:33:31.312 INFO:teuthology.orchestra.run.smithi008.stdout:stretch_mode_enabled false 2024-02-10T07:33:31.312 INFO:teuthology.orchestra.run.smithi008.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-02-10T07:33:31.313 INFO:teuthology.orchestra.run.smithi008.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 70 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-02-10T07:33:31.313 INFO:teuthology.orchestra.run.smithi008.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-02-10T07:33:31.313 INFO:teuthology.orchestra.run.smithi008.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-02-10T07:33:31.313 INFO:teuthology.orchestra.run.smithi008.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 72 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-02-10T07:33:31.313 INFO:teuthology.orchestra.run.smithi008.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 74 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-02-10T07:33:31.313 INFO:teuthology.orchestra.run.smithi008.stdout:max_osd 8 2024-02-10T07:33:31.313 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 up in weight 1 up_from 8 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6802/1442795455,v1:172.21.15.8:6803/1442795455] [v2:172.21.15.8:6804/1442795455,v1:172.21.15.8:6805/1442795455] exists,up 63abc4f1-1c83-4e03-8592-959c51d10847 2024-02-10T07:33:31.313 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6810/3278758893,v1:172.21.15.8:6811/3278758893] [v2:172.21.15.8:6812/3278758893,v1:172.21.15.8:6813/3278758893] exists,up 88307389-bfa8-402e-92b9-9fa5094262c4 2024-02-10T07:33:31.313 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2 up in weight 1 up_from 18 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6818/481314802,v1:172.21.15.8:6819/481314802] [v2:172.21.15.8:6820/481314802,v1:172.21.15.8:6821/481314802] exists,up c6120003-9ab5-4cf5-87ac-6351ed2c6226 2024-02-10T07:33:31.313 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 up in weight 1 up_from 26 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651] [v2:172.21.15.8:6828/3692331651,v1:172.21.15.8:6829/3692331651] exists,up 6ca8f6f3-013a-48cb-9083-be37c1767489 2024-02-10T07:33:31.313 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4 up in weight 1 up_from 32 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6800/3997132504,v1:172.21.15.170:6801/3997132504] [v2:172.21.15.170:6802/3997132504,v1:172.21.15.170:6803/3997132504] exists,up aa667766-e57f-447b-9a31-e3f3e61ac346 2024-02-10T07:33:31.313 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5 up in weight 1 up_from 40 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245] [v2:172.21.15.170:6810/2364068245,v1:172.21.15.170:6811/2364068245] exists,up 2e9ab14d-1aff-4638-9f41-311f6b5f6770 2024-02-10T07:33:31.313 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6 up in weight 1 up_from 50 up_thru 70 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6816/1764436984,v1:172.21.15.170:6817/1764436984] [v2:172.21.15.170:6818/1764436984,v1:172.21.15.170:6819/1764436984] exists,up 61970c81-4b79-4743-a379-50d5d102b373 2024-02-10T07:33:31.313 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7 up in weight 1 up_from 60 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6824/4048577717,v1:172.21.15.170:6825/4048577717] [v2:172.21.15.170:6826/4048577717,v1:172.21.15.170:6827/4048577717] exists,up 2e303441-b454-4750-8c5a-96c5348129a8 2024-02-10T07:33:31.314 INFO:teuthology.orchestra.run.smithi008.stdout:pg_upmap_items 4.7 [1,2] 2024-02-10T07:33:31.314 INFO:teuthology.orchestra.run.smithi008.stdout:pg_upmap_items 4.9 [1,2] 2024-02-10T07:33:31.314 INFO:teuthology.orchestra.run.smithi008.stdout:pg_upmap_items 4.13 [1,2] 2024-02-10T07:33:31.314 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:0/2210906095 expires 2024-02-11T07:32:31.651879+0000 2024-02-10T07:33:31.314 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:6800/1412156979 expires 2024-02-11T07:27:16.371197+0000 2024-02-10T07:33:31.314 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:0/2515892714 expires 2024-02-11T07:26:49.500037+0000 2024-02-10T07:33:31.314 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:6801/4001702773 expires 2024-02-11T07:26:49.500037+0000 2024-02-10T07:33:31.314 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:6801/1412156979 expires 2024-02-11T07:27:16.371197+0000 2024-02-10T07:33:31.314 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:0/1462056527 expires 2024-02-11T07:27:16.371197+0000 2024-02-10T07:33:31.314 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:6800/4001702773 expires 2024-02-11T07:26:49.500037+0000 2024-02-10T07:33:31.314 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:0/1905866501 expires 2024-02-11T07:26:49.500037+0000 2024-02-10T07:33:31.314 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:0/2381650878 expires 2024-02-11T07:27:16.371197+0000 2024-02-10T07:33:31.314 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:0/802075678 expires 2024-02-11T07:26:49.500037+0000 2024-02-10T07:33:31.314 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:6801/2677792951 expires 2024-02-11T07:32:31.651879+0000 2024-02-10T07:33:31.315 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:0/3905205183 expires 2024-02-11T07:32:31.651879+0000 2024-02-10T07:33:31.315 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:0/3314827874 expires 2024-02-11T07:32:31.651879+0000 2024-02-10T07:33:31.315 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:0/1624888680 expires 2024-02-11T07:32:31.651879+0000 2024-02-10T07:33:31.315 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:0/2789652913 expires 2024-02-11T07:27:16.371197+0000 2024-02-10T07:33:31.315 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:0/4074987889 expires 2024-02-11T07:32:31.651879+0000 2024-02-10T07:33:31.315 INFO:teuthology.orchestra.run.smithi008.stdout:blocklist 172.21.15.8:6800/2677792951 expires 2024-02-11T07:32:31.651879+0000 2024-02-10T07:33:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:31 smithi170 ceph-mon[33394]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:31 smithi170 ceph-mon[33394]: Deploying daemon node-exporter.a on smithi008 2024-02-10T07:33:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:31 smithi170 ceph-mon[33394]: 5.2 scrub starts 2024-02-10T07:33:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:31 smithi170 ceph-mon[33394]: 5.2 scrub ok 2024-02-10T07:33:32.082 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:31 smithi008 bash[76956]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-10T07:33:32.330 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6802/1442795455,v1:172.21.15.8:6803/1442795455] [v2:172.21.15.8:6804/1442795455,v1:172.21.15.8:6805/1442795455] exists,up 63abc4f1-1c83-4e03-8592-959c51d10847', 'osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6810/3278758893,v1:172.21.15.8:6811/3278758893] [v2:172.21.15.8:6812/3278758893,v1:172.21.15.8:6813/3278758893] exists,up 88307389-bfa8-402e-92b9-9fa5094262c4', 'osd.2 up in weight 1 up_from 18 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6818/481314802,v1:172.21.15.8:6819/481314802] [v2:172.21.15.8:6820/481314802,v1:172.21.15.8:6821/481314802] exists,up c6120003-9ab5-4cf5-87ac-6351ed2c6226', 'osd.3 up in weight 1 up_from 26 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.8:6826/3692331651,v1:172.21.15.8:6827/3692331651] [v2:172.21.15.8:6828/3692331651,v1:172.21.15.8:6829/3692331651] exists,up 6ca8f6f3-013a-48cb-9083-be37c1767489', 'osd.4 up in weight 1 up_from 32 up_thru 77 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6800/3997132504,v1:172.21.15.170:6801/3997132504] [v2:172.21.15.170:6802/3997132504,v1:172.21.15.170:6803/3997132504] exists,up aa667766-e57f-447b-9a31-e3f3e61ac346', 'osd.5 up in weight 1 up_from 40 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6808/2364068245,v1:172.21.15.170:6809/2364068245] [v2:172.21.15.170:6810/2364068245,v1:172.21.15.170:6811/2364068245] exists,up 2e9ab14d-1aff-4638-9f41-311f6b5f6770', 'osd.6 up in weight 1 up_from 50 up_thru 70 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6816/1764436984,v1:172.21.15.170:6817/1764436984] [v2:172.21.15.170:6818/1764436984,v1:172.21.15.170:6819/1764436984] exists,up 61970c81-4b79-4743-a379-50d5d102b373', 'osd.7 up in weight 1 up_from 60 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6824/4048577717,v1:172.21.15.170:6825/4048577717] [v2:172.21.15.170:6826/4048577717,v1:172.21.15.170:6827/4048577717] exists,up 2e303441-b454-4750-8c5a-96c5348129a8'] 2024-02-10T07:33:32.331 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon mon.a config show 2024-02-10T07:33:32.357 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[29816]: 3.e deep-scrub starts 2024-02-10T07:33:32.357 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[29816]: 3.e deep-scrub ok 2024-02-10T07:33:32.357 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[29816]: 3.3 scrub starts 2024-02-10T07:33:32.357 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[29816]: 3.3 scrub ok 2024-02-10T07:33:32.357 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2896290944' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-10T07:33:32.358 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[29816]: osdmap e93: 8 total, 8 up, 8 in 2024-02-10T07:33:32.358 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:32.358 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:32.358 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[40743]: 3.e deep-scrub starts 2024-02-10T07:33:32.358 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[40743]: 3.e deep-scrub ok 2024-02-10T07:33:32.358 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[40743]: 3.3 scrub starts 2024-02-10T07:33:32.358 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[40743]: 3.3 scrub ok 2024-02-10T07:33:32.358 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2896290944' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-10T07:33:32.358 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[40743]: osdmap e93: 8 total, 8 up, 8 in 2024-02-10T07:33:32.358 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:32.358 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:32.577 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:32 smithi170 ceph-mon[33394]: 3.e deep-scrub starts 2024-02-10T07:33:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:32 smithi170 ceph-mon[33394]: 3.e deep-scrub ok 2024-02-10T07:33:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:32 smithi170 ceph-mon[33394]: 3.3 scrub starts 2024-02-10T07:33:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:32 smithi170 ceph-mon[33394]: 3.3 scrub ok 2024-02-10T07:33:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:32 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2896290944' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-10T07:33:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:32 smithi170 ceph-mon[33394]: osdmap e93: 8 total, 8 up, 8 in 2024-02-10T07:33:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:32.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:33.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:33:32 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:33:33.284 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:33 smithi008 ceph-mon[29816]: 6.11 scrub starts 2024-02-10T07:33:33.285 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:33 smithi008 ceph-mon[29816]: 6.11 scrub ok 2024-02-10T07:33:33.285 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:33 smithi008 ceph-mon[29816]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:33.285 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:33 smithi008 ceph-mon[29816]: 5.16 scrub starts 2024-02-10T07:33:33.285 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:33 smithi008 ceph-mon[29816]: 5.16 scrub ok 2024-02-10T07:33:33.285 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:33 smithi008 ceph-mon[40743]: 6.11 scrub starts 2024-02-10T07:33:33.285 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:33 smithi008 ceph-mon[40743]: 6.11 scrub ok 2024-02-10T07:33:33.285 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:33 smithi008 ceph-mon[40743]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:33.285 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:33 smithi008 ceph-mon[40743]: 5.16 scrub starts 2024-02-10T07:33:33.285 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:33 smithi008 ceph-mon[40743]: 5.16 scrub ok 2024-02-10T07:33:33.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:33 smithi170 ceph-mon[33394]: 6.11 scrub starts 2024-02-10T07:33:33.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:33 smithi170 ceph-mon[33394]: 6.11 scrub ok 2024-02-10T07:33:33.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:33 smithi170 ceph-mon[33394]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 248 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:33.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:33 smithi170 ceph-mon[33394]: 5.16 scrub starts 2024-02-10T07:33:33.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:33 smithi170 ceph-mon[33394]: 5.16 scrub ok 2024-02-10T07:33:34.216 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:33 smithi008 bash[76956]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-10T07:33:34.216 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:34 smithi008 bash[76956]: Writing manifest to image destination 2024-02-10T07:33:34.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:34 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:34 smithi008 ceph-mon[29816]: 5.1a scrub starts 2024-02-10T07:33:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:34 smithi008 ceph-mon[29816]: 5.1a scrub ok 2024-02-10T07:33:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:34 smithi008 ceph-mon[29816]: 5.3 scrub starts 2024-02-10T07:33:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:34 smithi008 ceph-mon[29816]: 5.3 scrub ok 2024-02-10T07:33:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:34 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:34 smithi008 ceph-mon[40743]: 5.1a scrub starts 2024-02-10T07:33:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:34 smithi008 ceph-mon[40743]: 5.1a scrub ok 2024-02-10T07:33:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:34 smithi008 ceph-mon[40743]: 5.3 scrub starts 2024-02-10T07:33:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:34 smithi008 ceph-mon[40743]: 5.3 scrub ok 2024-02-10T07:33:34.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:34 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:34.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:34 smithi170 ceph-mon[33394]: 5.1a scrub starts 2024-02-10T07:33:34.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:34 smithi170 ceph-mon[33394]: 5.1a scrub ok 2024-02-10T07:33:34.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:34 smithi170 ceph-mon[33394]: 5.3 scrub starts 2024-02-10T07:33:34.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:34 smithi170 ceph-mon[33394]: 5.3 scrub ok 2024-02-10T07:33:35.030 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:34 smithi008 podman[76956]: 2024-02-10 07:33:34.519337332 +0000 UTC m=+3.341776447 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-10T07:33:35.030 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:34 smithi008 podman[76956]: 2024-02-10 07:33:34.728341363 +0000 UTC m=+3.550780466 container create 521ef0a9ea22a179bd36f96098b20f8be9d25b9c76f254e1d6d80baaf78765be (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:35.384 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-mon[29816]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:33:35.384 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-mon[40743]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:33:35.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:35 smithi170 ceph-mon[33394]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:33:35.702 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 podman[76956]: 2024-02-10 07:33:35.384292111 +0000 UTC m=+4.206731211 container init 521ef0a9ea22a179bd36f96098b20f8be9d25b9c76f254e1d6d80baaf78765be (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:35.702 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 podman[76956]: 2024-02-10 07:33:35.387382703 +0000 UTC m=+4.209821806 container start 521ef0a9ea22a179bd36f96098b20f8be9d25b9c76f254e1d6d80baaf78765be (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:33:35.702 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.391Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-10T07:33:35.702 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.391Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-10T07:33:35.702 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.391Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-10T07:33:35.702 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.391Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T07:33:35.702 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.392Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-10T07:33:35.702 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.392Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-10T07:33:35.702 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.392Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-10T07:33:35.702 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.392Z caller=node_exporter.go:117 level=info collector=arp 2024-02-10T07:33:35.703 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.392Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-10T07:33:35.703 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.392Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-10T07:33:35.703 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.392Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-10T07:33:35.703 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.392Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-10T07:33:35.703 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.392Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-10T07:33:35.703 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.392Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-10T07:33:35.703 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.392Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-10T07:33:35.703 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.392Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-10T07:33:35.704 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.392Z caller=node_exporter.go:117 level=info collector=edac 2024-02-10T07:33:35.704 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.392Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-10T07:33:35.704 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-10T07:33:35.704 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-10T07:33:35.704 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-10T07:33:35.704 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-10T07:33:35.704 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-10T07:33:35.704 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-10T07:33:35.704 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-10T07:33:35.704 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-10T07:33:35.704 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-10T07:33:35.704 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-10T07:33:35.704 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-10T07:33:35.704 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-10T07:33:35.705 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-10T07:33:35.705 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-10T07:33:35.705 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-10T07:33:35.705 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=os 2024-02-10T07:33:35.705 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-10T07:33:35.705 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-10T07:33:35.705 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-10T07:33:35.705 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-10T07:33:35.705 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-10T07:33:35.705 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-10T07:33:35.705 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-10T07:33:35.705 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=stat 2024-02-10T07:33:35.705 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-10T07:33:35.705 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-10T07:33:35.706 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-10T07:33:35.706 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=time 2024-02-10T07:33:35.706 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-10T07:33:35.706 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=uname 2024-02-10T07:33:35.706 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-10T07:33:35.706 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-10T07:33:35.706 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-10T07:33:35.706 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-10T07:33:35.706 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a[77151]: ts=2024-02-10T07:33:35.393Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-10T07:33:35.706 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 bash[76956]: 521ef0a9ea22a179bd36f96098b20f8be9d25b9c76f254e1d6d80baaf78765be 2024-02-10T07:33:35.706 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:33:35 smithi008 systemd[1]: Started Ceph node-exporter.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:33:35.747 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:33:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "name": "mon.a", 2024-02-10T07:33:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster": "ceph", 2024-02-10T07:33:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-02-10T07:33:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "admin_socket_mode": "", 2024-02-10T07:33:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-02-10T07:33:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_client_required": "cephx, none", 2024-02-10T07:33:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_cluster_required": "cephx", 2024-02-10T07:33:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_debug": "true", 2024-02-10T07:33:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-02-10T07:33:35.747 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-02-10T07:33:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_service_required": "cephx", 2024-02-10T07:33:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_service_ticket_ttl": "240.000000", 2024-02-10T07:33:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "auth_supported": "", 2024-02-10T07:33:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_aio": "true", 2024-02-10T07:33:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_aio_max_queue_depth": "1024", 2024-02-10T07:33:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_aio_poll_ms": "250", 2024-02-10T07:33:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_aio_reap_max": "16", 2024-02-10T07:33:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_async_discard": "false", 2024-02-10T07:33:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_block_size": "4096", 2024-02-10T07:33:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_debug_aio": "false", 2024-02-10T07:33:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-02-10T07:33:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-02-10T07:33:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_debug_inflight_ios": "false", 2024-02-10T07:33:35.748 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_enable_discard": "false", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_flock_retry": "3", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_flock_retry_interval": "0.100000", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_inject_crash": "0", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_inject_crash_flush_delay": "2", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_ioring": "false", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_ioring_hipri": "false", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_ioring_sqthread_poll": "false", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_read_buffer_alignment": "4096", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bdev_type": "", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_alloc_size": "1048576", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_allocator": "hybrid", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_buffered_io": "true", 2024-02-10T07:33:35.749 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_check_for_zeros": "false", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_check_volume_selector_often": "false", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_compact_log_sync": "false", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_log_compact_min_size": "16777216", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_log_replay_check_allocations": "true", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_max_log_runway": "4194304", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_max_prefetch": "1048576", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_min_flush_size": "524288", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_min_log_runway": "1048576", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_replay_recovery": "false", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_shared_alloc_size": "65536", 2024-02-10T07:33:35.750 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_sync_write": "false", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_allocation_from_file": "true", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_allocator": "hybrid", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_blobid_prealloc": "10240", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_create": "true", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_db_create": "false", 2024-02-10T07:33:35.751 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_db_path": "", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_db_size": "0", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_path": "", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_preallocate_file": "false", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_size": "107374182400", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_wal_create": "false", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_wal_path": "", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_block_wal_size": "100663296", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_bluefs": "true", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_bluefs_env_mirror": "false", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-02-10T07:33:35.752 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_autotune": "true", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_size": "0", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_type": "2q", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_clone_cow": "true", 2024-02-10T07:33:35.753 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_algorithm": "snappy", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_max_blob_size": "0", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_min_blob_size": "0", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_mode": "none", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_csum_type": "crc32c", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_enforce_settings": "default", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_freelist": "false", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_fsck_abort": "false", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-02-10T07:33:35.754 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_inject_read_err": "false", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_legacy_omap": "false", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_prefill": "0.000000", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_small_allocations": "0", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_default_buffered_read": "true", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_default_buffered_write": "false", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_deferred_batch_ops": "0", 2024-02-10T07:33:35.755 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_elastic_shared_blobs": "true", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fail_eio": "false", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_on_mkfs": "true", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-02-10T07:33:35.756 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_on_mount": "false", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_on_umount": "false", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_ignore_data_csum": "false", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_kvbackend": "rocksdb", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-02-10T07:33:35.757 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_log_op_age": "5.000000", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_max_alloc_size": "0", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_max_blob_size": "0", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_max_defer_interval": "3.000000", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_max_deferred_txc": "32", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_min_alloc_size": "0", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_nid_prealloc": "1024", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_prefer_deferred_size": "0", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_qfsck_on_mount": "true", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_retry_disk_reads": "3", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_rocksdb_cf": "true", 2024-02-10T07:33:35.758 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-02-10T07:33:35.759 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-02-10T07:33:35.759 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_rocksdb_options_annex": "", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_spdk_coremask": "0x1", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_spdk_io_sleep": "5", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_spdk_max_io_completion": "0", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_spdk_mem": "512", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_sync_submit_transaction": "false", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_throttle_bytes": "67108864", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_throttle_cost_per_io": "0", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_tracing": "false", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-02-10T07:33:35.760 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_volume_selection_reserved": "0", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_zero_block_detection": "false", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_action_update_interval": "2", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_mount_timeout": "10", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-02-10T07:33:35.761 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-02-10T07:33:35.762 INFO:teuthology.orchestra.run.smithi008.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-02-10T07:33:35.762 INFO:teuthology.orchestra.run.smithi008.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-02-10T07:33:35.762 INFO:teuthology.orchestra.run.smithi008.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-02-10T07:33:35.762 INFO:teuthology.orchestra.run.smithi008.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-02-10T07:33:35.762 INFO:teuthology.orchestra.run.smithi008.stdout: "cephx_cluster_require_signatures": "false", 2024-02-10T07:33:35.762 INFO:teuthology.orchestra.run.smithi008.stdout: "cephx_cluster_require_version": "2", 2024-02-10T07:33:35.762 INFO:teuthology.orchestra.run.smithi008.stdout: "cephx_require_signatures": "false", 2024-02-10T07:33:35.762 INFO:teuthology.orchestra.run.smithi008.stdout: "cephx_require_version": "2", 2024-02-10T07:33:35.762 INFO:teuthology.orchestra.run.smithi008.stdout: "cephx_service_require_signatures": "false", 2024-02-10T07:33:35.762 INFO:teuthology.orchestra.run.smithi008.stdout: "cephx_service_require_version": "2", 2024-02-10T07:33:35.762 INFO:teuthology.orchestra.run.smithi008.stdout: "cephx_sign_messages": "true", 2024-02-10T07:33:35.763 INFO:teuthology.orchestra.run.smithi008.stdout: "chdir": "", 2024-02-10T07:33:35.763 INFO:teuthology.orchestra.run.smithi008.stdout: "client_acl_type": "", 2024-02-10T07:33:35.763 INFO:teuthology.orchestra.run.smithi008.stdout: "client_asio_thread_count": "2", 2024-02-10T07:33:35.763 INFO:teuthology.orchestra.run.smithi008.stdout: "client_cache_mid": "0.750000", 2024-02-10T07:33:35.763 INFO:teuthology.orchestra.run.smithi008.stdout: "client_cache_size": "16384", 2024-02-10T07:33:35.763 INFO:teuthology.orchestra.run.smithi008.stdout: "client_caps_release_delay": "5", 2024-02-10T07:33:35.763 INFO:teuthology.orchestra.run.smithi008.stdout: "client_check_pool_perm": "true", 2024-02-10T07:33:35.763 INFO:teuthology.orchestra.run.smithi008.stdout: "client_collect_and_send_global_metrics": "false", 2024-02-10T07:33:35.763 INFO:teuthology.orchestra.run.smithi008.stdout: "client_debug_force_sync_read": "false", 2024-02-10T07:33:35.763 INFO:teuthology.orchestra.run.smithi008.stdout: "client_debug_getattr_caps": "false", 2024-02-10T07:33:35.763 INFO:teuthology.orchestra.run.smithi008.stdout: "client_debug_inject_tick_delay": "0", 2024-02-10T07:33:35.763 INFO:teuthology.orchestra.run.smithi008.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-02-10T07:33:35.763 INFO:teuthology.orchestra.run.smithi008.stdout: "client_die_on_failed_remount": "false", 2024-02-10T07:33:35.763 INFO:teuthology.orchestra.run.smithi008.stdout: "client_dirsize_rbytes": "true", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_force_lazyio": "false", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_fs": "", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_inject_fixed_oldest_tid": "false", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_inject_release_failure": "false", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_max_inline_size": "4096", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_max_retries_on_remount_failure": "5", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_mds_namespace": "", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_metadata": "", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_mount_gid": "-1", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_mount_timeout": "300", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_mount_uid": "-1", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_mountpoint": "/", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_notify_timeout": "10", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_oc": "true", 2024-02-10T07:33:35.764 INFO:teuthology.orchestra.run.smithi008.stdout: "client_oc_max_dirty": "104857600", 2024-02-10T07:33:35.765 INFO:teuthology.orchestra.run.smithi008.stdout: "client_oc_max_dirty_age": "5.000000", 2024-02-10T07:33:35.765 INFO:teuthology.orchestra.run.smithi008.stdout: "client_oc_max_objects": "1000", 2024-02-10T07:33:35.765 INFO:teuthology.orchestra.run.smithi008.stdout: "client_oc_size": "209715200", 2024-02-10T07:33:35.765 INFO:teuthology.orchestra.run.smithi008.stdout: "client_oc_target_dirty": "8388608", 2024-02-10T07:33:35.765 INFO:teuthology.orchestra.run.smithi008.stdout: "client_permissions": "true", 2024-02-10T07:33:35.765 INFO:teuthology.orchestra.run.smithi008.stdout: "client_quota": "true", 2024-02-10T07:33:35.765 INFO:teuthology.orchestra.run.smithi008.stdout: "client_quota_df": "true", 2024-02-10T07:33:35.765 INFO:teuthology.orchestra.run.smithi008.stdout: "client_readahead_max_bytes": "0", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "client_readahead_max_periods": "4", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "client_readahead_min": "131072", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "client_reconnect_stale": "false", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "client_shutdown_timeout": "30", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "client_snapdir": ".snap", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "client_tick_interval": "1", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "client_trace": "", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "client_try_dentry_invalidate": "false", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "client_use_faked_inos": "false", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "client_use_random_mds": "false", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "clog_to_graylog": "false", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "clog_to_graylog_port": "12201", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "clog_to_monitors": "default=true", 2024-02-10T07:33:35.766 INFO:teuthology.orchestra.run.smithi008.stdout: "clog_to_syslog": "false", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "clog_to_syslog_level": "info", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "-", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_network": "", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_network_interface": "", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "compressor_zlib_isal": "false", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "compressor_zlib_level": "5", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "compressor_zlib_winsize": "-15", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "compressor_zstd_level": "1", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "crimson_alien_op_num_threads": "6", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "crimson_alien_thread_cpu_cores": "", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "crimson_osd_obc_lru_size": "512", 2024-02-10T07:33:35.767 INFO:teuthology.orchestra.run.smithi008.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "crimson_seastar_cpu_cores": "", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_location": "", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_location_hook": "", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_location_hook_timeout": "10", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "daemonize": "false", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "daos_pool": "tank", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "dbstore_db_name_prefix": "dbstore", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_alienstore": "0/5", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_allow_any_pool_priority": "false", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_asok": "1/5", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_asok_assert_abort": "false", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_asserts_on_shutdown": "true", 2024-02-10T07:33:35.768 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_auth": "1/5", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_bdev": "1/3", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_bluefs": "1/5", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_bluestore": "1/5", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_buffer": "0/1", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_ceph_exporter": "1/5", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_cephfs_mirror": "0/5", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_cephsqlite": "0/5", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_client": "0/5", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_compressor": "1/5", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_context": "0/1", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_crimson_interrupt": "0/5", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_crush": "1/1", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_crypto": "1/5", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_cyanstore": "0/5", 2024-02-10T07:33:35.769 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_deliberately_leak_memory": "false", 2024-02-10T07:33:35.770 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_disable_randomized_ping": "false", 2024-02-10T07:33:35.770 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_dpdk": "1/5", 2024-02-10T07:33:35.770 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_eventtrace": "1/5", 2024-02-10T07:33:35.770 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_filer": "0/1", 2024-02-10T07:33:35.770 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_filestore": "1/3", 2024-02-10T07:33:35.770 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_finisher": "1/1", 2024-02-10T07:33:35.770 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_fuse": "1/5", 2024-02-10T07:33:35.770 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_heartbeat_testing_span": "0", 2024-02-10T07:33:35.770 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_heartbeatmap": "1/5", 2024-02-10T07:33:35.770 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_immutable_obj_cache": "0/5", 2024-02-10T07:33:35.770 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_javaclient": "1/5", 2024-02-10T07:33:35.770 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_journal": "1/3", 2024-02-10T07:33:35.770 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_journaler": "0/5", 2024-02-10T07:33:35.770 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_kstore": "1/5", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_lockdep": "0/1", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mclock": "1/5", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mds": "1/5", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mds_balancer": "1/5", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mds_locker": "1/5", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mds_log": "1/5", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mds_log_expire": "1/5", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mds_migrator": "1/5", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_memstore": "1/5", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mgr": "2/5", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mgrc": "1/5", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_mon": "20/20", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_monc": "0/10", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_ms": "1/1", 2024-02-10T07:33:35.771 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_none": "0/5", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_objclass": "0/5", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_objectcacher": "0/5", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_objecter": "0/1", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_optracker": "0/5", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_osd": "1/5", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_paxos": "20/20", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_perfcounter": "1/5", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_prioritycache": "1/5", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rados": "0/5", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rbd": "0/5", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rbd_mirror": "0/5", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rbd_pwl": "0/5", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rbd_replay": "0/5", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_refs": "0/0", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_reserver": "1/1", 2024-02-10T07:33:35.772 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rgw": "1/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rgw_access": "1/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rgw_datacache": "1/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rgw_dbstore": "1/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rgw_flight": "1/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rgw_lifecycle": "1/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rgw_sync": "1/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_rocksdb": "4/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore": "0/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_backref": "0/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_cache": "0/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_cleaner": "0/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_device": "0/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_epm": "0/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-02-10T07:33:35.773 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_journal": "0/5", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_lba": "0/5", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_odata": "0/5", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_omap": "0/5", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_onode": "0/5", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_t": "0/5", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_seastore_tm": "0/5", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_striper": "0/1", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_test": "0/5", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_throttle": "1/1", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_timer": "0/1", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_tp": "0/5", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "debug_trace": "1/5", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "device_failure_prediction_mode": "none", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-02-10T07:33:35.774 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-02-10T07:33:35.775 INFO:teuthology.orchestra.run.smithi008.stdout: "err_to_graylog": "false", 2024-02-10T07:33:35.775 INFO:teuthology.orchestra.run.smithi008.stdout: "err_to_journald": "false", 2024-02-10T07:33:35.775 INFO:teuthology.orchestra.run.smithi008.stdout: "err_to_stderr": "true", 2024-02-10T07:33:35.775 INFO:teuthology.orchestra.run.smithi008.stdout: "err_to_syslog": "false", 2024-02-10T07:33:35.775 INFO:teuthology.orchestra.run.smithi008.stdout: "event_tracing": "false", 2024-02-10T07:33:35.775 INFO:teuthology.orchestra.run.smithi008.stdout: "exporter_addr": "0.0.0.0", 2024-02-10T07:33:35.775 INFO:teuthology.orchestra.run.smithi008.stdout: "exporter_http_port": "9926", 2024-02-10T07:33:35.775 INFO:teuthology.orchestra.run.smithi008.stdout: "exporter_prio_limit": "5", 2024-02-10T07:33:35.775 INFO:teuthology.orchestra.run.smithi008.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-02-10T07:33:35.775 INFO:teuthology.orchestra.run.smithi008.stdout: "exporter_sort_metrics": "true", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "exporter_stats_period": "5", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "fake_statfs_for_testing": "0", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "fatal_signal_handlers": "true", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "filer_max_purge_ops": "10", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "filer_max_truncate_ops": "128", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_apply_finisher_threads": "1", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_blackhole": "false", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_btrfs_clone_range": "true", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_btrfs_snap": "true", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_caller_concurrency": "10", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_collect_device_partition_information": "true", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_commit_timeout": "600.000000", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_debug_inject_read_err": "false", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_debug_omap_check": "false", 2024-02-10T07:33:35.776 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_debug_random_read_err": "0.000000", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_debug_verify_split": "false", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_dump_file": "", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_fadvise": "true", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_fail_eio": "true", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_fd_cache_shards": "16", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_fd_cache_size": "128", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_fiemap": "false", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_fiemap_threshold": "4096", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_index_retry_probability": "0.000000", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_inject_stall": "0", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_journal_parallel": "false", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_journal_trailing": "false", 2024-02-10T07:33:35.777 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_journal_writeahead": "false", 2024-02-10T07:33:35.778 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_kill_at": "0", 2024-02-10T07:33:35.778 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-02-10T07:33:35.778 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattr_size": "0", 2024-02-10T07:33:35.778 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-02-10T07:33:35.778 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-02-10T07:33:35.778 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-02-10T07:33:35.778 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattrs": "0", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattrs_other": "2", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_sync_interval": "5.000000", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_xattr_value_size": "0", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_merge_threshold": "-10", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_min_sync_interval": "0.010000", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_odsync_write": "false", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_omap_backend": "rocksdb", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_omap_backend_path": "", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_omap_header_cache_size": "1024", 2024-02-10T07:33:35.779 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_ondisk_finisher_threads": "1", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_op_thread_timeout": "60", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_op_threads": "2", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_punch_hole": "false", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_max_bytes": "104857600", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_queue_max_ops": "50", 2024-02-10T07:33:35.780 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_seek_data_hole": "false", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_sloppy_crc": "false", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_splice": "false", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_split_multiple": "2", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_split_rand_factor": "20", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_update_to": "1000", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_enable": "true", 2024-02-10T07:33:35.781 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_xfs_extsize": "false", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "filestore_zfs_snap": "false", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "fio_dir": "/tmp/fio", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "fsid": "8c8244e6-c7e5-11ee-95b8-87774f69a715", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_allow_other": "true", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_atomic_o_trunc": "true", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_big_writes": "true", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_debug": "false", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_default_permissions": "false", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_disable_pagecache": "false", 2024-02-10T07:33:35.782 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_max_write": "0", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_multithreaded": "true", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_require_active_mds": "true", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_set_user_groups": "true", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_splice_move": "true", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_splice_read": "true", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_splice_write": "true", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_syncfs_on_mksnap": "true", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "fuse_use_invalidate_cb": "true", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "gss_target_name": "ceph", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_file": "", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_inject_failure": "0", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_interval": "5", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "host": "smithi008", 2024-02-10T07:33:35.783 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_max_size": "1073741824", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "immutable_object_cache_watermark": "0.900000", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "inject_early_sigterm": "false", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "jaeger_agent_port": "6799", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "jaeger_tracing_enable": "false", 2024-02-10T07:33:35.784 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_aio": "true", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_align_min_size": "65536", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_block_align": "true", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_block_size": "4096", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_dio": "true", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_discard": "false", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_force_aio": "false", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_ignore_corruption": "false", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_max_write_bytes": "10485760", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_max_write_entries": "100", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_replay_from": "0", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_throttle_high_multiple": "0.000000", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_throttle_max_multiple": "0.000000", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_write_header_frequency": "0", 2024-02-10T07:33:35.785 INFO:teuthology.orchestra.run.smithi008.stdout: "journal_zero_on_create": "false", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "journaler_prefetch_periods": "10", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "journaler_prezero_periods": "5", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "journaler_write_head_interval": "15", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "key": "", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "keyfile": "", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_backend": "rocksdb", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_default_stripe_size": "65536", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_fsck_on_mount": "false", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_fsck_on_mount_deep": "true", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_max_bytes": "67108864", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_max_ops": "512", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_nid_prealloc": "1024", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_onode_map_size": "1024", 2024-02-10T07:33:35.786 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_sync_submit_transaction": "false", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "kstore_sync_transaction": "false", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "librados_thread_count": "2", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "lockdep": "false", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "lockdep_force_backtrace": "false", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "log_coarse_timestamps": "true", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "log_flush_on_exit": "false", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "log_graylog_host": "127.0.0.1", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "log_graylog_port": "12201", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "log_max_new": "1000", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "log_max_recent": "10000", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "log_stderr_prefix": "", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "log_stop_at_utilization": "0.970000", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "log_to_file": "true", 2024-02-10T07:33:35.787 INFO:teuthology.orchestra.run.smithi008.stdout: "log_to_graylog": "false", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "log_to_journald": "false", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "log_to_stderr": "false", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "log_to_syslog": "false", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "max_rotating_auth_attempts": "10", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_action_on_write_error": "1", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_alternate_name_max": "8192", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_asio_thread_count": "2", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_export_pin": "true", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_fragment_dirs": "true", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_fragment_interval": "5", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_fragment_size_max": "100000", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_idle_threshold": "0.000000", 2024-02-10T07:33:35.788 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_interval": "10", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_max": "-1", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_max_until": "-1", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_merge_size": "50", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_midchunk": "0.300000", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_min_rebalance": "0.100000", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_min_start": "0.200000", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_minchunk": "0.001000", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_mode": "0", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_need_max": "1.200000", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_need_min": "0.800000", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_overload_epochs": "2", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_sample_interval": "3.000000", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_split_bits": "3", 2024-02-10T07:33:35.789 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_split_rd": "25000.000000", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_split_size": "10000", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_split_wr": "10000.000000", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_target_decay": "10.000000", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_beacon_grace": "15.000000", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_beacon_interval": "4.000000", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_beacon_mon_down_grace": "60", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_memory_limit": "4294967296", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_mid": "0.700000", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_release_free_interval": "10", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_reservation": "0.050000", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_trim_interval": "1", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cache_trim_threshold": "262144", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-02-10T07:33:35.790 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_client_delegate_inos_pct": "50", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_client_prealloc_inos": "1000", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_connect_bootstrapping": "false", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_damage_table_max_entries": "10000", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_debug_auth_pins": "false", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_debug_frag": "false", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_debug_scatterstat": "false", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_debug_subtrees": "false", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_decay_halflife": "5.000000", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_default_dir_hash": "2", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_defer_session_stale": "true", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_deny_all_reconnect": "false", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dir_keys_per_op": "16384", 2024-02-10T07:33:35.791 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dir_max_commit_size": "10", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dir_max_entries": "0", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dir_prefetch": "true", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dirstat_min_interval": "1.000000", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dump_cache_after_rejoin": "false", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dump_cache_on_map": "false", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dump_cache_threshold_file": "0", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_early_reply": "true", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_enable_op_tracker": "true", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_enforce_unique_name": "true", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_export_ephemeral_distributed": "true", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_export_ephemeral_random": "true", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-02-10T07:33:35.792 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_forward_all_requests_to_auth": "false", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_health_cache_threshold": "1.500000", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_health_summarize_threshold": "10", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_heartbeat_grace": "15.000000", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_heartbeat_reset_grace": "1000", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_inject_health_dummy": "false", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_inject_migrator_session_race": "false", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-02-10T07:33:35.793 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_join_fs": "", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_journal_format": "1", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_create_at": "0", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_export_at": "0", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_import_at": "0", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_journal_at": "0", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_journal_expire_at": "0", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_journal_replay_at": "0", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_link_at": "0", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_mdstable_at": "0", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_openc_at": "0", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_rename_at": "0", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_shutdown_at": "0", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_event_large_threshold": "524288", 2024-02-10T07:33:35.794 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_events_per_segment": "1024", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_major_segment_event_ratio": "12", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_max_events": "-1", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_max_segments": "128", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_pause": "false", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_segment_size": "0", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_skip_corrupt_events": "false", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_skip_unbounded_events": "false", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_trim_threshold": "128", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_log_warn_factor": "2.000000", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_caps_per_client": "1048576", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_completed_flushes": "100000", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_completed_requests": "100000", 2024-02-10T07:33:35.795 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_export_size": "20971520", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_file_recover": "32", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_purge_files": "64", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_purge_ops": "8192", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_max_snaps_per_dir": "100", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_metrics_update_interval": "2", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_min_caps_per_client": "100", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_min_caps_working_set": "10000", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_numa_node": "-1", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_op_complaint_time": "30.000000", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_op_history_duration": "600", 2024-02-10T07:33:35.796 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_op_history_size": "20", 2024-02-10T07:33:35.797 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_op_history_slow_op_size": "20", 2024-02-10T07:33:35.797 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_op_history_slow_op_threshold": "10", 2024-02-10T07:33:35.797 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_op_log_threshold": "5", 2024-02-10T07:33:35.797 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_ping_grace": "15", 2024-02-10T07:33:35.797 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_ping_interval": "5", 2024-02-10T07:33:35.797 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-02-10T07:33:35.797 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-02-10T07:33:35.797 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_recall_max_caps": "30000", 2024-02-10T07:33:35.797 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-02-10T07:33:35.797 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_recall_max_decay_threshold": "131072", 2024-02-10T07:33:35.797 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-02-10T07:33:35.797 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_recall_warning_threshold": "262144", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_reconnect_timeout": "45.000000", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_replay_interval": "1.000000", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_root_ino_gid": "0", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_root_ino_uid": "0", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_blocklist_on_evict": "true", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_blocklist_on_timeout": "true", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_session_metadata_threshold": "16777216", 2024-02-10T07:33:35.798 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_shutdown_check": "0", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_skip_ino": "0", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_sleep_rank_change": "0.000000", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_snap_max_uid": "4294967294", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_snap_min_uid": "0", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_snap_rstat": "false", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_standby_replay_damaged": "false", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_symlink_recovery": "true", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_task_status_update_interval": "2.000000", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_thrash_exports": "0", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_thrash_fragments": "0", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_tick_interval": "5.000000", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_valgrind_exit": "false", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_verify_backtrace": "1", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_verify_scatter": "false", 2024-02-10T07:33:35.799 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_wipe_ino_prealloc": "false", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_wipe_sessions": "false", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "mempool_debug": "false", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "memstore_debug_omit_block_device_write": "false", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "memstore_device_bytes": "1073741824", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "memstore_page_set": "false", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "memstore_page_size": "65536", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_client_bytes": "134217728", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_client_messages": "512", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_connect_retry_interval": "1.000000", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_disabled_modules": "", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-02-10T07:33:35.800 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_max_pg_num_change": "128", 2024-02-10T07:33:35.801 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_mds_bytes": "134217728", 2024-02-10T07:33:35.801 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_mds_messages": "128", 2024-02-10T07:33:35.801 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-02-10T07:33:35.801 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_mon_bytes": "134217728", 2024-02-10T07:33:35.801 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_mon_messages": "128", 2024-02-10T07:33:35.802 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_osd_bytes": "536870912", 2024-02-10T07:33:35.802 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_osd_messages": "8192", 2024-02-10T07:33:35.802 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_pool": "true", 2024-02-10T07:33:35.802 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_service_beacon_grace": "60.000000", 2024-02-10T07:33:35.802 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_standby_modules": "true", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_stats_period": "5", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_stats_threshold": "5", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_tick_period": "2", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_accept_timeout_factor": "2.000000", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_allow_pool_delete": "true", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_allow_pool_size_one": "false", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_auth_validate_all_caps": "true", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_bytes": "104857600", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_directed_command_retry": "2", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_hunt_interval": "3.000000", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-02-10T07:33:35.803 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_hunt_parallel": "3", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_log_interval": "1.000000", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_ping_interval": "10.000000", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_ping_timeout": "30.000000", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_client_target_rank": "-1", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_clock_drift_allowed": "1.000000", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_file_level": "debug", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_file": "true", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_graylog": "false", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-02-10T07:33:35.804 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_journald": "true", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_stderr": "false", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_compact_on_bootstrap": "false", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_compact_on_start": "false", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_compact_on_trim": "true", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_con_tracker_persist_interval": "10", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_con_tracker_score_halflife": "43200", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_config_key_max_entry_size": "65536", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_cpu_threads": "4", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_crush_min_required_version": "hammer", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_daemon_bytes": "419430400", 2024-02-10T07:33:35.805 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_data_avail_crit": "5", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_data_avail_warn": "5", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_data_size_warn": "16106127360", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_block_osdmap_trim": "false", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_dump_json": "false", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_dump_transactions": "false", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_extra_checks": "false", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_no_require_reef": "false", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_no_require_squid": "false", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_delta_reset_interval": "10.000000", 2024-02-10T07:33:35.806 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_dns_srv_name": "ceph-mon", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_down_mkfs_grace": "60", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_election_default_strategy": "1", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_election_timeout": "5.000000", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_elector_ping_divisor": "2", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_elector_ping_timeout": "2.000000", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_enable_op_tracker": "true", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_fake_pool_delete": "false", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_force_quorum_join": "false", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_globalid_prealloc": "10000", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_health_detail_to_clog": "true", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_health_log_update_period": "5", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_health_max_detail": "50", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_health_to_clog": "true", 2024-02-10T07:33:35.807 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_health_to_clog_interval": "600", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_host": "[v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0]", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_host_override": "", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_initial_members": "", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_keyvaluedb": "rocksdb", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_lease": "5.000000", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_log_full_interval": "50", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_log_max": "10000", 2024-02-10T07:33:35.808 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_log_max_summary": "50", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_log_entries_per_event": "4096", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_log_epochs": "500", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_mdsmap_epochs": "500", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_mgrmap_epochs": "500", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_osd": "10000", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_pg_per_osd": "10000", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_pool_pg_num": "65536", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mds_force_trim_to": "0", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mds_skip_sanity": "false", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_memory_autotune": "true", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_memory_target": "2147483648", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mgr_beacon_grace": "30", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-02-10T07:33:35.809 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mgr_digest_period": "5", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mgr_inactive_grace": "60", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mgr_mkfs_grace": "240", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_min_osdmap_epochs": "50", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_op_complaint_time": "30", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_op_history_duration": "600", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_op_history_size": "20", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_op_history_slow_op_size": "20", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_op_history_slow_op_threshold": "10", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_op_log_threshold": "5", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_auto_mark_in": "false", 2024-02-10T07:33:35.810 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_auto_mark_new_in": "true", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_cache_size": "500", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_cache_size_min": "134217728", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_crush_smoke_test": "true", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_destroyed_out_interval": "600", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_down_out_interval": "600", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_force_trim_to": "0", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_full_ratio": "0.950000", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_laggy_halflife": "3600", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_laggy_max_interval": "300", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_laggy_weight": "0.300000", 2024-02-10T07:33:35.811 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_max_creating_pgs": "1024", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_max_initial_pgs": "1024", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_min_down_reporters": "2", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_prime_pg_temp": "true", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_report_timeout": "900", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_warn_num_repaired": "10", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osd_warn_op_age": "32.000000", 2024-02-10T07:33:35.812 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osdmap_full_prune_interval": "2", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osdmap_full_prune_min": "15", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pg_stuck_threshold": "60", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pg_warn_min_objects": "10000", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pg_warn_min_per_osd": "0", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pool_quota_crit_threshold": "0", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_pool_quota_warn_threshold": "0", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_probe_timeout": "2.000000", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_reweight_max_change": "0.050000", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_reweight_max_osds": "4", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-02-10T07:33:35.813 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_scrub_interval": "86400", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_scrub_max_keys": "100", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_scrub_timeout": "300", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_session_timeout": "300", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_smart_report_timeout": "5", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_stat_smooth_intervals": "6", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_stretch_pool_min_size": "2", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_stretch_pool_size": "4", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_subscribe_interval": "86400.000000", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_sync_debug": "false", 2024-02-10T07:33:35.814 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_sync_max_payload_keys": "2000", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_sync_max_payload_size": "1048576", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_sync_provider_kill_at": "0", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_sync_requester_kill_at": "0", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_sync_timeout": "60.000000", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_target_pg_per_osd": "100", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_tick_interval": "5", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_timecheck_interval": "300.000000", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_use_min_delay_socket": "false", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_older_version_delay": "604800", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_filestore_osds": "true", 2024-02-10T07:33:35.815 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_misplaced": "false", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_older_version": "true", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_pool_no_app": "true", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_on_too_few_osds": "false", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-02-10T07:33:35.816 INFO:teuthology.orchestra.run.smithi008.stdout: "monmap": "", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "motr_tracing_enabled": "false", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_op_threads": "3", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_buffer_size": "131072", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_cm": "false", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_device_name": "", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_dscp": "96", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_gid_idx": "0", 2024-02-10T07:33:35.817 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_local_gid": "", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_polling_us": "1000", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_port_num": "1", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_roce_ver": "1", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_send_buffers": "1024", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_sl": "3", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_support_srq": "true", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_rdma_type": "ib", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_async_reap_threshold": "5", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_before_connect": "false", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_ipv4": "true", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_ipv6": "false", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_msgr1": "true", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_msgr2": "true", 2024-02-10T07:33:35.818 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_port_max": "7568", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_port_min": "6800", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_prefer_ipv4": "false", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_retry_count": "3", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_bind_retry_delay": "5", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_blackhole_client": "false", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_blackhole_mds": "false", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_blackhole_mgr": "false", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_blackhole_mon": "false", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_blackhole_osd": "false", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_client_mode": "crc secure", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_cluster_mode": "crc secure", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_cluster_type": "", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_compress_secure": "false", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_connection_idle_timeout": "900", 2024-02-10T07:33:35.819 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_connection_ready_timeout": "10", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_crc_data": "true", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_crc_header": "true", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_die_on_bad_msg": "false", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_die_on_bug": "true", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_die_on_old_message": "true", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_die_on_skipped_message": "false", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_die_on_unhandled_msg": "false", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_coremask": "0xF", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_devs_allowlist": "", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_enable_tso": "true", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-02-10T07:33:35.820 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_hugepages": "", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_hw_flow_control": "true", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_lro": "true", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_memory_channel": "4", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_pmd": "", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_port_id": "0", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dump_corrupt_message_level": "1", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_dump_on_send": "false", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_initial_backoff": "0.200000", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_inject_delay_max": "1.000000", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_inject_delay_probability": "0.000000", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_inject_delay_type": "", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_inject_internal_delays": "0.000000", 2024-02-10T07:33:35.821 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_inject_network_congestion": "0", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_inject_socket_failures": "0", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_learn_addr_from_peer": "true", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_max_accept_failures": "4", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_max_backoff": "15.000000", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_mon_client_mode": "secure crc", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_mon_cluster_mode": "secure crc", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_mon_service_mode": "secure crc", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_osd_compress_min_size": "1024", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_osd_compress_mode": "none", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_osd_compression_algorithm": "snappy", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_pq_min_cost": "65536", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_public_type": "", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_service_mode": "crc secure", 2024-02-10T07:33:35.822 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_tcp_listen_backlog": "512", 2024-02-10T07:33:35.823 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_tcp_nodelay": "true", 2024-02-10T07:33:35.823 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-02-10T07:33:35.823 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_tcp_rcvbuf": "0", 2024-02-10T07:33:35.823 INFO:teuthology.orchestra.run.smithi008.stdout: "ms_type": "async+posix", 2024-02-10T07:33:35.823 INFO:teuthology.orchestra.run.smithi008.stdout: "no_config_file": "false", 2024-02-10T07:33:35.823 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_completion_locks_per_session": "32", 2024-02-10T07:33:35.823 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_debug_inject_relock_delay": "false", 2024-02-10T07:33:35.823 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_inflight_op_bytes": "104857600", 2024-02-10T07:33:35.823 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_inflight_ops": "1024", 2024-02-10T07:33:35.823 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_inject_no_watch_ping": "false", 2024-02-10T07:33:35.823 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-02-10T07:33:35.824 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_tick_interval": "5.000000", 2024-02-10T07:33:35.824 INFO:teuthology.orchestra.run.smithi008.stdout: "objecter_timeout": "10.000000", 2024-02-10T07:33:35.824 INFO:teuthology.orchestra.run.smithi008.stdout: "objectstore_blackhole": "false", 2024-02-10T07:33:35.824 INFO:teuthology.orchestra.run.smithi008.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-02-10T07:33:35.824 INFO:teuthology.orchestra.run.smithi008.stdout: "openssl_engine_opts": "", 2024-02-10T07:33:35.824 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_agent_delay_time": "5.000000", 2024-02-10T07:33:35.824 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_agent_hist_halflife": "1000", 2024-02-10T07:33:35.824 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_agent_max_low_ops": "2", 2024-02-10T07:33:35.824 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_agent_max_ops": "4", 2024-02-10T07:33:35.824 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-02-10T07:33:35.824 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_agent_quantize_effort": "0.100000", 2024-02-10T07:33:35.824 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_agent_slop": "0.020000", 2024-02-10T07:33:35.824 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-02-10T07:33:35.825 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_allow_recovery_below_min_size": "true", 2024-02-10T07:33:35.825 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_asio_thread_count": "2", 2024-02-10T07:33:35.825 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_async_recovery_min_cost": "100", 2024-02-10T07:33:35.825 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_auto_mark_unfound_lost": "false", 2024-02-10T07:33:35.825 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_backfill_retry_interval": "30.000000", 2024-02-10T07:33:35.825 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_backfill_scan_max": "512", 2024-02-10T07:33:35.825 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_backfill_scan_min": "64", 2024-02-10T07:33:35.825 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_backoff_on_degraded": "false", 2024-02-10T07:33:35.825 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_backoff_on_peering": "false", 2024-02-10T07:33:35.825 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_backoff_on_unfound": "true", 2024-02-10T07:33:35.825 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_beacon_report_interval": "300", 2024-02-10T07:33:35.825 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_bench_duration": "30", 2024-02-10T07:33:35.825 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-02-10T07:33:35.826 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_bench_max_block_size": "67108864", 2024-02-10T07:33:35.826 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_bench_small_size_max_iops": "100", 2024-02-10T07:33:35.826 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_blkin_trace_all": "false", 2024-02-10T07:33:35.826 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_blocked_scrub_grace_period": "120", 2024-02-10T07:33:35.827 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-02-10T07:33:35.827 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-02-10T07:33:35.827 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-02-10T07:33:35.827 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_check_for_log_corruption": "false", 2024-02-10T07:33:35.827 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-02-10T07:33:35.827 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-10T07:33:35.828 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-02-10T07:33:35.828 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-10T07:33:35.828 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_class_update_on_start": "true", 2024-02-10T07:33:35.829 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_client_message_cap": "256", 2024-02-10T07:33:35.829 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_client_message_size_cap": "524288000", 2024-02-10T07:33:35.829 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_client_op_priority": "63", 2024-02-10T07:33:35.829 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_client_watch_timeout": "30", 2024-02-10T07:33:35.830 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_command_max_records": "256", 2024-02-10T07:33:35.830 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_compact_on_start": "false", 2024-02-10T07:33:35.830 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-02-10T07:33:35.830 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_crush_chooseleaf_type": "1", 2024-02-10T07:33:35.830 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_crush_initial_weight": "-1.000000", 2024-02-10T07:33:35.831 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_crush_update_on_start": "true", 2024-02-10T07:33:35.831 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_crush_update_weight_set": "true", 2024-02-10T07:33:35.831 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-02-10T07:33:35.831 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-02-10T07:33:35.831 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-02-10T07:33:35.831 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_drop_ping_duration": "0", 2024-02-10T07:33:35.831 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-02-10T07:33:35.832 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_feed_pullee": "-1", 2024-02-10T07:33:35.832 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-02-10T07:33:35.832 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-02-10T07:33:35.832 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-02-10T07:33:35.832 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_misdirected_ops": "false", 2024-02-10T07:33:35.832 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_no_acting_change": "false", 2024-02-10T07:33:35.832 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_no_purge_strays": "false", 2024-02-10T07:33:35.833 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_op_order": "false", 2024-02-10T07:33:35.833 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_pg_log_writeout": "false", 2024-02-10T07:33:35.833 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_pretend_recovery_active": "false", 2024-02-10T07:33:35.833 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-02-10T07:33:35.833 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-02-10T07:33:35.833 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_shutdown": "false", 2024-02-10T07:33:35.833 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-02-10T07:33:35.834 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-02-10T07:33:35.834 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_trim_objects": "false", 2024-02-10T07:33:35.834 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_verify_cached_snaps": "false", 2024-02-10T07:33:35.834 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_verify_missing_on_start": "false", 2024-02-10T07:33:35.834 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_verify_snaps": "false", 2024-02-10T07:33:35.834 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-02-10T07:33:35.834 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-02-10T07:33:35.835 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_deep_scrub_keys": "1024", 2024-02-10T07:33:35.835 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-02-10T07:33:35.835 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-02-10T07:33:35.835 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-02-10T07:33:35.835 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_deep_scrub_stride": "524288", 2024-02-10T07:33:35.835 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-02-10T07:33:35.835 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_default_data_pool_replay_window": "45", 2024-02-10T07:33:35.836 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_default_notify_timeout": "30", 2024-02-10T07:33:35.836 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_delete_sleep": "0.000000", 2024-02-10T07:33:35.836 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-02-10T07:33:35.836 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-02-10T07:33:35.836 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-02-10T07:33:35.837 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_discard_disconnected_ops": "true", 2024-02-10T07:33:35.837 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_enable_op_tracker": "true", 2024-02-10T07:33:35.837 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-02-10T07:33:35.837 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_extblkdev_plugins": "vdo", 2024-02-10T07:33:35.837 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-02-10T07:33:35.837 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-02-10T07:33:35.838 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_fast_info": "true", 2024-02-10T07:33:35.838 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_fast_shutdown": "true", 2024-02-10T07:33:35.838 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-02-10T07:33:35.838 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_fast_shutdown_timeout": "15", 2024-02-10T07:33:35.838 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-02-10T07:33:35.838 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-02-10T07:33:35.838 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-02-10T07:33:35.839 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_function_tracing": "false", 2024-02-10T07:33:35.839 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_heartbeat_grace": "20", 2024-02-10T07:33:35.839 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_heartbeat_interval": "6", 2024-02-10T07:33:35.839 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-02-10T07:33:35.839 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_heartbeat_min_peers": "10", 2024-02-10T07:33:35.839 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_heartbeat_min_size": "2000", 2024-02-10T07:33:35.839 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_heartbeat_stale": "600", 2024-02-10T07:33:35.839 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-02-10T07:33:35.840 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_hit_set_max_size": "100000", 2024-02-10T07:33:35.840 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_hit_set_min_size": "1000", 2024-02-10T07:33:35.840 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-02-10T07:33:35.841 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-02-10T07:33:35.841 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-02-10T07:33:35.841 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-02-10T07:33:35.841 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-02-10T07:33:35.841 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_journal_flush_on_shutdown": "true", 2024-02-10T07:33:35.841 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_journal_size": "5120", 2024-02-10T07:33:35.841 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_kill_backfill_at": "0", 2024-02-10T07:33:35.841 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_loop_before_reset_tphandle": "64", 2024-02-10T07:33:35.842 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_map_cache_size": "50", 2024-02-10T07:33:35.842 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_map_dedup": "true", 2024-02-10T07:33:35.842 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_map_message_max": "40", 2024-02-10T07:33:35.843 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_map_message_max_bytes": "10485760", 2024-02-10T07:33:35.843 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_map_share_max_epochs": "40", 2024-02-10T07:33:35.843 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_attr_name_len": "100", 2024-02-10T07:33:35.843 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_attr_size": "0", 2024-02-10T07:33:35.843 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_backfills": "1", 2024-02-10T07:33:35.843 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_markdown_count": "5", 2024-02-10T07:33:35.843 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_markdown_period": "600", 2024-02-10T07:33:35.844 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_object_name_len": "2048", 2024-02-10T07:33:35.844 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_object_namespace_len": "256", 2024-02-10T07:33:35.845 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_object_size": "134217728", 2024-02-10T07:33:35.845 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-02-10T07:33:35.845 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_omap_entries_per_request": "1024", 2024-02-10T07:33:35.845 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_pg_blocked_by": "16", 2024-02-10T07:33:35.845 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_pg_log_entries": "10000", 2024-02-10T07:33:35.845 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-02-10T07:33:35.846 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_pgls": "1024", 2024-02-10T07:33:35.846 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_push_cost": "8388608", 2024-02-10T07:33:35.846 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_push_objects": "10", 2024-02-10T07:33:35.846 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_scrubs": "3", 2024-02-10T07:33:35.846 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-02-10T07:33:35.846 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_trimming_pgs": "2", 2024-02-10T07:33:35.846 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_write_op_reply_len": "64", 2024-02-10T07:33:35.846 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_max_write_size": "90", 2024-02-10T07:33:35.846 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-02-10T07:33:35.847 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-02-10T07:33:35.847 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-02-10T07:33:35.847 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-02-10T07:33:35.847 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-02-10T07:33:35.847 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-02-10T07:33:35.847 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-02-10T07:33:35.847 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_override_recovery_settings": "false", 2024-02-10T07:33:35.847 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_profile": "balanced", 2024-02-10T07:33:35.847 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-02-10T07:33:35.847 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-02-10T07:33:35.847 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-02-10T07:33:35.847 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-02-10T07:33:35.847 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-02-10T07:33:35.847 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-02-10T07:33:35.848 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-02-10T07:33:35.848 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-02-10T07:33:35.848 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-02-10T07:33:35.848 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-02-10T07:33:35.848 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mclock_skip_benchmark": "false", 2024-02-10T07:33:35.848 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_memory_base": "805306368", 2024-02-10T07:33:35.848 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_memory_cache_min": "134217728", 2024-02-10T07:33:35.848 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-02-10T07:33:35.848 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-02-10T07:33:35.848 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_memory_target": "4294967296", 2024-02-10T07:33:35.848 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_memory_target_autotune": "false", 2024-02-10T07:33:35.848 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-02-10T07:33:35.849 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_min_pg_log_entries": "250", 2024-02-10T07:33:35.849 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_min_recovery_priority": "0", 2024-02-10T07:33:35.849 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mon_heartbeat_interval": "30", 2024-02-10T07:33:35.849 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-02-10T07:33:35.849 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mon_report_interval": "5", 2024-02-10T07:33:35.849 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mon_report_max_in_flight": "2", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_num_cache_shards": "32", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_num_op_tracker_shard": "32", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_numa_auto_affinity": "true", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_numa_node": "-1", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_numa_prefer_iface": "true", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_objecter_finishers": "1", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_objectstore": "bluestore", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_objectstore_fuse": "false", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_objectstore_tracing": "false", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_complaint_time": "30.000000", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_history_duration": "600", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_history_size": "20", 2024-02-10T07:33:35.850 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_history_slow_op_size": "20", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_log_threshold": "5", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_num_shards": "0", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_num_shards_hdd": "5", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_num_shards_ssd": "8", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_num_threads_per_shard": "0", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_pq_min_cost": "65536", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_queue": "mclock_scheduler", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_queue_cut_off": "high", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_thread_suicide_timeout": "150", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_op_thread_timeout": "15", 2024-02-10T07:33:35.851 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_open_classes_on_start": "true", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_os_flags": "0", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_peering_op_priority": "255", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_delete_cost": "1048576", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_delete_priority": "5", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_log_dups_tracked": "3000", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_log_trim_max": "10000", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_log_trim_min": "100", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_object_context_cache_count": "64", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pg_stat_report_interval_max_seconds": "500", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-02-10T07:33:35.852 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_crimson": "false", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_crush_rule": "-1", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_ec_fast_read": "false", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_flag_bulk": "false", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_flag_nodelete": "false", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_flags": "0", 2024-02-10T07:33:35.853 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-02-10T07:33:35.854 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_min_size": "0", 2024-02-10T07:33:35.854 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-02-10T07:33:35.854 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_pg_num": "32", 2024-02-10T07:33:35.854 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_pgp_num": "0", 2024-02-10T07:33:35.854 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-02-10T07:33:35.854 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_read_ratio": "70", 2024-02-10T07:33:35.854 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_size": "3", 2024-02-10T07:33:35.854 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_default_type": "replicated", 2024-02-10T07:33:35.854 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-02-10T07:33:35.854 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pool_use_gmt_hitset": "true", 2024-02-10T07:33:35.854 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_push_per_object_cost": "1000", 2024-02-10T07:33:35.854 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_read_ec_check_for_errors": "false", 2024-02-10T07:33:35.855 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recover_clone_overlap": "true", 2024-02-10T07:33:35.855 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recover_clone_overlap_limit": "10", 2024-02-10T07:33:35.855 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_cost": "20971520", 2024-02-10T07:33:35.855 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_delay_start": "0.000000", 2024-02-10T07:33:35.855 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_max_active": "0", 2024-02-10T07:33:35.856 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_max_active_hdd": "3", 2024-02-10T07:33:35.856 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_max_active_ssd": "10", 2024-02-10T07:33:35.856 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_max_chunk": "8388608", 2024-02-10T07:33:35.856 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-02-10T07:33:35.856 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_max_single_start": "1", 2024-02-10T07:33:35.856 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_op_priority": "3", 2024-02-10T07:33:35.856 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_op_warn_multiple": "16", 2024-02-10T07:33:35.856 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_priority": "5", 2024-02-10T07:33:35.856 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_retry_interval": "30.000000", 2024-02-10T07:33:35.856 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_sleep": "0.000000", 2024-02-10T07:33:35.856 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-02-10T07:33:35.856 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-02-10T07:33:35.856 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-02-10T07:33:35.856 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_repair_during_recovery": "false", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_requested_scrub_priority": "120", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_rollback_to_cluster_snap": "", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_auto_repair": "false", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_begin_hour": "0", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_begin_week_day": "0", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_chunk_max": "25", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_chunk_min": "5", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_cost": "52428800", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_during_recovery": "false", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_end_hour": "0", 2024-02-10T07:33:35.857 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_end_week_day": "0", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_event_cost": "4096", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_invalid_stats": "true", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_load_threshold": "0.500000", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_max_interval": "604800.000000", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_max_preemptions": "5", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_min_interval": "86400.000000", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_priority": "5", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_reservation_timeout": "300000", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_sleep": "0.000000", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-02-10T07:33:35.858 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_shutdown_pgref_assert": "false", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_skip_data_digest": "false", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_smart_report_timeout": "5", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_snap_trim_cost": "1048576", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_snap_trim_priority": "5", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_snap_trim_sleep": "0.000000", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_stats_update_period_scrubbing": "15", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_target_transaction_size": "30", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-02-10T07:33:35.859 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-02-10T07:33:35.860 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-02-10T07:33:35.860 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-02-10T07:33:35.860 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-02-10T07:33:35.860 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_default_cache_mode": "writeback", 2024-02-10T07:33:35.860 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-02-10T07:33:35.860 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-02-10T07:33:35.860 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_tracing": "false", 2024-02-10T07:33:35.860 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_use_stale_snap": "false", 2024-02-10T07:33:35.860 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-02-10T07:33:35.860 INFO:teuthology.orchestra.run.smithi008.stdout: "osdc_blkin_trace_all": "false", 2024-02-10T07:33:35.860 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_kill_at": "0", 2024-02-10T07:33:35.860 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_max_join_drift": "10", 2024-02-10T07:33:35.860 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_min": "500", 2024-02-10T07:33:35.860 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_min_wait": "0.050000", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_propose_interval": "1.000000", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_service_trim_max": "500", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_service_trim_max_multiplier": "20", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_service_trim_min": "10", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_stash_full_interval": "25", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_trim_max": "500", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "paxos_trim_min": "250", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "perf": "true", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "pid_file": "", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "-", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrv": "", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "public_bind_addr": "-", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "public_network": "", 2024-02-10T07:33:35.861 INFO:teuthology.orchestra.run.smithi008.stdout: "public_network_interface": "", 2024-02-10T07:33:35.862 INFO:teuthology.orchestra.run.smithi008.stdout: "qat_compressor_enabled": "false", 2024-02-10T07:33:35.862 INFO:teuthology.orchestra.run.smithi008.stdout: "qat_compressor_session_max_number": "256", 2024-02-10T07:33:35.862 INFO:teuthology.orchestra.run.smithi008.stdout: "rados_mon_op_timeout": "0", 2024-02-10T07:33:35.862 INFO:teuthology.orchestra.run.smithi008.stdout: "rados_osd_op_timeout": "0", 2024-02-10T07:33:35.862 INFO:teuthology.orchestra.run.smithi008.stdout: "rados_tracing": "false", 2024-02-10T07:33:35.862 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_atime_update_interval": "60", 2024-02-10T07:33:35.862 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-02-10T07:33:35.862 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_balance_parent_reads": "false", 2024-02-10T07:33:35.862 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_balance_snap_reads": "false", 2024-02-10T07:33:35.862 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_blkin_trace_all": "false", 2024-02-10T07:33:35.862 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_blocklist_expire_seconds": "0", 2024-02-10T07:33:35.863 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_blocklist_on_break_lock": "true", 2024-02-10T07:33:35.863 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache": "true", 2024-02-10T07:33:35.863 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_block_writes_upfront": "false", 2024-02-10T07:33:35.863 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_max_dirty": "25165824", 2024-02-10T07:33:35.863 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-02-10T07:33:35.863 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_max_dirty_object": "0", 2024-02-10T07:33:35.863 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_policy": "writearound", 2024-02-10T07:33:35.863 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_size": "33554432", 2024-02-10T07:33:35.863 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_target_dirty": "16777216", 2024-02-10T07:33:35.863 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-02-10T07:33:35.863 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_clone_copy_on_read": "false", 2024-02-10T07:33:35.863 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_compression_hint": "none", 2024-02-10T07:33:35.863 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_concurrent_management_ops": "10", 2024-02-10T07:33:35.863 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-02-10T07:33:35.864 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_clone_format": "auto", 2024-02-10T07:33:35.864 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_data_pool": "", 2024-02-10T07:33:35.864 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_features": "61", 2024-02-10T07:33:35.864 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_format": "2", 2024-02-10T07:33:35.864 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_map_options": "", 2024-02-10T07:33:35.864 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_order": "22", 2024-02-10T07:33:35.864 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_pool": "rbd", 2024-02-10T07:33:35.864 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-02-10T07:33:35.864 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_stripe_count": "0", 2024-02-10T07:33:35.865 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_default_stripe_unit": "0", 2024-02-10T07:33:35.865 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_disable_zero_copy_writes": "true", 2024-02-10T07:33:35.865 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_discard_granularity_bytes": "65536", 2024-02-10T07:33:35.865 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-02-10T07:33:35.865 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_enable_alloc_hint": "true", 2024-02-10T07:33:35.865 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-02-10T07:33:35.865 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_io_scheduler": "simple", 2024-02-10T07:33:35.865 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-02-10T07:33:35.865 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_commit_age": "5.000000", 2024-02-10T07:33:35.865 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-02-10T07:33:35.865 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-02-10T07:33:35.865 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-02-10T07:33:35.865 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-02-10T07:33:35.865 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_object_flush_interval": "0", 2024-02-10T07:33:35.866 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-02-10T07:33:35.866 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-02-10T07:33:35.866 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_order": "24", 2024-02-10T07:33:35.866 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_pool": "", 2024-02-10T07:33:35.866 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_journal_splay_width": "4", 2024-02-10T07:33:35.866 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_localize_parent_reads": "false", 2024-02-10T07:33:35.866 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_localize_snap_reads": "false", 2024-02-10T07:33:35.866 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-02-10T07:33:35.866 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-02-10T07:33:35.866 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-02-10T07:33:35.866 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_image_policy_type": "simple", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_memory_autotune": "true", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_memory_base": "805306368", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-02-10T07:33:35.867 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_memory_target": "4294967296", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirroring_delete_delay": "0", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirroring_replay_delay": "0", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_move_to_trash_on_remove": "false", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_mtime_update_interval": "60", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_non_blocking_aio": "true", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_op_thread_timeout": "60", 2024-02-10T07:33:35.868 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_op_threads": "1", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_parent_cache_enabled": "false", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_persistent_cache_mode": "disabled", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_persistent_cache_path": "/tmp", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_persistent_cache_size": "1073741824", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_plugins": "", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_bps_burst": "0", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_bps_limit": "0", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_exclude_ops": "0", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_iops_burst": "0", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_iops_limit": "0", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_read_bps_burst": "0", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-02-10T07:33:35.869 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_read_bps_limit": "0", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_read_iops_burst": "0", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_read_iops_limit": "0", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_schedule_tick_min": "50", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_write_bps_burst": "0", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_write_bps_limit": "0", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_write_iops_burst": "0", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_qos_write_iops_limit": "0", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_quiesce_notification_attempts": "10", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_read_from_replica_policy": "default", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_readahead_max_bytes": "524288", 2024-02-10T07:33:35.870 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_readahead_trigger_requests": "10", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_request_timed_out_seconds": "30", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_skip_partial_discard": "true", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_tracing": "false", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_validate_names": "true", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd_validate_pool": "true", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "restapi_base_url": "", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "restapi_log_level": "", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_acl_grants_max_num": "100", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_admin_entry": "admin", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_backend_store": "rados", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_barbican_url": "", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_beast_enable_async": "true", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_counters_cache": "false", 2024-02-10T07:33:35.871 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-02-10T07:33:35.872 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-02-10T07:33:35.872 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-02-10T07:33:35.872 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_index_max_aio": "128", 2024-02-10T07:33:35.872 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-02-10T07:33:35.872 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-02-10T07:33:35.872 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_quota_ttl": "600", 2024-02-10T07:33:35.872 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-02-10T07:33:35.872 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_cache_enabled": "true", 2024-02-10T07:33:35.872 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_cache_expiry_interval": "900", 2024-02-10T07:33:35.872 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_cache_lru_size": "10000", 2024-02-10T07:33:35.872 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_config_store": "rados", 2024-02-10T07:33:35.872 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_content_length_compat": "false", 2024-02-10T07:33:35.872 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_copy_obj_progress": "true", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_cors_rules_max_num": "100", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_cross_domain_policy": "", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_default_encryption_key": "", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_addr": "", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_ca_path": "", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_client_cert": "", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_client_key": "", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_password": "", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_kmip_username": "", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_require_ssl": "true", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-02-10T07:33:35.873 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_suppress_logs": "true", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_addr": "", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_auth": "token", 2024-02-10T07:33:35.874 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_namespace": "", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_prefix": "", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_token_file": "", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_curl_buffersize": "524288", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_curl_low_speed_limit": "1024", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_curl_low_speed_time": "300", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_curl_tcp_keepalive": "0", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-02-10T07:33:35.875 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-02-10T07:33:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_l1_fadvise": "4", 2024-02-10T07:33:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-02-10T07:33:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-02-10T07:33:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-02-10T07:33:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d4n_host": "127.0.0.1", 2024-02-10T07:33:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_d4n_port": "6379", 2024-02-10T07:33:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-02-10T07:33:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data_log_changes_size": "1000", 2024-02-10T07:33:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data_log_num_shards": "128", 2024-02-10T07:33:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-02-10T07:33:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data_log_window": "30", 2024-02-10T07:33:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data_notify_interval_msec": "0", 2024-02-10T07:33:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data_sync_poll_interval": "20", 2024-02-10T07:33:35.876 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_data_sync_spawn_window": "20", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_default_data_log_backing": "fifo", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_default_region_info_oid": "default.region", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_defer_to_bucket_acls": "", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-02-10T07:33:35.877 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-02-10T07:33:35.878 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-02-10T07:33:35.878 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_data_res": "500.000000", 2024-02-10T07:33:35.878 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-02-10T07:33:35.878 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-02-10T07:33:35.878 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-02-10T07:33:35.878 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-02-10T07:33:35.878 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dns_name": "", 2024-02-10T07:33:35.878 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dns_s3website_name": "", 2024-02-10T07:33:35.878 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_dynamic_resharding": "true", 2024-02-10T07:33:35.878 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-02-10T07:33:35.878 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enable_gc_threads": "true", 2024-02-10T07:33:35.878 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enable_lc_threads": "true", 2024-02-10T07:33:35.878 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enable_ops_log": "false", 2024-02-10T07:33:35.878 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enable_quota_threads": "true", 2024-02-10T07:33:35.879 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enable_static_website": "false", 2024-02-10T07:33:35.879 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enable_usage_log": "false", 2024-02-10T07:33:35.879 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_enforce_swift_acls": "true", 2024-02-10T07:33:35.879 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_exit_timeout_secs": "120", 2024-02-10T07:33:35.879 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_expose_bucket": "false", 2024-02-10T07:33:35.879 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_extended_http_attrs": "", 2024-02-10T07:33:35.879 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_filter": "none", 2024-02-10T07:33:35.879 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-02-10T07:33:35.879 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_frontends": "beast port=7480", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_max_concurrent_io": "10", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_max_deferred": "50", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_max_objs": "32", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_max_queue_size": "134213632", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_max_trim_chunk": "16", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_obj_min_wait": "7200", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_processor_max_time": "3600", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_gc_processor_period": "3600", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_get_obj_window_size": "16777216", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_healthcheck_disabling_path": "", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ignore_get_invalid_range": "false", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_init_timeout": "300", 2024-02-10T07:33:35.880 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_kafka_connection_idle": "30", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_kafka_sleep_timeout": "10", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_domain": "", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_password": "", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_password_path": "", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_project": "", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_tenant": "", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_token": "", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_token_path": "", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_admin_user": "", 2024-02-10T07:33:35.881 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_api_version": "2", 2024-02-10T07:33:35.882 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_barbican_domain": "", 2024-02-10T07:33:35.882 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_barbican_password": "", 2024-02-10T07:33:35.882 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_barbican_project": "", 2024-02-10T07:33:35.882 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_barbican_tenant": "", 2024-02-10T07:33:35.882 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_barbican_user": "", 2024-02-10T07:33:35.882 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-02-10T07:33:35.882 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_implicit_tenants": "false", 2024-02-10T07:33:35.882 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-02-10T07:33:35.882 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_service_token_enabled": "false", 2024-02-10T07:33:35.882 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_token_cache_size": "10000", 2024-02-10T07:33:35.882 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_url": "", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_keystone_verify_ssl": "true", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lc_debug_interval": "-1", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lc_lock_max_time": "90", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lc_max_objs": "32", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lc_max_rules": "1000", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lc_max_worker": "3", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lc_max_wp_worker": "3", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lc_thread_delay": "0", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ldap_dnattr": "uid", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ldap_searchfilter": "", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ldap_uri": "ldaps://", 2024-02-10T07:33:35.883 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-02-10T07:33:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-02-10T07:33:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-02-10T07:33:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_log_http_headers": "", 2024-02-10T07:33:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_log_nonexistent_bucket": "false", 2024-02-10T07:33:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-02-10T07:33:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_log_object_name_utc": "false", 2024-02-10T07:33:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-02-10T07:33:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-02-10T07:33:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_attr_name_len": "0", 2024-02-10T07:33:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_attr_size": "0", 2024-02-10T07:33:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_attrs_num_in_req": "0", 2024-02-10T07:33:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_chunk_size": "4194304", 2024-02-10T07:33:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_concurrent_requests": "1024", 2024-02-10T07:33:35.884 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-02-10T07:33:35.885 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_dynamic_shards": "1999", 2024-02-10T07:33:35.885 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_listing_results": "1000", 2024-02-10T07:33:35.885 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_notify_retries": "10", 2024-02-10T07:33:35.885 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_objs_per_shard": "100000", 2024-02-10T07:33:35.885 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_put_param_size": "1048576", 2024-02-10T07:33:35.885 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_put_size": "5368709120", 2024-02-10T07:33:35.885 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_max_slo_entries": "1000", 2024-02-10T07:33:35.885 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_md_log_max_shards": "64", 2024-02-10T07:33:35.885 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_md_notify_interval_msec": "200", 2024-02-10T07:33:35.885 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_meta_sync_poll_interval": "20", 2024-02-10T07:33:35.885 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_meta_sync_spawn_window": "20", 2024-02-10T07:33:35.885 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-02-10T07:33:35.885 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_mp_lock_max_time": "600", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_multipart_min_part_size": "5242880", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_fhcache_size": "2017", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_lru_lanes": "5", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_max_gc": "300", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_run_gc_threads": "false", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_run_lc_threads": "false", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_run_quota_threads": "false", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_run_sync_thread": "false", 2024-02-10T07:33:35.886 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-02-10T07:33:35.887 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-02-10T07:33:35.887 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_num_async_rados_threads": "32", 2024-02-10T07:33:35.887 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_num_control_oids": "8", 2024-02-10T07:33:35.887 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_numa_node": "-1", 2024-02-10T07:33:35.887 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_obj_stripe_size": "4194304", 2024-02-10T07:33:35.887 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-02-10T07:33:35.887 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_objexp_chunk_size": "100", 2024-02-10T07:33:35.887 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_objexp_gc_interval": "600", 2024-02-10T07:33:35.887 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_objexp_hints_num_shards": "127", 2024-02-10T07:33:35.887 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-02-10T07:33:35.887 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-02-10T07:33:35.887 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_op_thread_timeout": "600", 2024-02-10T07:33:35.887 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_op_tracing": "false", 2024-02-10T07:33:35.888 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_opa_token": "", 2024-02-10T07:33:35.888 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_opa_url": "", 2024-02-10T07:33:35.888 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_opa_verify_ssl": "true", 2024-02-10T07:33:35.888 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-02-10T07:33:35.888 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-02-10T07:33:35.888 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ops_log_rados": "false", 2024-02-10T07:33:35.888 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_ops_log_socket_path": "", 2024-02-10T07:33:35.888 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-02-10T07:33:35.888 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-02-10T07:33:35.888 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-02-10T07:33:35.888 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_period_push_interval": "2.000000", 2024-02-10T07:33:35.888 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_period_push_interval_max": "30.000000", 2024-02-10T07:33:35.888 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_period_root_pool": ".rgw.root", 2024-02-10T07:33:35.888 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-02-10T07:33:35.889 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-02-10T07:33:35.889 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_posix_cache_lanes": "3", 2024-02-10T07:33:35.889 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-02-10T07:33:35.889 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_posix_cache_max_buckets": "100", 2024-02-10T07:33:35.889 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_posix_cache_partitions": "3", 2024-02-10T07:33:35.889 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-02-10T07:33:35.889 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_print_continue": "true", 2024-02-10T07:33:35.889 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_print_prohibited_content_length": "false", 2024-02-10T07:33:35.889 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-02-10T07:33:35.889 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-02-10T07:33:35.889 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-02-10T07:33:35.889 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-02-10T07:33:35.889 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_rados_tracing": "false", 2024-02-10T07:33:35.889 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_realm": "", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_realm_id": "", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_region": "", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_region_root_pool": ".rgw.root", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_relaxed_region_enforcement": "false", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_request_uri": "", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_reshard_batch_size": "64", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_reshard_max_aio": "128", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_reshard_num_logs": "16", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_reshard_thread_interval": "600", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_resolve_cname": "false", 2024-02-10T07:33:35.890 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_rest_getusage_op_compat": "false", 2024-02-10T07:33:35.891 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_run_sync_thread": "true", 2024-02-10T07:33:35.891 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-02-10T07:33:35.891 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_auth_use_keystone": "false", 2024-02-10T07:33:35.891 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_auth_use_ldap": "false", 2024-02-10T07:33:35.891 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_auth_use_rados": "true", 2024-02-10T07:33:35.891 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_auth_use_sts": "false", 2024-02-10T07:33:35.891 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-02-10T07:33:35.891 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_s3_success_create_obj_status": "0", 2024-02-10T07:33:35.891 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-02-10T07:33:35.891 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_scheduler_type": "throttler", 2024-02-10T07:33:35.891 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_script_uri": "", 2024-02-10T07:33:35.891 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_service_provider_name": "", 2024-02-10T07:33:35.891 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-02-10T07:33:35.891 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sts_client_id": "", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sts_client_secret": "", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sts_entry": "sts", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sts_key": "sts", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sts_max_session_duration": "43200", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sts_min_session_duration": "900", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sts_token_introspection_url": "", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_account_in_url": "false", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_auth_entry": "auth", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_auth_url": "", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_custom_header": "", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_enforce_content_length": "false", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_need_stats": "true", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_tenant_name": "", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_token_expiration": "86400", 2024-02-10T07:33:35.892 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_url": "", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_url_prefix": "swift", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_swift_versioning_enabled": "false", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_lease_period": "120", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_log_trim_interval": "1200", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_obj_etag_verify": "false", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_trace_history_size": "4096", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-02-10T07:33:35.893 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_thread_pool_size": "512", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_topic_persistency_max_retries": "0", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_topic_require_publish_policy": "false", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_comment": "", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_createby": "", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_encoding": "", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_flag": "false", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_max_size": "5368709120", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_origin": "", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_sha_unit": "524288", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_torrent_tracker": "", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_trust_forwarded_https": "false", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_usage_log_tick_interval": "30", 2024-02-10T07:33:35.894 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_usage_max_shards": "32", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_usage_max_user_shards": "1", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_use_opa_authz": "false", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_counters_cache": "false", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_counters_cache_size": "10000", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_default_quota_max_size": "-1", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_max_buckets": "1000", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_policies_max_num": "100", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_quota_sync_interval": "86400", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_user_unique_email": "true", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_verify_ssl": "true", 2024-02-10T07:33:35.895 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_website_routing_rules_max_num": "50", 2024-02-10T07:33:35.896 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_zone": "", 2024-02-10T07:33:35.896 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_zone_id": "", 2024-02-10T07:33:35.896 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-02-10T07:33:35.896 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_zonegroup": "", 2024-02-10T07:33:35.896 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_zonegroup_id": "", 2024-02-10T07:33:35.896 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-02-10T07:33:35.896 INFO:teuthology.orchestra.run.smithi008.stdout: "rgwlc_auto_session_clear": "true", 2024-02-10T07:33:35.896 INFO:teuthology.orchestra.run.smithi008.stdout: "rgwlc_skip_bucket_step": "false", 2024-02-10T07:33:35.896 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_block_size": "4096", 2024-02-10T07:33:35.896 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-02-10T07:33:35.896 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-02-10T07:33:35.896 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-02-10T07:33:35.896 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-02-10T07:33:35.896 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cache_shard_bits": "4", 2024-02-10T07:33:35.897 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cache_size": "536870912", 2024-02-10T07:33:35.897 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cache_type": "binned_lru", 2024-02-10T07:33:35.897 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-02-10T07:33:35.897 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-02-10T07:33:35.897 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-02-10T07:33:35.897 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_collect_compaction_stats": "false", 2024-02-10T07:33:35.897 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_collect_extended_stats": "false", 2024-02-10T07:33:35.897 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_collect_memory_stats": "false", 2024-02-10T07:33:35.897 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-02-10T07:33:35.897 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_index_type": "binary_search", 2024-02-10T07:33:35.897 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_log_to_ceph_log": "true", 2024-02-10T07:33:35.897 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_metadata_block_size": "4096", 2024-02-10T07:33:35.897 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_partition_filters": "false", 2024-02-10T07:33:35.897 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_perf": "false", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "rotating_keys_renewal_timeout": "10", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "run_dir": "/var/run/ceph", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_block_create": "true", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_cache_lru_size": "67108864", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_cbjournal_size": "5368709120", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_data_delta_based_overwrite": "0", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_default_max_object_size": "16777216", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_device_size": "53687091200", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_journal_batch_capacity": "16", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-02-10T07:33:35.898 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_journal_iodepth_limit": "5", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_main_device_type": "SSD", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_max_concurrent_transactions": "8", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "seastore_segment_size": "67108864", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "set_keepcaps": "false", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "setgroup": "ceph", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "setuser": "ceph", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "setuser_match_path": "", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_misplaced_ratio": "0.050000", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "thp": "false", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "threadpool_default_timeout": "60", 2024-02-10T07:33:35.899 INFO:teuthology.orchestra.run.smithi008.stdout: "threadpool_empty_queue_max_wait": "2", 2024-02-10T07:33:35.900 INFO:teuthology.orchestra.run.smithi008.stdout: "throttler_perf_counter": "true" 2024-02-10T07:33:35.900 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:33:36.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:36 smithi008 ceph-mon[29816]: 6.b scrub starts 2024-02-10T07:33:36.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:36 smithi008 ceph-mon[29816]: 6.b scrub ok 2024-02-10T07:33:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:36 smithi008 ceph-mon[29816]: 4.18 scrub starts 2024-02-10T07:33:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:36 smithi008 ceph-mon[29816]: 4.18 scrub ok 2024-02-10T07:33:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:36 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:36 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:36 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:36.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:36 smithi008 ceph-mon[40743]: 6.b scrub starts 2024-02-10T07:33:36.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:36 smithi008 ceph-mon[40743]: 6.b scrub ok 2024-02-10T07:33:36.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:36 smithi008 ceph-mon[40743]: 4.18 scrub starts 2024-02-10T07:33:36.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:36 smithi008 ceph-mon[40743]: 4.18 scrub ok 2024-02-10T07:33:36.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:36 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:36.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:36 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:36.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:36 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:36.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:36 smithi170 ceph-mon[33394]: 6.b scrub starts 2024-02-10T07:33:36.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:36 smithi170 ceph-mon[33394]: 6.b scrub ok 2024-02-10T07:33:36.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:36 smithi170 ceph-mon[33394]: 4.18 scrub starts 2024-02-10T07:33:36.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:36 smithi170 ceph-mon[33394]: 4.18 scrub ok 2024-02-10T07:33:36.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:36 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:36.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:36 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:36.587 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:36 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:36.883 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:36 smithi170 bash[64870]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-10T07:33:37.133 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-02-10T07:33:37.423 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:37.545 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:37 smithi008 ceph-mon[29816]: Deploying daemon node-exporter.b on smithi170 2024-02-10T07:33:37.545 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:37 smithi008 ceph-mon[29816]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:33:37.546 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:37 smithi008 ceph-mon[29816]: 5.7 scrub starts 2024-02-10T07:33:37.546 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:37 smithi008 ceph-mon[29816]: 5.7 scrub ok 2024-02-10T07:33:37.546 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:37 smithi008 ceph-mon[29816]: 4.3 scrub starts 2024-02-10T07:33:37.546 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:37 smithi008 ceph-mon[29816]: 4.3 scrub ok 2024-02-10T07:33:37.546 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:37 smithi008 ceph-mon[40743]: Deploying daemon node-exporter.b on smithi170 2024-02-10T07:33:37.546 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:37 smithi008 ceph-mon[40743]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:33:37.546 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:37 smithi008 ceph-mon[40743]: 5.7 scrub starts 2024-02-10T07:33:37.546 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:37 smithi008 ceph-mon[40743]: 5.7 scrub ok 2024-02-10T07:33:37.546 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:37 smithi008 ceph-mon[40743]: 4.3 scrub starts 2024-02-10T07:33:37.546 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:37 smithi008 ceph-mon[40743]: 4.3 scrub ok 2024-02-10T07:33:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:37 smithi170 ceph-mon[33394]: Deploying daemon node-exporter.b on smithi170 2024-02-10T07:33:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:37 smithi170 ceph-mon[33394]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:33:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:37 smithi170 ceph-mon[33394]: 5.7 scrub starts 2024-02-10T07:33:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:37 smithi170 ceph-mon[33394]: 5.7 scrub ok 2024-02-10T07:33:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:37 smithi170 ceph-mon[33394]: 4.3 scrub starts 2024-02-10T07:33:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:37 smithi170 ceph-mon[33394]: 4.3 scrub ok 2024-02-10T07:33:37.633 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:37 smithi170 bash[64870]: Getting image source signatures 2024-02-10T07:33:37.633 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:37 smithi170 bash[64870]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-10T07:33:37.633 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:37 smithi170 bash[64870]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-10T07:33:37.633 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:37 smithi170 bash[64870]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-10T07:33:38.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:38 smithi008 ceph-mon[29816]: 4.1 scrub starts 2024-02-10T07:33:38.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:38 smithi008 ceph-mon[29816]: 4.1 scrub ok 2024-02-10T07:33:38.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:38 smithi008 ceph-mon[29816]: 4.6 scrub starts 2024-02-10T07:33:38.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:38 smithi008 ceph-mon[29816]: 4.6 scrub ok 2024-02-10T07:33:38.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:38 smithi008 ceph-mon[40743]: 4.1 scrub starts 2024-02-10T07:33:38.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:38 smithi008 ceph-mon[40743]: 4.1 scrub ok 2024-02-10T07:33:38.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:38 smithi008 ceph-mon[40743]: 4.6 scrub starts 2024-02-10T07:33:38.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:38 smithi008 ceph-mon[40743]: 4.6 scrub ok 2024-02-10T07:33:38.604 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:38 smithi170 ceph-mon[33394]: 4.1 scrub starts 2024-02-10T07:33:38.605 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:38 smithi170 ceph-mon[33394]: 4.1 scrub ok 2024-02-10T07:33:38.605 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:38 smithi170 ceph-mon[33394]: 4.6 scrub starts 2024-02-10T07:33:38.605 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:38 smithi170 ceph-mon[33394]: 4.6 scrub ok 2024-02-10T07:33:38.882 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:38 smithi170 bash[64870]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-10T07:33:38.883 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:38 smithi170 bash[64870]: Writing manifest to image destination 2024-02-10T07:33:38.900 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:33:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:33:39.121 INFO:teuthology.orchestra.run.smithi008.stdout:mon.a: {} 2024-02-10T07:33:39.132 INFO:teuthology.orchestra.run.smithi008.stdout:mon.b: {} 2024-02-10T07:33:39.141 INFO:teuthology.orchestra.run.smithi008.stdout:mon.c: {} 2024-02-10T07:33:39.143 INFO:teuthology.orchestra.run.smithi008.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-10T07:33:39.143 INFO:teuthology.orchestra.run.smithi008.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-10T07:33:39.143 INFO:teuthology.orchestra.run.smithi008.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-10T07:33:39.369 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-mon[33394]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:33:39.369 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/25939747' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:33:39.370 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 podman[64870]: 2024-02-10 07:33:39.020253546 +0000 UTC m=+2.331020560 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-10T07:33:39.370 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 podman[64870]: 2024-02-10 07:33:39.118967392 +0000 UTC m=+2.429734401 container create dc68e79a865eb180681ca6b7e9e5654042761d7294ed5f320d6de9f7c834d1c9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:33:39.370 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 podman[64870]: 2024-02-10 07:33:39.362872466 +0000 UTC m=+2.673639475 container init dc68e79a865eb180681ca6b7e9e5654042761d7294ed5f320d6de9f7c834d1c9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:33:39.370 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 podman[64870]: 2024-02-10 07:33:39.366069017 +0000 UTC m=+2.676836028 container start dc68e79a865eb180681ca6b7e9e5654042761d7294ed5f320d6de9f7c834d1c9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:33:39.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:39 smithi008 ceph-mon[29816]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:33:39.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:39 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/25939747' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:33:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:39 smithi008 ceph-mon[40743]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:33:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:39 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/25939747' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:33:39.633 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.369Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-10T07:33:39.633 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.369Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-10T07:33:39.633 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.370Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-10T07:33:39.633 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.370Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T07:33:39.633 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.370Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-10T07:33:39.633 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.370Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-10T07:33:39.633 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-10T07:33:39.634 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=arp 2024-02-10T07:33:39.634 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-10T07:33:39.634 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-10T07:33:39.634 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-10T07:33:39.634 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-10T07:33:39.634 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-10T07:33:39.634 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-10T07:33:39.634 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-10T07:33:39.634 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-10T07:33:39.634 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=edac 2024-02-10T07:33:39.634 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-10T07:33:39.635 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-10T07:33:39.635 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-10T07:33:39.635 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-10T07:33:39.635 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-10T07:33:39.635 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-10T07:33:39.635 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-10T07:33:39.635 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-10T07:33:39.635 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-10T07:33:39.635 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-10T07:33:39.635 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-10T07:33:39.635 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-10T07:33:39.636 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-10T07:33:39.636 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-10T07:33:39.636 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-10T07:33:39.636 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-10T07:33:39.636 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=os 2024-02-10T07:33:39.636 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-10T07:33:39.636 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-10T07:33:39.636 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-10T07:33:39.636 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-10T07:33:39.636 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-10T07:33:39.636 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-10T07:33:39.636 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-10T07:33:39.636 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=stat 2024-02-10T07:33:39.636 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-10T07:33:39.637 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-10T07:33:39.637 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-10T07:33:39.637 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=time 2024-02-10T07:33:39.637 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-10T07:33:39.637 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=uname 2024-02-10T07:33:39.637 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-10T07:33:39.637 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-10T07:33:39.637 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-10T07:33:39.637 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-10T07:33:39.637 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b[64922]: ts=2024-02-10T07:33:39.371Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-10T07:33:39.637 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 bash[64870]: dc68e79a865eb180681ca6b7e9e5654042761d7294ed5f320d6de9f7c834d1c9 2024-02-10T07:33:39.637 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:33:39 smithi170 systemd[1]: Started Ceph node-exporter.b for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:33:39.903 INFO:teuthology.run_tasks:Running task full_sequential... 2024-02-10T07:33:39.913 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-10T07:33:39.915 INFO:tasks.radosbench:Beginning radosbench... 2024-02-10T07:33:39.915 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-02-10T07:33:39.915 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-02-10T07:33:39.916 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-02-10T07:33:39.916 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T07:33:39.916 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:33:39.916 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-02-10T07:33:39.916 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight-by-utilization 110 3.0 2024-02-10T07:33:39.917 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-02-10T07:33:39.917 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:39.917 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-02-10T07:33:39.918 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:33:39.918 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-02-10T07:33:39.918 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-10T07:33:39.918 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-02-10T07:33:39.919 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:33:39.944 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:39.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:39.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:40.046 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:40.070 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:40.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:40.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:40.197 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:40.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:40.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:40.206 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:40.210 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:40.299 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:40.325 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:40.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:40.426 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:33:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:40.450 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:40.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:33:40.477 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:40 smithi170 ceph-mon[33394]: 3.8 scrub starts 2024-02-10T07:33:40.477 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:40 smithi170 ceph-mon[33394]: 3.8 scrub ok 2024-02-10T07:33:40.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:40 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:40 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:40 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:40 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:40 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:40 smithi170 ceph-mon[33394]: 6.18 scrub starts 2024-02-10T07:33:40.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:40 smithi170 ceph-mon[33394]: 6.18 scrub ok 2024-02-10T07:33:40.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:40 smithi170 ceph-mon[33394]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:40.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:40 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:33:40.487 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[29816]: 3.8 scrub starts 2024-02-10T07:33:40.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[29816]: 3.8 scrub ok 2024-02-10T07:33:40.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[29816]: 6.18 scrub starts 2024-02-10T07:33:40.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[29816]: 6.18 scrub ok 2024-02-10T07:33:40.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[29816]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:40.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:33:40.488 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[40743]: 3.8 scrub starts 2024-02-10T07:33:40.489 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[40743]: 3.8 scrub ok 2024-02-10T07:33:40.489 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.489 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.489 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.489 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.489 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:40.489 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[40743]: 6.18 scrub starts 2024-02-10T07:33:40.489 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[40743]: 6.18 scrub ok 2024-02-10T07:33:40.489 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[40743]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:40.489 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:40 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:33:40.552 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:40.576 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:40.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:40.678 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:40.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:40.704 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:40.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:40.805 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:40.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:40.830 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:40.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:40.931 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:40.956 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:40.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:41.057 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:41.082 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:41.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:41.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:41.184 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:41.208 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:41.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:41.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:41.310 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:33:41.340 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:41.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:33:41.441 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:41.465 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:41.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:41.566 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:41.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:41 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:33:41.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:41 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:33:41.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:41 smithi008 ceph-mon[29816]: 6.1a scrub starts 2024-02-10T07:33:41.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:41 smithi008 ceph-mon[29816]: 6.1a scrub ok 2024-02-10T07:33:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:41 smithi008 ceph-mon[29816]: osdmap e94: 8 total, 8 up, 8 in 2024-02-10T07:33:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:41 smithi008 ceph-mon[29816]: 3.1 scrub starts 2024-02-10T07:33:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:41 smithi008 ceph-mon[29816]: 3.1 scrub ok 2024-02-10T07:33:41.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:41 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:33:41.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:41 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:33:41.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:41 smithi008 ceph-mon[40743]: 6.1a scrub starts 2024-02-10T07:33:41.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:41 smithi008 ceph-mon[40743]: 6.1a scrub ok 2024-02-10T07:33:41.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:41 smithi008 ceph-mon[40743]: osdmap e94: 8 total, 8 up, 8 in 2024-02-10T07:33:41.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:41 smithi008 ceph-mon[40743]: 3.1 scrub starts 2024-02-10T07:33:41.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:41 smithi008 ceph-mon[40743]: 3.1 scrub ok 2024-02-10T07:33:41.583 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:41 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:33:41.583 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:41 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:33:41.583 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:41 smithi170 ceph-mon[33394]: 6.1a scrub starts 2024-02-10T07:33:41.583 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:41 smithi170 ceph-mon[33394]: 6.1a scrub ok 2024-02-10T07:33:41.583 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:41 smithi170 ceph-mon[33394]: osdmap e94: 8 total, 8 up, 8 in 2024-02-10T07:33:41.583 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:41 smithi170 ceph-mon[33394]: 3.1 scrub starts 2024-02-10T07:33:41.583 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:41 smithi170 ceph-mon[33394]: 3.1 scrub ok 2024-02-10T07:33:41.592 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:41.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:41.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:41.694 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:33:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:41.722 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:41.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:33:41.823 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:41.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:41.848 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:41.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:41.949 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:41.974 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:41.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:42.075 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:42.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:42.100 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:42.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:42.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:42.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:42.225 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:42.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:42.326 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:42.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:42.351 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:42.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:42.414 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:42.415 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:42.445 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:42.451 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:42.477 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:42.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:42.542 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:42 smithi008 ceph-mon[29816]: 6.7 deep-scrub starts 2024-02-10T07:33:42.542 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:42 smithi008 ceph-mon[29816]: 6.7 deep-scrub ok 2024-02-10T07:33:42.542 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:42 smithi008 ceph-mon[29816]: 4.c scrub starts 2024-02-10T07:33:42.542 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:42 smithi008 ceph-mon[29816]: 4.c scrub ok 2024-02-10T07:33:42.542 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:42 smithi008 ceph-mon[29816]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:42.543 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:42 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:42.543 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:42 smithi008 ceph-mon[40743]: 6.7 deep-scrub starts 2024-02-10T07:33:42.543 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:42 smithi008 ceph-mon[40743]: 6.7 deep-scrub ok 2024-02-10T07:33:42.543 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:42 smithi008 ceph-mon[40743]: 4.c scrub starts 2024-02-10T07:33:42.543 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:42 smithi008 ceph-mon[40743]: 4.c scrub ok 2024-02-10T07:33:42.543 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:42 smithi008 ceph-mon[40743]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:42.543 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:42 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:42.559 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:42 smithi170 ceph-mon[33394]: 6.7 deep-scrub starts 2024-02-10T07:33:42.560 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:42 smithi170 ceph-mon[33394]: 6.7 deep-scrub ok 2024-02-10T07:33:42.560 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:42 smithi170 ceph-mon[33394]: 4.c scrub starts 2024-02-10T07:33:42.560 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:42 smithi170 ceph-mon[33394]: 4.c scrub ok 2024-02-10T07:33:42.560 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:42 smithi170 ceph-mon[33394]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:42.560 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:42 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:42.579 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:42.603 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:42.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:42.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:42.778 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:42.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:42.803 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:42.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:42.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:33:42 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:33:42.904 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:42.927 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:42.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:42.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:43.029 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:43.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:43.053 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:43.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:43.154 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:43.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:43.179 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:43.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:43.280 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:43.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:43.307 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:43.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:43.408 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:43.433 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:43.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:43.434 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:43 smithi008 ceph-mon[40743]: 3.5 scrub starts 2024-02-10T07:33:43.434 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:43 smithi008 ceph-mon[40743]: 3.5 scrub ok 2024-02-10T07:33:43.434 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:43 smithi008 ceph-mon[40743]: 3.f scrub starts 2024-02-10T07:33:43.434 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:43 smithi008 ceph-mon[40743]: 3.f scrub ok 2024-02-10T07:33:43.434 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:43 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:43.434 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:43 smithi008 ceph-mon[29816]: 3.5 scrub starts 2024-02-10T07:33:43.434 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:43 smithi008 ceph-mon[29816]: 3.5 scrub ok 2024-02-10T07:33:43.434 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:43 smithi008 ceph-mon[29816]: 3.f scrub starts 2024-02-10T07:33:43.434 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:43 smithi008 ceph-mon[29816]: 3.f scrub ok 2024-02-10T07:33:43.434 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:43 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:43.534 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:43.558 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:43.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:43 smithi170 ceph-mon[33394]: 3.5 scrub starts 2024-02-10T07:33:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:43 smithi170 ceph-mon[33394]: 3.5 scrub ok 2024-02-10T07:33:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:43 smithi170 ceph-mon[33394]: 3.f scrub starts 2024-02-10T07:33:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:43 smithi170 ceph-mon[33394]: 3.f scrub ok 2024-02-10T07:33:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:43 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:43.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:43.685 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:43.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:43.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:43.787 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:43.811 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:43.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:43.912 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:43.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:43.936 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:43.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:44.037 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:44.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:44.063 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:44.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:44.165 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:44.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:44.191 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:44.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:44.293 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:44.318 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:44.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:44.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:44 smithi008 ceph-mon[29816]: 5.1b scrub starts 2024-02-10T07:33:44.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:44 smithi008 ceph-mon[29816]: 5.1b scrub ok 2024-02-10T07:33:44.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:44 smithi008 ceph-mon[29816]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:44.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:44 smithi008 ceph-mon[40743]: 5.1b scrub starts 2024-02-10T07:33:44.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:44 smithi008 ceph-mon[40743]: 5.1b scrub ok 2024-02-10T07:33:44.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:44 smithi008 ceph-mon[40743]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:44.419 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:44.444 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:44.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:44.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:44.546 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:44.570 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:44.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:44.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:44.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:44 smithi170 ceph-mon[33394]: 5.1b scrub starts 2024-02-10T07:33:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:44 smithi170 ceph-mon[33394]: 5.1b scrub ok 2024-02-10T07:33:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:44 smithi170 ceph-mon[33394]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:33:44.672 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:44.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:44.696 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:44.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:44.797 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:44.820 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:44.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:44.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:44.922 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:44.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:44.948 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:44.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:45.049 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:45.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:45.075 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:45.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:45.096 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:33:45.115 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:33:45.135 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:33:45.154 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:33:45.174 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:33:45.176 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:45.196 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:33:45.199 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:45.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:45.221 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:33:45.243 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:33:45.244 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:33:45.244 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:33:45.244 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:33:45.244 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:33:45.244 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:33:45.244 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:33:45.244 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:33:45.244 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:33:45.301 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:45.325 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:45.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:45.330 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:45 smithi008 ceph-mon[29816]: 6.12 deep-scrub starts 2024-02-10T07:33:45.330 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:45 smithi008 ceph-mon[29816]: 6.12 deep-scrub ok 2024-02-10T07:33:45.330 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:45 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:45.330 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:45 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:45.330 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:45 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/263875890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:33:45.330 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:45 smithi008 ceph-mon[29816]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:45.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:45 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'57454'}"}]: dispatch 2024-02-10T07:33:45.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:45 smithi008 ceph-mon[40743]: 6.12 deep-scrub starts 2024-02-10T07:33:45.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:45 smithi008 ceph-mon[40743]: 6.12 deep-scrub ok 2024-02-10T07:33:45.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:45 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:45.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:45 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:45.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:45 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/263875890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:33:45.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:45 smithi008 ceph-mon[40743]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:45.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:45 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'57454'}"}]: dispatch 2024-02-10T07:33:45.427 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:45.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:45.452 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:45.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:45.488 INFO:teuthology.orchestra.run.smithi008.stdout:moved 6 / 396 (1.51515%) 2024-02-10T07:33:45.488 INFO:teuthology.orchestra.run.smithi008.stdout:avg 49.5 2024-02-10T07:33:45.488 INFO:teuthology.orchestra.run.smithi008.stdout:stddev 6.30476 -> 6.10328 (expected baseline 6.58122) 2024-02-10T07:33:45.488 INFO:teuthology.orchestra.run.smithi008.stdout:min osd.2 with 39 -> 39 pgs (0.787879 -> 0.787879 * mean) 2024-02-10T07:33:45.488 INFO:teuthology.orchestra.run.smithi008.stdout:max osd.4 with 58 -> 58 pgs (1.17172 -> 1.17172 * mean) 2024-02-10T07:33:45.489 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:33:45.489 INFO:teuthology.orchestra.run.smithi008.stdout:oload 110 2024-02-10T07:33:45.489 INFO:teuthology.orchestra.run.smithi008.stdout:max_change 3 2024-02-10T07:33:45.489 INFO:teuthology.orchestra.run.smithi008.stdout:max_change_osds 4 2024-02-10T07:33:45.489 INFO:teuthology.orchestra.run.smithi008.stdout:average_utilization 0.0003 2024-02-10T07:33:45.489 INFO:teuthology.orchestra.run.smithi008.stdout:overload_utilization 0.0004 2024-02-10T07:33:45.489 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 weight 1.0000 -> 0.8767 2024-02-10T07:33:45.553 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:45.577 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:45.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:45.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:45 smithi170 ceph-mon[33394]: 6.12 deep-scrub starts 2024-02-10T07:33:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:45 smithi170 ceph-mon[33394]: 6.12 deep-scrub ok 2024-02-10T07:33:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:45 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:45 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:33:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:45 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/263875890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:33:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:45 smithi170 ceph-mon[33394]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:45 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'57454'}"}]: dispatch 2024-02-10T07:33:45.679 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:45.703 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:45.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:45.805 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:45.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:45.830 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:45.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:45.931 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:45.955 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:45.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:46.056 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:46.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:46.081 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:46.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:46.181 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:46.204 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:46.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:46.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:46.307 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:46.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:46.332 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:46.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:46.433 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:46.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:46.458 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:46.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:46.559 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:46.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:46 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'57454'}"}]': finished 2024-02-10T07:33:46.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:46 smithi008 ceph-mon[29816]: osdmap e95: 8 total, 8 up, 8 in 2024-02-10T07:33:46.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:46 smithi008 ceph-mon[29816]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:33:46.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:46 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'57454'}"}]': finished 2024-02-10T07:33:46.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:46 smithi008 ceph-mon[40743]: osdmap e95: 8 total, 8 up, 8 in 2024-02-10T07:33:46.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:46 smithi008 ceph-mon[40743]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:33:46.582 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:46.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:46.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:46.684 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:46.686 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-02-10T07:33:46.686 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight-by-utilization 1000 0.05 2024-02-10T07:33:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:46.750 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:46.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:46.851 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:46.876 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:46.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:46.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:46.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:46 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'57454'}"}]': finished 2024-02-10T07:33:46.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:46 smithi170 ceph-mon[33394]: osdmap e95: 8 total, 8 up, 8 in 2024-02-10T07:33:46.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:46 smithi170 ceph-mon[33394]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:33:46.978 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:33:47.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:47.002 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:47.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:33:47.103 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:47.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:47.127 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:47.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:47.228 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:47.256 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:47.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:47.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:47.358 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:47.384 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:47.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:47.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:47.486 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:47.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:47.512 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:47.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:47.522 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:47 smithi008 ceph-mon[40743]: 6.d scrub starts 2024-02-10T07:33:47.522 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:47 smithi008 ceph-mon[40743]: 6.d scrub ok 2024-02-10T07:33:47.522 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:47 smithi008 ceph-mon[40743]: 4.10 deep-scrub starts 2024-02-10T07:33:47.522 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:47 smithi008 ceph-mon[40743]: 4.10 deep-scrub ok 2024-02-10T07:33:47.522 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:47 smithi008 ceph-mon[40743]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T07:33:47.522 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:47 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:47.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:47 smithi008 ceph-mon[29816]: 6.d scrub starts 2024-02-10T07:33:47.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:47 smithi008 ceph-mon[29816]: 6.d scrub ok 2024-02-10T07:33:47.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:47 smithi008 ceph-mon[29816]: 4.10 deep-scrub starts 2024-02-10T07:33:47.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:47 smithi008 ceph-mon[29816]: 4.10 deep-scrub ok 2024-02-10T07:33:47.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:47 smithi008 ceph-mon[29816]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T07:33:47.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:47 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:47.613 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:47.640 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:47.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:47.741 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:33:47.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:47.768 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:47.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:33:47.869 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:47 smithi170 ceph-mon[33394]: 6.d scrub starts 2024-02-10T07:33:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:47 smithi170 ceph-mon[33394]: 6.d scrub ok 2024-02-10T07:33:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:47 smithi170 ceph-mon[33394]: 4.10 deep-scrub starts 2024-02-10T07:33:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:47 smithi170 ceph-mon[33394]: 4.10 deep-scrub ok 2024-02-10T07:33:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:47 smithi170 ceph-mon[33394]: osdmap e96: 8 total, 8 up, 8 in 2024-02-10T07:33:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:47 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:33:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:47.896 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:47.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:47.997 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:48.023 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:48.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:48.124 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:48.154 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:48.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:48.169 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:33:48.255 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:48.279 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:48.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:48.380 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:48.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:48.405 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:48.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:48.506 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:33:48.529 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:48.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:33:48.631 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:33:48.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:48.656 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:48.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:33:48.757 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:48.770 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:48 smithi008 ceph-mon[29816]: 6.19 scrub starts 2024-02-10T07:33:48.770 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:48 smithi008 ceph-mon[29816]: 6.19 scrub ok 2024-02-10T07:33:48.770 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:48 smithi008 ceph-mon[29816]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 30 B/s, 2 objects/s recovering 2024-02-10T07:33:48.770 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:48 smithi008 ceph-mon[29816]: 3.1b deep-scrub starts 2024-02-10T07:33:48.770 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:48 smithi008 ceph-mon[29816]: 3.1b deep-scrub ok 2024-02-10T07:33:48.770 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:48 smithi008 ceph-mon[40743]: 6.19 scrub starts 2024-02-10T07:33:48.771 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:48 smithi008 ceph-mon[40743]: 6.19 scrub ok 2024-02-10T07:33:48.771 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:48 smithi008 ceph-mon[40743]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 30 B/s, 2 objects/s recovering 2024-02-10T07:33:48.771 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:48 smithi008 ceph-mon[40743]: 3.1b deep-scrub starts 2024-02-10T07:33:48.771 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:48 smithi008 ceph-mon[40743]: 3.1b deep-scrub ok 2024-02-10T07:33:48.785 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:48.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:48.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:48.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:48 smithi170 ceph-mon[33394]: 6.19 scrub starts 2024-02-10T07:33:48.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:48 smithi170 ceph-mon[33394]: 6.19 scrub ok 2024-02-10T07:33:48.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:48 smithi170 ceph-mon[33394]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 30 B/s, 2 objects/s recovering 2024-02-10T07:33:48.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:48 smithi170 ceph-mon[33394]: 3.1b deep-scrub starts 2024-02-10T07:33:48.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:48 smithi170 ceph-mon[33394]: 3.1b deep-scrub ok 2024-02-10T07:33:48.886 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:48.915 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:48.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:48.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:49.018 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:49.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:49.047 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:49.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:49.076 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:33:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:33:49.148 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:49.174 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:49.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:49.276 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:49.301 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:49.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:49.402 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:33:49.431 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:49.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:49.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:33:49.460 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:33:49.460 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:33:49.509 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:33:49.533 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:49.558 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:49.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:49.659 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:49.687 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:49.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:49.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:49.698 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[29816]: 6.a scrub starts 2024-02-10T07:33:49.698 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[29816]: 6.a scrub ok 2024-02-10T07:33:49.698 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[29816]: 4.1a scrub starts 2024-02-10T07:33:49.698 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[29816]: 4.1a scrub ok 2024-02-10T07:33:49.699 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[29816]: 4.14 deep-scrub starts 2024-02-10T07:33:49.699 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[29816]: 4.14 deep-scrub ok 2024-02-10T07:33:49.699 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[29816]: 5.1e scrub starts 2024-02-10T07:33:49.699 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[29816]: 5.1e scrub ok 2024-02-10T07:33:49.699 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[29816]: 6.3 scrub starts 2024-02-10T07:33:49.699 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[29816]: 6.3 scrub ok 2024-02-10T07:33:49.699 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4000978990' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:33:49.699 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[40743]: 6.a scrub starts 2024-02-10T07:33:49.699 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[40743]: 6.a scrub ok 2024-02-10T07:33:49.699 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[40743]: 4.1a scrub starts 2024-02-10T07:33:49.699 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[40743]: 4.1a scrub ok 2024-02-10T07:33:49.699 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[40743]: 4.14 deep-scrub starts 2024-02-10T07:33:49.699 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[40743]: 4.14 deep-scrub ok 2024-02-10T07:33:49.700 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[40743]: 5.1e scrub starts 2024-02-10T07:33:49.700 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[40743]: 5.1e scrub ok 2024-02-10T07:33:49.700 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[40743]: 6.3 scrub starts 2024-02-10T07:33:49.700 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[40743]: 6.3 scrub ok 2024-02-10T07:33:49.700 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:49 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4000978990' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:33:49.789 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:49.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:49.818 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:49.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:49 smithi170 ceph-mon[33394]: 6.a scrub starts 2024-02-10T07:33:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:49 smithi170 ceph-mon[33394]: 6.a scrub ok 2024-02-10T07:33:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:49 smithi170 ceph-mon[33394]: 4.1a scrub starts 2024-02-10T07:33:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:49 smithi170 ceph-mon[33394]: 4.1a scrub ok 2024-02-10T07:33:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:49 smithi170 ceph-mon[33394]: 4.14 deep-scrub starts 2024-02-10T07:33:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:49 smithi170 ceph-mon[33394]: 4.14 deep-scrub ok 2024-02-10T07:33:49.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:49 smithi170 ceph-mon[33394]: 5.1e scrub starts 2024-02-10T07:33:49.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:49 smithi170 ceph-mon[33394]: 5.1e scrub ok 2024-02-10T07:33:49.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:49 smithi170 ceph-mon[33394]: 6.3 scrub starts 2024-02-10T07:33:49.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:49 smithi170 ceph-mon[33394]: 6.3 scrub ok 2024-02-10T07:33:49.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:49 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4000978990' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:33:49.919 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:49.947 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:49.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:49.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:50.049 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:50.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:50.079 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:50.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:50.180 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:50.207 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:50.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:50.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:50.310 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:50.336 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:50.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:50.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:50.346 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-10T07:33:50.438 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:50.466 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:50.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:50.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:50.569 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:50.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:50 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:33:50.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:50 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4000978990' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:33:50.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:50 smithi008 ceph-mon[29816]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T07:33:50.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:50 smithi008 ceph-mon[29816]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 36 B/s, 3 objects/s recovering 2024-02-10T07:33:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:50 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:33:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:50 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4000978990' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:33:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:50 smithi008 ceph-mon[40743]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T07:33:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:50 smithi008 ceph-mon[40743]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 36 B/s, 3 objects/s recovering 2024-02-10T07:33:50.592 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:50.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:50.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:50.694 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:50.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:50.718 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:50.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:50.819 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:50.849 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:50.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:50.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:50 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:33:50.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:50 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4000978990' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:33:50.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:50 smithi170 ceph-mon[33394]: osdmap e97: 8 total, 8 up, 8 in 2024-02-10T07:33:50.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:50 smithi170 ceph-mon[33394]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 36 B/s, 3 objects/s recovering 2024-02-10T07:33:50.950 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:50.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:50.974 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:50.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:51.075 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:51.099 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:51.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:51.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:51.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:51.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:51.230 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:51.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:51.332 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:51.358 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:51.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:51.459 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:51.486 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:51.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:51.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:51.588 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:33:51.612 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:51.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:33:51.714 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:51.739 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:51.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:51.823 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:51.824 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:51.840 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:51.868 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:51.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:51.970 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:51.998 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:51.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:52.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:52.100 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:52.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:52.126 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:52.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:52.227 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:52.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:52.253 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:52.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:52.354 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:52.381 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:52.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:52.482 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:52.507 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:52.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:52.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:52.609 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:33:52.633 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:52.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:52.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:33:52.676 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'unique_pool_0' created 2024-02-10T07:33:52.735 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:52.761 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:52.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:52.863 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:52.888 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:52.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:52.990 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:53.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:53.016 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:53.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:53.033 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:52 smithi008 ceph-mon[29816]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 495 B/s rd, 0 op/s; 35 B/s, 3 objects/s recovering 2024-02-10T07:33:53.033 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:52 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2277684466' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-10T07:33:53.034 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:52 smithi008 ceph-mon[40743]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 495 B/s rd, 0 op/s; 35 B/s, 3 objects/s recovering 2024-02-10T07:33:53.034 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:52 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2277684466' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-10T07:33:53.117 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:53.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:33:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:33:53.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:52 smithi170 ceph-mon[33394]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 495 B/s rd, 0 op/s; 35 B/s, 3 objects/s recovering 2024-02-10T07:33:53.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:52 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2277684466' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-10T07:33:53.144 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:53.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:53.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:53.246 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:53.255 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:33:53.272 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:53.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:53.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:53.374 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:53.399 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:53.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:53.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:53.501 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:53.528 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:53.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:53.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:53.630 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:53.655 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:53.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:53.756 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:53.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:53.780 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:53.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:53.881 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:53.910 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:53.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:53.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:53.966 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:53 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2277684466' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-10T07:33:53.966 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:53 smithi008 ceph-mon[29816]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T07:33:53.966 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:53 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:53.966 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:53 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2277684466' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-10T07:33:53.967 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:53 smithi008 ceph-mon[40743]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T07:33:53.967 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:53 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:53.999 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-02-10T07:33:54.012 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:54.039 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:54.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:54.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:54.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:53 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2277684466' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-10T07:33:54.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:53 smithi170 ceph-mon[33394]: osdmap e98: 8 total, 8 up, 8 in 2024-02-10T07:33:54.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:53 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:33:54.141 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:54.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:54.170 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:54.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:54.271 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:54.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:54.301 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:54.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:54.305 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:54.345 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:33:54.365 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:33:54.384 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:33:54.402 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:54.404 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:33:54.424 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:33:54.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:54.430 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:54.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:54.445 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:33:54.465 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:33:54.486 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:33:54.487 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:33:54.487 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:33:54.487 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:33:54.487 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:33:54.487 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:33:54.487 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:33:54.487 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:33:54.487 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:33:54.531 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:54.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:54.557 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:54.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:54.659 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:54.686 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:54.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:54.788 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:54.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:54.815 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:54.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:54.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:54 smithi008 ceph-mon[29816]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T07:33:54.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:54 smithi008 ceph-mon[29816]: pgmap v61: 148 pgs: 12 creating+peering, 132 active+clean, 4 unknown; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:54.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:54 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:33:54.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:54 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:33:54.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:54 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3467239627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:33:54.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:54 smithi008 ceph-mon[40743]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T07:33:54.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:54 smithi008 ceph-mon[40743]: pgmap v61: 148 pgs: 12 creating+peering, 132 active+clean, 4 unknown; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:54.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:54 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:33:54.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:54 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:33:54.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:54 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3467239627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:33:54.916 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:54.945 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:54.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:55.047 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:33:55.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:55.072 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:55.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:33:55.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:54 smithi170 ceph-mon[33394]: osdmap e99: 8 total, 8 up, 8 in 2024-02-10T07:33:55.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:54 smithi170 ceph-mon[33394]: pgmap v61: 148 pgs: 12 creating+peering, 132 active+clean, 4 unknown; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:55.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:54 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:33:55.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:54 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:33:55.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:54 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3467239627' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:33:55.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:55.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:55.201 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:55.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:55.303 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:55.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:55.330 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:55.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:55.431 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:55.458 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:55.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:55.559 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:55.588 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:55.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:55.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:55.690 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:55.718 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:55.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:55.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:55.820 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:55.847 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:55.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:55.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:55.950 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:33:55.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:55.975 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:55.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:33:56.054 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:55 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:33:56.054 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:55 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:33:56.054 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:55 smithi008 ceph-mon[29816]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:33:56.054 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:55 smithi008 ceph-mon[29816]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T07:33:56.054 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:55 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:33:56.054 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:55 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:33:56.055 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:55 smithi008 ceph-mon[40743]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:33:56.055 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:55 smithi008 ceph-mon[40743]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T07:33:56.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:56.106 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:56.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:56.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:56.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:55 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:33:56.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:55 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:33:56.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:55 smithi170 ceph-mon[33394]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:33:56.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:55 smithi170 ceph-mon[33394]: osdmap e100: 8 total, 8 up, 8 in 2024-02-10T07:33:56.208 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:33:56.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:56.232 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:56.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:33:56.333 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:56.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:56.357 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:56.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:56.458 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:56.486 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:56.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:56.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:56.588 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:56.613 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:56.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:56.714 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:56.734 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:56 smithi008 ceph-mon[29816]: pgmap v63: 148 pgs: 12 creating+peering, 132 active+clean, 4 unknown; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:56.734 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:56 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:33:56.735 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:56 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:33:56.735 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:56 smithi008 ceph-mon[40743]: pgmap v63: 148 pgs: 12 creating+peering, 132 active+clean, 4 unknown; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:56.735 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:56 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:33:56.735 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:56 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:33:56.739 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:56.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:56.841 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:56.868 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:56.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:56.971 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:33:56.986 INFO:teuthology.orchestra.run.smithi008.stdout:moved 0 / 444 (0%) 2024-02-10T07:33:56.986 INFO:teuthology.orchestra.run.smithi008.stdout:avg 55.5 2024-02-10T07:33:56.986 INFO:teuthology.orchestra.run.smithi008.stdout:stddev 6.57647 -> 6.57647 (expected baseline 6.96868) 2024-02-10T07:33:56.986 INFO:teuthology.orchestra.run.smithi008.stdout:min osd.2 with 46 -> 46 pgs (0.828829 -> 0.828829 * mean) 2024-02-10T07:33:56.986 INFO:teuthology.orchestra.run.smithi008.stdout:max osd.3 with 66 -> 66 pgs (1.18919 -> 1.18919 * mean) 2024-02-10T07:33:56.986 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:33:56.986 INFO:teuthology.orchestra.run.smithi008.stdout:oload 1000 2024-02-10T07:33:56.986 INFO:teuthology.orchestra.run.smithi008.stdout:max_change 0.05 2024-02-10T07:33:56.986 INFO:teuthology.orchestra.run.smithi008.stdout:max_change_osds 4 2024-02-10T07:33:56.986 INFO:teuthology.orchestra.run.smithi008.stdout:average_utilization 0.0003 2024-02-10T07:33:56.987 INFO:teuthology.orchestra.run.smithi008.stdout:overload_utilization 0.0034 2024-02-10T07:33:56.989 INFO:teuthology.orchestra.run.smithi008.stderr:no change 2024-02-10T07:33:56.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:56.996 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:56.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:33:57.034 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:56 smithi170 ceph-mon[33394]: pgmap v63: 148 pgs: 12 creating+peering, 132 active+clean, 4 unknown; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:33:57.035 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:56 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:33:57.035 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:56 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:33:57.097 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:57.126 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:57.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:57.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:57.228 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:57.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:57.252 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:57.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:57.354 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:57.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:57.379 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:57.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:57.480 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:57.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:57.504 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:57.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:57.606 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:57.630 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:57.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:57.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:57.732 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:33:57.755 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:57.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:57.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:33:57.783 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:57 smithi008 ceph-mon[40743]: osdmap e101: 8 total, 8 up, 8 in 2024-02-10T07:33:57.783 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:57 smithi008 ceph-mon[40743]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:57.784 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:57 smithi008 ceph-mon[29816]: osdmap e101: 8 total, 8 up, 8 in 2024-02-10T07:33:57.784 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:57 smithi008 ceph-mon[29816]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:57.810 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-02-10T07:33:57.810 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight-by-utilization 1000 1.0 2024-02-10T07:33:57.857 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:57.882 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:57.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:57.983 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:33:58.008 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:58.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:58.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:33:58.087 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:33:58.111 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:58.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:57 smithi170 ceph-mon[33394]: osdmap e101: 8 total, 8 up, 8 in 2024-02-10T07:33:58.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:57 smithi170 ceph-mon[33394]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:33:58.154 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:58.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:58.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:58.256 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:58.281 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:58.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:58.382 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:58.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:58.407 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:58.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:58.508 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:58.518 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:58.518 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:58.532 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:58.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:58.634 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:33:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:58.658 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:58.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:33:58.759 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:58.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:58.784 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:58.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:58.885 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:58.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:58.909 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:58.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:58.924 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:33:58 smithi008 ceph-mon[29816]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-02-10T07:33:58.924 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:33:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:33:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:33:58.925 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:33:58 smithi008 ceph-mon[40743]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-02-10T07:33:58.986 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:33:59.011 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:59.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:59.036 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:59.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:59.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:33:58 smithi170 ceph-mon[33394]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-02-10T07:33:59.137 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:59.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:59.162 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:59.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:59.263 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:59.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:59.291 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:59.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:59.392 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:33:59.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:59.418 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:59.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:33:59.519 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:33:59.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:59.546 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:33:59.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:33:59.647 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:33:59.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:59.671 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:59.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:33:59.772 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:59.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:59.797 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:59.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:33:59.898 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:33:59.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:33:59.923 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:33:59.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:00.024 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:00.052 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:00.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:00.153 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:00.177 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:00.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:00.278 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:00.306 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:00.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:00.408 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:00.437 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:00.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:00.538 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:00.564 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:00.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:00.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:00.666 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:00.691 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:00.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:00.792 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:00.820 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:00.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:00.922 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:00.946 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:00.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:00.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:01.048 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:01.077 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:01.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:01.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:00 smithi008 ceph-mon[29816]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-02-10T07:34:01.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:00 smithi008 ceph-mon[40743]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-02-10T07:34:01.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:00 smithi170 ceph-mon[33394]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-02-10T07:34:01.178 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:01.206 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:01.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:01.307 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:01.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:01.336 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:01.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:01.437 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:01.464 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:01.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:01.566 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:01.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:01.594 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:01.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:01.695 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:01.719 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:01.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:01.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:01.820 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:01.848 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:01.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:01.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:01.950 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:01.977 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:01.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:01.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:02.080 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:02.107 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:02.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:02.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:02.209 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:02.237 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:02.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:02.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:02.275 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:34:02.275 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:34:02.275 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-02-10T07:34:02.275 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-02-10T07:34:02.276 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:34:02.339 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:02.367 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:02.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:02.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:02.469 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:02.496 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:02.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:02.598 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:02.626 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:02.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:02.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:02.728 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:02.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:02.753 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:02.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:02.854 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:02.883 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:02.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:02.984 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:03.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:03.010 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:03.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:03.111 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:03.112 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:02 smithi008 ceph-mon[29816]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:34:03.112 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:02 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:03.112 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:02 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:03.113 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:02 smithi008 ceph-mon[40743]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:34:03.113 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:02 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:03.113 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:02 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:03.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:34:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:34:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:02 smithi170 ceph-mon[33394]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:34:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:02 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:03.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:02 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:03.139 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:03.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:03.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:03.241 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:03.268 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:03.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:03.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:03.318 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:34:03.372 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:03.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:03.396 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:03.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:03.497 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:03.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:03.522 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:03.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:03.623 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:03.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:03.648 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:03.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:03.749 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:03.777 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:03.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:03.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:03.880 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:03.905 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:03.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:04.007 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:04.034 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:04.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:04.077 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:03 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:04.078 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:03 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:04.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:03 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:04.136 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:04.161 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:04.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:04.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:04.263 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:04.286 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:04.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:04.388 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:04.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:04.412 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:04.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:04.514 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:04.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:04.539 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:04.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:04.640 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:04.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:04.664 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:04.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:04.765 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:04.793 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:04.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:04.849 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:34:04.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:04.923 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:04.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:04.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:05.025 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:05.052 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:05.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:05.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:04 smithi170 ceph-mon[33394]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:34:05.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:04 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3922236647' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:34:05.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:05.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:05.154 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:05.182 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:05.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:05.201 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:04 smithi008 ceph-mon[29816]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:34:05.201 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:04 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3922236647' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:34:05.201 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:05.202 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:05.202 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:04 smithi008 ceph-mon[40743]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:34:05.202 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:04 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3922236647' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:34:05.202 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:05.202 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:05.283 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:05.311 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:05.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:05.413 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:05.437 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:05.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:05.539 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:05.563 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:05.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:05.664 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:05.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:05.688 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:05.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:05.716 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-10T07:34:05.789 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:05.818 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:05.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:05.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:05.920 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:05.947 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:05.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:05.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:05.968 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:05 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:34:05.968 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:05 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3922236647' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:34:05.968 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:05 smithi008 ceph-mon[29816]: osdmap e102: 8 total, 8 up, 8 in 2024-02-10T07:34:05.968 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:05 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:05.968 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:05 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:34:05.968 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:05 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:05.969 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:05 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:34:05.969 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:05 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3922236647' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:34:05.969 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:05 smithi008 ceph-mon[40743]: osdmap e102: 8 total, 8 up, 8 in 2024-02-10T07:34:05.969 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:05 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:05.969 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:05 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:34:05.969 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:05 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:06.049 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:06.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:06.074 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:06.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:05 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:34:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:05 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3922236647' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:34:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:05 smithi170 ceph-mon[33394]: osdmap e102: 8 total, 8 up, 8 in 2024-02-10T07:34:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:05 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:34:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:05 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:34:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:05 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:06.175 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:06.199 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:06.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:06.300 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:06.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:06.328 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:06.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:06.429 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:06.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:06.453 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:06.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:06.554 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:06.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:06.583 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:06.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:06.685 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:06.712 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:06.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:06.814 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:06.840 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:06.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:06.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:06.942 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:06.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:06.967 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:06.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:07.068 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:07.096 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:07.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:07.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:07.099 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:06 smithi008 ceph-mon[29816]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T07:34:07.099 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:06 smithi008 ceph-mon[29816]: Reconfiguring daemon alertmanager.a on smithi008 2024-02-10T07:34:07.099 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:06 smithi008 ceph-mon[29816]: pgmap v70: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:34:07.099 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:06 smithi008 ceph-mon[40743]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T07:34:07.099 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:06 smithi008 ceph-mon[40743]: Reconfiguring daemon alertmanager.a on smithi008 2024-02-10T07:34:07.099 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:06 smithi008 ceph-mon[40743]: pgmap v70: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:34:07.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:06 smithi170 ceph-mon[33394]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-10T07:34:07.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:06 smithi170 ceph-mon[33394]: Reconfiguring daemon alertmanager.a on smithi008 2024-02-10T07:34:07.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:06 smithi170 ceph-mon[33394]: pgmap v70: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:34:07.198 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:07.226 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:07.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:07.321 INFO:teuthology.orchestra.run.smithi008.stdout:moved 0 / 444 (0%) 2024-02-10T07:34:07.322 INFO:teuthology.orchestra.run.smithi008.stdout:avg 55.5 2024-02-10T07:34:07.322 INFO:teuthology.orchestra.run.smithi008.stdout:stddev 6.57647 -> 6.57647 (expected baseline 6.96868) 2024-02-10T07:34:07.322 INFO:teuthology.orchestra.run.smithi008.stdout:min osd.2 with 46 -> 46 pgs (0.828829 -> 0.828829 * mean) 2024-02-10T07:34:07.322 INFO:teuthology.orchestra.run.smithi008.stdout:max osd.3 with 66 -> 66 pgs (1.18919 -> 1.18919 * mean) 2024-02-10T07:34:07.322 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:34:07.322 INFO:teuthology.orchestra.run.smithi008.stdout:oload 1000 2024-02-10T07:34:07.322 INFO:teuthology.orchestra.run.smithi008.stdout:max_change 1 2024-02-10T07:34:07.322 INFO:teuthology.orchestra.run.smithi008.stdout:max_change_osds 4 2024-02-10T07:34:07.322 INFO:teuthology.orchestra.run.smithi008.stdout:average_utilization 0.0003 2024-02-10T07:34:07.322 INFO:teuthology.orchestra.run.smithi008.stdout:overload_utilization 0.0034 2024-02-10T07:34:07.324 INFO:teuthology.orchestra.run.smithi008.stderr:no change 2024-02-10T07:34:07.328 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:07.356 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:07.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:07.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:07.485 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:07.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:07.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:07.587 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:07.592 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:34:07.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:07.614 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:07.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:07.715 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:07.744 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:07.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:07.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:07.845 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:07.871 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:07.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:07.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:07.973 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:07.997 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:07.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:07.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:08.005 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:07 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:34:08.005 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:07 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:34:08.005 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:07 smithi008 ceph-mon[29816]: osdmap e103: 8 total, 8 up, 8 in 2024-02-10T07:34:08.005 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:07 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3111006363' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:34:08.005 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:07 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:34:08.006 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:07 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:34:08.006 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:07 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:34:08.006 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:07 smithi008 ceph-mon[40743]: osdmap e103: 8 total, 8 up, 8 in 2024-02-10T07:34:08.006 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:07 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3111006363' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:34:08.006 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:07 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:34:08.031 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-02-10T07:34:08.031 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight-by-utilization 1000 0.05 2024-02-10T07:34:08.099 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:08.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:07 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:34:08.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:07 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:34:08.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:07 smithi170 ceph-mon[33394]: osdmap e103: 8 total, 8 up, 8 in 2024-02-10T07:34:08.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:07 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3111006363' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:34:08.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:07 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:34:08.137 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:08.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:08.240 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:08.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:08.266 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:08.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:08.336 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:34:08.367 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:08.403 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:08.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:08.413 INFO:teuthology.orchestra.run.smithi008.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-02-10T07:34:08.504 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:08.529 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:08.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:08.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:08.631 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:08.654 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:08.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:08.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:08.757 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:08.780 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:08.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:08.882 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:08.907 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:08.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:09.008 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:09.033 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:09.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:09.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:09.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:34:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:34:09.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:08 smithi008 ceph-mon[29816]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:09.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:08 smithi008 ceph-mon[29816]: pgmap v72: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:09.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:08 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:34:09.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:08 smithi008 ceph-mon[29816]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T07:34:09.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:08 smithi008 ceph-mon[40743]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:09.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:08 smithi008 ceph-mon[40743]: pgmap v72: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:09.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:08 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:34:09.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:08 smithi008 ceph-mon[40743]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T07:34:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:08 smithi170 ceph-mon[33394]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:08 smithi170 ceph-mon[33394]: pgmap v72: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:34:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:08 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:34:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:08 smithi170 ceph-mon[33394]: osdmap e104: 8 total, 8 up, 8 in 2024-02-10T07:34:09.134 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:09.159 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:09.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:09.260 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:09.284 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:09.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:09.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:09.386 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:09.401 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:34:09.410 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:09.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:09.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:09.427 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:34:09.446 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:34:09.466 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:34:09.486 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:34:09.511 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:34:09.512 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:09.530 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:34:09.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:09.536 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:09.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:09.550 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:34:09.551 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:34:09.551 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:34:09.551 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:34:09.551 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:34:09.552 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:34:09.552 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:34:09.552 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:34:09.552 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:34:09.637 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:09.662 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:09.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:09.763 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:09.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:09.787 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:09.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:09.888 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:09.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:09.913 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:09.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:10.014 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:10.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:10.039 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:10.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:10.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:09 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/885263320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:34:10.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:09 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/885263320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:34:10.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:09 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/885263320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:34:10.140 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:10.142 DEBUG:teuthology.orchestra.run.smithi008:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-02-10T07:34:10.144 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-10T07:34:10.144 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-10T07:34:10.163 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:10.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:10.201 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:hints = 1 2024-02-10T07:34:10.201 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-10T07:34:10.202 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Object prefix: benchmark_data_smithi008_82046 2024-02-10T07:34:10.202 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:34:10.202 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 0 0 0 0 0 0 - 0 2024-02-10T07:34:10.265 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:10.306 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:10.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:10.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:10.408 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:10.448 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:10.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:10.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:10.550 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:10.583 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:10.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:10.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:10.685 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:10.727 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:10.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:10.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:10.829 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:10.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:10.869 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:10.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:10.970 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:11.004 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:11.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:11.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:11.106 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:11.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:10 smithi170 ceph-mon[33394]: pgmap v74: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:11.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:10 smithi170 ceph-mon[33394]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:34:11.148 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:11.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:11.202 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 1 16 3499 3483 217.675 217.688 0.00518126 0.00457743 2024-02-10T07:34:11.250 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:11.285 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:11.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:11.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:11.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:10 smithi008 ceph-mon[29816]: pgmap v74: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:11.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:10 smithi008 ceph-mon[29816]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:34:11.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:10 smithi008 ceph-mon[40743]: pgmap v74: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:34:11.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:10 smithi008 ceph-mon[40743]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:34:11.386 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:11.420 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:11.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:11.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:11.522 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:11.564 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:11.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:11.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:11.665 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:11.698 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:11.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:11.800 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:11.841 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:11.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:11.943 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:11.978 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:11.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:11.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:12.080 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:12.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:12.126 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:12.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:12.202 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 2 15 6741 6726 210.165 202.688 0.00398433 0.00474922 2024-02-10T07:34:12.227 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:12.270 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:12.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:12.370 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:12.414 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:12.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:12.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:12.515 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:12.548 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:12.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:12.650 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:12.691 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:12.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:12.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:12.755 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:34:12.793 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:12.822 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:12.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:12.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:12 smithi008 ceph-mon[29816]: pgmap v75: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-02-10T07:34:12.832 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:34:12.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:12 smithi008 ceph-mon[40743]: pgmap v75: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-02-10T07:34:12.924 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:12.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:12.964 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:12.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:13.065 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:13.110 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:13.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:13.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:13.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:12 smithi170 ceph-mon[33394]: pgmap v75: 148 pgs: 148 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-02-10T07:34:13.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:34:12 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:34:13.202 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 3 16 9703 9687 201.792 185.062 0.00603393 0.00494722 2024-02-10T07:34:13.212 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:13.252 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:13.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:13.354 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:13.398 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:13.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:13.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:13.500 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:13.536 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:34:13.543 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:13.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:13.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:13.647 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:13.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:13.673 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:13.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:13.774 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:13.804 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:13.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:13.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:13.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:13 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:13.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:13 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:13.906 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:13.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:13.968 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:13.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:14.069 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:14.118 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:14.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:14.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:13 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:14.202 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 4 16 12325 12309 192.309 163.875 0.00566502 0.00519151 2024-02-10T07:34:14.220 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:14.253 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:14.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:14.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:14.356 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:14.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:14.395 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:14.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:14.496 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:14.535 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:14.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:14.638 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:14.668 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:14.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:14.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:14.770 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:14.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:14.817 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:14.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:14.913 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:14 smithi008 ceph-mon[29816]: pgmap v76: 148 pgs: 148 active+clean; 515 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s 2024-02-10T07:34:14.913 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:14 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:34:14.913 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:14 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:34:14.913 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:14 smithi008 ceph-mon[40743]: pgmap v76: 148 pgs: 148 active+clean; 515 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s 2024-02-10T07:34:14.913 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:14 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:34:14.913 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:14 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:34:14.919 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:14.949 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:14.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:14.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:15.051 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:15.093 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:15.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:15.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:15.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:14 smithi170 ceph-mon[33394]: pgmap v76: 148 pgs: 148 active+clean; 515 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s 2024-02-10T07:34:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:14 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:34:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:14 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:34:15.195 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:15.203 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 5 16 14918 14902 186.258 162.062 0.00405877 0.00535985 2024-02-10T07:34:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:15.234 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:15.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:15.335 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:15.370 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:15.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:15.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:15.473 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:15.507 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:15.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:15.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:15.609 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:15.641 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:15.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:15.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:15.743 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:15.777 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:15.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:15.879 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:15.908 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:15 smithi008 ceph-mon[29816]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T07:34:15.908 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:15 smithi008 ceph-mon[40743]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T07:34:15.914 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:15.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:16.015 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:16.057 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:16.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:16.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:16.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:15 smithi170 ceph-mon[33394]: osdmap e105: 8 total, 8 up, 8 in 2024-02-10T07:34:16.159 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:16.202 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 6 16 17569 17553 182.827 165.688 0.00354075 0.00546152 2024-02-10T07:34:16.204 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:16.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:16.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:16.306 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:16.339 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:16.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:16.440 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:16.479 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:16.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:16.581 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:16.621 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:16.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:16.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:16.723 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:16.753 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:16.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:16.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:16 smithi008 ceph-mon[29816]: pgmap v78: 148 pgs: 148 active+clean; 515 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-02-10T07:34:16.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:16 smithi008 ceph-mon[40743]: pgmap v78: 148 pgs: 148 active+clean; 515 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-02-10T07:34:16.855 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:16.885 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:16.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:16.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:16.986 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:17.025 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:17.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:17.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:17.127 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:17.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:16 smithi170 ceph-mon[33394]: pgmap v78: 148 pgs: 148 active+clean; 515 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-02-10T07:34:17.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:17.170 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:17.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:17.202 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 7 16 19887 19871 177.404 144.875 0.00255276 0.00562993 2024-02-10T07:34:17.271 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:17.310 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:17.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:17.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:17.412 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:17.454 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:17.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:17.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:17.556 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:17.588 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:17.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:17.621 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:34:17.621 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:34:17.689 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:17.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:17.734 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:17.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:17.835 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:17.854 INFO:teuthology.orchestra.run.smithi008.stdout:moved 3 / 444 (0.675676%) 2024-02-10T07:34:17.854 INFO:teuthology.orchestra.run.smithi008.stdout:avg 55.5 2024-02-10T07:34:17.854 INFO:teuthology.orchestra.run.smithi008.stdout:stddev 6.57647 -> 6.8374 (expected baseline 6.96868) 2024-02-10T07:34:17.854 INFO:teuthology.orchestra.run.smithi008.stdout:min osd.2 with 46 -> 44 pgs (0.828829 -> 0.792793 * mean) 2024-02-10T07:34:17.854 INFO:teuthology.orchestra.run.smithi008.stdout:max osd.3 with 66 -> 66 pgs (1.18919 -> 1.18919 * mean) 2024-02-10T07:34:17.854 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:34:17.855 INFO:teuthology.orchestra.run.smithi008.stdout:oload 1000 2024-02-10T07:34:17.855 INFO:teuthology.orchestra.run.smithi008.stdout:max_change 0.05 2024-02-10T07:34:17.855 INFO:teuthology.orchestra.run.smithi008.stdout:max_change_osds 4 2024-02-10T07:34:17.855 INFO:teuthology.orchestra.run.smithi008.stdout:average_utilization 0.0021 2024-02-10T07:34:17.855 INFO:teuthology.orchestra.run.smithi008.stdout:overload_utilization 0.0212 2024-02-10T07:34:17.855 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 weight 0.8767 -> 0.9267 2024-02-10T07:34:17.879 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:17.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:17.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:17.980 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:18.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:18.005 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:18.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:18.106 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:18.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:18.131 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:18.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:18.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:17 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:18.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:17 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:18.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:17 smithi170 ceph-mon[33394]: from='client.24746 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:18.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:17 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'60730'}"}]: dispatch 2024-02-10T07:34:18.202 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 8 16 21615 21599 168.727 108 0.00168346 0.00569589 2024-02-10T07:34:18.233 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:18.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:18.258 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:18.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:18.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:17 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:18.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:17 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:18.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:17 smithi008 ceph-mon[40743]: from='client.24746 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:18.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:17 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'60730'}"}]: dispatch 2024-02-10T07:34:18.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:17 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:18.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:17 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:18.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:17 smithi008 ceph-mon[29816]: from='client.24746 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:18.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:17 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'60730'}"}]: dispatch 2024-02-10T07:34:18.359 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:18.381 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:18.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:18.483 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:18.513 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:18.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:18.615 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:18.645 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:18.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:18.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:18.749 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:18.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:18.784 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:18.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:18.885 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:18.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:18.925 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:18.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:19.026 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:19.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:19.061 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:19.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:19.084 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:34:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:34:19.085 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:18 smithi008 ceph-mon[29816]: pgmap v79: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-02-10T07:34:19.085 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'60730'}"}]': finished 2024-02-10T07:34:19.085 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:18 smithi008 ceph-mon[29816]: osdmap e106: 8 total, 8 up, 8 in 2024-02-10T07:34:19.085 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:18 smithi008 ceph-mon[29816]: osdmap e107: 8 total, 8 up, 8 in 2024-02-10T07:34:19.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:18 smithi008 ceph-mon[40743]: pgmap v79: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-02-10T07:34:19.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'60730'}"}]': finished 2024-02-10T07:34:19.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:18 smithi008 ceph-mon[40743]: osdmap e106: 8 total, 8 up, 8 in 2024-02-10T07:34:19.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:18 smithi008 ceph-mon[40743]: osdmap e107: 8 total, 8 up, 8 in 2024-02-10T07:34:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:18 smithi170 ceph-mon[33394]: pgmap v79: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-02-10T07:34:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'60730'}"}]': finished 2024-02-10T07:34:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:18 smithi170 ceph-mon[33394]: osdmap e106: 8 total, 8 up, 8 in 2024-02-10T07:34:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:18 smithi170 ceph-mon[33394]: osdmap e107: 8 total, 8 up, 8 in 2024-02-10T07:34:19.162 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:19.190 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:19.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:19.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:19.202 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 9 16 21615 21599 149.979 0 - 0.00569589 2024-02-10T07:34:19.292 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:19.322 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:19.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:19.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:19.424 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:19.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:19.457 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:19.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:19.558 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:19.587 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:19.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:19.689 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:19.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:19.717 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:19.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:19.818 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:19.831 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:19 smithi008 systemd[1]: Stopping Ceph alertmanager.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:34:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:19.848 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:19.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:19.949 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:19.976 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:19.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:19.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:20.078 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:20.106 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:20.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:20.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:20.202 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 10 16 21615 21599 134.981 0 - 0.00569589 2024-02-10T07:34:20.208 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:20.248 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:20.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:20.350 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:20.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:20.377 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:20.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:20.478 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:20.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:20.502 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:20.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:20.603 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:20.634 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:20.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:20.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:20.737 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:20.770 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:20.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:20.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:20.872 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:20.905 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:20.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:21.007 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:21.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:21.032 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:21.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:21.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:20 smithi170 ceph-mon[33394]: pgmap v82: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-10T07:34:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:20 smithi170 ceph-mon[33394]: osdmap e108: 8 total, 8 up, 8 in 2024-02-10T07:34:21.133 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:21.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:21.166 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:21.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:21.202 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 11 16 23149 23133 131.425 31.9583 0.00645069 0.00760113 2024-02-10T07:34:21.267 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:21.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:21.293 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:21.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:21.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:20 smithi008 ceph-mon[29816]: pgmap v82: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-10T07:34:21.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:20 smithi008 ceph-mon[29816]: osdmap e108: 8 total, 8 up, 8 in 2024-02-10T07:34:21.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:20 smithi008 ceph-mon[40743]: pgmap v82: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-10T07:34:21.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:20 smithi008 ceph-mon[40743]: osdmap e108: 8 total, 8 up, 8 in 2024-02-10T07:34:21.395 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:21.427 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:21.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:21.529 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:21.560 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:21.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:21.662 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:21.687 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:21.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:21.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:21.788 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:21.828 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:21.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:21.929 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:21.961 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:21.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:22.063 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:22.094 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:22.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:22.196 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:22.203 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 12 16 24663 24647 128.358 94.625 0.0062341 0.0077828 2024-02-10T07:34:22.226 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:22.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:22.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:22.328 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:22.361 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:22.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:22.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:22.463 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:22.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:22.491 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:22.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:22.592 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:22.624 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:22.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:22.726 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:22.759 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:22.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:22.860 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:22.891 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:22.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:22.945 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:22 smithi008 ceph-mon[29816]: osdmap e109: 8 total, 8 up, 8 in 2024-02-10T07:34:22.945 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:22 smithi008 ceph-mon[29816]: pgmap v85: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail 2024-02-10T07:34:22.945 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:22 smithi008 ceph-mon[40743]: osdmap e109: 8 total, 8 up, 8 in 2024-02-10T07:34:22.946 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:22 smithi008 ceph-mon[40743]: pgmap v85: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail 2024-02-10T07:34:22.993 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:23.024 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:23.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:23.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:23.126 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:23.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:22 smithi170 ceph-mon[33394]: osdmap e109: 8 total, 8 up, 8 in 2024-02-10T07:34:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:22 smithi170 ceph-mon[33394]: pgmap v85: 148 pgs: 148 active+clean; 1.3 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail 2024-02-10T07:34:23.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:34:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:34:23.160 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:23.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:23.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:23.203 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 13 16 26475 26459 127.195 113.25 0.00487163 0.0078532 2024-02-10T07:34:23.261 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:23.287 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:23.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:23.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:23.389 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:23.429 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:23.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:23.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:23.531 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:23.562 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:23.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:23.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:23.664 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:23.695 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:23.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:23.797 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:23.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:23.837 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:23.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:23.938 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:23.976 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:23.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:23.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:24.078 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:24.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:23 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:24.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:23 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:24.110 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:24.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:24.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:24.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:23 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:24.203 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 14 16 28000 27984 124.917 95.3125 0.00347744 0.00798695 2024-02-10T07:34:24.212 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:24.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:24.238 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:24.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:24.339 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:24.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:24.372 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:24.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:24.473 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:24.512 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:24.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:24.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:24.614 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:24.646 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:24.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:24.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:24.748 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:24.785 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:24.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:24.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:24.887 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:24.918 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:24.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:25.007 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:24 smithi008 ceph-mon[29816]: pgmap v86: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 877 B/s rd, 47 MiB/s wr, 745 op/s; 1/76812 objects degraded (0.001%); 2357/76812 objects misplaced (3.069%); 5.0 MiB/s, 80 objects/s recovering 2024-02-10T07:34:25.008 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:24 smithi008 ceph-mon[40743]: pgmap v86: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 877 B/s rd, 47 MiB/s wr, 745 op/s; 1/76812 objects degraded (0.001%); 2357/76812 objects misplaced (3.069%); 5.0 MiB/s, 80 objects/s recovering 2024-02-10T07:34:25.020 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:25.063 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-10T07:34:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:25.088 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:25.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:25.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:24 smithi170 ceph-mon[33394]: pgmap v86: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 877 B/s rd, 47 MiB/s wr, 745 op/s; 1/76812 objects degraded (0.001%); 2357/76812 objects misplaced (3.069%); 5.0 MiB/s, 80 objects/s recovering 2024-02-10T07:34:25.190 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:25.203 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 15 16 29585 29569 123.192 99.0625 0.0100723 0.00810857 2024-02-10T07:34:25.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:25.220 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:25.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:25.322 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:25.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:25.349 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:25.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:25.450 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:25.500 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:25.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:25.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:25.602 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:25.650 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:25.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:25.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:25.753 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:25.799 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:25.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:25.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:25.901 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:25.930 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:25.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:25.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:26.032 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:26.068 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:25 smithi008 ceph-mon[29816]: Health check failed: Degraded data redundancy: 1/76812 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:34:26.068 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:25 smithi008 ceph-mon[40743]: Health check failed: Degraded data redundancy: 1/76812 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:34:26.074 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:26.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:26.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:26.123 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-02-10T07:34:26.124 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd test-reweight-by-utilization 110 0.05 2024-02-10T07:34:26.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:25 smithi170 ceph-mon[33394]: Health check failed: Degraded data redundancy: 1/76812 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:34:26.137 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:34:26.177 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:26.204 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 16 16 30929 30913 120.735 84 0.00844238 0.00827552 2024-02-10T07:34:26.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:26.206 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:26.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:26.307 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:26.333 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:34:26.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:26.334 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:26.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:26.335 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:26 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[75760]: ts=2024-02-10T07:34:26.295Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-10T07:34:26.435 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:26.464 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:26.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:26.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:26.566 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:26.594 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:26.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:26.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:26.683 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:34:26.688 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:26 smithi008 podman[83397]: 2024-02-10 07:34:26.528698704 +0000 UTC m=+0.403356844 container died 583cb0039d939852601fa18a520037a41bc64c3f7830f21129acbfbae668cbc2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:26.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:26.725 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:26.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:26.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:26.827 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:26.856 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:26.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:26.958 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:26.987 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:26.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:27.033 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:26 smithi170 ceph-mon[33394]: pgmap v87: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 748 B/s rd, 40 MiB/s wr, 636 op/s; 1/76812 objects degraded (0.001%); 2357/76812 objects misplaced (3.069%); 4.3 MiB/s, 68 objects/s recovering 2024-02-10T07:34:27.034 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:26 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1392487236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:34:27.035 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:26 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:34:27.035 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:26 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:34:27.035 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:26 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:34:27.035 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:26 smithi170 ceph-mon[33394]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T07:34:27.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:26 smithi008 ceph-mon[29816]: pgmap v87: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 748 B/s rd, 40 MiB/s wr, 636 op/s; 1/76812 objects degraded (0.001%); 2357/76812 objects misplaced (3.069%); 4.3 MiB/s, 68 objects/s recovering 2024-02-10T07:34:27.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:26 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1392487236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:34:27.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:26 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:34:27.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:26 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:34:27.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:26 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:34:27.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:26 smithi008 ceph-mon[29816]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T07:34:27.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:26 smithi008 ceph-mon[40743]: pgmap v87: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 748 B/s rd, 40 MiB/s wr, 636 op/s; 1/76812 objects degraded (0.001%); 2357/76812 objects misplaced (3.069%); 4.3 MiB/s, 68 objects/s recovering 2024-02-10T07:34:27.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:26 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1392487236' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:34:27.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:26 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:34:27.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:26 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:34:27.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:26 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:34:27.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:26 smithi008 ceph-mon[40743]: osdmap e110: 8 total, 8 up, 8 in 2024-02-10T07:34:27.090 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:27.133 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:27.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:27.204 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 17 15 32369 32354 118.931 90.0625 0.00929361 0.00840269 2024-02-10T07:34:27.235 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:27.267 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:27.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:27.369 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:27.407 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:27.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:27.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:27.509 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:27.549 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:27.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:27.651 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:27.694 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:27.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:27.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:27.796 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:27.831 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:27.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:27.933 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:27.975 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:27.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:28.077 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:28.110 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:28.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:28.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:28.204 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 18 16 33987 33971 117.938 101.062 0.00976079 0.00847309 2024-02-10T07:34:28.208 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:28 smithi008 podman[83397]: 2024-02-10 07:34:28.041259187 +0000 UTC m=+1.915917324 container cleanup 583cb0039d939852601fa18a520037a41bc64c3f7830f21129acbfbae668cbc2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:28.208 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:28 smithi008 bash[83397]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T07:34:28.211 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:28.260 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:28.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:28.362 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:28.405 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:28.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:28.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:28.507 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:28.539 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:28.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:28.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:28.641 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:28.680 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:28.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:28.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:28.782 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:28.821 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:28.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:28.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:28.907 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:28 smithi008 ceph-mon[29816]: 4.1b scrub starts 2024-02-10T07:34:28.907 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:28 smithi008 ceph-mon[29816]: 4.1b scrub ok 2024-02-10T07:34:28.907 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:28 smithi008 ceph-mon[29816]: pgmap v89: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.9 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 1/94101 objects degraded (0.001%); 1382/94101 objects misplaced (1.469%); 11 MiB/s, 184 objects/s recovering 2024-02-10T07:34:28.908 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:28 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:34:28.908 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:28 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:34:28.908 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:28 smithi008 ceph-mon[29816]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T07:34:28.908 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:34:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:34:28.908 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:28 smithi008 ceph-mon[40743]: 4.1b scrub starts 2024-02-10T07:34:28.908 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:28 smithi008 ceph-mon[40743]: 4.1b scrub ok 2024-02-10T07:34:28.908 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:28 smithi008 ceph-mon[40743]: pgmap v89: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.9 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 1/94101 objects degraded (0.001%); 1382/94101 objects misplaced (1.469%); 11 MiB/s, 184 objects/s recovering 2024-02-10T07:34:28.909 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:28 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:34:28.909 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:28 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:34:28.909 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:28 smithi008 ceph-mon[40743]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T07:34:28.923 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:28.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:28.992 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:28.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:29.093 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:28 smithi170 ceph-mon[33394]: 4.1b scrub starts 2024-02-10T07:34:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:28 smithi170 ceph-mon[33394]: 4.1b scrub ok 2024-02-10T07:34:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:28 smithi170 ceph-mon[33394]: pgmap v89: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 1.9 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s; 1/94101 objects degraded (0.001%); 1382/94101 objects misplaced (1.469%); 11 MiB/s, 184 objects/s recovering 2024-02-10T07:34:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:28 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:34:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:28 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:34:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:28 smithi170 ceph-mon[33394]: osdmap e111: 8 total, 8 up, 8 in 2024-02-10T07:34:29.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:29.140 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:29.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:29.205 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 19 16 35630 35614 117.135 102.688 0.0116079 0.00853071 2024-02-10T07:34:29.241 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:29.272 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:29.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:29.373 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:29.405 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:29.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:29.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:29.506 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:29.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:29.549 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:29.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:29.650 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:29.680 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:29.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:29.689 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:29 smithi008 podman[83455]: 2024-02-10 07:34:29.412028875 +0000 UTC m=+3.071228877 container remove 583cb0039d939852601fa18a520037a41bc64c3f7830f21129acbfbae668cbc2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:29.689 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:29 smithi008 podman[83455]: 2024-02-10 07:34:29.488556058 +0000 UTC m=+3.147756061 volume remove e097244bafa46188b9bcc3334d1545736e2ef8f40385c3601cc293e1121de9fc 2024-02-10T07:34:29.693 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:34:29.782 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:29.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:29.822 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:29.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:29.923 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:29.948 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:29.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:29.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:30.025 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:29 smithi008 ceph-mon[29816]: 5.e scrub starts 2024-02-10T07:34:30.025 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:29 smithi008 ceph-mon[29816]: 5.e scrub ok 2024-02-10T07:34:30.025 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:29 smithi008 ceph-mon[40743]: 5.e scrub starts 2024-02-10T07:34:30.025 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:29 smithi008 ceph-mon[40743]: 5.e scrub ok 2024-02-10T07:34:30.050 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:30.074 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:30.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:30.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:29 smithi170 ceph-mon[33394]: 5.e scrub starts 2024-02-10T07:34:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:29 smithi170 ceph-mon[33394]: 5.e scrub ok 2024-02-10T07:34:30.176 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:30.201 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:30.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:30.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:30.204 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:34:30.205420+0000 min lat: 0.00141219 max lat: 2.36731 avg lat: 0.00854204 2024-02-10T07:34:30.204 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:34:30.204 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 20 16 36778 36762 114.865 71.75 0.00330114 0.00854204 2024-02-10T07:34:30.303 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:30.327 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:30.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:30.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:30.429 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:30.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:30.452 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:30.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:30.553 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:30.577 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:30.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:30.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:30.679 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:30.703 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:30.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:30.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:30.805 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:30.848 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:30.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:30.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:30.950 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:30.994 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:30.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:31.005 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:34:31.029 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:34:31.030 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:30 smithi008 ceph-mon[29816]: pgmap v91: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.0 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 1/99912 objects degraded (0.001%); 1382/99912 objects misplaced (1.383%); 11 MiB/s, 180 objects/s recovering 2024-02-10T07:34:31.030 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:30 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 1/99912 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:34:31.030 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:30 smithi008 ceph-mon[29816]: osdmap e112: 8 total, 8 up, 8 in 2024-02-10T07:34:31.030 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:30 smithi008 ceph-mon[29816]: 6.17 scrub starts 2024-02-10T07:34:31.030 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:30 smithi008 ceph-mon[29816]: 6.17 scrub ok 2024-02-10T07:34:31.030 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:30 smithi008 ceph-mon[40743]: pgmap v91: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.0 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 1/99912 objects degraded (0.001%); 1382/99912 objects misplaced (1.383%); 11 MiB/s, 180 objects/s recovering 2024-02-10T07:34:31.030 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:30 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 1/99912 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:34:31.030 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:30 smithi008 ceph-mon[40743]: osdmap e112: 8 total, 8 up, 8 in 2024-02-10T07:34:31.031 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:30 smithi008 ceph-mon[40743]: 6.17 scrub starts 2024-02-10T07:34:31.031 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:30 smithi008 ceph-mon[40743]: 6.17 scrub ok 2024-02-10T07:34:31.074 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:34:31.096 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:31.120 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:34:31.136 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:30 smithi170 ceph-mon[33394]: pgmap v91: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.0 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s; 1/99912 objects degraded (0.001%); 1382/99912 objects misplaced (1.383%); 11 MiB/s, 180 objects/s recovering 2024-02-10T07:34:31.136 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:30 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 1/99912 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:34:31.136 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:30 smithi170 ceph-mon[33394]: osdmap e112: 8 total, 8 up, 8 in 2024-02-10T07:34:31.136 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:30 smithi170 ceph-mon[33394]: 6.17 scrub starts 2024-02-10T07:34:31.136 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:30 smithi170 ceph-mon[33394]: 6.17 scrub ok 2024-02-10T07:34:31.144 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:31.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:31.159 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:34:31.200 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:34:31.204 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 21 16 37337 37321 111.059 34.9375 0.0055861 0.00899713 2024-02-10T07:34:31.238 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:34:31.246 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:31.277 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:31.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:31.279 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:34:31.320 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:34:31.329 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:34:31.330 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:34:31.330 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:34:31.330 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:34:31.330 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:34:31.330 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:34:31.330 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:34:31.330 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:34:31.379 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:31.418 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:31.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:31.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:31.520 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:31.548 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:31.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:31.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:31.650 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:31.681 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:31.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:31.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:31.784 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:31.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:31.835 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:31.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:31.913 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[29816]: 6.8 scrub starts 2024-02-10T07:34:31.913 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[29816]: 6.8 scrub ok 2024-02-10T07:34:31.914 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[29816]: osdmap e113: 8 total, 8 up, 8 in 2024-02-10T07:34:31.914 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3113073394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:34:31.914 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:34:31.914 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:34:31.914 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[29816]: 4.8 scrub starts 2024-02-10T07:34:31.914 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[29816]: 4.8 scrub ok 2024-02-10T07:34:31.914 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[29816]: osdmap e114: 8 total, 8 up, 8 in 2024-02-10T07:34:31.914 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[40743]: 6.8 scrub starts 2024-02-10T07:34:31.914 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[40743]: 6.8 scrub ok 2024-02-10T07:34:31.914 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[40743]: osdmap e113: 8 total, 8 up, 8 in 2024-02-10T07:34:31.914 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3113073394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:34:31.915 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:34:31.915 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:34:31.915 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[40743]: 4.8 scrub starts 2024-02-10T07:34:31.915 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[40743]: 4.8 scrub ok 2024-02-10T07:34:31.915 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:31 smithi008 ceph-mon[40743]: osdmap e114: 8 total, 8 up, 8 in 2024-02-10T07:34:31.915 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:31 smithi008 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-10T07:34:31.915 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:31 smithi008 systemd[1]: Stopped Ceph alertmanager.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:34:31.915 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:31 smithi008 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@alertmanager.a.service: Consumed 1.216s CPU time. 2024-02-10T07:34:31.915 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:31 smithi008 systemd[1]: Starting Ceph alertmanager.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:34:31.936 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:31.976 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:31.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:31.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:32.078 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:32.115 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:32.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:32.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:31 smithi170 ceph-mon[33394]: 6.8 scrub starts 2024-02-10T07:34:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:31 smithi170 ceph-mon[33394]: 6.8 scrub ok 2024-02-10T07:34:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:31 smithi170 ceph-mon[33394]: osdmap e113: 8 total, 8 up, 8 in 2024-02-10T07:34:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:31 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3113073394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:34:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:31 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:34:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:31 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:34:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:31 smithi170 ceph-mon[33394]: 4.8 scrub starts 2024-02-10T07:34:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:31 smithi170 ceph-mon[33394]: 4.8 scrub ok 2024-02-10T07:34:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:31 smithi170 ceph-mon[33394]: osdmap e114: 8 total, 8 up, 8 in 2024-02-10T07:34:32.204 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 22 16 38897 38881 110.442 97.5 0.00647893 0.00904805 2024-02-10T07:34:32.217 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:32.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:32.258 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:32.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:32.359 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:32.388 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:32.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:32.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:32.490 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:32.531 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:32.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:32.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:32.633 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:32.676 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:32.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:32.778 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:32.807 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:32.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:32.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:32.909 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:32.953 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:32.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:32.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:33.024 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:32 smithi008 ceph-mon[29816]: 2.0 scrub starts 2024-02-10T07:34:33.024 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:32 smithi008 ceph-mon[29816]: 2.0 scrub ok 2024-02-10T07:34:33.024 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:32 smithi008 ceph-mon[29816]: pgmap v95: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.0 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 30 MiB/s wr, 484 op/s; 1/99912 objects degraded (0.001%); 1382/99912 objects misplaced (1.383%) 2024-02-10T07:34:33.024 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:33.024 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:33.025 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:32 smithi008 ceph-mon[40743]: 2.0 scrub starts 2024-02-10T07:34:33.025 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:32 smithi008 ceph-mon[40743]: 2.0 scrub ok 2024-02-10T07:34:33.025 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:32 smithi008 ceph-mon[40743]: pgmap v95: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.0 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 30 MiB/s wr, 484 op/s; 1/99912 objects degraded (0.001%); 1382/99912 objects misplaced (1.383%) 2024-02-10T07:34:33.025 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:33.025 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:33.055 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:33.105 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:33.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:33.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:33.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:34:32 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:34:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:32 smithi170 ceph-mon[33394]: 2.0 scrub starts 2024-02-10T07:34:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:32 smithi170 ceph-mon[33394]: 2.0 scrub ok 2024-02-10T07:34:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:32 smithi170 ceph-mon[33394]: pgmap v95: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 2.0 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 30 MiB/s wr, 484 op/s; 1/99912 objects degraded (0.001%); 1382/99912 objects misplaced (1.383%) 2024-02-10T07:34:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:33.205 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 23 16 40554 40538 110.143 103.562 0.00331374 0.00907285 2024-02-10T07:34:33.207 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:33.238 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:33.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:33.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:33.340 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:33.371 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:33.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:33.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:33.473 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:33.503 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:33.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:33.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:33.605 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:33.636 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:33.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:33.737 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:33.766 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:33.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:33.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:33.868 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:33.894 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:33 smithi008 ceph-mon[29816]: 6.15 scrub starts 2024-02-10T07:34:33.894 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:33 smithi008 ceph-mon[29816]: 6.15 scrub ok 2024-02-10T07:34:33.894 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:33 smithi008 ceph-mon[29816]: 6.10 scrub starts 2024-02-10T07:34:33.895 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:33 smithi008 ceph-mon[29816]: 6.10 scrub ok 2024-02-10T07:34:33.895 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:33 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:33.895 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:33 smithi008 ceph-mon[29816]: 6.2 scrub starts 2024-02-10T07:34:33.895 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:33 smithi008 ceph-mon[29816]: 6.2 scrub ok 2024-02-10T07:34:33.895 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:33 smithi008 ceph-mon[40743]: 6.15 scrub starts 2024-02-10T07:34:33.895 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:33 smithi008 ceph-mon[40743]: 6.15 scrub ok 2024-02-10T07:34:33.897 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:33.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:33.999 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:34.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:34.030 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:34.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:34.131 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:34.182 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:34.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:34.206 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 24 16 42238 42222 109.937 105.25 0.0124585 0.00909088 2024-02-10T07:34:34.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:33 smithi008 ceph-mon[40743]: 6.10 scrub starts 2024-02-10T07:34:34.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:33 smithi008 ceph-mon[40743]: 6.10 scrub ok 2024-02-10T07:34:34.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:33 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:34.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:33 smithi008 ceph-mon[40743]: 6.2 scrub starts 2024-02-10T07:34:34.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:33 smithi008 ceph-mon[40743]: 6.2 scrub ok 2024-02-10T07:34:34.284 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:34.331 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:34.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:34.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:34.339 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:34:34.339 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:34:34.340 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.340 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDPGPush(7.2 114/107 [PushOp(7:4d204b54:::benchmark_data_smithi008_82046_object17223:head, version: 105'1155, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:4d204b54:::benchmark_data_smithi008_82046_object17223:head@105'1155, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-10T07:34:34.340 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:34.337369+0000", 2024-02-10T07:34:34.340 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.001122987, 2024-02-10T07:34:34.340 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0011767780000000001, 2024-02-10T07:34:34.340 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:34:34.341 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:34:34.341 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:34:34.341 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.341 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:34:34.341 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.337369+0000", 2024-02-10T07:34:34.341 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:34:34.341 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.341 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.341 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:34:34.341 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.337369+0000", 2024-02-10T07:34:34.341 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:34:34.341 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.341 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.342 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:34:34.342 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.337370+0000", 2024-02-10T07:34:34.342 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6599999999999995e-07 2024-02-10T07:34:34.342 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.342 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.342 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:34:34.342 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.337403+0000", 2024-02-10T07:34:34.342 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3435000000000001e-05 2024-02-10T07:34:34.342 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.342 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.342 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:34:34.343 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.337428+0000", 2024-02-10T07:34:34.343 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4564000000000001e-05 2024-02-10T07:34:34.343 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.343 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.343 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:34:34.343 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.337446+0000", 2024-02-10T07:34:34.343 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7931000000000002e-05 2024-02-10T07:34:34.343 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.343 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.344 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:34:34.344 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.337672+0000", 2024-02-10T07:34:34.345 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00022650099999999999 2024-02-10T07:34:34.345 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.345 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.345 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:34:34.345 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.337686+0000", 2024-02-10T07:34:34.345 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4270000000000001e-05 2024-02-10T07:34:34.346 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:34.346 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:34.346 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:34.346 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.346 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.346 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:42424 7.b 7:dd93a251:::benchmark_data_smithi008_82046_object42423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-02-10T07:34:34.346 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:34.337914+0000", 2024-02-10T07:34:34.346 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.00057796599999999996, 2024-02-10T07:34:34.346 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00071632399999999995, 2024-02-10T07:34:34.346 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:34:34.346 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:34:34.347 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:34:34.347 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:34:34.347 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:34:34.347 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 42424 2024-02-10T07:34:34.347 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.347 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:34:34.347 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.347 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:34:34.347 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.337914+0000", 2024-02-10T07:34:34.347 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:34:34.348 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.348 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.348 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:34:34.348 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.337914+0000", 2024-02-10T07:34:34.348 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:34:34.348 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.348 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.349 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:34:34.349 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.337915+0000", 2024-02-10T07:34:34.349 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3430000000000001e-06 2024-02-10T07:34:34.349 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.349 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.349 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:34:34.349 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.337939+0000", 2024-02-10T07:34:34.349 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3541e-05 2024-02-10T07:34:34.349 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.349 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:34:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.337952+0000", 2024-02-10T07:34:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2995e-05 2024-02-10T07:34:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:34:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.337960+0000", 2024-02-10T07:34:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0590000000000003e-06 2024-02-10T07:34:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:34:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338020+0000", 2024-02-10T07:34:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0109999999999999e-05 2024-02-10T07:34:34.350 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.351 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.351 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:34:34.351 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338267+0000", 2024-02-10T07:34:34.351 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00024678900000000003 2024-02-10T07:34:34.351 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.351 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.351 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:34:34.351 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338393+0000", 2024-02-10T07:34:34.351 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012653999999999999 2024-02-10T07:34:34.352 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:34.352 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:34.352 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:34.352 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.352 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.352 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:42415 7.4 e114/98 7:2b4bff87:::benchmark_data_smithi008_82046_object42414:head v 114'2573, mlcod=114'2570)", 2024-02-10T07:34:34.352 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:34.338144+0000", 2024-02-10T07:34:34.352 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.00034807300000000001, 2024-02-10T07:34:34.352 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00056429199999999998, 2024-02-10T07:34:34.352 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:34:34.353 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:34:34.353 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:34:34.353 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.353 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:34:34.353 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338144+0000", 2024-02-10T07:34:34.353 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:34:34.353 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.353 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.353 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:34:34.353 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338144+0000", 2024-02-10T07:34:34.353 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:34:34.353 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.354 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.354 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:34:34.354 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338144+0000", 2024-02-10T07:34:34.354 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5799999999999999e-07 2024-02-10T07:34:34.354 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.354 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.354 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:34:34.355 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338168+0000", 2024-02-10T07:34:34.355 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3839999999999999e-05 2024-02-10T07:34:34.355 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.355 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.355 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:34:34.355 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338178+0000", 2024-02-10T07:34:34.355 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6849999999999996e-06 2024-02-10T07:34:34.355 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.355 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.355 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:34:34.355 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338185+0000", 2024-02-10T07:34:34.355 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6510000000000004e-06 2024-02-10T07:34:34.356 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.356 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.356 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:34:34.356 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338624+0000", 2024-02-10T07:34:34.356 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00043890200000000002 2024-02-10T07:34:34.356 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.356 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.356 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:34:34.356 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338643+0000", 2024-02-10T07:34:34.357 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9587000000000001e-05 2024-02-10T07:34:34.357 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:34.357 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:34.357 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:34.357 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.357 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.357 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:42417 7.d e114/98 7:bda3e939:::benchmark_data_smithi008_82046_object42416:head v 114'2576, mlcod=114'2572)", 2024-02-10T07:34:34.357 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:34.338227+0000", 2024-02-10T07:34:34.357 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.00026515700000000002, 2024-02-10T07:34:34.357 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00055076300000000001, 2024-02-10T07:34:34.357 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:34:34.357 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:34:34.358 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:34:34.358 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.358 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:34:34.358 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338227+0000", 2024-02-10T07:34:34.358 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:34:34.358 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.358 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.358 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:34:34.358 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338227+0000", 2024-02-10T07:34:34.358 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:34:34.359 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.359 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.359 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:34:34.359 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338228+0000", 2024-02-10T07:34:34.359 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1699999999999997e-07 2024-02-10T07:34:34.359 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.359 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.360 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:34:34.360 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338258+0000", 2024-02-10T07:34:34.360 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9921e-05 2024-02-10T07:34:34.360 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.360 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.360 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:34:34.361 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338267+0000", 2024-02-10T07:34:34.361 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9229999999999997e-06 2024-02-10T07:34:34.361 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.361 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.362 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:34:34.362 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338274+0000", 2024-02-10T07:34:34.362 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7170000000000002e-06 2024-02-10T07:34:34.362 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.362 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.362 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:34:34.362 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338337+0000", 2024-02-10T07:34:34.362 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2482000000000002e-05 2024-02-10T07:34:34.362 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:34.362 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:34.363 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:34:34.363 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:34.338356+0000", 2024-02-10T07:34:34.363 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9028000000000001e-05 2024-02-10T07:34:34.363 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:34.363 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:34.363 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:34.363 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:34.364 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:34:34.364 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 4 2024-02-10T07:34:34.364 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:34:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:33 smithi170 ceph-mon[33394]: 6.15 scrub starts 2024-02-10T07:34:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:33 smithi170 ceph-mon[33394]: 6.15 scrub ok 2024-02-10T07:34:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:33 smithi170 ceph-mon[33394]: 6.10 scrub starts 2024-02-10T07:34:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:33 smithi170 ceph-mon[33394]: 6.10 scrub ok 2024-02-10T07:34:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:33 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:33 smithi170 ceph-mon[33394]: 6.2 scrub starts 2024-02-10T07:34:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:33 smithi170 ceph-mon[33394]: 6.2 scrub ok 2024-02-10T07:34:34.433 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:34.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:34.465 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:34.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:34.566 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:34.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:34.595 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:34.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:34.697 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:34.739 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:34.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:34.841 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:34.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:34.882 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:34.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:34.983 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:35.012 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:35.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:35.076 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:34:35.114 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:35.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:35.164 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:35.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:35.205 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 25 16 43795 43779 109.432 97.3125 0.00454184 0.00913168 2024-02-10T07:34:35.265 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:35.295 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:35.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:35.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:35.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:34 smithi008 ceph-mon[29816]: 3.12 scrub starts 2024-02-10T07:34:35.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:34 smithi008 ceph-mon[29816]: 3.12 scrub ok 2024-02-10T07:34:35.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:34 smithi008 ceph-mon[29816]: pgmap v96: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.4 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 863 B/s rd, 70 MiB/s wr, 1.11k op/s; 1/119709 objects degraded (0.001%); 409/119709 objects misplaced (0.342%); 10 MiB/s, 164 objects/s recovering 2024-02-10T07:34:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:34 smithi008 ceph-mon[40743]: 3.12 scrub starts 2024-02-10T07:34:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:34 smithi008 ceph-mon[40743]: 3.12 scrub ok 2024-02-10T07:34:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:34 smithi008 ceph-mon[40743]: pgmap v96: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.4 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 863 B/s rd, 70 MiB/s wr, 1.11k op/s; 1/119709 objects degraded (0.001%); 409/119709 objects misplaced (0.342%); 10 MiB/s, 164 objects/s recovering 2024-02-10T07:34:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:34 smithi170 ceph-mon[33394]: 3.12 scrub starts 2024-02-10T07:34:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:34 smithi170 ceph-mon[33394]: 3.12 scrub ok 2024-02-10T07:34:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:34 smithi170 ceph-mon[33394]: pgmap v96: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.4 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 863 B/s rd, 70 MiB/s wr, 1.11k op/s; 1/119709 objects degraded (0.001%); 409/119709 objects misplaced (0.342%); 10 MiB/s, 164 objects/s recovering 2024-02-10T07:34:35.397 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:35.440 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:35.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:35.542 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:35.573 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:35.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:35.675 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:35.706 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:35.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:35.807 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:35.849 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:35.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:35.950 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:36.003 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:36.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:36.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:36.106 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:36.130 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:36.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:36.205 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 26 16 45192 45176 108.581 87.3125 0.00412261 0.0092038 2024-02-10T07:34:36.232 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:36.262 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:36.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:36.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:36.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:35 smithi008 ceph-mon[29816]: 4.1f scrub starts 2024-02-10T07:34:36.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:35 smithi008 ceph-mon[29816]: 4.1f scrub ok 2024-02-10T07:34:36.332 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:36 smithi008 podman[84367]: 2024-02-10 07:34:35.967083871 +0000 UTC m=+1.418586717 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-02-10T07:34:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:35 smithi008 ceph-mon[40743]: 4.1f scrub starts 2024-02-10T07:34:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:35 smithi008 ceph-mon[40743]: 4.1f scrub ok 2024-02-10T07:34:36.364 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:35 smithi170 ceph-mon[33394]: 4.1f scrub starts 2024-02-10T07:34:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:35 smithi170 ceph-mon[33394]: 4.1f scrub ok 2024-02-10T07:34:36.391 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:36.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:36.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:36.493 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:36.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:36.520 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:36.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:36.621 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:36.664 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:36.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:36.766 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:36.790 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:36.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:36.892 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:36.932 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:36.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:36.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:37.034 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:37.075 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:37.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:37.177 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:37.205 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 27 16 46393 46377 107.339 75.0625 0.0128997 0.00930954 2024-02-10T07:34:37.217 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:37.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:37.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:37.319 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:37.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:36 smithi008 ceph-mon[29816]: 4.16 scrub starts 2024-02-10T07:34:37.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:36 smithi008 ceph-mon[29816]: 4.16 scrub ok 2024-02-10T07:34:37.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:36 smithi008 ceph-mon[29816]: pgmap v97: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.4 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s; 1/119709 objects degraded (0.001%); 409/119709 objects misplaced (0.342%); 10 MiB/s, 162 objects/s recovering 2024-02-10T07:34:37.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:36 smithi008 ceph-mon[29816]: 3.a scrub starts 2024-02-10T07:34:37.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:36 smithi008 ceph-mon[29816]: 3.a scrub ok 2024-02-10T07:34:37.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:36 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 1/119709 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:34:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:36 smithi008 ceph-mon[40743]: 4.16 scrub starts 2024-02-10T07:34:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:36 smithi008 ceph-mon[40743]: 4.16 scrub ok 2024-02-10T07:34:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:36 smithi008 ceph-mon[40743]: pgmap v97: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.4 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s; 1/119709 objects degraded (0.001%); 409/119709 objects misplaced (0.342%); 10 MiB/s, 162 objects/s recovering 2024-02-10T07:34:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:36 smithi008 ceph-mon[40743]: 3.a scrub starts 2024-02-10T07:34:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:36 smithi008 ceph-mon[40743]: 3.a scrub ok 2024-02-10T07:34:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:36 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 1/119709 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:34:37.348 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:37.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:36 smithi170 ceph-mon[33394]: 4.16 scrub starts 2024-02-10T07:34:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:36 smithi170 ceph-mon[33394]: 4.16 scrub ok 2024-02-10T07:34:37.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:36 smithi170 ceph-mon[33394]: pgmap v97: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.4 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s; 1/119709 objects degraded (0.001%); 409/119709 objects misplaced (0.342%); 10 MiB/s, 162 objects/s recovering 2024-02-10T07:34:37.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:36 smithi170 ceph-mon[33394]: 3.a scrub starts 2024-02-10T07:34:37.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:36 smithi170 ceph-mon[33394]: 3.a scrub ok 2024-02-10T07:34:37.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:36 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 1/119709 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:34:37.450 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:37.480 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:37.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:37.582 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:37.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:37.607 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:37.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:37.708 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:37.749 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:37.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:37.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:37.831 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:37 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:34:37.420+0000 7f404514a700 -1 mgr.server reply reply (3) No such process no change 2024-02-10T07:34:37.852 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:37.892 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:37.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:37.997 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:38.033 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:38.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:38.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:38.135 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:38.159 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:38.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:38.174 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:37 smithi008 ceph-mon[29816]: 3.16 scrub starts 2024-02-10T07:34:38.175 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:37 smithi008 ceph-mon[29816]: 3.16 scrub ok 2024-02-10T07:34:38.175 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:37 smithi008 ceph-mon[29816]: 3.1a scrub starts 2024-02-10T07:34:38.175 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:37 smithi008 ceph-mon[29816]: 3.1a scrub ok 2024-02-10T07:34:38.175 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:37 smithi008 ceph-mon[40743]: 3.16 scrub starts 2024-02-10T07:34:38.175 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:37 smithi008 ceph-mon[40743]: 3.16 scrub ok 2024-02-10T07:34:38.175 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:37 smithi008 ceph-mon[40743]: 3.1a scrub starts 2024-02-10T07:34:38.175 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:37 smithi008 ceph-mon[40743]: 3.1a scrub ok 2024-02-10T07:34:38.176 INFO:teuthology.orchestra.run.smithi008.stdout:moved 5 / 444 (1.12613%) 2024-02-10T07:34:38.176 INFO:teuthology.orchestra.run.smithi008.stdout:avg 55.5 2024-02-10T07:34:38.176 INFO:teuthology.orchestra.run.smithi008.stdout:stddev 6.8374 -> 6.63325 (expected baseline 6.96868) 2024-02-10T07:34:38.176 INFO:teuthology.orchestra.run.smithi008.stdout:min osd.2 with 44 -> 45 pgs (0.792793 -> 0.810811 * mean) 2024-02-10T07:34:38.176 INFO:teuthology.orchestra.run.smithi008.stdout:max osd.3 with 66 -> 65 pgs (1.18919 -> 1.17117 * mean) 2024-02-10T07:34:38.176 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:34:38.176 INFO:teuthology.orchestra.run.smithi008.stdout:oload 110 2024-02-10T07:34:38.176 INFO:teuthology.orchestra.run.smithi008.stdout:max_change 0.05 2024-02-10T07:34:38.176 INFO:teuthology.orchestra.run.smithi008.stdout:max_change_osds 4 2024-02-10T07:34:38.176 INFO:teuthology.orchestra.run.smithi008.stdout:average_utilization 0.0106 2024-02-10T07:34:38.177 INFO:teuthology.orchestra.run.smithi008.stdout:overload_utilization 0.0117 2024-02-10T07:34:38.177 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3 weight 1.0000 -> 0.9500 2024-02-10T07:34:38.177 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0 weight 1.0000 -> 0.9500 2024-02-10T07:34:38.177 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1 weight 0.9267 -> 0.9433 2024-02-10T07:34:38.177 INFO:teuthology.orchestra.run.smithi008.stderr:no change 2024-02-10T07:34:38.205 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 28 16 47545 47529 106.077 72 0.00210335 0.00935794 2024-02-10T07:34:38.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:37 smithi170 ceph-mon[33394]: 3.16 scrub starts 2024-02-10T07:34:38.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:37 smithi170 ceph-mon[33394]: 3.16 scrub ok 2024-02-10T07:34:38.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:37 smithi170 ceph-mon[33394]: 3.1a scrub starts 2024-02-10T07:34:38.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:37 smithi170 ceph-mon[33394]: 3.1a scrub ok 2024-02-10T07:34:38.261 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:38.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:38.285 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:38.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:38.386 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:38.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:38.412 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:38.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:38.515 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:38.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:38.540 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:38.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:38.641 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:38.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:38.666 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:38.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:38.767 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:38.792 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:38.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:38.893 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:38.919 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:38.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:38.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:39.021 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:39.061 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:39.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:39.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:39.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:34:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:34:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-mon[29816]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-mon[29816]: pgmap v98: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 1/129138 objects degraded (0.001%); 409/129138 objects misplaced (0.317%); 7.7 MiB/s, 122 objects/s recovering 2024-02-10T07:34:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-mon[29816]: 4.a scrub starts 2024-02-10T07:34:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-mon[29816]: 4.a scrub ok 2024-02-10T07:34:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-mon[29816]: osdmap e115: 8 total, 8 up, 8 in 2024-02-10T07:34:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-mon[29816]: 5.18 scrub starts 2024-02-10T07:34:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-mon[29816]: 5.18 scrub ok 2024-02-10T07:34:39.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-mon[40743]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-mon[40743]: pgmap v98: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 1/129138 objects degraded (0.001%); 409/129138 objects misplaced (0.317%); 7.7 MiB/s, 122 objects/s recovering 2024-02-10T07:34:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-mon[40743]: 4.a scrub starts 2024-02-10T07:34:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-mon[40743]: 4.a scrub ok 2024-02-10T07:34:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-mon[40743]: osdmap e115: 8 total, 8 up, 8 in 2024-02-10T07:34:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-mon[40743]: 5.18 scrub starts 2024-02-10T07:34:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:38 smithi008 ceph-mon[40743]: 5.18 scrub ok 2024-02-10T07:34:39.165 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:39.195 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:39.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:39.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:39.206 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 29 16 47858 47842 103.094 19.5625 0.0101372 0.00969429 2024-02-10T07:34:39.297 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:39.336 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:39.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:39.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:38 smithi170 ceph-mon[33394]: from='client.14859 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T07:34:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:38 smithi170 ceph-mon[33394]: pgmap v98: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 1/129138 objects degraded (0.001%); 409/129138 objects misplaced (0.317%); 7.7 MiB/s, 122 objects/s recovering 2024-02-10T07:34:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:38 smithi170 ceph-mon[33394]: 4.a scrub starts 2024-02-10T07:34:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:38 smithi170 ceph-mon[33394]: 4.a scrub ok 2024-02-10T07:34:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:38 smithi170 ceph-mon[33394]: osdmap e115: 8 total, 8 up, 8 in 2024-02-10T07:34:39.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:38 smithi170 ceph-mon[33394]: 5.18 scrub starts 2024-02-10T07:34:39.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:38 smithi170 ceph-mon[33394]: 5.18 scrub ok 2024-02-10T07:34:39.437 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:39.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:39.461 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:39.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:39.562 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:39.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:39.586 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:39.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:39.687 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:39.712 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:39.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:39.813 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:39.841 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:39.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:39.943 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:39.968 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:39.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:40.068 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:40.107 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:40.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:40.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:40.206 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 30 16 48857 48841 101.738 62.4375 0.00247296 0.00982054 2024-02-10T07:34:40.210 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:40.240 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:40.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:40.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:40.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:39 smithi008 ceph-mon[29816]: 5.4 scrub starts 2024-02-10T07:34:40.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:39 smithi008 ceph-mon[29816]: 5.4 scrub ok 2024-02-10T07:34:40.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:39 smithi008 ceph-mon[29816]: 5.1f scrub starts 2024-02-10T07:34:40.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:39 smithi008 ceph-mon[29816]: 5.1f scrub ok 2024-02-10T07:34:40.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:39 smithi008 ceph-mon[29816]: osdmap e116: 8 total, 8 up, 8 in 2024-02-10T07:34:40.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:39 smithi008 ceph-mon[40743]: 5.4 scrub starts 2024-02-10T07:34:40.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:39 smithi008 ceph-mon[40743]: 5.4 scrub ok 2024-02-10T07:34:40.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:39 smithi008 ceph-mon[40743]: 5.1f scrub starts 2024-02-10T07:34:40.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:39 smithi008 ceph-mon[40743]: 5.1f scrub ok 2024-02-10T07:34:40.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:39 smithi008 ceph-mon[40743]: osdmap e116: 8 total, 8 up, 8 in 2024-02-10T07:34:40.343 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:40.376 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:40.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:40.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:39 smithi170 ceph-mon[33394]: 5.4 scrub starts 2024-02-10T07:34:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:39 smithi170 ceph-mon[33394]: 5.4 scrub ok 2024-02-10T07:34:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:39 smithi170 ceph-mon[33394]: 5.1f scrub starts 2024-02-10T07:34:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:39 smithi170 ceph-mon[33394]: 5.1f scrub ok 2024-02-10T07:34:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:39 smithi170 ceph-mon[33394]: osdmap e116: 8 total, 8 up, 8 in 2024-02-10T07:34:40.477 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:40.514 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:34:40.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:40.538 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:40.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:40.555 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-10T07:34:40.639 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:40.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:40.691 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:40.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:40.792 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:40.836 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:40.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:40.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:40.938 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:40.989 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:40.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:41.091 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:41.120 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:41.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:41.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:41.206 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 31 15 50354 50339 101.476 93.625 0.00931559 0.00984935 2024-02-10T07:34:41.222 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:41.262 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:41.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:41.301 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[29816]: 6.6 scrub starts 2024-02-10T07:34:41.301 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[29816]: 6.6 scrub ok 2024-02-10T07:34:41.301 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[29816]: 6.9 scrub starts 2024-02-10T07:34:41.301 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[29816]: 6.9 scrub ok 2024-02-10T07:34:41.302 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[29816]: pgmap v101: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 1/141453 objects degraded (0.001%); 11/141453 objects misplaced (0.008%); 11 MiB/s, 171 objects/s recovering 2024-02-10T07:34:41.302 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[29816]: 5.8 scrub starts 2024-02-10T07:34:41.302 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[29816]: 5.8 scrub ok 2024-02-10T07:34:41.302 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[29816]: 4.2 scrub starts 2024-02-10T07:34:41.302 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[29816]: 4.2 scrub ok 2024-02-10T07:34:41.302 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[40743]: 6.6 scrub starts 2024-02-10T07:34:41.302 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[40743]: 6.6 scrub ok 2024-02-10T07:34:41.302 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[40743]: 6.9 scrub starts 2024-02-10T07:34:41.302 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[40743]: 6.9 scrub ok 2024-02-10T07:34:41.302 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[40743]: pgmap v101: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 1/141453 objects degraded (0.001%); 11/141453 objects misplaced (0.008%); 11 MiB/s, 171 objects/s recovering 2024-02-10T07:34:41.303 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[40743]: 5.8 scrub starts 2024-02-10T07:34:41.303 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[40743]: 5.8 scrub ok 2024-02-10T07:34:41.303 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[40743]: 4.2 scrub starts 2024-02-10T07:34:41.303 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:40 smithi008 ceph-mon[40743]: 4.2 scrub ok 2024-02-10T07:34:41.303 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:41 smithi008 podman[84367]: 2024-02-10 07:34:41.183363663 +0000 UTC m=+6.634866481 volume create 6f52805fe07df4a3c3f4ceec93c4773fd3e35d9ac95517bd57890e8ee651164f 2024-02-10T07:34:41.364 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:40 smithi170 ceph-mon[33394]: 6.6 scrub starts 2024-02-10T07:34:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:40 smithi170 ceph-mon[33394]: 6.6 scrub ok 2024-02-10T07:34:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:40 smithi170 ceph-mon[33394]: 6.9 scrub starts 2024-02-10T07:34:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:40 smithi170 ceph-mon[33394]: 6.9 scrub ok 2024-02-10T07:34:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:40 smithi170 ceph-mon[33394]: pgmap v101: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 1/141453 objects degraded (0.001%); 11/141453 objects misplaced (0.008%); 11 MiB/s, 171 objects/s recovering 2024-02-10T07:34:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:40 smithi170 ceph-mon[33394]: 5.8 scrub starts 2024-02-10T07:34:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:40 smithi170 ceph-mon[33394]: 5.8 scrub ok 2024-02-10T07:34:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:40 smithi170 ceph-mon[33394]: 4.2 scrub starts 2024-02-10T07:34:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:40 smithi170 ceph-mon[33394]: 4.2 scrub ok 2024-02-10T07:34:41.393 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:41.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:41.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:41.495 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:41.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:41.527 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:41.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:41.559 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:41 smithi008 podman[84367]: 2024-02-10 07:34:41.301108373 +0000 UTC m=+6.752611198 container create e3bc4fcf4b68cd4f0371ec05657febd21721e4ee3cef7dd31b8be95ed0fbe53c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:41.628 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:41.658 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:41.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:41.760 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:41.804 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:41.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:41.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:41.906 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:41.949 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:41.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:42.050 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:42.083 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:42.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:42.184 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:42.206 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 32 16 51921 51905 101.364 97.875 0.00606384 0.00985951 2024-02-10T07:34:42.212 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:41 smithi008 ceph-mon[29816]: 4.17 scrub starts 2024-02-10T07:34:42.212 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:41 smithi008 ceph-mon[29816]: 4.17 scrub ok 2024-02-10T07:34:42.212 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:41 smithi008 ceph-mon[29816]: 4.5 scrub starts 2024-02-10T07:34:42.212 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:41 smithi008 ceph-mon[29816]: 4.5 scrub ok 2024-02-10T07:34:42.212 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:41 smithi008 ceph-mon[29816]: 5.6 scrub starts 2024-02-10T07:34:42.212 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:41 smithi008 ceph-mon[29816]: 5.6 scrub ok 2024-02-10T07:34:42.212 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:41 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 1/141453 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:34:42.213 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:41 smithi008 ceph-mon[40743]: 4.17 scrub starts 2024-02-10T07:34:42.213 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:41 smithi008 ceph-mon[40743]: 4.17 scrub ok 2024-02-10T07:34:42.213 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:41 smithi008 ceph-mon[40743]: 4.5 scrub starts 2024-02-10T07:34:42.213 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:41 smithi008 ceph-mon[40743]: 4.5 scrub ok 2024-02-10T07:34:42.213 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:41 smithi008 ceph-mon[40743]: 5.6 scrub starts 2024-02-10T07:34:42.213 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:41 smithi008 ceph-mon[40743]: 5.6 scrub ok 2024-02-10T07:34:42.213 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:41 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 1/141453 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:34:42.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:42.230 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:42.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:42.331 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:42.377 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:42.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:41 smithi170 ceph-mon[33394]: 4.17 scrub starts 2024-02-10T07:34:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:41 smithi170 ceph-mon[33394]: 4.17 scrub ok 2024-02-10T07:34:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:41 smithi170 ceph-mon[33394]: 4.5 scrub starts 2024-02-10T07:34:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:41 smithi170 ceph-mon[33394]: 4.5 scrub ok 2024-02-10T07:34:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:41 smithi170 ceph-mon[33394]: 5.6 scrub starts 2024-02-10T07:34:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:41 smithi170 ceph-mon[33394]: 5.6 scrub ok 2024-02-10T07:34:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:41 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 1/141453 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:34:42.478 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:42.518 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:42.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:42.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:42.582 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:42 smithi008 podman[84367]: 2024-02-10 07:34:42.212360849 +0000 UTC m=+7.663863678 container init e3bc4fcf4b68cd4f0371ec05657febd21721e4ee3cef7dd31b8be95ed0fbe53c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:42.582 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:42 smithi008 podman[84367]: 2024-02-10 07:34:42.221590204 +0000 UTC m=+7.673093027 container start e3bc4fcf4b68cd4f0371ec05657febd21721e4ee3cef7dd31b8be95ed0fbe53c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:42.582 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[84970]: ts=2024-02-10T07:34:42.246Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-10T07:34:42.582 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[84970]: ts=2024-02-10T07:34:42.246Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-10T07:34:42.582 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[84970]: ts=2024-02-10T07:34:42.255Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.8 port=9094 2024-02-10T07:34:42.582 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[84970]: ts=2024-02-10T07:34:42.261Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T07:34:42.582 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[84970]: ts=2024-02-10T07:34:42.308Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T07:34:42.582 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[84970]: ts=2024-02-10T07:34:42.309Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T07:34:42.582 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[84970]: ts=2024-02-10T07:34:42.313Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-10T07:34:42.582 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[84970]: ts=2024-02-10T07:34:42.313Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-10T07:34:42.583 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:42 smithi008 bash[84367]: e3bc4fcf4b68cd4f0371ec05657febd21721e4ee3cef7dd31b8be95ed0fbe53c 2024-02-10T07:34:42.583 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:42 smithi008 systemd[1]: Started Ceph alertmanager.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:34:42.620 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:42.650 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:42.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:42.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:42.752 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:42.794 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:42.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:42.896 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:42.936 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:42.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:42.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:43.038 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:43.075 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:43.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[29816]: 3.6 scrub starts 2024-02-10T07:34:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[29816]: 3.6 scrub ok 2024-02-10T07:34:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[29816]: pgmap v102: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 906 op/s; 1/141453 objects degraded (0.001%); 11/141453 objects misplaced (0.008%); 3.1 MiB/s, 49 objects/s recovering 2024-02-10T07:34:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[29816]: 5.11 scrub starts 2024-02-10T07:34:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[29816]: 5.11 scrub ok 2024-02-10T07:34:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[29816]: 6.14 scrub starts 2024-02-10T07:34:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[29816]: 6.14 scrub ok 2024-02-10T07:34:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[29816]: 5.1 scrub starts 2024-02-10T07:34:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[29816]: 5.1 scrub ok 2024-02-10T07:34:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[40743]: 3.6 scrub starts 2024-02-10T07:34:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[40743]: 3.6 scrub ok 2024-02-10T07:34:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[40743]: pgmap v102: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 906 op/s; 1/141453 objects degraded (0.001%); 11/141453 objects misplaced (0.008%); 3.1 MiB/s, 49 objects/s recovering 2024-02-10T07:34:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[40743]: 5.11 scrub starts 2024-02-10T07:34:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[40743]: 5.11 scrub ok 2024-02-10T07:34:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[40743]: 6.14 scrub starts 2024-02-10T07:34:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[40743]: 6.14 scrub ok 2024-02-10T07:34:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[40743]: 5.1 scrub starts 2024-02-10T07:34:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[40743]: 5.1 scrub ok 2024-02-10T07:34:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:42 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:43.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:34:42 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:34:43.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:42 smithi170 ceph-mon[33394]: 3.6 scrub starts 2024-02-10T07:34:43.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:42 smithi170 ceph-mon[33394]: 3.6 scrub ok 2024-02-10T07:34:43.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:42 smithi170 ceph-mon[33394]: pgmap v102: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 147 active+clean; 2.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 906 op/s; 1/141453 objects degraded (0.001%); 11/141453 objects misplaced (0.008%); 3.1 MiB/s, 49 objects/s recovering 2024-02-10T07:34:43.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:42 smithi170 ceph-mon[33394]: 5.11 scrub starts 2024-02-10T07:34:43.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:42 smithi170 ceph-mon[33394]: 5.11 scrub ok 2024-02-10T07:34:43.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:42 smithi170 ceph-mon[33394]: 6.14 scrub starts 2024-02-10T07:34:43.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:42 smithi170 ceph-mon[33394]: 6.14 scrub ok 2024-02-10T07:34:43.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:42 smithi170 ceph-mon[33394]: 5.1 scrub starts 2024-02-10T07:34:43.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:42 smithi170 ceph-mon[33394]: 5.1 scrub ok 2024-02-10T07:34:43.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:42 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:43.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:42 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:43.178 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:43.206 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 33 16 53313 53297 100.928 87 0.0107928 0.00990236 2024-02-10T07:34:43.215 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:43.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:43.317 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:43.356 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:43.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:43.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:43.368 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:34:43.439 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:34:43.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:43.489 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:43.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:43.592 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:43.627 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:43.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:43.729 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:43.773 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:43.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:43.875 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:43.910 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:43.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:43.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:44.012 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:44.056 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:44.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:44.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:44.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:43 smithi008 ceph-mon[29816]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T07:34:44.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:43 smithi008 ceph-mon[29816]: Reconfiguring daemon prometheus.a on smithi170 2024-02-10T07:34:44.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:43 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:44.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:43 smithi008 ceph-mon[29816]: 3.10 scrub starts 2024-02-10T07:34:44.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:43 smithi008 ceph-mon[29816]: 3.10 scrub ok 2024-02-10T07:34:44.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:43 smithi008 ceph-mon[29816]: 5.f scrub starts 2024-02-10T07:34:44.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:43 smithi008 ceph-mon[29816]: 5.f scrub ok 2024-02-10T07:34:44.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:43 smithi008 ceph-mon[40743]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T07:34:44.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:43 smithi008 ceph-mon[40743]: Reconfiguring daemon prometheus.a on smithi170 2024-02-10T07:34:44.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:43 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:44.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:43 smithi008 ceph-mon[40743]: 3.10 scrub starts 2024-02-10T07:34:44.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:43 smithi008 ceph-mon[40743]: 3.10 scrub ok 2024-02-10T07:34:44.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:43 smithi008 ceph-mon[40743]: 5.f scrub starts 2024-02-10T07:34:44.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:43 smithi008 ceph-mon[40743]: 5.f scrub ok 2024-02-10T07:34:44.101 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:43 smithi170 ceph-mon[33394]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-10T07:34:44.102 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:43 smithi170 ceph-mon[33394]: Reconfiguring daemon prometheus.a on smithi170 2024-02-10T07:34:44.102 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:43 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:44.102 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:43 smithi170 ceph-mon[33394]: 3.10 scrub starts 2024-02-10T07:34:44.102 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:43 smithi170 ceph-mon[33394]: 3.10 scrub ok 2024-02-10T07:34:44.102 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:43 smithi170 ceph-mon[33394]: 5.f scrub starts 2024-02-10T07:34:44.102 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:43 smithi170 ceph-mon[33394]: 5.f scrub ok 2024-02-10T07:34:44.157 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:44.195 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:44.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:44.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:44.206 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 34 16 54804 54788 100.7 93.1875 0.00678751 0.00992509 2024-02-10T07:34:44.297 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:44.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:44.384 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:44.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:44.485 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:44.523 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:44.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:44.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:44.582 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[84970]: ts=2024-02-10T07:34:44.261Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000065581s 2024-02-10T07:34:44.626 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:44.667 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:44.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:44.769 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:44.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:44.808 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:44.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:44.909 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:44.949 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:44.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:44.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:45.051 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:45.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[29816]: 6.1c scrub starts 2024-02-10T07:34:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[29816]: 6.1c scrub ok 2024-02-10T07:34:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[29816]: pgmap v103: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 3.2 MiB/s, 51 objects/s recovering 2024-02-10T07:34:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[29816]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/141453 objects degraded (0.001%), 1 pg degraded) 2024-02-10T07:34:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[29816]: 6.f scrub starts 2024-02-10T07:34:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[29816]: 6.f scrub ok 2024-02-10T07:34:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[29816]: 2.2 scrub starts 2024-02-10T07:34:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[29816]: 2.2 scrub ok 2024-02-10T07:34:45.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[40743]: 6.1c scrub starts 2024-02-10T07:34:45.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[40743]: 6.1c scrub ok 2024-02-10T07:34:45.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[40743]: pgmap v103: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 3.2 MiB/s, 51 objects/s recovering 2024-02-10T07:34:45.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[40743]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/141453 objects degraded (0.001%), 1 pg degraded) 2024-02-10T07:34:45.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[40743]: 6.f scrub starts 2024-02-10T07:34:45.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[40743]: 6.f scrub ok 2024-02-10T07:34:45.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[40743]: 2.2 scrub starts 2024-02-10T07:34:45.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:44 smithi008 ceph-mon[40743]: 2.2 scrub ok 2024-02-10T07:34:45.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:45.084 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:45.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:45.151 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:44 smithi170 ceph-mon[33394]: 6.1c scrub starts 2024-02-10T07:34:45.151 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:44 smithi170 ceph-mon[33394]: 6.1c scrub ok 2024-02-10T07:34:45.151 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:44 smithi170 ceph-mon[33394]: pgmap v103: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 3.2 MiB/s, 51 objects/s recovering 2024-02-10T07:34:45.151 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:44 smithi170 ceph-mon[33394]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/141453 objects degraded (0.001%), 1 pg degraded) 2024-02-10T07:34:45.151 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:44 smithi170 ceph-mon[33394]: 6.f scrub starts 2024-02-10T07:34:45.152 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:44 smithi170 ceph-mon[33394]: 6.f scrub ok 2024-02-10T07:34:45.152 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:44 smithi170 ceph-mon[33394]: 2.2 scrub starts 2024-02-10T07:34:45.152 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:44 smithi170 ceph-mon[33394]: 2.2 scrub ok 2024-02-10T07:34:45.185 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:45.206 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 35 16 56266 56250 100.434 91.375 0.0102094 0.00995184 2024-02-10T07:34:45.218 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:45.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:45.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:45.320 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:45.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:45.352 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:45.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:45.453 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:45.486 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:45.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:45.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:45.588 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:45.616 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:45.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:45.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:45.718 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:45.749 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:45.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:45.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:45.851 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:45.882 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:45.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:45.984 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:46.026 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:46.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:46.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:46.128 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:46.174 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:46.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:46.206 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 36 16 57685 57669 100.107 88.6875 0.00473667 0.00998368 2024-02-10T07:34:46.275 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:46.306 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:45 smithi008 ceph-mon[29816]: 1.0 scrub starts 2024-02-10T07:34:46.306 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:45 smithi008 ceph-mon[29816]: 1.0 scrub ok 2024-02-10T07:34:46.306 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:45 smithi008 ceph-mon[29816]: 6.1f scrub starts 2024-02-10T07:34:46.306 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:45 smithi008 ceph-mon[29816]: 6.1f scrub ok 2024-02-10T07:34:46.307 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:45 smithi008 ceph-mon[40743]: 1.0 scrub starts 2024-02-10T07:34:46.307 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:45 smithi008 ceph-mon[40743]: 1.0 scrub ok 2024-02-10T07:34:46.307 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:45 smithi008 ceph-mon[40743]: 6.1f scrub starts 2024-02-10T07:34:46.307 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:45 smithi008 ceph-mon[40743]: 6.1f scrub ok 2024-02-10T07:34:46.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:46.308 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:46.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:45 smithi170 ceph-mon[33394]: 1.0 scrub starts 2024-02-10T07:34:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:45 smithi170 ceph-mon[33394]: 1.0 scrub ok 2024-02-10T07:34:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:45 smithi170 ceph-mon[33394]: 6.1f scrub starts 2024-02-10T07:34:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:45 smithi170 ceph-mon[33394]: 6.1f scrub ok 2024-02-10T07:34:46.410 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:46.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:46.462 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:46.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:46.563 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:46.593 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:46.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:46.696 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:46.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:46.736 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:46.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:46.838 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:46.867 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:46.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:46.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:46.969 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:47.010 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:47.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:47.111 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:47.144 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:47.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:47.206 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 37 16 58952 58936 99.5416 79.1875 0.00650832 0.0100406 2024-02-10T07:34:47.246 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:47.286 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:47.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:47.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[29816]: 5.5 scrub starts 2024-02-10T07:34:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[29816]: 5.5 scrub ok 2024-02-10T07:34:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[29816]: pgmap v104: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s; 3.2 MiB/s, 51 objects/s recovering 2024-02-10T07:34:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:34:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:34:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[29816]: osdmap e117: 8 total, 8 up, 8 in 2024-02-10T07:34:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[29816]: 4.1c scrub starts 2024-02-10T07:34:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[29816]: 4.1c scrub ok 2024-02-10T07:34:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[29816]: osdmap e118: 8 total, 8 up, 8 in 2024-02-10T07:34:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1984108827' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:34:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:34:47.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:47.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[40743]: 5.5 scrub starts 2024-02-10T07:34:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[40743]: 5.5 scrub ok 2024-02-10T07:34:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[40743]: pgmap v104: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s; 3.2 MiB/s, 51 objects/s recovering 2024-02-10T07:34:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:34:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:34:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[40743]: osdmap e117: 8 total, 8 up, 8 in 2024-02-10T07:34:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[40743]: 4.1c scrub starts 2024-02-10T07:34:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[40743]: 4.1c scrub ok 2024-02-10T07:34:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[40743]: osdmap e118: 8 total, 8 up, 8 in 2024-02-10T07:34:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1984108827' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:34:47.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:34:47.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:47.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:46 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:47.383 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 systemd[1]: Stopping Ceph prometheus.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:34:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:46 smithi170 ceph-mon[33394]: 5.5 scrub starts 2024-02-10T07:34:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:46 smithi170 ceph-mon[33394]: 5.5 scrub ok 2024-02-10T07:34:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:46 smithi170 ceph-mon[33394]: pgmap v104: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s; 3.2 MiB/s, 51 objects/s recovering 2024-02-10T07:34:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:46 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:34:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:46 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:34:47.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:46 smithi170 ceph-mon[33394]: osdmap e117: 8 total, 8 up, 8 in 2024-02-10T07:34:47.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:46 smithi170 ceph-mon[33394]: 4.1c scrub starts 2024-02-10T07:34:47.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:46 smithi170 ceph-mon[33394]: 4.1c scrub ok 2024-02-10T07:34:47.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:46 smithi170 ceph-mon[33394]: osdmap e118: 8 total, 8 up, 8 in 2024-02-10T07:34:47.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:46 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1984108827' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:34:47.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:46 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:34:47.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:46 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:47.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:46 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:34:47.387 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:47.417 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:47.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:47.519 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:47.554 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:47.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:47.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:47.656 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:47.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:47.682 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:47.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:47.692 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:34:47.783 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:47.815 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:47.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:47.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:47.835 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T07:34:47.835 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:34:47.836 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-10T07:34:47.836 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-10T07:34:47.837 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[62705]: ts=2024-02-10T07:34:47.501Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T07:34:47.837 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[62705]: ts=2024-02-10T07:34:47.501Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-10T07:34:47.837 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[62705]: ts=2024-02-10T07:34:47.501Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-10T07:34:47.838 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[62705]: ts=2024-02-10T07:34:47.501Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T07:34:47.838 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[62705]: ts=2024-02-10T07:34:47.501Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-10T07:34:47.838 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[62705]: ts=2024-02-10T07:34:47.501Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T07:34:47.838 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[62705]: ts=2024-02-10T07:34:47.501Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-10T07:34:47.838 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[62705]: ts=2024-02-10T07:34:47.501Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-10T07:34:47.838 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[62705]: ts=2024-02-10T07:34:47.501Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-10T07:34:47.839 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[62705]: ts=2024-02-10T07:34:47.502Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T07:34:47.839 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[62705]: ts=2024-02-10T07:34:47.502Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-10T07:34:47.839 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[62705]: ts=2024-02-10T07:34:47.502Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-10T07:34:47.839 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 podman[70250]: 2024-02-10 07:34:47.511670736 +0000 UTC m=+0.150104402 container died 8ec86518b472b4d3665b0bdc836a361473181a4b122efb27157b8fb804267f8b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:47.917 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:47.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:47.969 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:47.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:48.070 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:48.100 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:48.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:48.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:48.136 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 podman[70250]: 2024-02-10 07:34:47.943229869 +0000 UTC m=+0.581663529 container cleanup 8ec86518b472b4d3665b0bdc836a361473181a4b122efb27157b8fb804267f8b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:48.136 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:47 smithi170 bash[70250]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T07:34:48.136 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-mon[33394]: 4.f scrub starts 2024-02-10T07:34:48.136 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-mon[33394]: 4.f scrub ok 2024-02-10T07:34:48.136 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-mon[33394]: 4.19 scrub starts 2024-02-10T07:34:48.137 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-mon[33394]: 4.19 scrub ok 2024-02-10T07:34:48.137 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-mon[33394]: 6.0 scrub starts 2024-02-10T07:34:48.137 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-mon[33394]: 6.0 scrub ok 2024-02-10T07:34:48.137 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-mon[33394]: 3.13 scrub starts 2024-02-10T07:34:48.137 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-mon[33394]: 3.13 scrub ok 2024-02-10T07:34:48.137 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-mon[33394]: 5.2 scrub starts 2024-02-10T07:34:48.137 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-mon[33394]: 5.2 scrub ok 2024-02-10T07:34:48.137 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:34:48.137 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:34:48.137 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:34:48.137 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:47 smithi170 ceph-mon[33394]: osdmap e119: 8 total, 8 up, 8 in 2024-02-10T07:34:48.202 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:48.207 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 38 15 60265 60250 99.0825 82.125 0.0134059 0.0100879 2024-02-10T07:34:48.235 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:48.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:48.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:48.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[29816]: 4.f scrub starts 2024-02-10T07:34:48.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[29816]: 4.f scrub ok 2024-02-10T07:34:48.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[29816]: 4.19 scrub starts 2024-02-10T07:34:48.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[29816]: 4.19 scrub ok 2024-02-10T07:34:48.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[29816]: 6.0 scrub starts 2024-02-10T07:34:48.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[29816]: 6.0 scrub ok 2024-02-10T07:34:48.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[29816]: 3.13 scrub starts 2024-02-10T07:34:48.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[29816]: 3.13 scrub ok 2024-02-10T07:34:48.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[29816]: 5.2 scrub starts 2024-02-10T07:34:48.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[29816]: 5.2 scrub ok 2024-02-10T07:34:48.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:34:48.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:34:48.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:34:48.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[29816]: osdmap e119: 8 total, 8 up, 8 in 2024-02-10T07:34:48.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[40743]: 4.f scrub starts 2024-02-10T07:34:48.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[40743]: 4.f scrub ok 2024-02-10T07:34:48.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[40743]: 4.19 scrub starts 2024-02-10T07:34:48.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[40743]: 4.19 scrub ok 2024-02-10T07:34:48.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[40743]: 6.0 scrub starts 2024-02-10T07:34:48.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[40743]: 6.0 scrub ok 2024-02-10T07:34:48.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[40743]: 3.13 scrub starts 2024-02-10T07:34:48.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[40743]: 3.13 scrub ok 2024-02-10T07:34:48.337 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[40743]: 5.2 scrub starts 2024-02-10T07:34:48.337 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[40743]: 5.2 scrub ok 2024-02-10T07:34:48.337 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:34:48.337 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:34:48.337 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:34:48.337 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:47 smithi008 ceph-mon[40743]: osdmap e119: 8 total, 8 up, 8 in 2024-02-10T07:34:48.337 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:48.386 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:48.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:48.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:48.487 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:48.489 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:48 smithi170 podman[70277]: 2024-02-10 07:34:48.203606097 +0000 UTC m=+0.689680208 container remove 8ec86518b472b4d3665b0bdc836a361473181a4b122efb27157b8fb804267f8b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:48.544 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:48.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:48.645 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:48.685 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:48.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:48.758 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:48 smithi170 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-10T07:34:48.758 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:48 smithi170 systemd[1]: Stopped Ceph prometheus.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:34:48.758 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:48 smithi170 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@prometheus.a.service: Consumed 1.299s CPU time. 2024-02-10T07:34:48.758 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:48 smithi170 systemd[1]: Starting Ceph prometheus.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:34:48.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:48.817 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:48.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:48.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:48.919 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:48.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:48.949 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:48.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:49.049 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:49.050 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:48 smithi170 podman[70448]: 2024-02-10 07:34:48.666041829 +0000 UTC m=+0.027114102 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-02-10T07:34:49.050 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:48 smithi170 podman[70448]: 2024-02-10 07:34:48.796543754 +0000 UTC m=+0.157616070 container create 78459537e7c8487a980f395a10ee3dec052a7412d81f3b409456d11d156888e0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:49.051 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:48 smithi170 ceph-mon[33394]: 4.11 scrub starts 2024-02-10T07:34:49.051 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:48 smithi170 ceph-mon[33394]: 4.11 scrub ok 2024-02-10T07:34:49.051 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:48 smithi170 ceph-mon[33394]: pgmap v108: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.45k op/s; 128 KiB/s, 1 objects/s recovering 2024-02-10T07:34:49.051 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:48 smithi170 ceph-mon[33394]: 4.12 scrub starts 2024-02-10T07:34:49.051 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:48 smithi170 ceph-mon[33394]: 4.12 scrub ok 2024-02-10T07:34:49.089 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:49.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:49.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:49.190 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:49.207 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 39 16 61630 61614 98.7276 85.25 0.0140898 0.0101231 2024-02-10T07:34:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:49.242 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:49.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:49.301 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:48 smithi008 ceph-mon[29816]: 4.11 scrub starts 2024-02-10T07:34:49.301 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:48 smithi008 ceph-mon[29816]: 4.11 scrub ok 2024-02-10T07:34:49.301 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:48 smithi008 ceph-mon[29816]: pgmap v108: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.45k op/s; 128 KiB/s, 1 objects/s recovering 2024-02-10T07:34:49.301 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:48 smithi008 ceph-mon[29816]: 4.12 scrub starts 2024-02-10T07:34:49.301 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:48 smithi008 ceph-mon[29816]: 4.12 scrub ok 2024-02-10T07:34:49.302 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:48 smithi008 ceph-mon[40743]: 4.11 scrub starts 2024-02-10T07:34:49.302 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:48 smithi008 ceph-mon[40743]: 4.11 scrub ok 2024-02-10T07:34:49.302 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:48 smithi008 ceph-mon[40743]: pgmap v108: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.45k op/s; 128 KiB/s, 1 objects/s recovering 2024-02-10T07:34:49.302 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:48 smithi008 ceph-mon[40743]: 4.12 scrub starts 2024-02-10T07:34:49.302 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:48 smithi008 ceph-mon[40743]: 4.12 scrub ok 2024-02-10T07:34:49.344 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:49.385 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:49.386 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 podman[70448]: 2024-02-10 07:34:49.049913152 +0000 UTC m=+0.410985429 container init 78459537e7c8487a980f395a10ee3dec052a7412d81f3b409456d11d156888e0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:49.386 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 podman[70448]: 2024-02-10 07:34:49.053974767 +0000 UTC m=+0.415047040 container start 78459537e7c8487a980f395a10ee3dec052a7412d81f3b409456d11d156888e0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:34:49.387 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.095Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-02-10T07:34:49.387 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.095Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-02-10T07:34:49.388 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.095Z caller=main.go:570 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi170 (none))" 2024-02-10T07:34:49.388 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.095Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T07:34:49.388 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.095Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T07:34:49.388 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.097Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T07:34:49.388 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.098Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-02-10T07:34:49.388 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.099Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-02-10T07:34:49.389 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.099Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-02-10T07:34:49.389 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.105Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T07:34:49.389 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.105Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.93µs 2024-02-10T07:34:49.389 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.105Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T07:34:49.389 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.105Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-02-10T07:34:49.389 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.106Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-02-10T07:34:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:49.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:49.391 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.106Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=32.479µs wal_replay_duration=1.291365ms wbl_replay_duration=142ns total_replay_duration=1.395602ms 2024-02-10T07:34:49.392 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.107Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T07:34:49.392 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.107Z caller=main.go:1029 level=info msg="TSDB started" 2024-02-10T07:34:49.392 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.107Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T07:34:49.392 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.134Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=27.309264ms db_storage=1.112µs remote_storage=1.44µs web_handler=412ns query_engine=922ns scrape=512.584µs scrape_sd=90.207µs notify=9.758µs notify_sd=8.489µs rules=26.383943ms tracing=8.331µs 2024-02-10T07:34:49.392 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.135Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-02-10T07:34:49.392 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:34:49.135Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-02-10T07:34:49.393 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 bash[70448]: 78459537e7c8487a980f395a10ee3dec052a7412d81f3b409456d11d156888e0 2024-02-10T07:34:49.393 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:34:49 smithi170 systemd[1]: Started Ceph prometheus.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:34:49.492 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:49.539 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:49.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:49.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:49.579 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Bus STOPPING 2024-02-10T07:34:49.579 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:34:49.579 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Bus STOPPED 2024-02-10T07:34:49.579 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Bus STARTING 2024-02-10T07:34:49.579 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Serving on http://:::9283 2024-02-10T07:34:49.579 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Bus STARTED 2024-02-10T07:34:49.580 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Bus STOPPING 2024-02-10T07:34:49.580 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:34:49.580 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Bus STOPPED 2024-02-10T07:34:49.580 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Bus STARTING 2024-02-10T07:34:49.580 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Serving on http://:::9283 2024-02-10T07:34:49.580 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Bus STARTED 2024-02-10T07:34:49.581 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Bus STOPPING 2024-02-10T07:34:49.581 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:34:49.581 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Bus STOPPED 2024-02-10T07:34:49.581 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Bus STARTING 2024-02-10T07:34:49.643 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:49.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:49.694 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:49.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:49.795 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:49.827 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Serving on http://:::9283 2024-02-10T07:34:49.827 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:34:49] ENGINE Bus STARTED 2024-02-10T07:34:49.829 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:49.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:49.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:49.931 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:49.966 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:49.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:49.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:50.068 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:50.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: 3.0 scrub starts 2024-02-10T07:34:50.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: 3.0 scrub ok 2024-02-10T07:34:50.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:34:50.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T07:34:50.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:50.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T07:34:50.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: 3.1d deep-scrub starts 2024-02-10T07:34:50.084 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:34:50.084 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: 3.1d deep-scrub ok 2024-02-10T07:34:50.084 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi170.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T07:34:50.084 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.084 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:50.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: 3.0 scrub starts 2024-02-10T07:34:50.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: 3.0 scrub ok 2024-02-10T07:34:50.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:34:50.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T07:34:50.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:50.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T07:34:50.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: 3.1d deep-scrub starts 2024-02-10T07:34:50.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:34:50.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: 3.1d deep-scrub ok 2024-02-10T07:34:50.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi170.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T07:34:50.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:50.102 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:50.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:50.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:50.205 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:50.210 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:34:50.210321+0000 min lat: 0.00141219 max lat: 2.36731 avg lat: 0.0101912 2024-02-10T07:34:50.210 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:34:50.210 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 40 16 62786 62770 98.0593 72.25 0.0263989 0.0101912 2024-02-10T07:34:50.237 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:34:50.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:50.270 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:50.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:50.371 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: 3.0 scrub starts 2024-02-10T07:34:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: 3.0 scrub ok 2024-02-10T07:34:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:34:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T07:34:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T07:34:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: 3.1d deep-scrub starts 2024-02-10T07:34:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:34:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: 3.1d deep-scrub ok 2024-02-10T07:34:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi170.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T07:34:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:34:50.387 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:34:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:50.423 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:50.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:50.436 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:34:50.491 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:34:50.524 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:50.542 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:34:50.548 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:34:50.548 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:34:50.558 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:50.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:50.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:50.592 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:34:50.641 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:34:50.659 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:50.685 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:34:50.703 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:50.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:50.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:50.730 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:34:50.736 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:34:50.736 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:34:50.736 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:34:50.736 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:34:50.736 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:34:50.736 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:34:50.736 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:34:50.736 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:34:50.806 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:50.848 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:50.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:50.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:50.950 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:50.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:50.984 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:50.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:51.085 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:51.122 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:51.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:51.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:50 smithi170 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:34:51.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:50 smithi170 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T07:34:51.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:50 smithi170 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:51.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:50 smithi170 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T07:34:51.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:50 smithi170 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:34:51.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:50 smithi170 ceph-mon[33394]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi170.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T07:34:51.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:50 smithi170 ceph-mon[33394]: pgmap v109: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-10T07:34:51.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:50 smithi170 ceph-mon[33394]: 5.b scrub starts 2024-02-10T07:34:51.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:50 smithi170 ceph-mon[33394]: 5.b scrub ok 2024-02-10T07:34:51.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:50 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1357005308' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:34:51.210 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 41 16 63996 63980 97.512 75.625 0.0107737 0.01025 2024-02-10T07:34:51.224 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:51.260 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:51.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:51.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[29816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:34:51.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[29816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T07:34:51.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[29816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:51.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[29816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T07:34:51.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[29816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:34:51.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[29816]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi170.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T07:34:51.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[29816]: pgmap v109: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-10T07:34:51.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[29816]: 5.b scrub starts 2024-02-10T07:34:51.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[29816]: 5.b scrub ok 2024-02-10T07:34:51.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1357005308' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:34:51.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[40743]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-10T07:34:51.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[40743]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi008.front.sepia.ceph.com:9093"}]: dispatch 2024-02-10T07:34:51.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[40743]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-10T07:34:51.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[40743]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-02-10T07:34:51.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[40743]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-10T07:34:51.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[40743]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi170.front.sepia.ceph.com:3000"}]: dispatch 2024-02-10T07:34:51.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[40743]: pgmap v109: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-10T07:34:51.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[40743]: 5.b scrub starts 2024-02-10T07:34:51.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[40743]: 5.b scrub ok 2024-02-10T07:34:51.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:50 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1357005308' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:34:51.363 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:51.399 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:51.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:51.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:51.460 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:34:51.502 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:51.530 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:51.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:51.633 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:51.682 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:51.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:51.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:51.784 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:51.838 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:51.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:51.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:51.874 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-10T07:34:51.939 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:51.969 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:51.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:52.071 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:52.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:51 smithi008 ceph-mon[40743]: 6.1e scrub starts 2024-02-10T07:34:52.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:51 smithi008 ceph-mon[40743]: 6.1e scrub ok 2024-02-10T07:34:52.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:51 smithi008 ceph-mon[40743]: osdmap e120: 8 total, 8 up, 8 in 2024-02-10T07:34:52.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:51 smithi008 ceph-mon[29816]: 6.1e scrub starts 2024-02-10T07:34:52.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:51 smithi008 ceph-mon[29816]: 6.1e scrub ok 2024-02-10T07:34:52.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:51 smithi008 ceph-mon[29816]: osdmap e120: 8 total, 8 up, 8 in 2024-02-10T07:34:52.103 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:52.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:52.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:52.205 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:52.210 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 42 16 65246 65230 97.0492 78.125 0.0183108 0.0102984 2024-02-10T07:34:52.234 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:52.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:52.336 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:52.372 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:52.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:52.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:51 smithi170 ceph-mon[33394]: 6.1e scrub starts 2024-02-10T07:34:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:51 smithi170 ceph-mon[33394]: 6.1e scrub ok 2024-02-10T07:34:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:51 smithi170 ceph-mon[33394]: osdmap e120: 8 total, 8 up, 8 in 2024-02-10T07:34:52.475 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:52.506 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:52.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:52.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:52.581 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[84970]: ts=2024-02-10T07:34:52.262Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001860251s 2024-02-10T07:34:52.608 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:52.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:52.665 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:52.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:52.766 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:52.804 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:52.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:52.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:34:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:34:52.906 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:52.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:52.945 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:52.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:53.047 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:53.077 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:53.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:53.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:53.179 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:53.210 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 43 16 66576 66560 96.725 83.125 0.0135862 0.010333 2024-02-10T07:34:53.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:53.221 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:53.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:53.322 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[29816]: 4.1e scrub starts 2024-02-10T07:34:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[29816]: 4.1e scrub ok 2024-02-10T07:34:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[29816]: 6.5 scrub starts 2024-02-10T07:34:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[29816]: 6.5 scrub ok 2024-02-10T07:34:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[29816]: pgmap v111: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 891 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-10T07:34:53.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[29816]: 2.1 scrub starts 2024-02-10T07:34:53.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[29816]: 2.1 scrub ok 2024-02-10T07:34:53.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[29816]: 4.13 deep-scrub starts 2024-02-10T07:34:53.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[29816]: 4.13 deep-scrub ok 2024-02-10T07:34:53.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[40743]: 4.1e scrub starts 2024-02-10T07:34:53.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[40743]: 4.1e scrub ok 2024-02-10T07:34:53.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[40743]: 6.5 scrub starts 2024-02-10T07:34:53.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[40743]: 6.5 scrub ok 2024-02-10T07:34:53.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[40743]: pgmap v111: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 891 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-10T07:34:53.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[40743]: 2.1 scrub starts 2024-02-10T07:34:53.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[40743]: 2.1 scrub ok 2024-02-10T07:34:53.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[40743]: 4.13 deep-scrub starts 2024-02-10T07:34:53.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:52 smithi008 ceph-mon[40743]: 4.13 deep-scrub ok 2024-02-10T07:34:53.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:53.357 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:53.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:52 smithi170 ceph-mon[33394]: 4.1e scrub starts 2024-02-10T07:34:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:52 smithi170 ceph-mon[33394]: 4.1e scrub ok 2024-02-10T07:34:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:52 smithi170 ceph-mon[33394]: 6.5 scrub starts 2024-02-10T07:34:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:52 smithi170 ceph-mon[33394]: 6.5 scrub ok 2024-02-10T07:34:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:52 smithi170 ceph-mon[33394]: pgmap v111: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 891 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-10T07:34:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:52 smithi170 ceph-mon[33394]: 2.1 scrub starts 2024-02-10T07:34:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:52 smithi170 ceph-mon[33394]: 2.1 scrub ok 2024-02-10T07:34:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:52 smithi170 ceph-mon[33394]: 4.13 deep-scrub starts 2024-02-10T07:34:53.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:52 smithi170 ceph-mon[33394]: 4.13 deep-scrub ok 2024-02-10T07:34:53.458 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:53.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:53.501 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:53.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:53.602 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:53.685 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:53.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:53.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:53.786 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:53.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:53.824 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:53.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:53.925 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:53.965 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:34:53.971 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:53.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:53.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:54.006 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:34:54.074 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:54.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:54.120 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:54.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:54.121 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:53 smithi008 ceph-mon[29816]: 3.11 scrub starts 2024-02-10T07:34:54.121 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:53 smithi008 ceph-mon[29816]: 3.11 scrub ok 2024-02-10T07:34:54.121 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:53 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:54.122 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:53 smithi008 ceph-mon[29816]: 5.a scrub starts 2024-02-10T07:34:54.122 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:54 smithi008 ceph-mon[29816]: 5.a scrub ok 2024-02-10T07:34:54.122 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:54 smithi008 ceph-mon[40743]: 3.11 scrub starts 2024-02-10T07:34:54.122 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:54 smithi008 ceph-mon[40743]: 3.11 scrub ok 2024-02-10T07:34:54.122 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:54 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:54.122 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:54 smithi008 ceph-mon[40743]: 5.a scrub starts 2024-02-10T07:34:54.122 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:54 smithi008 ceph-mon[40743]: 5.a scrub ok 2024-02-10T07:34:54.210 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 44 16 67782 67766 96.2397 75.375 0.00783536 0.0103851 2024-02-10T07:34:54.222 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:54.253 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:54.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:54.355 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:54 smithi170 ceph-mon[33394]: 3.11 scrub starts 2024-02-10T07:34:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:54 smithi170 ceph-mon[33394]: 3.11 scrub ok 2024-02-10T07:34:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:54 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:34:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:54 smithi170 ceph-mon[33394]: 5.a scrub starts 2024-02-10T07:34:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:54 smithi170 ceph-mon[33394]: 5.a scrub ok 2024-02-10T07:34:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:54.405 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:54.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:54.506 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:54.538 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:54.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:54.640 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:54.694 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:54.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:54.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:54.797 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:54.828 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:54.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:54.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:54.930 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:54.985 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:54.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:55.008 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:34:55.080 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:55 smithi008 ceph-mon[29816]: pgmap v112: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 119 MiB/s wr, 1.90k op/s 2024-02-10T07:34:55.080 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:55 smithi008 ceph-mon[29816]: 6.4 deep-scrub starts 2024-02-10T07:34:55.080 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:55 smithi008 ceph-mon[29816]: 6.4 deep-scrub ok 2024-02-10T07:34:55.080 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:55 smithi008 ceph-mon[40743]: pgmap v112: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 119 MiB/s wr, 1.90k op/s 2024-02-10T07:34:55.080 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:55 smithi008 ceph-mon[40743]: 6.4 deep-scrub starts 2024-02-10T07:34:55.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:55 smithi008 ceph-mon[40743]: 6.4 deep-scrub ok 2024-02-10T07:34:55.087 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:55.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:55.116 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:55.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:55.212 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 45 16 68997 68981 95.7843 75.9375 0.0110222 0.0104335 2024-02-10T07:34:55.217 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:55.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:55.245 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:55.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:55.346 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:55 smithi170 ceph-mon[33394]: pgmap v112: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 119 MiB/s wr, 1.90k op/s 2024-02-10T07:34:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:55 smithi170 ceph-mon[33394]: 6.4 deep-scrub starts 2024-02-10T07:34:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:55 smithi170 ceph-mon[33394]: 6.4 deep-scrub ok 2024-02-10T07:34:55.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:55.413 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:55.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:55.514 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:55.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:55.555 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:55.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:34:55.656 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:55.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:55.689 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:55.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:55.693 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:34:55.694 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:34:55.694 INFO:teuthology.orchestra.run.smithi008.stdout: "epoch": 120, 2024-02-10T07:34:55.694 INFO:teuthology.orchestra.run.smithi008.stdout: "fsid": "8c8244e6-c7e5-11ee-95b8-87774f69a715", 2024-02-10T07:34:55.695 INFO:teuthology.orchestra.run.smithi008.stdout: "created": "2024-02-10T07:26:21.316239+0000", 2024-02-10T07:34:55.695 INFO:teuthology.orchestra.run.smithi008.stdout: "modified": "2024-02-10T07:34:51.675847+0000", 2024-02-10T07:34:55.695 INFO:teuthology.orchestra.run.smithi008.stdout: "last_up_change": "2024-02-10T07:31:36.599653+0000", 2024-02-10T07:34:55.695 INFO:teuthology.orchestra.run.smithi008.stdout: "last_in_change": "2024-02-10T07:31:19.283633+0000", 2024-02-10T07:34:55.695 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-10T07:34:55.696 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_num": 5799936, 2024-02-10T07:34:55.696 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_set": [ 2024-02-10T07:34:55.696 INFO:teuthology.orchestra.run.smithi008.stdout: "pglog_hardlimit", 2024-02-10T07:34:55.696 INFO:teuthology.orchestra.run.smithi008.stdout: "purged_snapdirs", 2024-02-10T07:34:55.696 INFO:teuthology.orchestra.run.smithi008.stdout: "recovery_deletes", 2024-02-10T07:34:55.696 INFO:teuthology.orchestra.run.smithi008.stdout: "sortbitwise" 2024-02-10T07:34:55.697 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:34:55.697 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_version": 18, 2024-02-10T07:34:55.697 INFO:teuthology.orchestra.run.smithi008.stdout: "full_ratio": 0.94999998807907104, 2024-02-10T07:34:55.697 INFO:teuthology.orchestra.run.smithi008.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-10T07:34:55.697 INFO:teuthology.orchestra.run.smithi008.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-10T07:34:55.698 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_snapshot": "", 2024-02-10T07:34:55.698 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_max": 7, 2024-02-10T07:34:55.698 INFO:teuthology.orchestra.run.smithi008.stdout: "max_osd": 8, 2024-02-10T07:34:55.698 INFO:teuthology.orchestra.run.smithi008.stdout: "require_min_compat_client": "luminous", 2024-02-10T07:34:55.699 INFO:teuthology.orchestra.run.smithi008.stdout: "min_compat_client": "luminous", 2024-02-10T07:34:55.699 INFO:teuthology.orchestra.run.smithi008.stdout: "require_osd_release": "squid", 2024-02-10T07:34:55.699 INFO:teuthology.orchestra.run.smithi008.stdout: "allow_crimson": false, 2024-02-10T07:34:55.699 INFO:teuthology.orchestra.run.smithi008.stdout: "pools": [ 2024-02-10T07:34:55.699 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.699 INFO:teuthology.orchestra.run.smithi008.stdout: "pool": 1, 2024-02-10T07:34:55.700 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_name": ".mgr", 2024-02-10T07:34:55.700 INFO:teuthology.orchestra.run.smithi008.stdout: "create_time": "2024-02-10T07:29:32.704872+0000", 2024-02-10T07:34:55.700 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": 1, 2024-02-10T07:34:55.700 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_names": "hashpspool", 2024-02-10T07:34:55.700 INFO:teuthology.orchestra.run.smithi008.stdout: "type": 1, 2024-02-10T07:34:55.700 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 3, 2024-02-10T07:34:55.701 INFO:teuthology.orchestra.run.smithi008.stdout: "min_size": 2, 2024-02-10T07:34:55.701 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_rule": 0, 2024-02-10T07:34:55.701 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_count": 0, 2024-02-10T07:34:55.701 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_target": 0, 2024-02-10T07:34:55.701 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T07:34:55.701 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T07:34:55.701 INFO:teuthology.orchestra.run.smithi008.stdout: "object_hash": 2, 2024-02-10T07:34:55.702 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_mode": "off", 2024-02-10T07:34:55.703 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num": 1, 2024-02-10T07:34:55.703 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num": 1, 2024-02-10T07:34:55.703 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num_target": 1, 2024-02-10T07:34:55.703 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_target": 1, 2024-02-10T07:34:55.704 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_pending": 1, 2024-02-10T07:34:55.704 INFO:teuthology.orchestra.run.smithi008.stdout: "last_pg_merge_meta": { 2024-02-10T07:34:55.704 INFO:teuthology.orchestra.run.smithi008.stdout: "source_pgid": "0.0", 2024-02-10T07:34:55.704 INFO:teuthology.orchestra.run.smithi008.stdout: "ready_epoch": 0, 2024-02-10T07:34:55.704 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_started": 0, 2024-02-10T07:34:55.704 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_clean": 0, 2024-02-10T07:34:55.705 INFO:teuthology.orchestra.run.smithi008.stdout: "source_version": "0'0", 2024-02-10T07:34:55.705 INFO:teuthology.orchestra.run.smithi008.stdout: "target_version": "0'0" 2024-02-10T07:34:55.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.705 INFO:teuthology.orchestra.run.smithi008.stdout: "last_change": "21", 2024-02-10T07:34:55.705 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend": "0", 2024-02-10T07:34:55.705 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T07:34:55.706 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T07:34:55.706 INFO:teuthology.orchestra.run.smithi008.stdout: "auid": 0, 2024-02-10T07:34:55.706 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_mode": "selfmanaged", 2024-02-10T07:34:55.706 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_seq": 0, 2024-02-10T07:34:55.706 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_epoch": 0, 2024-02-10T07:34:55.706 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_snaps": [], 2024-02-10T07:34:55.707 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps": "[]", 2024-02-10T07:34:55.707 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_bytes": 0, 2024-02-10T07:34:55.707 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_objects": 0, 2024-02-10T07:34:55.708 INFO:teuthology.orchestra.run.smithi008.stdout: "tiers": [], 2024-02-10T07:34:55.708 INFO:teuthology.orchestra.run.smithi008.stdout: "tier_of": -1, 2024-02-10T07:34:55.708 INFO:teuthology.orchestra.run.smithi008.stdout: "read_tier": -1, 2024-02-10T07:34:55.708 INFO:teuthology.orchestra.run.smithi008.stdout: "write_tier": -1, 2024-02-10T07:34:55.708 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_mode": "none", 2024-02-10T07:34:55.708 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_bytes": 0, 2024-02-10T07:34:55.709 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_objects": 0, 2024-02-10T07:34:55.709 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T07:34:55.709 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T07:34:55.709 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T07:34:55.709 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_flush_age": 0, 2024-02-10T07:34:55.709 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_evict_age": 0, 2024-02-10T07:34:55.709 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profile": "", 2024-02-10T07:34:55.710 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_params": { 2024-02-10T07:34:55.710 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "none" 2024-02-10T07:34:55.710 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.710 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_period": 0, 2024-02-10T07:34:55.710 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_count": 0, 2024-02-10T07:34:55.710 INFO:teuthology.orchestra.run.smithi008.stdout: "use_gmt_hitset": true, 2024-02-10T07:34:55.711 INFO:teuthology.orchestra.run.smithi008.stdout: "min_read_recency_for_promote": 0, 2024-02-10T07:34:55.711 INFO:teuthology.orchestra.run.smithi008.stdout: "min_write_recency_for_promote": 0, 2024-02-10T07:34:55.711 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T07:34:55.711 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_search_last_n": 0, 2024-02-10T07:34:55.711 INFO:teuthology.orchestra.run.smithi008.stdout: "grade_table": [], 2024-02-10T07:34:55.711 INFO:teuthology.orchestra.run.smithi008.stdout: "stripe_width": 0, 2024-02-10T07:34:55.712 INFO:teuthology.orchestra.run.smithi008.stdout: "expected_num_objects": 0, 2024-02-10T07:34:55.712 INFO:teuthology.orchestra.run.smithi008.stdout: "fast_read": false, 2024-02-10T07:34:55.712 INFO:teuthology.orchestra.run.smithi008.stdout: "options": { 2024-02-10T07:34:55.712 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_max": 32, 2024-02-10T07:34:55.712 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_min": 1 2024-02-10T07:34:55.713 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.713 INFO:teuthology.orchestra.run.smithi008.stdout: "application_metadata": { 2024-02-10T07:34:55.713 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": {} 2024-02-10T07:34:55.713 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.713 INFO:teuthology.orchestra.run.smithi008.stdout: "read_balance": { 2024-02-10T07:34:55.713 INFO:teuthology.orchestra.run.smithi008.stdout: "score_type": "Fair distribution", 2024-02-10T07:34:55.713 INFO:teuthology.orchestra.run.smithi008.stdout: "score_acting": 7.8899998664855957, 2024-02-10T07:34:55.714 INFO:teuthology.orchestra.run.smithi008.stdout: "score_stable": 7.8899998664855957, 2024-02-10T07:34:55.714 INFO:teuthology.orchestra.run.smithi008.stdout: "optimal_score": 0.37999999523162842, 2024-02-10T07:34:55.714 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_acting": 3, 2024-02-10T07:34:55.714 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_stable": 3, 2024-02-10T07:34:55.714 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity_weighted": 1, 2024-02-10T07:34:55.714 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity": 1, 2024-02-10T07:34:55.715 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity_weighted": 1 2024-02-10T07:34:55.715 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.715 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.715 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.715 INFO:teuthology.orchestra.run.smithi008.stdout: "pool": 2, 2024-02-10T07:34:55.715 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_name": "datapool", 2024-02-10T07:34:55.715 INFO:teuthology.orchestra.run.smithi008.stdout: "create_time": "2024-02-10T07:31:49.150741+0000", 2024-02-10T07:34:55.715 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": 8193, 2024-02-10T07:34:55.716 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-10T07:34:55.716 INFO:teuthology.orchestra.run.smithi008.stdout: "type": 1, 2024-02-10T07:34:55.716 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 3, 2024-02-10T07:34:55.716 INFO:teuthology.orchestra.run.smithi008.stdout: "min_size": 2, 2024-02-10T07:34:55.716 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_rule": 0, 2024-02-10T07:34:55.716 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_count": 0, 2024-02-10T07:34:55.717 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_target": 0, 2024-02-10T07:34:55.717 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T07:34:55.717 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T07:34:55.717 INFO:teuthology.orchestra.run.smithi008.stdout: "object_hash": 2, 2024-02-10T07:34:55.717 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_mode": "off", 2024-02-10T07:34:55.717 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num": 3, 2024-02-10T07:34:55.718 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num": 3, 2024-02-10T07:34:55.718 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num_target": 3, 2024-02-10T07:34:55.718 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_target": 3, 2024-02-10T07:34:55.718 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_pending": 3, 2024-02-10T07:34:55.718 INFO:teuthology.orchestra.run.smithi008.stdout: "last_pg_merge_meta": { 2024-02-10T07:34:55.718 INFO:teuthology.orchestra.run.smithi008.stdout: "source_pgid": "0.0", 2024-02-10T07:34:55.719 INFO:teuthology.orchestra.run.smithi008.stdout: "ready_epoch": 0, 2024-02-10T07:34:55.719 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_started": 0, 2024-02-10T07:34:55.719 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_clean": 0, 2024-02-10T07:34:55.719 INFO:teuthology.orchestra.run.smithi008.stdout: "source_version": "0'0", 2024-02-10T07:34:55.719 INFO:teuthology.orchestra.run.smithi008.stdout: "target_version": "0'0" 2024-02-10T07:34:55.719 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.719 INFO:teuthology.orchestra.run.smithi008.stdout: "last_change": "70", 2024-02-10T07:34:55.720 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend": "0", 2024-02-10T07:34:55.720 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T07:34:55.720 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T07:34:55.720 INFO:teuthology.orchestra.run.smithi008.stdout: "auid": 0, 2024-02-10T07:34:55.720 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_mode": "selfmanaged", 2024-02-10T07:34:55.720 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_seq": 2, 2024-02-10T07:34:55.721 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_epoch": 70, 2024-02-10T07:34:55.721 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_snaps": [], 2024-02-10T07:34:55.721 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps": "[]", 2024-02-10T07:34:55.721 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_bytes": 0, 2024-02-10T07:34:55.721 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_objects": 0, 2024-02-10T07:34:55.721 INFO:teuthology.orchestra.run.smithi008.stdout: "tiers": [], 2024-02-10T07:34:55.721 INFO:teuthology.orchestra.run.smithi008.stdout: "tier_of": -1, 2024-02-10T07:34:55.722 INFO:teuthology.orchestra.run.smithi008.stdout: "read_tier": -1, 2024-02-10T07:34:55.722 INFO:teuthology.orchestra.run.smithi008.stdout: "write_tier": -1, 2024-02-10T07:34:55.722 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_mode": "none", 2024-02-10T07:34:55.722 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_bytes": 0, 2024-02-10T07:34:55.722 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_objects": 0, 2024-02-10T07:34:55.722 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T07:34:55.723 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T07:34:55.723 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T07:34:55.723 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_flush_age": 0, 2024-02-10T07:34:55.723 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_evict_age": 0, 2024-02-10T07:34:55.724 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profile": "", 2024-02-10T07:34:55.724 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_params": { 2024-02-10T07:34:55.724 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "none" 2024-02-10T07:34:55.724 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.724 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_period": 0, 2024-02-10T07:34:55.724 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_count": 0, 2024-02-10T07:34:55.724 INFO:teuthology.orchestra.run.smithi008.stdout: "use_gmt_hitset": true, 2024-02-10T07:34:55.725 INFO:teuthology.orchestra.run.smithi008.stdout: "min_read_recency_for_promote": 0, 2024-02-10T07:34:55.725 INFO:teuthology.orchestra.run.smithi008.stdout: "min_write_recency_for_promote": 0, 2024-02-10T07:34:55.725 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T07:34:55.725 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_search_last_n": 0, 2024-02-10T07:34:55.725 INFO:teuthology.orchestra.run.smithi008.stdout: "grade_table": [], 2024-02-10T07:34:55.725 INFO:teuthology.orchestra.run.smithi008.stdout: "stripe_width": 0, 2024-02-10T07:34:55.725 INFO:teuthology.orchestra.run.smithi008.stdout: "expected_num_objects": 0, 2024-02-10T07:34:55.725 INFO:teuthology.orchestra.run.smithi008.stdout: "fast_read": false, 2024-02-10T07:34:55.725 INFO:teuthology.orchestra.run.smithi008.stdout: "options": {}, 2024-02-10T07:34:55.725 INFO:teuthology.orchestra.run.smithi008.stdout: "application_metadata": { 2024-02-10T07:34:55.725 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd": {} 2024-02-10T07:34:55.725 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: "read_balance": { 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: "score_type": "Fair distribution", 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: "score_acting": 2.6500000953674316, 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: "score_stable": 2.6500000953674316, 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: "optimal_score": 0.87999999523162842, 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_acting": 2.3299999237060547, 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_stable": 2.3299999237060547, 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity_weighted": 1, 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity": 1, 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity_weighted": 1 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: "pool": 3, 2024-02-10T07:34:55.726 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_name": ".rgw.root", 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "create_time": "2024-02-10T07:31:49.831683+0000", 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": 1, 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_names": "hashpspool", 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "type": 1, 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 3, 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "min_size": 2, 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_rule": 0, 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_count": 0, 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_target": 0, 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "object_hash": 2, 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_mode": "off", 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num": 32, 2024-02-10T07:34:55.727 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num": 32, 2024-02-10T07:34:55.728 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num_target": 32, 2024-02-10T07:34:55.728 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_target": 32, 2024-02-10T07:34:55.728 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_pending": 32, 2024-02-10T07:34:55.728 INFO:teuthology.orchestra.run.smithi008.stdout: "last_pg_merge_meta": { 2024-02-10T07:34:55.728 INFO:teuthology.orchestra.run.smithi008.stdout: "source_pgid": "0.0", 2024-02-10T07:34:55.728 INFO:teuthology.orchestra.run.smithi008.stdout: "ready_epoch": 0, 2024-02-10T07:34:55.728 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_started": 0, 2024-02-10T07:34:55.728 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_clean": 0, 2024-02-10T07:34:55.728 INFO:teuthology.orchestra.run.smithi008.stdout: "source_version": "0'0", 2024-02-10T07:34:55.728 INFO:teuthology.orchestra.run.smithi008.stdout: "target_version": "0'0" 2024-02-10T07:34:55.728 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.728 INFO:teuthology.orchestra.run.smithi008.stdout: "last_change": "68", 2024-02-10T07:34:55.730 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend": "0", 2024-02-10T07:34:55.730 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T07:34:55.730 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T07:34:55.730 INFO:teuthology.orchestra.run.smithi008.stdout: "auid": 0, 2024-02-10T07:34:55.730 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_mode": "selfmanaged", 2024-02-10T07:34:55.730 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_seq": 0, 2024-02-10T07:34:55.730 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_epoch": 0, 2024-02-10T07:34:55.730 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_snaps": [], 2024-02-10T07:34:55.731 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps": "[]", 2024-02-10T07:34:55.731 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_bytes": 0, 2024-02-10T07:34:55.731 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_objects": 0, 2024-02-10T07:34:55.731 INFO:teuthology.orchestra.run.smithi008.stdout: "tiers": [], 2024-02-10T07:34:55.731 INFO:teuthology.orchestra.run.smithi008.stdout: "tier_of": -1, 2024-02-10T07:34:55.731 INFO:teuthology.orchestra.run.smithi008.stdout: "read_tier": -1, 2024-02-10T07:34:55.731 INFO:teuthology.orchestra.run.smithi008.stdout: "write_tier": -1, 2024-02-10T07:34:55.731 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_mode": "none", 2024-02-10T07:34:55.731 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_bytes": 0, 2024-02-10T07:34:55.732 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_objects": 0, 2024-02-10T07:34:55.732 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T07:34:55.732 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T07:34:55.732 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T07:34:55.732 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_flush_age": 0, 2024-02-10T07:34:55.732 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_evict_age": 0, 2024-02-10T07:34:55.732 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profile": "", 2024-02-10T07:34:55.732 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_params": { 2024-02-10T07:34:55.733 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "none" 2024-02-10T07:34:55.733 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.733 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_period": 0, 2024-02-10T07:34:55.733 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_count": 0, 2024-02-10T07:34:55.733 INFO:teuthology.orchestra.run.smithi008.stdout: "use_gmt_hitset": true, 2024-02-10T07:34:55.733 INFO:teuthology.orchestra.run.smithi008.stdout: "min_read_recency_for_promote": 0, 2024-02-10T07:34:55.733 INFO:teuthology.orchestra.run.smithi008.stdout: "min_write_recency_for_promote": 0, 2024-02-10T07:34:55.733 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T07:34:55.733 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_search_last_n": 0, 2024-02-10T07:34:55.733 INFO:teuthology.orchestra.run.smithi008.stdout: "grade_table": [], 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: "stripe_width": 0, 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: "expected_num_objects": 0, 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: "fast_read": false, 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: "options": {}, 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: "application_metadata": { 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw": {} 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: "read_balance": { 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: "score_type": "Fair distribution", 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: "score_acting": 1.5, 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: "score_stable": 1.5, 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: "optimal_score": 1, 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_acting": 1.5, 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_stable": 1.5, 2024-02-10T07:34:55.734 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity_weighted": 1, 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity": 1, 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity_weighted": 1 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: "pool": 4, 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_name": "default.rgw.log", 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: "create_time": "2024-02-10T07:31:51.622544+0000", 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": 1, 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_names": "hashpspool", 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: "type": 1, 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 3, 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: "min_size": 2, 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_rule": 0, 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_count": 0, 2024-02-10T07:34:55.735 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_target": 0, 2024-02-10T07:34:55.736 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T07:34:55.736 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T07:34:55.736 INFO:teuthology.orchestra.run.smithi008.stdout: "object_hash": 2, 2024-02-10T07:34:55.736 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_mode": "off", 2024-02-10T07:34:55.736 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num": 32, 2024-02-10T07:34:55.736 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num": 32, 2024-02-10T07:34:55.736 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num_target": 32, 2024-02-10T07:34:55.736 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_target": 32, 2024-02-10T07:34:55.736 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_pending": 32, 2024-02-10T07:34:55.736 INFO:teuthology.orchestra.run.smithi008.stdout: "last_pg_merge_meta": { 2024-02-10T07:34:55.736 INFO:teuthology.orchestra.run.smithi008.stdout: "source_pgid": "0.0", 2024-02-10T07:34:55.736 INFO:teuthology.orchestra.run.smithi008.stdout: "ready_epoch": 0, 2024-02-10T07:34:55.736 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_started": 0, 2024-02-10T07:34:55.737 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_clean": 0, 2024-02-10T07:34:55.737 INFO:teuthology.orchestra.run.smithi008.stdout: "source_version": "0'0", 2024-02-10T07:34:55.737 INFO:teuthology.orchestra.run.smithi008.stdout: "target_version": "0'0" 2024-02-10T07:34:55.737 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.737 INFO:teuthology.orchestra.run.smithi008.stdout: "last_change": "70", 2024-02-10T07:34:55.737 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend": "0", 2024-02-10T07:34:55.737 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T07:34:55.738 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T07:34:55.738 INFO:teuthology.orchestra.run.smithi008.stdout: "auid": 0, 2024-02-10T07:34:55.738 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_mode": "selfmanaged", 2024-02-10T07:34:55.738 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_seq": 0, 2024-02-10T07:34:55.738 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_epoch": 0, 2024-02-10T07:34:55.738 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_snaps": [], 2024-02-10T07:34:55.738 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps": "[]", 2024-02-10T07:34:55.738 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_bytes": 0, 2024-02-10T07:34:55.738 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_objects": 0, 2024-02-10T07:34:55.738 INFO:teuthology.orchestra.run.smithi008.stdout: "tiers": [], 2024-02-10T07:34:55.738 INFO:teuthology.orchestra.run.smithi008.stdout: "tier_of": -1, 2024-02-10T07:34:55.738 INFO:teuthology.orchestra.run.smithi008.stdout: "read_tier": -1, 2024-02-10T07:34:55.738 INFO:teuthology.orchestra.run.smithi008.stdout: "write_tier": -1, 2024-02-10T07:34:55.739 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_mode": "none", 2024-02-10T07:34:55.739 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_bytes": 0, 2024-02-10T07:34:55.739 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_objects": 0, 2024-02-10T07:34:55.739 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T07:34:55.739 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T07:34:55.739 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T07:34:55.739 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_flush_age": 0, 2024-02-10T07:34:55.739 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_evict_age": 0, 2024-02-10T07:34:55.739 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profile": "", 2024-02-10T07:34:55.739 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_params": { 2024-02-10T07:34:55.740 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "none" 2024-02-10T07:34:55.740 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.740 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_period": 0, 2024-02-10T07:34:55.740 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_count": 0, 2024-02-10T07:34:55.740 INFO:teuthology.orchestra.run.smithi008.stdout: "use_gmt_hitset": true, 2024-02-10T07:34:55.740 INFO:teuthology.orchestra.run.smithi008.stdout: "min_read_recency_for_promote": 0, 2024-02-10T07:34:55.741 INFO:teuthology.orchestra.run.smithi008.stdout: "min_write_recency_for_promote": 0, 2024-02-10T07:34:55.741 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T07:34:55.741 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_search_last_n": 0, 2024-02-10T07:34:55.741 INFO:teuthology.orchestra.run.smithi008.stdout: "grade_table": [], 2024-02-10T07:34:55.741 INFO:teuthology.orchestra.run.smithi008.stdout: "stripe_width": 0, 2024-02-10T07:34:55.741 INFO:teuthology.orchestra.run.smithi008.stdout: "expected_num_objects": 0, 2024-02-10T07:34:55.741 INFO:teuthology.orchestra.run.smithi008.stdout: "fast_read": false, 2024-02-10T07:34:55.742 INFO:teuthology.orchestra.run.smithi008.stdout: "options": {}, 2024-02-10T07:34:55.742 INFO:teuthology.orchestra.run.smithi008.stdout: "application_metadata": { 2024-02-10T07:34:55.742 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw": {} 2024-02-10T07:34:55.742 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.742 INFO:teuthology.orchestra.run.smithi008.stdout: "read_balance": { 2024-02-10T07:34:55.742 INFO:teuthology.orchestra.run.smithi008.stdout: "score_type": "Fair distribution", 2024-02-10T07:34:55.743 INFO:teuthology.orchestra.run.smithi008.stdout: "score_acting": 2.25, 2024-02-10T07:34:55.743 INFO:teuthology.orchestra.run.smithi008.stdout: "score_stable": 2.25, 2024-02-10T07:34:55.743 INFO:teuthology.orchestra.run.smithi008.stdout: "optimal_score": 1, 2024-02-10T07:34:55.743 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_acting": 2.25, 2024-02-10T07:34:55.743 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_stable": 2.25, 2024-02-10T07:34:55.743 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity_weighted": 1, 2024-02-10T07:34:55.744 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity": 1, 2024-02-10T07:34:55.744 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity_weighted": 1 2024-02-10T07:34:55.744 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.744 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.744 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.744 INFO:teuthology.orchestra.run.smithi008.stdout: "pool": 5, 2024-02-10T07:34:55.745 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_name": "default.rgw.control", 2024-02-10T07:34:55.745 INFO:teuthology.orchestra.run.smithi008.stdout: "create_time": "2024-02-10T07:31:53.550827+0000", 2024-02-10T07:34:55.745 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": 1, 2024-02-10T07:34:55.745 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_names": "hashpspool", 2024-02-10T07:34:55.745 INFO:teuthology.orchestra.run.smithi008.stdout: "type": 1, 2024-02-10T07:34:55.746 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 3, 2024-02-10T07:34:55.746 INFO:teuthology.orchestra.run.smithi008.stdout: "min_size": 2, 2024-02-10T07:34:55.746 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_rule": 0, 2024-02-10T07:34:55.746 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_count": 0, 2024-02-10T07:34:55.746 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_target": 0, 2024-02-10T07:34:55.746 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T07:34:55.747 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T07:34:55.747 INFO:teuthology.orchestra.run.smithi008.stdout: "object_hash": 2, 2024-02-10T07:34:55.747 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_mode": "off", 2024-02-10T07:34:55.747 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num": 32, 2024-02-10T07:34:55.747 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num": 32, 2024-02-10T07:34:55.747 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num_target": 32, 2024-02-10T07:34:55.748 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_target": 32, 2024-02-10T07:34:55.748 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_pending": 32, 2024-02-10T07:34:55.748 INFO:teuthology.orchestra.run.smithi008.stdout: "last_pg_merge_meta": { 2024-02-10T07:34:55.748 INFO:teuthology.orchestra.run.smithi008.stdout: "source_pgid": "0.0", 2024-02-10T07:34:55.748 INFO:teuthology.orchestra.run.smithi008.stdout: "ready_epoch": 0, 2024-02-10T07:34:55.748 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_started": 0, 2024-02-10T07:34:55.748 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_clean": 0, 2024-02-10T07:34:55.749 INFO:teuthology.orchestra.run.smithi008.stdout: "source_version": "0'0", 2024-02-10T07:34:55.749 INFO:teuthology.orchestra.run.smithi008.stdout: "target_version": "0'0" 2024-02-10T07:34:55.749 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.749 INFO:teuthology.orchestra.run.smithi008.stdout: "last_change": "72", 2024-02-10T07:34:55.749 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend": "0", 2024-02-10T07:34:55.749 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T07:34:55.749 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T07:34:55.750 INFO:teuthology.orchestra.run.smithi008.stdout: "auid": 0, 2024-02-10T07:34:55.750 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_mode": "selfmanaged", 2024-02-10T07:34:55.750 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_seq": 0, 2024-02-10T07:34:55.750 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_epoch": 0, 2024-02-10T07:34:55.750 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_snaps": [], 2024-02-10T07:34:55.750 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps": "[]", 2024-02-10T07:34:55.750 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_bytes": 0, 2024-02-10T07:34:55.750 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_objects": 0, 2024-02-10T07:34:55.751 INFO:teuthology.orchestra.run.smithi008.stdout: "tiers": [], 2024-02-10T07:34:55.751 INFO:teuthology.orchestra.run.smithi008.stdout: "tier_of": -1, 2024-02-10T07:34:55.751 INFO:teuthology.orchestra.run.smithi008.stdout: "read_tier": -1, 2024-02-10T07:34:55.751 INFO:teuthology.orchestra.run.smithi008.stdout: "write_tier": -1, 2024-02-10T07:34:55.751 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_mode": "none", 2024-02-10T07:34:55.751 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_bytes": 0, 2024-02-10T07:34:55.752 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_objects": 0, 2024-02-10T07:34:55.752 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T07:34:55.752 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T07:34:55.752 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T07:34:55.752 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_flush_age": 0, 2024-02-10T07:34:55.752 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_evict_age": 0, 2024-02-10T07:34:55.753 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profile": "", 2024-02-10T07:34:55.753 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_params": { 2024-02-10T07:34:55.753 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "none" 2024-02-10T07:34:55.753 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.753 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_period": 0, 2024-02-10T07:34:55.753 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_count": 0, 2024-02-10T07:34:55.753 INFO:teuthology.orchestra.run.smithi008.stdout: "use_gmt_hitset": true, 2024-02-10T07:34:55.754 INFO:teuthology.orchestra.run.smithi008.stdout: "min_read_recency_for_promote": 0, 2024-02-10T07:34:55.754 INFO:teuthology.orchestra.run.smithi008.stdout: "min_write_recency_for_promote": 0, 2024-02-10T07:34:55.754 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T07:34:55.754 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_search_last_n": 0, 2024-02-10T07:34:55.754 INFO:teuthology.orchestra.run.smithi008.stdout: "grade_table": [], 2024-02-10T07:34:55.754 INFO:teuthology.orchestra.run.smithi008.stdout: "stripe_width": 0, 2024-02-10T07:34:55.755 INFO:teuthology.orchestra.run.smithi008.stdout: "expected_num_objects": 0, 2024-02-10T07:34:55.755 INFO:teuthology.orchestra.run.smithi008.stdout: "fast_read": false, 2024-02-10T07:34:55.755 INFO:teuthology.orchestra.run.smithi008.stdout: "options": {}, 2024-02-10T07:34:55.755 INFO:teuthology.orchestra.run.smithi008.stdout: "application_metadata": { 2024-02-10T07:34:55.755 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw": {} 2024-02-10T07:34:55.755 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.756 INFO:teuthology.orchestra.run.smithi008.stdout: "read_balance": { 2024-02-10T07:34:55.756 INFO:teuthology.orchestra.run.smithi008.stdout: "score_type": "Fair distribution", 2024-02-10T07:34:55.756 INFO:teuthology.orchestra.run.smithi008.stdout: "score_acting": 1.25, 2024-02-10T07:34:55.756 INFO:teuthology.orchestra.run.smithi008.stdout: "score_stable": 1.25, 2024-02-10T07:34:55.756 INFO:teuthology.orchestra.run.smithi008.stdout: "optimal_score": 1, 2024-02-10T07:34:55.756 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_acting": 1.25, 2024-02-10T07:34:55.756 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_stable": 1.25, 2024-02-10T07:34:55.757 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity_weighted": 1, 2024-02-10T07:34:55.757 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity": 1, 2024-02-10T07:34:55.757 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity_weighted": 1 2024-02-10T07:34:55.757 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.757 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.757 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.758 INFO:teuthology.orchestra.run.smithi008.stdout: "pool": 6, 2024-02-10T07:34:55.758 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_name": "default.rgw.meta", 2024-02-10T07:34:55.758 INFO:teuthology.orchestra.run.smithi008.stdout: "create_time": "2024-02-10T07:31:55.600117+0000", 2024-02-10T07:34:55.758 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": 1, 2024-02-10T07:34:55.759 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_names": "hashpspool", 2024-02-10T07:34:55.759 INFO:teuthology.orchestra.run.smithi008.stdout: "type": 1, 2024-02-10T07:34:55.759 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 3, 2024-02-10T07:34:55.760 INFO:teuthology.orchestra.run.smithi008.stdout: "min_size": 2, 2024-02-10T07:34:55.760 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_rule": 0, 2024-02-10T07:34:55.760 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_count": 0, 2024-02-10T07:34:55.760 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_target": 0, 2024-02-10T07:34:55.760 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T07:34:55.760 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T07:34:55.760 INFO:teuthology.orchestra.run.smithi008.stdout: "object_hash": 2, 2024-02-10T07:34:55.760 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_mode": "off", 2024-02-10T07:34:55.760 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num": 32, 2024-02-10T07:34:55.760 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num": 32, 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num_target": 32, 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_target": 32, 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_pending": 32, 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: "last_pg_merge_meta": { 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: "source_pgid": "0.0", 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: "ready_epoch": 0, 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_started": 0, 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_clean": 0, 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: "source_version": "0'0", 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: "target_version": "0'0" 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: "last_change": "74", 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend": "0", 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T07:34:55.761 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "auid": 0, 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_mode": "selfmanaged", 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_seq": 0, 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_epoch": 0, 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_snaps": [], 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps": "[]", 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_bytes": 0, 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_objects": 0, 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "tiers": [], 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "tier_of": -1, 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "read_tier": -1, 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "write_tier": -1, 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_mode": "none", 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_bytes": 0, 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_objects": 0, 2024-02-10T07:34:55.762 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_flush_age": 0, 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_evict_age": 0, 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profile": "", 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_params": { 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "none" 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_period": 0, 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_count": 0, 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "use_gmt_hitset": true, 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "min_read_recency_for_promote": 0, 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "min_write_recency_for_promote": 0, 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_search_last_n": 0, 2024-02-10T07:34:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: "grade_table": [], 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: "stripe_width": 0, 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: "expected_num_objects": 0, 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: "fast_read": false, 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: "options": { 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_bias": 4 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: "application_metadata": { 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw": {} 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: "read_balance": { 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: "score_type": "Fair distribution", 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: "score_acting": 1.75, 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: "score_stable": 1.75, 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: "optimal_score": 1, 2024-02-10T07:34:55.764 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_acting": 1.75, 2024-02-10T07:34:55.765 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_stable": 1.75, 2024-02-10T07:34:55.765 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity_weighted": 1, 2024-02-10T07:34:55.765 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity": 1, 2024-02-10T07:34:55.765 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity_weighted": 1 2024-02-10T07:34:55.765 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.765 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.765 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.765 INFO:teuthology.orchestra.run.smithi008.stdout: "pool": 7, 2024-02-10T07:34:55.765 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_name": "unique_pool_0", 2024-02-10T07:34:55.765 INFO:teuthology.orchestra.run.smithi008.stdout: "create_time": "2024-02-10T07:33:52.310148+0000", 2024-02-10T07:34:55.765 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": 1, 2024-02-10T07:34:55.765 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_names": "hashpspool", 2024-02-10T07:34:55.765 INFO:teuthology.orchestra.run.smithi008.stdout: "type": 1, 2024-02-10T07:34:55.765 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 3, 2024-02-10T07:34:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "min_size": 2, 2024-02-10T07:34:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_rule": 0, 2024-02-10T07:34:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_count": 0, 2024-02-10T07:34:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_target": 0, 2024-02-10T07:34:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T07:34:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T07:34:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "object_hash": 2, 2024-02-10T07:34:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_mode": "off", 2024-02-10T07:34:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num": 16, 2024-02-10T07:34:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num": 16, 2024-02-10T07:34:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num_target": 16, 2024-02-10T07:34:55.766 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_target": 16, 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_pending": 16, 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "last_pg_merge_meta": { 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "source_pgid": "0.0", 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "ready_epoch": 0, 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_started": 0, 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_clean": 0, 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "source_version": "0'0", 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "target_version": "0'0" 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "last_change": "104", 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend": "0", 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "auid": 0, 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_mode": "selfmanaged", 2024-02-10T07:34:55.767 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_seq": 0, 2024-02-10T07:34:55.768 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_epoch": 0, 2024-02-10T07:34:55.768 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_snaps": [], 2024-02-10T07:34:55.768 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps": "[]", 2024-02-10T07:34:55.768 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_bytes": 0, 2024-02-10T07:34:55.768 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_objects": 0, 2024-02-10T07:34:55.768 INFO:teuthology.orchestra.run.smithi008.stdout: "tiers": [], 2024-02-10T07:34:55.768 INFO:teuthology.orchestra.run.smithi008.stdout: "tier_of": -1, 2024-02-10T07:34:55.768 INFO:teuthology.orchestra.run.smithi008.stdout: "read_tier": -1, 2024-02-10T07:34:55.768 INFO:teuthology.orchestra.run.smithi008.stdout: "write_tier": -1, 2024-02-10T07:34:55.768 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_mode": "none", 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_bytes": 0, 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_objects": 0, 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_flush_age": 0, 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_evict_age": 0, 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profile": "", 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_params": { 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "none" 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_period": 0, 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_count": 0, 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: "use_gmt_hitset": true, 2024-02-10T07:34:55.769 INFO:teuthology.orchestra.run.smithi008.stdout: "min_read_recency_for_promote": 0, 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "min_write_recency_for_promote": 0, 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_search_last_n": 0, 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "grade_table": [], 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "stripe_width": 0, 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "expected_num_objects": 0, 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "fast_read": false, 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "options": {}, 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "application_metadata": { 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "rados": {} 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "read_balance": { 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "score_type": "Fair distribution", 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "score_acting": 2, 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "score_stable": 2, 2024-02-10T07:34:55.770 INFO:teuthology.orchestra.run.smithi008.stdout: "optimal_score": 1, 2024-02-10T07:34:55.771 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_acting": 2, 2024-02-10T07:34:55.771 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_stable": 2, 2024-02-10T07:34:55.771 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity_weighted": 1, 2024-02-10T07:34:55.771 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity": 1, 2024-02-10T07:34:55.771 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity_weighted": 1 2024-02-10T07:34:55.771 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.771 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.771 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:34:55.771 INFO:teuthology.orchestra.run.smithi008.stdout: "osds": [ 2024-02-10T07:34:55.771 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.771 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 0, 2024-02-10T07:34:55.771 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "63abc4f1-1c83-4e03-8592-959c51d10847", 2024-02-10T07:34:55.771 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:34:55.771 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:34:55.772 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 1, 2024-02-10T07:34:55.772 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:34:55.772 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:34:55.772 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:34:55.772 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 8, 2024-02-10T07:34:55.772 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 98, 2024-02-10T07:34:55.772 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 0, 2024-02-10T07:34:55.772 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:34:55.772 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:34:55.772 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.772 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.773 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.773 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6802", 2024-02-10T07:34:55.773 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:34:55.773 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.773 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.773 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.773 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6803", 2024-02-10T07:34:55.773 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:34:55.773 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.773 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.773 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.773 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:34:55.773 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.773 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6804", 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6805", 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.774 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6808", 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6809", 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6806", 2024-02-10T07:34:55.775 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:34:55.776 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.776 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.776 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.776 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6807", 2024-02-10T07:34:55.776 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:34:55.776 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.776 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.776 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.776 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.8:6803/1442795455", 2024-02-10T07:34:55.776 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.8:6805/1442795455", 2024-02-10T07:34:55.776 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.8:6809/1442795455", 2024-02-10T07:34:55.776 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.8:6807/1442795455", 2024-02-10T07:34:55.776 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:34:55.776 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 1, 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "88307389-bfa8-402e-92b9-9fa5094262c4", 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 0.926666259765625, 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 13, 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 98, 2024-02-10T07:34:55.777 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 0, 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6810", 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6811", 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.778 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6812", 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6813", 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:34:55.779 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6816", 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6817", 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.780 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6814", 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6815", 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.8:6811/3278758893", 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.8:6813/3278758893", 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.8:6817/3278758893", 2024-02-10T07:34:55.781 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.8:6815/3278758893", 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 2, 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "c6120003-9ab5-4cf5-87ac-6351ed2c6226", 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 1, 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 18, 2024-02-10T07:34:55.782 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 98, 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 0, 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6818", 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6819", 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.783 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6820", 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6821", 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.784 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6824", 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6825", 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:34:55.785 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6822", 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6823", 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.8:6819/481314802", 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.8:6821/481314802", 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.8:6825/481314802", 2024-02-10T07:34:55.786 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.8:6823/481314802", 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 3, 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "6ca8f6f3-013a-48cb-9083-be37c1767489", 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 1, 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 26, 2024-02-10T07:34:55.787 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 115, 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 0, 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6826", 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6827", 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.788 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6828", 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6829", 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.789 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6832", 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6833", 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.790 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6830", 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6831", 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.8:6827/3692331651", 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.8:6829/3692331651", 2024-02-10T07:34:55.791 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.8:6833/3692331651", 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.8:6831/3692331651", 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 4, 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "aa667766-e57f-447b-9a31-e3f3e61ac346", 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 1, 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:34:55.792 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:34:55.793 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 32, 2024-02-10T07:34:55.793 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 98, 2024-02-10T07:34:55.793 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 0, 2024-02-10T07:34:55.793 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:34:55.793 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:34:55.793 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:55.794 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.794 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6800", 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6801", 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.795 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6802", 2024-02-10T07:34:55.796 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:34:55.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.796 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.796 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6803", 2024-02-10T07:34:55.796 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:34:55.796 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.796 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.796 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:34:55.796 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.796 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.796 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6806", 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6807", 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6804", 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:34:55.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6805", 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.170:6801/3997132504", 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.170:6803/3997132504", 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.170:6807/3997132504", 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.170:6805/3997132504", 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:34:55.798 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.799 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 5, 2024-02-10T07:34:55.799 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "2e9ab14d-1aff-4638-9f41-311f6b5f6770", 2024-02-10T07:34:55.799 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:34:55.799 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:34:55.799 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 1, 2024-02-10T07:34:55.799 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:34:55.799 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:34:55.799 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:34:55.799 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 40, 2024-02-10T07:34:55.799 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 112, 2024-02-10T07:34:55.799 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 0, 2024-02-10T07:34:55.799 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:34:55.800 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:34:55.800 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.800 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.800 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6808", 2024-02-10T07:34:55.800 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 2364068245 2024-02-10T07:34:55.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.800 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.800 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6809", 2024-02-10T07:34:55.801 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 2364068245 2024-02-10T07:34:55.801 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.801 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.801 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:34:55.801 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.801 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.801 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6810", 2024-02-10T07:34:55.801 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 2364068245 2024-02-10T07:34:55.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.802 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.802 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6811", 2024-02-10T07:34:55.802 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 2364068245 2024-02-10T07:34:55.802 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.802 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.802 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:34:55.802 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.802 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.802 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6814", 2024-02-10T07:34:55.802 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 2364068245 2024-02-10T07:34:55.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.803 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.803 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6815", 2024-02-10T07:34:55.803 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 2364068245 2024-02-10T07:34:55.803 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.803 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.803 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.803 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:34:55.803 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.803 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.804 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6812", 2024-02-10T07:34:55.804 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 2364068245 2024-02-10T07:34:55.804 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.804 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.804 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.804 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6813", 2024-02-10T07:34:55.804 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 2364068245 2024-02-10T07:34:55.804 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.804 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.804 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.804 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.170:6809/2364068245", 2024-02-10T07:34:55.804 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.170:6811/2364068245", 2024-02-10T07:34:55.805 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.170:6815/2364068245", 2024-02-10T07:34:55.805 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.170:6813/2364068245", 2024-02-10T07:34:55.805 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:34:55.805 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:34:55.805 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:34:55.805 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.805 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 6, 2024-02-10T07:34:55.805 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "61970c81-4b79-4743-a379-50d5d102b373", 2024-02-10T07:34:55.805 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:34:55.805 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:34:55.805 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 1, 2024-02-10T07:34:55.805 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 50, 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 98, 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 0, 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6816", 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.806 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6817", 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6818", 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.807 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6819", 2024-02-10T07:34:55.808 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:34:55.808 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.808 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.808 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.808 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:34:55.808 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.808 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.808 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.808 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6822", 2024-02-10T07:34:55.808 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:34:55.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.809 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.809 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6823", 2024-02-10T07:34:55.809 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:34:55.809 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.809 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.809 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:34:55.809 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.809 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.809 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6820", 2024-02-10T07:34:55.809 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6821", 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.170:6817/1764436984", 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.170:6819/1764436984", 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.170:6823/1764436984", 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.170:6821/1764436984", 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:34:55.810 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 7, 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "2e303441-b454-4750-8c5a-96c5348129a8", 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 1, 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 60, 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 98, 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 0, 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:34:55.811 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6824", 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6825", 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.812 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6826", 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6827", 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.813 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6830", 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6831", 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:34:55.814 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6828", 2024-02-10T07:34:55.815 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:34:55.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.815 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:34:55.815 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6829", 2024-02-10T07:34:55.815 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:34:55.815 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.815 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.815 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.170:6825/4048577717", 2024-02-10T07:34:55.815 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.170:6827/4048577717", 2024-02-10T07:34:55.815 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.170:6831/4048577717", 2024-02-10T07:34:55.815 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.170:6829/4048577717", 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_xinfo": [ 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 0, 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:34:55.816 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:33:54.474360+0000", 2024-02-10T07:34:55.817 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 0 2024-02-10T07:34:55.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.817 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 1, 2024-02-10T07:34:55.817 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:34:55.817 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:34:55.817 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:34:55.817 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:34:55.817 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:34:55.817 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:33:56.015138+0000", 2024-02-10T07:34:55.817 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 0 2024-02-10T07:34:55.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.817 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 2, 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:34:31.124916+0000", 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 0 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 3, 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:34:55.818 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:33:54.174257+0000", 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 0 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 4, 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:34:06.657431+0000", 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 0 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.819 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 5, 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:34:27.733238+0000", 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 0 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 6, 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:34:55.820 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:34:45.914335+0000", 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 0 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 7, 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:34:13.750579+0000", 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 0 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:34:55.821 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_upmap": [], 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_upmap_items": [ 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: "pgid": "4.9", 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: "mappings": [ 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: "from": 1, 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: "to": 2 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: "pgid": "4.13", 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: "mappings": [ 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: "from": 1, 2024-02-10T07:34:55.822 INFO:teuthology.orchestra.run.smithi008.stdout: "to": 2 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_upmap_primaries": [], 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_temp": [], 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_temp": [], 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: "blocklist": { 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/2210906095": "2024-02-11T07:32:31.651879+0000", 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:6800/1412156979": "2024-02-11T07:27:16.371197+0000", 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/2515892714": "2024-02-11T07:26:49.500037+0000", 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:6801/4001702773": "2024-02-11T07:26:49.500037+0000", 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:6801/1412156979": "2024-02-11T07:27:16.371197+0000", 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/1462056527": "2024-02-11T07:27:16.371197+0000", 2024-02-10T07:34:55.823 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:6800/4001702773": "2024-02-11T07:26:49.500037+0000", 2024-02-10T07:34:55.824 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/1905866501": "2024-02-11T07:26:49.500037+0000", 2024-02-10T07:34:55.824 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/2381650878": "2024-02-11T07:27:16.371197+0000", 2024-02-10T07:34:55.824 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/802075678": "2024-02-11T07:26:49.500037+0000", 2024-02-10T07:34:55.824 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:6801/2677792951": "2024-02-11T07:32:31.651879+0000", 2024-02-10T07:34:55.824 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/3905205183": "2024-02-11T07:32:31.651879+0000", 2024-02-10T07:34:55.824 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/3314827874": "2024-02-11T07:32:31.651879+0000", 2024-02-10T07:34:55.824 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/1624888680": "2024-02-11T07:32:31.651879+0000", 2024-02-10T07:34:55.824 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/2789652913": "2024-02-11T07:27:16.371197+0000", 2024-02-10T07:34:55.824 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/4074987889": "2024-02-11T07:32:31.651879+0000", 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:6800/2677792951": "2024-02-11T07:32:31.651879+0000" 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: "range_blocklist": {}, 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profiles": { 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: "default": { 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: "crush-failure-domain": "osd", 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: "k": "2", 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: "m": "1", 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: "plugin": "jerasure", 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: "technique": "reed_sol_van" 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps_queue": [], 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: "new_removed_snaps": [], 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: "new_purged_snaps": [], 2024-02-10T07:34:55.825 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_node_flags": {}, 2024-02-10T07:34:55.826 INFO:teuthology.orchestra.run.smithi008.stdout: "device_class_flags": {}, 2024-02-10T07:34:55.826 INFO:teuthology.orchestra.run.smithi008.stdout: "stretch_mode": { 2024-02-10T07:34:55.826 INFO:teuthology.orchestra.run.smithi008.stdout: "stretch_mode_enabled": false, 2024-02-10T07:34:55.826 INFO:teuthology.orchestra.run.smithi008.stdout: "stretch_bucket_count": 0, 2024-02-10T07:34:55.826 INFO:teuthology.orchestra.run.smithi008.stdout: "degraded_stretch_mode": 0, 2024-02-10T07:34:55.826 INFO:teuthology.orchestra.run.smithi008.stdout: "recovering_stretch_mode": 0, 2024-02-10T07:34:55.826 INFO:teuthology.orchestra.run.smithi008.stdout: "stretch_mode_bucket": 0 2024-02-10T07:34:55.826 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:34:55.826 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:34:55.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:55.828 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:55.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:55.929 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:55.958 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:55.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:56.059 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:56.105 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:56.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:56.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:56.207 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:56.212 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 46 16 70165 70149 95.2889 73 0.011545 0.0104904 2024-02-10T07:34:56.246 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:56.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:56.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:56.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:56 smithi008 ceph-mon[29816]: 3.19 scrub starts 2024-02-10T07:34:56.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:56 smithi008 ceph-mon[29816]: 3.19 scrub ok 2024-02-10T07:34:56.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:56 smithi008 ceph-mon[29816]: 3.14 scrub starts 2024-02-10T07:34:56.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:56 smithi008 ceph-mon[29816]: 3.14 scrub ok 2024-02-10T07:34:56.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:56 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/860972093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T07:34:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:56 smithi008 ceph-mon[40743]: 3.19 scrub starts 2024-02-10T07:34:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:56 smithi008 ceph-mon[40743]: 3.19 scrub ok 2024-02-10T07:34:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:56 smithi008 ceph-mon[40743]: 3.14 scrub starts 2024-02-10T07:34:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:56 smithi008 ceph-mon[40743]: 3.14 scrub ok 2024-02-10T07:34:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:56 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/860972093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T07:34:56.333 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:56 smithi170 ceph-mon[33394]: 3.19 scrub starts 2024-02-10T07:34:56.333 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:56 smithi170 ceph-mon[33394]: 3.19 scrub ok 2024-02-10T07:34:56.333 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:56 smithi170 ceph-mon[33394]: 3.14 scrub starts 2024-02-10T07:34:56.333 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:56 smithi170 ceph-mon[33394]: 3.14 scrub ok 2024-02-10T07:34:56.334 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:56 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/860972093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T07:34:56.348 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:56.389 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:56.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:56.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:56.491 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:56.545 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:56.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:56.646 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:34:56.690 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:56.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:34:56.792 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:56.824 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:56.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:56.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:56.925 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:56.956 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:56.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:57.058 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:57.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:57 smithi008 ceph-mon[29816]: pgmap v113: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 638 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-02-10T07:34:57.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:57 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:57.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:57 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:57.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:57 smithi008 ceph-mon[40743]: pgmap v113: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 638 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-02-10T07:34:57.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:57 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:57.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:57 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:57.091 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:57.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:57.193 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:57.214 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 47 16 71407 71391 94.912 77.625 0.0232407 0.010531 2024-02-10T07:34:57.223 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:57.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:57.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:57.325 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:57.356 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:57.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:57 smithi170 ceph-mon[33394]: pgmap v113: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 638 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-02-10T07:34:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:57 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:57 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:34:57.458 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:57.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:57.488 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:57.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:57.590 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:57.647 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:34:57.647 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:34:57.652 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:57.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:57.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:57.754 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:34:57.786 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:57.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:34:57.888 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:57.918 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:57.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:57.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:58.020 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:58.050 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:58.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:58.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:58.152 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:58.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:58.188 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:58.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:58.213 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 48 16 72570 72554 94.4489 72.6875 0.0131335 0.0105826 2024-02-10T07:34:58.290 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:58.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:58.323 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:58.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:58.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:58 smithi008 ceph-mon[40743]: 4.e scrub starts 2024-02-10T07:34:58.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:58 smithi008 ceph-mon[40743]: 4.e scrub ok 2024-02-10T07:34:58.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:58 smithi008 ceph-mon[40743]: 3.17 scrub starts 2024-02-10T07:34:58.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:58 smithi008 ceph-mon[40743]: 3.17 scrub ok 2024-02-10T07:34:58.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:58 smithi008 ceph-mon[40743]: 3.15 scrub starts 2024-02-10T07:34:58.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:58 smithi008 ceph-mon[40743]: 3.15 scrub ok 2024-02-10T07:34:58.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:58 smithi008 ceph-mon[29816]: 4.e scrub starts 2024-02-10T07:34:58.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:58 smithi008 ceph-mon[29816]: 4.e scrub ok 2024-02-10T07:34:58.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:58 smithi008 ceph-mon[29816]: 3.17 scrub starts 2024-02-10T07:34:58.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:58 smithi008 ceph-mon[29816]: 3.17 scrub ok 2024-02-10T07:34:58.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:58 smithi008 ceph-mon[29816]: 3.15 scrub starts 2024-02-10T07:34:58.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:58 smithi008 ceph-mon[29816]: 3.15 scrub ok 2024-02-10T07:34:58.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:58 smithi170 ceph-mon[33394]: 4.e scrub starts 2024-02-10T07:34:58.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:58 smithi170 ceph-mon[33394]: 4.e scrub ok 2024-02-10T07:34:58.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:58 smithi170 ceph-mon[33394]: 3.17 scrub starts 2024-02-10T07:34:58.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:58 smithi170 ceph-mon[33394]: 3.17 scrub ok 2024-02-10T07:34:58.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:58 smithi170 ceph-mon[33394]: 3.15 scrub starts 2024-02-10T07:34:58.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:58 smithi170 ceph-mon[33394]: 3.15 scrub ok 2024-02-10T07:34:58.424 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:58.453 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:58.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:58.555 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:58.585 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:58.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:58.687 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:58.720 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:58.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:58.821 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:34:58.851 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:58.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:34:58.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:58.982 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:58.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:58.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:59.026 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:34:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:34:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:34:59.084 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:34:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:59.129 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:59.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:34:59.213 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 49 16 73843 73827 94.145 79.5625 0.0183434 0.010617 2024-02-10T07:34:59.222 INFO:tasks.thrashosds.thrasher:j is {'epoch': 120, 'fsid': '8c8244e6-c7e5-11ee-95b8-87774f69a715', 'created': '2024-02-10T07:26:21.316239+0000', 'modified': '2024-02-10T07:34:51.675847+0000', 'last_up_change': '2024-02-10T07:31:36.599653+0000', 'last_in_change': '2024-02-10T07:31:19.283633+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-10T07:29:32.704872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-10T07:31:49.150741+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-10T07:31:49.831683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-10T07:31:51.622544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-10T07:31:53.550827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-10T07:31:55.600117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-02-10T07:33:52.310148+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '63abc4f1-1c83-4e03-8592-959c51d10847', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 98, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6802', 'nonce': 1442795455}, {'type': 'v1', 'addr': '172.21.15.8:6803', 'nonce': 1442795455}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6804', 'nonce': 1442795455}, {'type': 'v1', 'addr': '172.21.15.8:6805', 'nonce': 1442795455}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6808', 'nonce': 1442795455}, {'type': 'v1', 'addr': '172.21.15.8:6809', 'nonce': 1442795455}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6806', 'nonce': 1442795455}, {'type': 'v1', 'addr': '172.21.15.8:6807', 'nonce': 1442795455}]}, 'public_addr': '172.21.15.8:6803/1442795455', 'cluster_addr': '172.21.15.8:6805/1442795455', 'heartbeat_back_addr': '172.21.15.8:6809/1442795455', 'heartbeat_front_addr': '172.21.15.8:6807/1442795455', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '88307389-bfa8-402e-92b9-9fa5094262c4', 'up': 1, 'in': 1, 'weight': 0.926666259765625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 98, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6810', 'nonce': 3278758893}, {'type': 'v1', 'addr': '172.21.15.8:6811', 'nonce': 3278758893}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6812', 'nonce': 3278758893}, {'type': 'v1', 'addr': '172.21.15.8:6813', 'nonce': 3278758893}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6816', 'nonce': 3278758893}, {'type': 'v1', 'addr': '172.21.15.8:6817', 'nonce': 3278758893}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6814', 'nonce': 3278758893}, {'type': 'v1', 'addr': '172.21.15.8:6815', 'nonce': 3278758893}]}, 'public_addr': '172.21.15.8:6811/3278758893', 'cluster_addr': '172.21.15.8:6813/3278758893', 'heartbeat_back_addr': '172.21.15.8:6817/3278758893', 'heartbeat_front_addr': '172.21.15.8:6815/3278758893', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c6120003-9ab5-4cf5-87ac-6351ed2c6226', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 98, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6818', 'nonce': 481314802}, {'type': 'v1', 'addr': '172.21.15.8:6819', 'nonce': 481314802}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6820', 'nonce': 481314802}, {'type': 'v1', 'addr': '172.21.15.8:6821', 'nonce': 481314802}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6824', 'nonce': 481314802}, {'type': 'v1', 'addr': '172.21.15.8:6825', 'nonce': 481314802}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6822', 'nonce': 481314802}, {'type': 'v1', 'addr': '172.21.15.8:6823', 'nonce': 481314802}]}, 'public_addr': '172.21.15.8:6819/481314802', 'cluster_addr': '172.21.15.8:6821/481314802', 'heartbeat_back_addr': '172.21.15.8:6825/481314802', 'heartbeat_front_addr': '172.21.15.8:6823/481314802', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6ca8f6f3-013a-48cb-9083-be37c1767489', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6826', 'nonce': 3692331651}, {'type': 'v1', 'addr': '172.21.15.8:6827', 'nonce': 3692331651}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6828', 'nonce': 3692331651}, {'type': 'v1', 'addr': '172.21.15.8:6829', 'nonce': 3692331651}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6832', 'nonce': 3692331651}, {'type': 'v1', 'addr': '172.21.15.8:6833', 'nonce': 3692331651}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6830', 'nonce': 3692331651}, {'type': 'v1', 'addr': '172.21.15.8:6831', 'nonce': 3692331651}]}, 'public_addr': '172.21.15.8:6827/3692331651', 'cluster_addr': '172.21.15.8:6829/3692331651', 'heartbeat_back_addr': '172.21.15.8:6833/3692331651', 'heartbeat_front_addr': '172.21.15.8:6831/3692331651', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'aa667766-e57f-447b-9a31-e3f3e61ac346', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 98, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6800', 'nonce': 3997132504}, {'type': 'v1', 'addr': '172.21.15.170:6801', 'nonce': 3997132504}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6802', 'nonce': 3997132504}, {'type': 'v1', 'addr': '172.21.15.170:6803', 'nonce': 3997132504}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6806', 'nonce': 3997132504}, {'type': 'v1', 'addr': '172.21.15.170:6807', 'nonce': 3997132504}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6804', 'nonce': 3997132504}, {'type': 'v1', 'addr': '172.21.15.170:6805', 'nonce': 3997132504}]}, 'public_addr': '172.21.15.170:6801/3997132504', 'cluster_addr': '172.21.15.170:6803/3997132504', 'heartbeat_back_addr': '172.21.15.170:6807/3997132504', 'heartbeat_front_addr': '172.21.15.170:6805/3997132504', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '2e9ab14d-1aff-4638-9f41-311f6b5f6770', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 112, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6808', 'nonce': 2364068245}, {'type': 'v1', 'addr': '172.21.15.170:6809', 'nonce': 2364068245}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6810', 'nonce': 2364068245}, {'type': 'v1', 'addr': '172.21.15.170:6811', 'nonce': 2364068245}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6814', 'nonce': 2364068245}, {'type': 'v1', 'addr': '172.21.15.170:6815', 'nonce': 2364068245}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6812', 'nonce': 2364068245}, {'type': 'v1', 'addr': '172.21.15.170:6813', 'nonce': 2364068245}]}, 'public_addr': '172.21.15.170:6809/2364068245', 'cluster_addr': '172.21.15.170:6811/2364068245', 'heartbeat_back_addr': '172.21.15.170:6815/2364068245', 'heartbeat_front_addr': '172.21.15.170:6813/2364068245', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '61970c81-4b79-4743-a379-50d5d102b373', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 98, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6816', 'nonce': 1764436984}, {'type': 'v1', 'addr': '172.21.15.170:6817', 'nonce': 1764436984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6818', 'nonce': 1764436984}, {'type': 'v1', 'addr': '172.21.15.170:6819', 'nonce': 1764436984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6822', 'nonce': 1764436984}, {'type': 'v1', 'addr': '172.21.15.170:6823', 'nonce': 1764436984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6820', 'nonce': 1764436984}, {'type': 'v1', 'addr': '172.21.15.170:6821', 'nonce': 1764436984}]}, 'public_addr': '172.21.15.170:6817/1764436984', 'cluster_addr': '172.21.15.170:6819/1764436984', 'heartbeat_back_addr': '172.21.15.170:6823/1764436984', 'heartbeat_front_addr': '172.21.15.170:6821/1764436984', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '2e303441-b454-4750-8c5a-96c5348129a8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 98, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6824', 'nonce': 4048577717}, {'type': 'v1', 'addr': '172.21.15.170:6825', 'nonce': 4048577717}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6826', 'nonce': 4048577717}, {'type': 'v1', 'addr': '172.21.15.170:6827', 'nonce': 4048577717}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6830', 'nonce': 4048577717}, {'type': 'v1', 'addr': '172.21.15.170:6831', 'nonce': 4048577717}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6828', 'nonce': 4048577717}, {'type': 'v1', 'addr': '172.21.15.170:6829', 'nonce': 4048577717}]}, 'public_addr': '172.21.15.170:6825/4048577717', 'cluster_addr': '172.21.15.170:6827/4048577717', 'heartbeat_back_addr': '172.21.15.170:6831/4048577717', 'heartbeat_front_addr': '172.21.15.170:6829/4048577717', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:33:54.474360+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:33:56.015138+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:34:31.124916+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:33:54.174257+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:34:06.657431+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:34:27.733238+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:34:45.914335+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:34:13.750579+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.9', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.8:0/2210906095': '2024-02-11T07:32:31.651879+0000', '172.21.15.8:6800/1412156979': '2024-02-11T07:27:16.371197+0000', '172.21.15.8:0/2515892714': '2024-02-11T07:26:49.500037+0000', '172.21.15.8:6801/4001702773': '2024-02-11T07:26:49.500037+0000', '172.21.15.8:6801/1412156979': '2024-02-11T07:27:16.371197+0000', '172.21.15.8:0/1462056527': '2024-02-11T07:27:16.371197+0000', '172.21.15.8:6800/4001702773': '2024-02-11T07:26:49.500037+0000', '172.21.15.8:0/1905866501': '2024-02-11T07:26:49.500037+0000', '172.21.15.8:0/2381650878': '2024-02-11T07:27:16.371197+0000', '172.21.15.8:0/802075678': '2024-02-11T07:26:49.500037+0000', '172.21.15.8:6801/2677792951': '2024-02-11T07:32:31.651879+0000', '172.21.15.8:0/3905205183': '2024-02-11T07:32:31.651879+0000', '172.21.15.8:0/3314827874': '2024-02-11T07:32:31.651879+0000', '172.21.15.8:0/1624888680': '2024-02-11T07:32:31.651879+0000', '172.21.15.8:0/2789652913': '2024-02-11T07:27:16.371197+0000', '172.21.15.8:0/4074987889': '2024-02-11T07:32:31.651879+0000', '172.21.15.8:6800/2677792951': '2024-02-11T07:32:31.651879+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-10T07:34:59.222 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:34:59.230 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:59.257 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:59.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:59.330 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:59 smithi008 ceph-mon[29816]: 3.d scrub starts 2024-02-10T07:34:59.330 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:59 smithi008 ceph-mon[29816]: 3.d scrub ok 2024-02-10T07:34:59.330 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:59 smithi008 ceph-mon[29816]: pgmap v114: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-10T07:34:59.330 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:59 smithi008 ceph-mon[29816]: 5.15 scrub starts 2024-02-10T07:34:59.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:34:59 smithi008 ceph-mon[29816]: 5.15 scrub ok 2024-02-10T07:34:59.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:59 smithi008 ceph-mon[40743]: 3.d scrub starts 2024-02-10T07:34:59.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:59 smithi008 ceph-mon[40743]: 3.d scrub ok 2024-02-10T07:34:59.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:59 smithi008 ceph-mon[40743]: pgmap v114: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-10T07:34:59.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:59 smithi008 ceph-mon[40743]: 5.15 scrub starts 2024-02-10T07:34:59.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:34:59 smithi008 ceph-mon[40743]: 5.15 scrub ok 2024-02-10T07:34:59.360 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:34:59.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:59 smithi170 ceph-mon[33394]: 3.d scrub starts 2024-02-10T07:34:59.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:59 smithi170 ceph-mon[33394]: 3.d scrub ok 2024-02-10T07:34:59.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:59 smithi170 ceph-mon[33394]: pgmap v114: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-10T07:34:59.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:59 smithi170 ceph-mon[33394]: 5.15 scrub starts 2024-02-10T07:34:59.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:34:59 smithi170 ceph-mon[33394]: 5.15 scrub ok 2024-02-10T07:34:59.385 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:59.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:59.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:34:59.487 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:34:59.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:59.513 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:34:59.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:34:59.608 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:34:59.614 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:59.669 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:59.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:59.770 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:34:59.817 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:59.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:59.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:34:59.919 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:34:59.961 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:34:59.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:34:59.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:00.063 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:00.104 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:00.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:00.175 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-10T07:35:00.206 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:00.216 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 50 16 74661 74645 93.279 51.125 0.024712 0.0107141 2024-02-10T07:35:00.233 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:00.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:00.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:00.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[40743]: 6.16 deep-scrub starts 2024-02-10T07:35:00.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[40743]: 6.16 deep-scrub ok 2024-02-10T07:35:00.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[40743]: 5.1c scrub starts 2024-02-10T07:35:00.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[40743]: 5.1c scrub ok 2024-02-10T07:35:00.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[40743]: 5.d scrub starts 2024-02-10T07:35:00.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[40743]: 5.d scrub ok 2024-02-10T07:35:00.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[40743]: 4.7 scrub starts 2024-02-10T07:35:00.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[40743]: 4.7 scrub ok 2024-02-10T07:35:00.292 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[29816]: 6.16 deep-scrub starts 2024-02-10T07:35:00.292 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[29816]: 6.16 deep-scrub ok 2024-02-10T07:35:00.292 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[29816]: 5.1c scrub starts 2024-02-10T07:35:00.292 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[29816]: 5.1c scrub ok 2024-02-10T07:35:00.292 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[29816]: 5.d scrub starts 2024-02-10T07:35:00.292 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[29816]: 5.d scrub ok 2024-02-10T07:35:00.292 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[29816]: 4.7 scrub starts 2024-02-10T07:35:00.292 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:00 smithi008 ceph-mon[29816]: 4.7 scrub ok 2024-02-10T07:35:00.335 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:00.362 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:00.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:00.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:00 smithi170 ceph-mon[33394]: 6.16 deep-scrub starts 2024-02-10T07:35:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:00 smithi170 ceph-mon[33394]: 6.16 deep-scrub ok 2024-02-10T07:35:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:00 smithi170 ceph-mon[33394]: 5.1c scrub starts 2024-02-10T07:35:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:00 smithi170 ceph-mon[33394]: 5.1c scrub ok 2024-02-10T07:35:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:00 smithi170 ceph-mon[33394]: 5.d scrub starts 2024-02-10T07:35:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:00 smithi170 ceph-mon[33394]: 5.d scrub ok 2024-02-10T07:35:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:00 smithi170 ceph-mon[33394]: 4.7 scrub starts 2024-02-10T07:35:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:00 smithi170 ceph-mon[33394]: 4.7 scrub ok 2024-02-10T07:35:00.464 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:00.494 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:00.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:00.595 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:00.625 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:00.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:00.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:00.727 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:00.759 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:00.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:00.860 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:00.888 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:00.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:00.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:00.990 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:01.030 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:01.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:01.051 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:35:01.131 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:01.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:01.174 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:01.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:01.216 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 51 16 75630 75614 92.6375 60.5625 0.0128376 0.0107898 2024-02-10T07:35:01.275 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:01.317 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:01.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:01.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[29816]: 5.12 scrub starts 2024-02-10T07:35:01.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[29816]: 5.12 scrub ok 2024-02-10T07:35:01.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[29816]: 3.9 scrub starts 2024-02-10T07:35:01.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[29816]: 3.9 scrub ok 2024-02-10T07:35:01.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[29816]: pgmap v115: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-10T07:35:01.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[29816]: 5.0 scrub starts 2024-02-10T07:35:01.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[29816]: 5.0 scrub ok 2024-02-10T07:35:01.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2078502257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:35:01.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:35:01.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[40743]: 5.12 scrub starts 2024-02-10T07:35:01.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[40743]: 5.12 scrub ok 2024-02-10T07:35:01.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[40743]: 3.9 scrub starts 2024-02-10T07:35:01.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[40743]: 3.9 scrub ok 2024-02-10T07:35:01.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[40743]: pgmap v115: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-10T07:35:01.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[40743]: 5.0 scrub starts 2024-02-10T07:35:01.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[40743]: 5.0 scrub ok 2024-02-10T07:35:01.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2078502257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:35:01.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:01 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:35:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:01 smithi170 ceph-mon[33394]: 5.12 scrub starts 2024-02-10T07:35:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:01 smithi170 ceph-mon[33394]: 5.12 scrub ok 2024-02-10T07:35:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:01 smithi170 ceph-mon[33394]: 3.9 scrub starts 2024-02-10T07:35:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:01 smithi170 ceph-mon[33394]: 3.9 scrub ok 2024-02-10T07:35:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:01 smithi170 ceph-mon[33394]: pgmap v115: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-10T07:35:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:01 smithi170 ceph-mon[33394]: 5.0 scrub starts 2024-02-10T07:35:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:01 smithi170 ceph-mon[33394]: 5.0 scrub ok 2024-02-10T07:35:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:01 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2078502257' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:35:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:01 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:35:01.420 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:01.448 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:01.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:01.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:01.550 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:01.587 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:01.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:01.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:01.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:01.715 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:01.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:01.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:01.817 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:01.847 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:01.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:01.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:01.949 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:01.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:01.991 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:01.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:02.092 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:02.122 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:02.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:02.213 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:02.217 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 52 15 76801 76786 92.2634 73.25 0.0196284 0.0108332 2024-02-10T07:35:02.224 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:02.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:02.257 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:02.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:02.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:02 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:35:02.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:02 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:35:02.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:02 smithi008 ceph-mon[29816]: osdmap e121: 8 total, 8 up, 8 in 2024-02-10T07:35:02.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:02 smithi008 ceph-mon[29816]: osdmap e122: 8 total, 8 up, 8 in 2024-02-10T07:35:02.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:02 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:02.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:02 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:35:02.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:02 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:35:02.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:02 smithi008 ceph-mon[40743]: osdmap e121: 8 total, 8 up, 8 in 2024-02-10T07:35:02.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:02 smithi008 ceph-mon[40743]: osdmap e122: 8 total, 8 up, 8 in 2024-02-10T07:35:02.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:02 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:02.359 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:02.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:02 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:35:02.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:02 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:35:02.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:02 smithi170 ceph-mon[33394]: osdmap e121: 8 total, 8 up, 8 in 2024-02-10T07:35:02.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:02 smithi170 ceph-mon[33394]: osdmap e122: 8 total, 8 up, 8 in 2024-02-10T07:35:02.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:02 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:02.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:02.412 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:02.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:02.513 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:02.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:02.543 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:02.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:02.644 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:02.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:02.684 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:02.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:02.785 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:02.827 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:02.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:02.929 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:02.960 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:02.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:02.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:03.061 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:35:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:35:03.062 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:03.092 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:03.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:03.194 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:03.217 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 53 16 77979 77963 91.9105 73.5625 0.0239285 0.0108735 2024-02-10T07:35:03.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:03.257 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:03.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:03.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:03 smithi008 ceph-mon[29816]: pgmap v118: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 749 op/s 2024-02-10T07:35:03.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:03 smithi008 ceph-mon[40743]: pgmap v118: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 749 op/s 2024-02-10T07:35:03.358 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:03.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:03 smithi170 ceph-mon[33394]: pgmap v118: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 749 op/s 2024-02-10T07:35:03.394 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:03.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:03.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:03.496 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:03.535 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:03.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:03.637 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:03.665 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:03.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:03.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:03.767 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:03.795 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:03.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:03.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:03.897 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:03.940 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:03.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:04.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:04.071 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:04.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:04.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:04.132 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:04 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:04.132 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:04 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:04.173 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:04.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:04.216 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:04.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:04.217 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 54 16 79169 79153 91.5858 74.375 0.0247786 0.0109131 2024-02-10T07:35:04.317 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:04.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:04.360 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:04.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:04.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:04 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:04.461 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:04.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:04.502 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:04.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:04.603 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:04.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:04.633 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:04.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:04.734 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:35:04.734 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:04.764 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:04.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:04.865 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:04.931 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:04.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:05.032 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:05.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:05.060 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:05.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:05.140 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:05 smithi008 ceph-mon[29816]: pgmap v119: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-02-10T07:35:05.141 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:05 smithi008 ceph-mon[40743]: pgmap v119: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-02-10T07:35:05.162 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:05.188 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:05.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:05.217 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 55 16 80274 80258 91.176 69.0625 0.0419296 0.0109611 2024-02-10T07:35:05.290 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:05.293 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:35:05.337 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:35:05.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:05.339 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:05.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:05.376 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:35:05.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:05 smithi170 ceph-mon[33394]: pgmap v119: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-02-10T07:35:05.423 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:35:05.440 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:05.467 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:05.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:05.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:05.474 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:35:05.522 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:35:05.564 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:35:05.574 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:05.615 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:35:05.618 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:35:05.618 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:35:05.618 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:35:05.618 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:35:05.619 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:35:05.619 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:35:05.619 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:35:05.619 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:35:05.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:05.625 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:05.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:05.726 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:05.770 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:05.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:05.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:05.872 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:05.924 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:05.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:06.025 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:06.053 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:06.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:06.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:06.155 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:06.199 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:06.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:06.217 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 56 16 81274 81258 90.6638 62.5 0.0183912 0.0110248 2024-02-10T07:35:06.243 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:35:06.246 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:35:06.301 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:06.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:06 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3056166149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:06.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:06 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3056166149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:06.342 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:06.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:06.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:06.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:06 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3056166149' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:06.444 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:06.505 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:06.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:06.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:06.607 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:06.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:06.639 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:06.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:06.695 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:06.700 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:06.740 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:06.792 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:06.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:06.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:06.893 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:06.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:06.940 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:06.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:07.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:07.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:07.073 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:07.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:07.174 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:07.205 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:07.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:07.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:07.218 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 57 16 82408 82392 90.3166 70.875 0.00944233 0.0110664 2024-02-10T07:35:07.307 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:07.348 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:07.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:07.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:07 smithi170 ceph-mon[33394]: pgmap v120: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-02-10T07:35:07.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:07 smithi170 ceph-mon[33394]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:35:07.453 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:07.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:07.494 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:07.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:07.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:07 smithi008 ceph-mon[29816]: pgmap v120: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-02-10T07:35:07.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:07 smithi008 ceph-mon[29816]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:35:07.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:07 smithi008 ceph-mon[40743]: pgmap v120: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-02-10T07:35:07.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:07 smithi008 ceph-mon[40743]: from='client.24682 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:35:07.595 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:07.627 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:07.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:07.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:07.729 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:07.774 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:07.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:07.876 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:07.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:07.914 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:07.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:08.015 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:08.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:08.058 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:08.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:08.159 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:08.198 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:08.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:08.218 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 58 16 83445 83429 89.8769 64.8125 0.00840227 0.0111217 2024-02-10T07:35:08.301 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:08.340 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:08.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:08.441 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:08.473 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:35:08.478 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:08.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:08.580 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:08.607 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:08.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:08.709 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:08.737 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:08.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:08.839 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:08.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:08.869 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:08.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:08.970 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:09.001 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:09.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:09.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:09.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:35:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:35:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:35:09.103 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:09.128 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:09.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:09.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:09.218 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 59 16 84367 84351 89.3303 57.625 0.0190198 0.0111895 2024-02-10T07:35:09.230 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:09.262 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:09.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:09.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:09.364 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:09.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:09 smithi170 ceph-mon[33394]: pgmap v121: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 987 op/s 2024-02-10T07:35:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:09.405 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:09.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:09.506 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:09.546 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:09.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:09.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:09.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:09 smithi008 ceph-mon[29816]: pgmap v121: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 987 op/s 2024-02-10T07:35:09.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:09 smithi008 ceph-mon[40743]: pgmap v121: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 987 op/s 2024-02-10T07:35:09.652 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:09.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:09.684 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:09.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:09.785 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:09.816 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:09.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:09.918 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:09.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:09.950 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:09.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:10.051 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:10.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:10.106 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:10.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:10.208 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:10.220 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:35:10.218974+0000 min lat: 0.00141219 max lat: 2.36731 avg lat: 0.0112336 2024-02-10T07:35:10.220 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:35:10.221 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 60 16 85454 85438 88.9737 67.9375 0.0174104 0.0112336 2024-02-10T07:35:10.243 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:10.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:10.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:10.346 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:10.375 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:10.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:10.477 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:10.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:10.526 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:10.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:10.627 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:10.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:10.665 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:10.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:10.767 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:10.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:10.798 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:10.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:10.899 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:10.932 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:10.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:10.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:11.034 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:11.062 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:11.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:11.163 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:11.213 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:11.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:11.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:11.218 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 61 16 86448 86432 88.5335 62.125 0.00589211 0.0112898 2024-02-10T07:35:11.317 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:11.348 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:11.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:11.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:11.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:11 smithi170 ceph-mon[33394]: pgmap v122: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-02-10T07:35:11.411 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:11 smithi008 ceph-mon[29816]: pgmap v122: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-02-10T07:35:11.412 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:11 smithi008 ceph-mon[40743]: pgmap v122: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-02-10T07:35:11.451 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:11.459 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-02-10T07:35:05.711822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5073148252,"num_objects":77599,"num_object_clones":0,"num_object_copies":232797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77599,"num_whiteouts":0,"num_read":1117,"num_read_kb":963,"num_write":78016,"num_write_kb":4956125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2869,"num_bytes_recovered":186845648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77147,"ondisk_log_size":77147,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15282176,"kb_used_data":14791196,"kb_used_omap":12,"kb_used_meta":490867,"kb_avail":734679040,"statfs":{"total":767960285184,"available":752311336960,"internally_reserved":0,"allocated":15146184704,"data_stored":15142284164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":502648369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":582942720,"num_objects":8895,"num_object_clones":0,"num_object_copies":26685,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8895,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8895,"num_write_kb":569280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002602"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682077+0000","last_change":"2024-02-10T07:33:40.321091+0000","last_active":"2024-02-10T07:35:01.682077+0000","last_peered":"2024-02-10T07:35:01.682077+0000","last_clean":"2024-02-10T07:35:01.682077+0000","last_became_active":"2024-02-10T07:31:57.311360+0000","last_became_peered":"2024-02-10T07:31:57.311360+0000","last_unstale":"2024-02-10T07:35:01.682077+0000","last_undegraded":"2024-02-10T07:35:01.682077+0000","last_fullsized":"2024-02-10T07:35:01.682077+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:40.320971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:40.320971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:05.618404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":120,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.704781+0000","last_change":"2024-02-10T07:33:14.320677+0000","last_active":"2024-02-10T07:35:01.704781+0000","last_peered":"2024-02-10T07:35:01.704781+0000","last_clean":"2024-02-10T07:35:01.704781+0000","last_became_active":"2024-02-10T07:31:55.299946+0000","last_became_peered":"2024-02-10T07:31:55.299946+0000","last_unstale":"2024-02-10T07:35:01.704781+0000","last_undegraded":"2024-02-10T07:35:01.704781+0000","last_fullsized":"2024-02-10T07:35:01.704781+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:14.320588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:14.320588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":135,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.719209+0000","last_change":"2024-02-10T07:33:29.609321+0000","last_active":"2024-02-10T07:35:01.719209+0000","last_peered":"2024-02-10T07:35:01.719209+0000","last_clean":"2024-02-10T07:35:01.719209+0000","last_became_active":"2024-02-10T07:31:51.320873+0000","last_became_peered":"2024-02-10T07:31:51.320873+0000","last_unstale":"2024-02-10T07:35:01.719209+0000","last_undegraded":"2024-02-10T07:35:01.719209+0000","last_fullsized":"2024-02-10T07:35:01.719209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:33:29.609159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:29.609159+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:06.431287+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":142,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.681973+0000","last_change":"2024-02-10T07:33:35.276549+0000","last_active":"2024-02-10T07:35:01.681973+0000","last_peered":"2024-02-10T07:35:01.681973+0000","last_clean":"2024-02-10T07:35:01.681973+0000","last_became_active":"2024-02-10T07:31:53.300846+0000","last_became_peered":"2024-02-10T07:31:53.300846+0000","last_unstale":"2024-02-10T07:35:01.681973+0000","last_undegraded":"2024-02-10T07:35:01.681973+0000","last_fullsized":"2024-02-10T07:35:01.681973+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:33:35.276429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:35.276429+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.705360+0000","last_change":"2024-02-10T07:33:12.587948+0000","last_active":"2024-02-10T07:35:01.705360+0000","last_peered":"2024-02-10T07:35:01.705360+0000","last_clean":"2024-02-10T07:35:01.705360+0000","last_became_active":"2024-02-10T07:31:57.300056+0000","last_became_peered":"2024-02-10T07:31:57.300056+0000","last_unstale":"2024-02-10T07:35:01.705360+0000","last_undegraded":"2024-02-10T07:35:01.705360+0000","last_fullsized":"2024-02-10T07:35:01.705360+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:12.587860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:12.587860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682054+0000","last_change":"2024-02-10T07:34:38.285132+0000","last_active":"2024-02-10T07:35:01.682054+0000","last_peered":"2024-02-10T07:35:01.682054+0000","last_clean":"2024-02-10T07:35:01.682054+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:35:01.682054+0000","last_undegraded":"2024-02-10T07:35:01.682054+0000","last_fullsized":"2024-02-10T07:35:01.682054+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:38.285054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:38.285054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:07.608477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.702984+0000","last_change":"2024-02-10T07:33:24.458907+0000","last_active":"2024-02-10T07:35:01.702984+0000","last_peered":"2024-02-10T07:35:01.702984+0000","last_clean":"2024-02-10T07:35:01.702984+0000","last_became_active":"2024-02-10T07:31:51.323331+0000","last_became_peered":"2024-02-10T07:31:51.323331+0000","last_unstale":"2024-02-10T07:35:01.702984+0000","last_undegraded":"2024-02-10T07:35:01.702984+0000","last_fullsized":"2024-02-10T07:35:01.702984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:33:24.458825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":159,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.703244+0000","last_change":"2024-02-10T07:34:46.545956+0000","last_active":"2024-02-10T07:35:01.703244+0000","last_peered":"2024-02-10T07:35:01.703244+0000","last_clean":"2024-02-10T07:35:01.703244+0000","last_became_active":"2024-02-10T07:31:53.286957+0000","last_became_peered":"2024-02-10T07:31:53.286957+0000","last_unstale":"2024-02-10T07:35:01.703244+0000","last_undegraded":"2024-02-10T07:35:01.703244+0000","last_fullsized":"2024-02-10T07:35:01.703244+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:34:46.545852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:46.545852+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:55.758249+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":117,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.710057+0000","last_change":"2024-02-10T07:33:39.604426+0000","last_active":"2024-02-10T07:35:01.710057+0000","last_peered":"2024-02-10T07:35:01.710057+0000","last_clean":"2024-02-10T07:35:01.710057+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:35:01.710057+0000","last_undegraded":"2024-02-10T07:35:01.710057+0000","last_fullsized":"2024-02-10T07:35:01.710057+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:39.604346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:39.604346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:06.431485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":121,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708660+0000","last_change":"2024-02-10T07:33:43.115599+0000","last_active":"2024-02-10T07:35:01.708660+0000","last_peered":"2024-02-10T07:35:01.708660+0000","last_clean":"2024-02-10T07:35:01.708660+0000","last_became_active":"2024-02-10T07:31:55.297715+0000","last_became_peered":"2024-02-10T07:31:55.297715+0000","last_unstale":"2024-02-10T07:35:01.708660+0000","last_undegraded":"2024-02-10T07:35:01.708660+0000","last_fullsized":"2024-02-10T07:35:01.708660+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:43.115523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:43.115523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":144,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708159+0000","last_change":"2024-02-10T07:34:49.349481+0000","last_active":"2024-02-10T07:35:01.708159+0000","last_peered":"2024-02-10T07:35:01.708159+0000","last_clean":"2024-02-10T07:35:01.708159+0000","last_became_active":"2024-02-10T07:31:51.283229+0000","last_became_peered":"2024-02-10T07:31:51.283229+0000","last_unstale":"2024-02-10T07:35:01.708159+0000","last_undegraded":"2024-02-10T07:35:01.708159+0000","last_fullsized":"2024-02-10T07:35:01.708159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:34:49.349360+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:34:49.349360+0000","last_clean_scrub_stamp":"2024-02-10T07:34:49.349360+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:05.734497+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":142,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682628+0000","last_change":"2024-02-10T07:33:48.378628+0000","last_active":"2024-02-10T07:35:01.682628+0000","last_peered":"2024-02-10T07:35:01.682628+0000","last_clean":"2024-02-10T07:35:01.682628+0000","last_became_active":"2024-02-10T07:31:53.287858+0000","last_became_peered":"2024-02-10T07:31:53.287858+0000","last_unstale":"2024-02-10T07:35:01.682628+0000","last_undegraded":"2024-02-10T07:35:01.682628+0000","last_fullsized":"2024-02-10T07:35:01.682628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:33:48.378524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:48.378524+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":117,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.707233+0000","last_change":"2024-02-10T07:33:47.227552+0000","last_active":"2024-02-10T07:35:01.707233+0000","last_peered":"2024-02-10T07:35:01.707233+0000","last_clean":"2024-02-10T07:35:01.707233+0000","last_became_active":"2024-02-10T07:31:57.310088+0000","last_became_peered":"2024-02-10T07:31:57.310088+0000","last_unstale":"2024-02-10T07:35:01.707233+0000","last_undegraded":"2024-02-10T07:35:01.707233+0000","last_fullsized":"2024-02-10T07:35:01.707233+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:47.227448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:47.227448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:04.702854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":122,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.719575+0000","last_change":"2024-02-10T07:33:33.199150+0000","last_active":"2024-02-10T07:35:01.719575+0000","last_peered":"2024-02-10T07:35:01.719575+0000","last_clean":"2024-02-10T07:35:01.719575+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:35:01.719575+0000","last_undegraded":"2024-02-10T07:35:01.719575+0000","last_fullsized":"2024-02-10T07:35:01.719575+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:33.199031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:33.199031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":129,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.707726+0000","last_change":"2024-02-10T07:33:28.316764+0000","last_active":"2024-02-10T07:35:01.707726+0000","last_peered":"2024-02-10T07:35:01.707726+0000","last_clean":"2024-02-10T07:35:01.707726+0000","last_became_active":"2024-02-10T07:31:51.276350+0000","last_became_peered":"2024-02-10T07:31:51.276350+0000","last_unstale":"2024-02-10T07:35:01.707726+0000","last_undegraded":"2024-02-10T07:35:01.707726+0000","last_fullsized":"2024-02-10T07:35:01.707726+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:28.316679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:28.316679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":136,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682530+0000","last_change":"2024-02-10T07:34:27.386460+0000","last_active":"2024-02-10T07:35:01.682530+0000","last_peered":"2024-02-10T07:35:01.682530+0000","last_clean":"2024-02-10T07:35:01.682530+0000","last_became_active":"2024-02-10T07:31:53.292417+0000","last_became_peered":"2024-02-10T07:31:53.292417+0000","last_unstale":"2024-02-10T07:35:01.682530+0000","last_undegraded":"2024-02-10T07:35:01.682530+0000","last_fullsized":"2024-02-10T07:35:01.682530+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-10T07:34:27.386341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:27.386341+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:38.466835+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":126,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682972+0000","last_change":"2024-02-10T07:34:51.300380+0000","last_active":"2024-02-10T07:35:01.682972+0000","last_peered":"2024-02-10T07:35:01.682972+0000","last_clean":"2024-02-10T07:35:01.682972+0000","last_became_active":"2024-02-10T07:31:57.305613+0000","last_became_peered":"2024-02-10T07:31:57.305613+0000","last_unstale":"2024-02-10T07:35:01.682972+0000","last_undegraded":"2024-02-10T07:35:01.682972+0000","last_fullsized":"2024-02-10T07:35:01.682972+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:51.300266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:51.300266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:04.129482+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":120,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.713416+0000","last_change":"2024-02-10T07:33:19.252470+0000","last_active":"2024-02-10T07:35:01.713416+0000","last_peered":"2024-02-10T07:35:01.713416+0000","last_clean":"2024-02-10T07:35:01.713416+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:35:01.713416+0000","last_undegraded":"2024-02-10T07:35:01.713416+0000","last_fullsized":"2024-02-10T07:35:01.713416+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:19.252386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:19.252386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":129,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.710198+0000","last_change":"2024-02-10T07:33:47.723413+0000","last_active":"2024-02-10T07:35:01.710198+0000","last_peered":"2024-02-10T07:35:01.710198+0000","last_clean":"2024-02-10T07:35:01.710198+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:35:01.710198+0000","last_undegraded":"2024-02-10T07:35:01.710198+0000","last_fullsized":"2024-02-10T07:35:01.710198+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:33:47.723328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:06.431635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"89'22","reported_seq":165,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.713168+0000","last_change":"2024-02-10T07:34:46.082140+0000","last_active":"2024-02-10T07:35:01.713168+0000","last_peered":"2024-02-10T07:35:01.713168+0000","last_clean":"2024-02-10T07:35:01.713168+0000","last_became_active":"2024-02-10T07:31:53.295527+0000","last_became_peered":"2024-02-10T07:31:53.295527+0000","last_unstale":"2024-02-10T07:35:01.713168+0000","last_undegraded":"2024-02-10T07:35:01.713168+0000","last_fullsized":"2024-02-10T07:35:01.713168+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2024-02-10T07:34:46.081979+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:34:46.081979+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:06.591014+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.705294+0000","last_change":"2024-02-10T07:34:44.596132+0000","last_active":"2024-02-10T07:35:01.705294+0000","last_peered":"2024-02-10T07:35:01.705294+0000","last_clean":"2024-02-10T07:35:01.705294+0000","last_became_active":"2024-02-10T07:31:57.306315+0000","last_became_peered":"2024-02-10T07:31:57.306315+0000","last_unstale":"2024-02-10T07:35:01.705294+0000","last_undegraded":"2024-02-10T07:35:01.705294+0000","last_fullsized":"2024-02-10T07:35:01.705294+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:44.595328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:44.595328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:57.623977+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682676+0000","last_change":"2024-02-10T07:34:58.502489+0000","last_active":"2024-02-10T07:35:01.682676+0000","last_peered":"2024-02-10T07:35:01.682676+0000","last_clean":"2024-02-10T07:35:01.682676+0000","last_became_active":"2024-02-10T07:31:55.298474+0000","last_became_peered":"2024-02-10T07:31:55.298474+0000","last_unstale":"2024-02-10T07:35:01.682676+0000","last_undegraded":"2024-02-10T07:35:01.682676+0000","last_fullsized":"2024-02-10T07:35:01.682676+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:58.502372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:27.368799+0000","last_clean_scrub_stamp":"2024-02-10T07:34:58.502372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:08.046058+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":138,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682628+0000","last_change":"2024-02-10T07:34:37.329912+0000","last_active":"2024-02-10T07:35:01.682628+0000","last_peered":"2024-02-10T07:35:01.682628+0000","last_clean":"2024-02-10T07:35:01.682628+0000","last_became_active":"2024-02-10T07:31:51.321237+0000","last_became_peered":"2024-02-10T07:31:51.321237+0000","last_unstale":"2024-02-10T07:35:01.682628+0000","last_undegraded":"2024-02-10T07:35:01.682628+0000","last_fullsized":"2024-02-10T07:35:01.682628+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:37.329817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:37.329817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:39.334428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":149,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.702479+0000","last_change":"2024-02-10T07:33:13.556374+0000","last_active":"2024-02-10T07:35:01.702479+0000","last_peered":"2024-02-10T07:35:01.702479+0000","last_clean":"2024-02-10T07:35:01.702479+0000","last_became_active":"2024-02-10T07:31:53.294914+0000","last_became_peered":"2024-02-10T07:31:53.294914+0000","last_unstale":"2024-02-10T07:35:01.702479+0000","last_undegraded":"2024-02-10T07:35:01.702479+0000","last_fullsized":"2024-02-10T07:35:01.702479+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:33:13.556271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:13.556271+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":119,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.722119+0000","last_change":"2024-02-10T07:34:43.532210+0000","last_active":"2024-02-10T07:35:01.722119+0000","last_peered":"2024-02-10T07:35:01.722119+0000","last_clean":"2024-02-10T07:35:01.722119+0000","last_became_active":"2024-02-10T07:31:57.306892+0000","last_became_peered":"2024-02-10T07:31:57.306892+0000","last_unstale":"2024-02-10T07:35:01.722119+0000","last_undegraded":"2024-02-10T07:35:01.722119+0000","last_fullsized":"2024-02-10T07:35:01.722119+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:34:43.531966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:43.531966+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:59.235526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":179,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708929+0000","last_change":"2024-02-10T07:34:38.826446+0000","last_active":"2024-02-10T07:35:01.708929+0000","last_peered":"2024-02-10T07:35:01.708929+0000","last_clean":"2024-02-10T07:35:01.708929+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:35:01.708929+0000","last_undegraded":"2024-02-10T07:35:01.708929+0000","last_fullsized":"2024-02-10T07:35:01.708929+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:34:38.826344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:38.826344+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:47.498990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":137,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.704875+0000","last_change":"2024-02-10T07:34:54.328873+0000","last_active":"2024-02-10T07:35:01.704875+0000","last_peered":"2024-02-10T07:35:01.704875+0000","last_clean":"2024-02-10T07:35:01.704875+0000","last_became_active":"2024-02-10T07:31:51.282077+0000","last_became_peered":"2024-02-10T07:31:51.282077+0000","last_unstale":"2024-02-10T07:35:01.704875+0000","last_undegraded":"2024-02-10T07:35:01.704875+0000","last_fullsized":"2024-02-10T07:35:01.704875+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:34:54.328677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:54.328677+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:12.122414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":148,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.710509+0000","last_change":"2024-02-10T07:34:51.520784+0000","last_active":"2024-02-10T07:35:01.710509+0000","last_peered":"2024-02-10T07:35:01.710509+0000","last_clean":"2024-02-10T07:35:01.710509+0000","last_became_active":"2024-02-10T07:31:53.292936+0000","last_became_peered":"2024-02-10T07:31:53.292936+0000","last_unstale":"2024-02-10T07:35:01.710509+0000","last_undegraded":"2024-02-10T07:35:01.710509+0000","last_fullsized":"2024-02-10T07:35:01.710509+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:34:51.520532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:51.520532+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:59.847057+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":116,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.715817+0000","last_change":"2024-02-10T07:33:18.242898+0000","last_active":"2024-02-10T07:35:01.715817+0000","last_peered":"2024-02-10T07:35:01.715817+0000","last_clean":"2024-02-10T07:35:01.715817+0000","last_became_active":"2024-02-10T07:31:57.308820+0000","last_became_peered":"2024-02-10T07:31:57.308820+0000","last_unstale":"2024-02-10T07:35:01.715817+0000","last_undegraded":"2024-02-10T07:35:01.715817+0000","last_fullsized":"2024-02-10T07:35:01.715817+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:18.242759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:18.242759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":121,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.711084+0000","last_change":"2024-02-10T07:33:48.707506+0000","last_active":"2024-02-10T07:35:01.711084+0000","last_peered":"2024-02-10T07:35:01.711084+0000","last_clean":"2024-02-10T07:35:01.711084+0000","last_became_active":"2024-02-10T07:31:55.297595+0000","last_became_peered":"2024-02-10T07:31:55.297595+0000","last_unstale":"2024-02-10T07:35:01.711084+0000","last_undegraded":"2024-02-10T07:35:01.711084+0000","last_fullsized":"2024-02-10T07:35:01.711084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:48.707394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:48.707394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:13.816712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.702667+0000","last_change":"2024-02-10T07:33:16.558719+0000","last_active":"2024-02-10T07:35:01.702667+0000","last_peered":"2024-02-10T07:35:01.702667+0000","last_clean":"2024-02-10T07:35:01.702667+0000","last_became_active":"2024-02-10T07:31:51.285405+0000","last_became_peered":"2024-02-10T07:31:51.285405+0000","last_unstale":"2024-02-10T07:35:01.702667+0000","last_undegraded":"2024-02-10T07:35:01.702667+0000","last_fullsized":"2024-02-10T07:35:01.702667+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:16.558593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:16.558593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":152,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708837+0000","last_change":"2024-02-10T07:34:34.728508+0000","last_active":"2024-02-10T07:35:01.708837+0000","last_peered":"2024-02-10T07:35:01.708837+0000","last_clean":"2024-02-10T07:35:01.708837+0000","last_became_active":"2024-02-10T07:31:53.300871+0000","last_became_peered":"2024-02-10T07:31:53.300871+0000","last_unstale":"2024-02-10T07:35:01.708837+0000","last_undegraded":"2024-02-10T07:35:01.708837+0000","last_fullsized":"2024-02-10T07:35:01.708837+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:34:34.728407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:34.728407+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:00.562652+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.719468+0000","last_change":"2024-02-10T07:33:44.251598+0000","last_active":"2024-02-10T07:35:01.719468+0000","last_peered":"2024-02-10T07:35:01.719468+0000","last_clean":"2024-02-10T07:35:01.719468+0000","last_became_active":"2024-02-10T07:31:57.306776+0000","last_became_peered":"2024-02-10T07:31:57.306776+0000","last_unstale":"2024-02-10T07:35:01.719468+0000","last_undegraded":"2024-02-10T07:35:01.719468+0000","last_fullsized":"2024-02-10T07:35:01.719468+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:44.251492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:44.251492+0000","last_clean_scrub_stamp":"2024-02-10T07:33:44.251492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":129,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708057+0000","last_change":"2024-02-10T07:34:41.840128+0000","last_active":"2024-02-10T07:35:01.708057+0000","last_peered":"2024-02-10T07:35:01.708057+0000","last_clean":"2024-02-10T07:35:01.708057+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:35:01.708057+0000","last_undegraded":"2024-02-10T07:35:01.708057+0000","last_fullsized":"2024-02-10T07:35:01.708057+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:41.839991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:41.839991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:06.871399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":137,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.719404+0000","last_change":"2024-02-10T07:34:56.494219+0000","last_active":"2024-02-10T07:35:01.719404+0000","last_peered":"2024-02-10T07:35:01.719404+0000","last_clean":"2024-02-10T07:35:01.719404+0000","last_became_active":"2024-02-10T07:31:51.284022+0000","last_became_peered":"2024-02-10T07:31:51.284022+0000","last_unstale":"2024-02-10T07:35:01.719404+0000","last_undegraded":"2024-02-10T07:35:01.719404+0000","last_fullsized":"2024-02-10T07:35:01.719404+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:56.494106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:56.494106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:02.762479+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"95'6","reported_seq":134,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.702768+0000","last_change":"2024-02-10T07:33:46.438241+0000","last_active":"2024-02-10T07:35:01.702768+0000","last_peered":"2024-02-10T07:35:01.702768+0000","last_clean":"2024-02-10T07:35:01.702768+0000","last_became_active":"2024-02-10T07:31:53.295520+0000","last_became_peered":"2024-02-10T07:31:53.295520+0000","last_unstale":"2024-02-10T07:35:01.702768+0000","last_undegraded":"2024-02-10T07:35:01.702768+0000","last_fullsized":"2024-02-10T07:35:01.702768+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:33:46.438117+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:02.522079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.705606+0000","last_change":"2024-02-10T07:33:26.457321+0000","last_active":"2024-02-10T07:35:01.705606+0000","last_peered":"2024-02-10T07:35:01.705606+0000","last_clean":"2024-02-10T07:35:01.705606+0000","last_became_active":"2024-02-10T07:31:57.315738+0000","last_became_peered":"2024-02-10T07:31:57.315738+0000","last_unstale":"2024-02-10T07:35:01.705606+0000","last_undegraded":"2024-02-10T07:35:01.705606+0000","last_fullsized":"2024-02-10T07:35:01.705606+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:26.457244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:26.457244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":122,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.720853+0000","last_change":"2024-02-10T07:33:29.198369+0000","last_active":"2024-02-10T07:35:01.720853+0000","last_peered":"2024-02-10T07:35:01.720853+0000","last_clean":"2024-02-10T07:35:01.720853+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:35:01.720853+0000","last_undegraded":"2024-02-10T07:35:01.720853+0000","last_fullsized":"2024-02-10T07:35:01.720853+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:29.198265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:29.198265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":137,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708579+0000","last_change":"2024-02-10T07:34:37.328219+0000","last_active":"2024-02-10T07:35:01.708579+0000","last_peered":"2024-02-10T07:35:01.708579+0000","last_clean":"2024-02-10T07:35:01.708579+0000","last_became_active":"2024-02-10T07:31:51.279438+0000","last_became_peered":"2024-02-10T07:31:51.279438+0000","last_unstale":"2024-02-10T07:35:01.708579+0000","last_undegraded":"2024-02-10T07:35:01.708579+0000","last_fullsized":"2024-02-10T07:35:01.708579+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:37.328147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:34:37.328147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:02.438373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":153,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.703151+0000","last_change":"2024-02-10T07:34:47.537455+0000","last_active":"2024-02-10T07:35:01.703151+0000","last_peered":"2024-02-10T07:35:01.703151+0000","last_clean":"2024-02-10T07:35:01.703151+0000","last_became_active":"2024-02-10T07:31:53.287823+0000","last_became_peered":"2024-02-10T07:31:53.287823+0000","last_unstale":"2024-02-10T07:35:01.703151+0000","last_undegraded":"2024-02-10T07:35:01.703151+0000","last_fullsized":"2024-02-10T07:35:01.703151+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:34:47.537324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:47.537324+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:00.222329+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":117,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.709739+0000","last_change":"2024-02-10T07:34:32.584922+0000","last_active":"2024-02-10T07:35:01.709739+0000","last_peered":"2024-02-10T07:35:01.709739+0000","last_clean":"2024-02-10T07:35:01.709739+0000","last_became_active":"2024-02-10T07:31:57.313529+0000","last_became_peered":"2024-02-10T07:31:57.313529+0000","last_unstale":"2024-02-10T07:35:01.709739+0000","last_undegraded":"2024-02-10T07:35:01.709739+0000","last_fullsized":"2024-02-10T07:35:01.709739+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:32.584777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:32.584777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:54.431843+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":122,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.705276+0000","last_change":"2024-02-10T07:32:59.584373+0000","last_active":"2024-02-10T07:35:01.705276+0000","last_peered":"2024-02-10T07:35:01.705276+0000","last_clean":"2024-02-10T07:35:01.705276+0000","last_became_active":"2024-02-10T07:31:55.296670+0000","last_became_peered":"2024-02-10T07:31:55.296670+0000","last_unstale":"2024-02-10T07:35:01.705276+0000","last_undegraded":"2024-02-10T07:35:01.705276+0000","last_fullsized":"2024-02-10T07:35:01.705276+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:59.584219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:59.584219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":138,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.721279+0000","last_change":"2024-02-10T07:34:56.606543+0000","last_active":"2024-02-10T07:35:01.721279+0000","last_peered":"2024-02-10T07:35:01.721279+0000","last_clean":"2024-02-10T07:35:01.721279+0000","last_became_active":"2024-02-10T07:31:51.288165+0000","last_became_peered":"2024-02-10T07:31:51.288165+0000","last_unstale":"2024-02-10T07:35:01.721279+0000","last_undegraded":"2024-02-10T07:35:01.721279+0000","last_fullsized":"2024-02-10T07:35:01.721279+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:56.606253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:56.606253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:05.787229+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":117,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.707726+0000","last_change":"2024-02-10T07:34:47.925231+0000","last_active":"2024-02-10T07:35:01.707726+0000","last_peered":"2024-02-10T07:35:01.707726+0000","last_clean":"2024-02-10T07:35:01.707726+0000","last_became_active":"2024-02-10T07:33:46.501312+0000","last_became_peered":"2024-02-10T07:33:46.501312+0000","last_unstale":"2024-02-10T07:35:01.707726+0000","last_undegraded":"2024-02-10T07:35:01.707726+0000","last_fullsized":"2024-02-10T07:35:01.707726+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:34:47.925100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:47.925100+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:16.951245+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.702830+0000","last_change":"2024-02-10T07:33:31.451285+0000","last_active":"2024-02-10T07:35:01.702830+0000","last_peered":"2024-02-10T07:35:01.702830+0000","last_clean":"2024-02-10T07:35:01.702830+0000","last_became_active":"2024-02-10T07:31:57.315781+0000","last_became_peered":"2024-02-10T07:31:57.315781+0000","last_unstale":"2024-02-10T07:35:01.702830+0000","last_undegraded":"2024-02-10T07:35:01.702830+0000","last_fullsized":"2024-02-10T07:35:01.702830+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:31.451210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:31.451210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":128,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.705522+0000","last_change":"2024-02-10T07:34:59.399183+0000","last_active":"2024-02-10T07:35:01.705522+0000","last_peered":"2024-02-10T07:35:01.705522+0000","last_clean":"2024-02-10T07:35:01.705522+0000","last_became_active":"2024-02-10T07:31:55.301260+0000","last_became_peered":"2024-02-10T07:31:55.301260+0000","last_unstale":"2024-02-10T07:35:01.705522+0000","last_undegraded":"2024-02-10T07:35:01.705522+0000","last_fullsized":"2024-02-10T07:35:01.705522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:59.399046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:59.399046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:25.056822+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":138,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682397+0000","last_change":"2024-02-10T07:34:54.340934+0000","last_active":"2024-02-10T07:35:01.682397+0000","last_peered":"2024-02-10T07:35:01.682397+0000","last_clean":"2024-02-10T07:35:01.682397+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:35:01.682397+0000","last_undegraded":"2024-02-10T07:35:01.682397+0000","last_fullsized":"2024-02-10T07:35:01.682397+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:54.340863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:54.340863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:24.123274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"120'16","reported_seq":176,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682424+0000","last_change":"2024-02-10T07:34:52.295223+0000","last_active":"2024-02-10T07:35:01.682424+0000","last_peered":"2024-02-10T07:35:01.682424+0000","last_clean":"2024-02-10T07:35:01.682424+0000","last_became_active":"2024-02-10T07:32:18.389564+0000","last_became_peered":"2024-02-10T07:32:18.389564+0000","last_unstale":"2024-02-10T07:35:01.682424+0000","last_undegraded":"2024-02-10T07:35:01.682424+0000","last_fullsized":"2024-02-10T07:35:01.682424+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'16","last_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:34:52.295064+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:55.073746+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":125,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.709965+0000","last_change":"2024-02-10T07:34:58.488060+0000","last_active":"2024-02-10T07:35:01.709965+0000","last_peered":"2024-02-10T07:35:01.709965+0000","last_clean":"2024-02-10T07:35:01.709965+0000","last_became_active":"2024-02-10T07:31:57.318009+0000","last_became_peered":"2024-02-10T07:31:57.318009+0000","last_unstale":"2024-02-10T07:35:01.709965+0000","last_undegraded":"2024-02-10T07:35:01.709965+0000","last_fullsized":"2024-02-10T07:35:01.709965+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:34:58.487937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:01.955191+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":176,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708215+0000","last_change":"2024-02-10T07:34:58.311787+0000","last_active":"2024-02-10T07:35:01.708215+0000","last_peered":"2024-02-10T07:35:01.708215+0000","last_clean":"2024-02-10T07:35:01.708215+0000","last_became_active":"2024-02-10T07:31:55.297670+0000","last_became_peered":"2024-02-10T07:31:55.297670+0000","last_unstale":"2024-02-10T07:35:01.708215+0000","last_undegraded":"2024-02-10T07:35:01.708215+0000","last_fullsized":"2024-02-10T07:35:01.708215+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:34:58.311646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:58.311646+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:23.772344+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":138,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.719440+0000","last_change":"2024-02-10T07:34:46.575053+0000","last_active":"2024-02-10T07:35:01.719440+0000","last_peered":"2024-02-10T07:35:01.719440+0000","last_clean":"2024-02-10T07:35:01.719440+0000","last_became_active":"2024-02-10T07:31:51.320644+0000","last_became_peered":"2024-02-10T07:31:51.320644+0000","last_unstale":"2024-02-10T07:35:01.719440+0000","last_undegraded":"2024-02-10T07:35:01.719440+0000","last_fullsized":"2024-02-10T07:35:01.719440+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:46.574930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:34:46.574930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:58.089207+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"96'15","reported_seq":182,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.705915+0000","last_change":"2024-02-10T07:34:18.901515+0000","last_active":"2024-02-10T07:35:01.705915+0000","last_peered":"2024-02-10T07:35:01.705915+0000","last_clean":"2024-02-10T07:35:01.705915+0000","last_became_active":"2024-02-10T07:34:18.894619+0000","last_became_peered":"2024-02-10T07:34:18.894619+0000","last_unstale":"2024-02-10T07:35:01.705915+0000","last_undegraded":"2024-02-10T07:35:01.705915+0000","last_fullsized":"2024-02-10T07:35:01.705915+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'15","last_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:33:48.458885+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.683255+0000","last_change":"2024-02-10T07:34:30.335810+0000","last_active":"2024-02-10T07:35:01.683255+0000","last_peered":"2024-02-10T07:35:01.683255+0000","last_clean":"2024-02-10T07:35:01.683255+0000","last_became_active":"2024-02-10T07:31:57.307583+0000","last_became_peered":"2024-02-10T07:31:57.307583+0000","last_unstale":"2024-02-10T07:35:01.683255+0000","last_undegraded":"2024-02-10T07:35:01.683255+0000","last_fullsized":"2024-02-10T07:35:01.683255+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:30.335721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:30.335721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:51.756306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":169,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.706089+0000","last_change":"2024-02-10T07:33:28.459581+0000","last_active":"2024-02-10T07:35:01.706089+0000","last_peered":"2024-02-10T07:35:01.706089+0000","last_clean":"2024-02-10T07:35:01.706089+0000","last_became_active":"2024-02-10T07:31:55.294594+0000","last_became_peered":"2024-02-10T07:31:55.294594+0000","last_unstale":"2024-02-10T07:35:01.706089+0000","last_undegraded":"2024-02-10T07:35:01.706089+0000","last_fullsized":"2024-02-10T07:35:01.706089+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:33:28.459467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:28.459467+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":137,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.711261+0000","last_change":"2024-02-10T07:34:33.603966+0000","last_active":"2024-02-10T07:35:01.711261+0000","last_peered":"2024-02-10T07:35:01.711261+0000","last_clean":"2024-02-10T07:35:01.711261+0000","last_became_active":"2024-02-10T07:31:51.282353+0000","last_became_peered":"2024-02-10T07:31:51.282353+0000","last_unstale":"2024-02-10T07:35:01.711261+0000","last_undegraded":"2024-02-10T07:35:01.711261+0000","last_fullsized":"2024-02-10T07:35:01.711261+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:33.603844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:33.603844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:00.993478+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":141,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708297+0000","last_change":"2024-02-10T07:33:09.243608+0000","last_active":"2024-02-10T07:35:01.708297+0000","last_peered":"2024-02-10T07:35:01.708297+0000","last_clean":"2024-02-10T07:35:01.708297+0000","last_became_active":"2024-02-10T07:31:53.285777+0000","last_became_peered":"2024-02-10T07:31:53.285777+0000","last_unstale":"2024-02-10T07:35:01.708297+0000","last_undegraded":"2024-02-10T07:35:01.708297+0000","last_fullsized":"2024-02-10T07:35:01.708297+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:33:09.243500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:09.243500+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":125,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.713482+0000","last_change":"2024-02-10T07:34:42.090173+0000","last_active":"2024-02-10T07:35:01.713482+0000","last_peered":"2024-02-10T07:35:01.713482+0000","last_clean":"2024-02-10T07:35:01.713482+0000","last_became_active":"2024-02-10T07:31:57.308615+0000","last_became_peered":"2024-02-10T07:31:57.308615+0000","last_unstale":"2024-02-10T07:35:01.713482+0000","last_undegraded":"2024-02-10T07:35:01.713482+0000","last_fullsized":"2024-02-10T07:35:01.713482+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:42.090037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:34:42.090037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:50.668927+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":122,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.705774+0000","last_change":"2024-02-10T07:33:17.587557+0000","last_active":"2024-02-10T07:35:01.705774+0000","last_peered":"2024-02-10T07:35:01.705774+0000","last_clean":"2024-02-10T07:35:01.705774+0000","last_became_active":"2024-02-10T07:31:55.297431+0000","last_became_peered":"2024-02-10T07:31:55.297431+0000","last_unstale":"2024-02-10T07:35:01.705774+0000","last_undegraded":"2024-02-10T07:35:01.705774+0000","last_fullsized":"2024-02-10T07:35:01.705774+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:33:17.587433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":138,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.719433+0000","last_change":"2024-02-10T07:34:52.665968+0000","last_active":"2024-02-10T07:35:01.719433+0000","last_peered":"2024-02-10T07:35:01.719433+0000","last_clean":"2024-02-10T07:35:01.719433+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:35:01.719433+0000","last_undegraded":"2024-02-10T07:35:01.719433+0000","last_fullsized":"2024-02-10T07:35:01.719433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:52.665846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:52.665846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:52.800299+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"89'13","reported_seq":153,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.710775+0000","last_change":"2024-02-10T07:34:35.582398+0000","last_active":"2024-02-10T07:35:01.710775+0000","last_peered":"2024-02-10T07:35:01.710775+0000","last_clean":"2024-02-10T07:35:01.710775+0000","last_became_active":"2024-02-10T07:31:53.295353+0000","last_became_peered":"2024-02-10T07:31:53.295353+0000","last_unstale":"2024-02-10T07:35:01.710775+0000","last_undegraded":"2024-02-10T07:35:01.710775+0000","last_fullsized":"2024-02-10T07:35:01.710775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-10T07:34:35.582276+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:34:35.582276+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:47.622072+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.721197+0000","last_change":"2024-02-10T07:34:32.515334+0000","last_active":"2024-02-10T07:35:01.721197+0000","last_peered":"2024-02-10T07:35:01.721197+0000","last_clean":"2024-02-10T07:35:01.721197+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:35:01.721197+0000","last_undegraded":"2024-02-10T07:35:01.721197+0000","last_fullsized":"2024-02-10T07:35:01.721197+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:32.515225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:32.515225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:59.065744+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":121,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708664+0000","last_change":"2024-02-10T07:33:32.294625+0000","last_active":"2024-02-10T07:35:01.708664+0000","last_peered":"2024-02-10T07:35:01.708664+0000","last_clean":"2024-02-10T07:35:01.708664+0000","last_became_active":"2024-02-10T07:31:55.296369+0000","last_became_peered":"2024-02-10T07:31:55.296369+0000","last_unstale":"2024-02-10T07:35:01.708664+0000","last_undegraded":"2024-02-10T07:35:01.708664+0000","last_fullsized":"2024-02-10T07:35:01.708664+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:32.294504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:32.294504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":137,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708884+0000","last_change":"2024-02-10T07:34:42.822759+0000","last_active":"2024-02-10T07:35:01.708884+0000","last_peered":"2024-02-10T07:35:01.708884+0000","last_clean":"2024-02-10T07:35:01.708884+0000","last_became_active":"2024-02-10T07:31:51.285159+0000","last_became_peered":"2024-02-10T07:31:51.285159+0000","last_unstale":"2024-02-10T07:35:01.708884+0000","last_undegraded":"2024-02-10T07:35:01.708884+0000","last_fullsized":"2024-02-10T07:35:01.708884+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:42.822615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:42.822615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:54.821259+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":135,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.705611+0000","last_change":"2024-02-10T07:34:40.532359+0000","last_active":"2024-02-10T07:35:01.705611+0000","last_peered":"2024-02-10T07:35:01.705611+0000","last_clean":"2024-02-10T07:35:01.705611+0000","last_became_active":"2024-02-10T07:31:53.296831+0000","last_became_peered":"2024-02-10T07:31:53.296831+0000","last_unstale":"2024-02-10T07:35:01.705611+0000","last_undegraded":"2024-02-10T07:35:01.705611+0000","last_fullsized":"2024-02-10T07:35:01.705611+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-10T07:34:40.531579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:40.531579+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:48.219679+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"122'4837","reported_seq":4895,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.673249+0000","last_change":"2024-02-10T07:33:53.679248+0000","last_active":"2024-02-10T07:35:02.673249+0000","last_peered":"2024-02-10T07:35:02.673249+0000","last_clean":"2024-02-10T07:35:02.673249+0000","last_became_active":"2024-02-10T07:33:53.678919+0000","last_became_peered":"2024-02-10T07:33:53.678919+0000","last_unstale":"2024-02-10T07:35:02.673249+0000","last_undegraded":"2024-02-10T07:35:02.673249+0000","last_fullsized":"2024-02-10T07:35:02.673249+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4837,"log_dups_size":0,"ondisk_log_size":4837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:17.561449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316997632,"num_objects":4837,"num_object_clones":0,"num_object_copies":14511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4837,"num_write_kb":309568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":117,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.707307+0000","last_change":"2024-02-10T07:33:48.258791+0000","last_active":"2024-02-10T07:35:01.707307+0000","last_peered":"2024-02-10T07:35:01.707307+0000","last_clean":"2024-02-10T07:35:01.707307+0000","last_became_active":"2024-02-10T07:31:57.311110+0000","last_became_peered":"2024-02-10T07:31:57.311110+0000","last_unstale":"2024-02-10T07:35:01.707307+0000","last_undegraded":"2024-02-10T07:35:01.707307+0000","last_fullsized":"2024-02-10T07:35:01.707307+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:48.258671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:48.258671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:02.137979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":142,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.721081+0000","last_change":"2024-02-10T07:33:42.202739+0000","last_active":"2024-02-10T07:35:01.721081+0000","last_peered":"2024-02-10T07:35:01.721081+0000","last_clean":"2024-02-10T07:35:01.721081+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:35:01.721081+0000","last_undegraded":"2024-02-10T07:35:01.721081+0000","last_fullsized":"2024-02-10T07:35:01.721081+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-10T07:33:42.202654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:42.202654+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":158,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.707089+0000","last_change":"2024-02-10T07:34:31.437585+0000","last_active":"2024-02-10T07:35:01.707089+0000","last_peered":"2024-02-10T07:35:01.707089+0000","last_clean":"2024-02-10T07:35:01.707089+0000","last_became_active":"2024-02-10T07:31:53.284817+0000","last_became_peered":"2024-02-10T07:31:53.284817+0000","last_unstale":"2024-02-10T07:35:01.707089+0000","last_undegraded":"2024-02-10T07:35:01.707089+0000","last_fullsized":"2024-02-10T07:35:01.707089+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:34:31.437421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:31.437421+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:37.912389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":169,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.719631+0000","last_change":"2024-02-10T07:33:28.244567+0000","last_active":"2024-02-10T07:35:01.719631+0000","last_peered":"2024-02-10T07:35:01.719631+0000","last_clean":"2024-02-10T07:35:01.719631+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:35:01.719631+0000","last_undegraded":"2024-02-10T07:35:01.719631+0000","last_fullsized":"2024-02-10T07:35:01.719631+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:33:28.244436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:28.244436+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"122'4852","reported_seq":4910,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.733827+0000","last_change":"2024-02-10T07:33:53.679379+0000","last_active":"2024-02-10T07:35:02.733827+0000","last_peered":"2024-02-10T07:35:02.733827+0000","last_clean":"2024-02-10T07:35:02.733827+0000","last_became_active":"2024-02-10T07:33:53.679099+0000","last_became_peered":"2024-02-10T07:33:53.679099+0000","last_unstale":"2024-02-10T07:35:02.733827+0000","last_undegraded":"2024-02-10T07:35:02.733827+0000","last_fullsized":"2024-02-10T07:35:02.733827+0000","mapping_epoch":98,"log_start":"104'300","ondisk_log_start":"104'300","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4552,"log_dups_size":0,"ondisk_log_size":4552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:11.224206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317980672,"num_objects":4852,"num_object_clones":0,"num_object_copies":14556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4852,"num_write_kb":310528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.705440+0000","last_change":"2024-02-10T07:33:34.504958+0000","last_active":"2024-02-10T07:35:01.705440+0000","last_peered":"2024-02-10T07:35:01.705440+0000","last_clean":"2024-02-10T07:35:01.705440+0000","last_became_active":"2024-02-10T07:31:57.314872+0000","last_became_peered":"2024-02-10T07:31:57.314872+0000","last_unstale":"2024-02-10T07:35:01.705440+0000","last_undegraded":"2024-02-10T07:35:01.705440+0000","last_fullsized":"2024-02-10T07:35:01.705440+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:34.504852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:34.504852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.720889+0000","last_change":"2024-02-10T07:33:31.170623+0000","last_active":"2024-02-10T07:35:01.720889+0000","last_peered":"2024-02-10T07:35:01.720889+0000","last_clean":"2024-02-10T07:35:01.720889+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:35:01.720889+0000","last_undegraded":"2024-02-10T07:35:01.720889+0000","last_fullsized":"2024-02-10T07:35:01.720889+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:33:31.170544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":167,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682388+0000","last_change":"2024-02-10T07:33:24.357572+0000","last_active":"2024-02-10T07:35:01.682388+0000","last_peered":"2024-02-10T07:35:01.682388+0000","last_clean":"2024-02-10T07:35:01.682388+0000","last_became_active":"2024-02-10T07:32:18.389409+0000","last_became_peered":"2024-02-10T07:32:18.389409+0000","last_unstale":"2024-02-10T07:35:01.682388+0000","last_undegraded":"2024-02-10T07:35:01.682388+0000","last_fullsized":"2024-02-10T07:35:01.682388+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:33:24.357485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:24.357485+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":129,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.712443+0000","last_change":"2024-02-10T07:34:40.067429+0000","last_active":"2024-02-10T07:35:01.712443+0000","last_peered":"2024-02-10T07:35:01.712443+0000","last_clean":"2024-02-10T07:35:01.712443+0000","last_became_active":"2024-02-10T07:31:55.303059+0000","last_became_peered":"2024-02-10T07:31:55.303059+0000","last_unstale":"2024-02-10T07:35:01.712443+0000","last_undegraded":"2024-02-10T07:35:01.712443+0000","last_fullsized":"2024-02-10T07:35:01.712443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:40.067288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:40.067288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:55.717664+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"122'4959","reported_seq":5017,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.750190+0000","last_change":"2024-02-10T07:33:53.680345+0000","last_active":"2024-02-10T07:35:02.750190+0000","last_peered":"2024-02-10T07:35:02.750190+0000","last_clean":"2024-02-10T07:35:02.750190+0000","last_became_active":"2024-02-10T07:33:53.680042+0000","last_became_peered":"2024-02-10T07:33:53.680042+0000","last_unstale":"2024-02-10T07:35:02.750190+0000","last_undegraded":"2024-02-10T07:35:02.750190+0000","last_fullsized":"2024-02-10T07:35:02.750190+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4959,"log_dups_size":0,"ondisk_log_size":4959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:22.564081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324993024,"num_objects":4959,"num_object_clones":0,"num_object_copies":14877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4959,"num_write_kb":317376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":126,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.719278+0000","last_change":"2024-02-10T07:34:30.506461+0000","last_active":"2024-02-10T07:35:01.719278+0000","last_peered":"2024-02-10T07:35:01.719278+0000","last_clean":"2024-02-10T07:35:01.719278+0000","last_became_active":"2024-02-10T07:31:57.304971+0000","last_became_peered":"2024-02-10T07:31:57.304971+0000","last_unstale":"2024-02-10T07:35:01.719278+0000","last_undegraded":"2024-02-10T07:35:01.719278+0000","last_fullsized":"2024-02-10T07:35:01.719278+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:30.506386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:30.506386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:57.021606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":129,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.712903+0000","last_change":"2024-02-10T07:34:50.188777+0000","last_active":"2024-02-10T07:35:01.712903+0000","last_peered":"2024-02-10T07:35:01.712903+0000","last_clean":"2024-02-10T07:35:01.712903+0000","last_became_active":"2024-02-10T07:31:55.298322+0000","last_became_peered":"2024-02-10T07:31:55.298322+0000","last_unstale":"2024-02-10T07:35:01.712903+0000","last_undegraded":"2024-02-10T07:35:01.712903+0000","last_fullsized":"2024-02-10T07:35:01.712903+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:50.188510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:50.188510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:09.879905+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":138,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.721505+0000","last_change":"2024-02-10T07:34:57.552605+0000","last_active":"2024-02-10T07:35:01.721505+0000","last_peered":"2024-02-10T07:35:01.721505+0000","last_clean":"2024-02-10T07:35:01.721505+0000","last_became_active":"2024-02-10T07:31:51.283433+0000","last_became_peered":"2024-02-10T07:31:51.283433+0000","last_unstale":"2024-02-10T07:35:01.721505+0000","last_undegraded":"2024-02-10T07:35:01.721505+0000","last_fullsized":"2024-02-10T07:35:01.721505+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:57.552495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:57.552495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:59.008097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":164,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.707379+0000","last_change":"2024-02-10T07:34:37.811083+0000","last_active":"2024-02-10T07:35:01.707379+0000","last_peered":"2024-02-10T07:35:01.707379+0000","last_clean":"2024-02-10T07:35:01.707379+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:35:01.707379+0000","last_undegraded":"2024-02-10T07:35:01.707379+0000","last_fullsized":"2024-02-10T07:35:01.707379+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-10T07:34:37.810931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:37.810931+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:59.912539+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"122'4841","reported_seq":4899,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.708060+0000","last_change":"2024-02-10T07:33:53.680187+0000","last_active":"2024-02-10T07:35:02.708060+0000","last_peered":"2024-02-10T07:35:02.708060+0000","last_clean":"2024-02-10T07:35:02.708060+0000","last_became_active":"2024-02-10T07:33:53.679911+0000","last_became_peered":"2024-02-10T07:33:53.679911+0000","last_unstale":"2024-02-10T07:35:02.708060+0000","last_undegraded":"2024-02-10T07:35:02.708060+0000","last_fullsized":"2024-02-10T07:35:02.708060+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:20.187848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317259776,"num_objects":4841,"num_object_clones":0,"num_object_copies":14523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4841,"num_write_kb":309824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":125,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.710536+0000","last_change":"2024-02-10T07:34:39.652304+0000","last_active":"2024-02-10T07:35:01.710536+0000","last_peered":"2024-02-10T07:35:01.710536+0000","last_clean":"2024-02-10T07:35:01.710536+0000","last_became_active":"2024-02-10T07:31:57.313394+0000","last_became_peered":"2024-02-10T07:31:57.313394+0000","last_unstale":"2024-02-10T07:35:01.710536+0000","last_undegraded":"2024-02-10T07:35:01.710536+0000","last_fullsized":"2024-02-10T07:35:01.710536+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:39.652060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:39.652060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:01.416562+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":129,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.712572+0000","last_change":"2024-02-10T07:34:53.160632+0000","last_active":"2024-02-10T07:35:01.712572+0000","last_peered":"2024-02-10T07:35:01.712572+0000","last_clean":"2024-02-10T07:35:01.712572+0000","last_became_active":"2024-02-10T07:31:55.298622+0000","last_became_peered":"2024-02-10T07:31:55.298622+0000","last_unstale":"2024-02-10T07:35:01.712572+0000","last_undegraded":"2024-02-10T07:35:01.712572+0000","last_fullsized":"2024-02-10T07:35:01.712572+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:53.160525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:53.160525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:12.352277+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":129,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.707597+0000","last_change":"2024-02-10T07:33:19.291637+0000","last_active":"2024-02-10T07:35:01.707597+0000","last_peered":"2024-02-10T07:35:01.707597+0000","last_clean":"2024-02-10T07:35:01.707597+0000","last_became_active":"2024-02-10T07:31:51.283208+0000","last_became_peered":"2024-02-10T07:31:51.283208+0000","last_unstale":"2024-02-10T07:35:01.707597+0000","last_undegraded":"2024-02-10T07:35:01.707597+0000","last_fullsized":"2024-02-10T07:35:01.707597+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:19.291556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:19.291556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":141,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.711400+0000","last_change":"2024-02-10T07:33:18.612946+0000","last_active":"2024-02-10T07:35:01.711400+0000","last_peered":"2024-02-10T07:35:01.711400+0000","last_clean":"2024-02-10T07:35:01.711400+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:35:01.711400+0000","last_undegraded":"2024-02-10T07:35:01.711400+0000","last_fullsized":"2024-02-10T07:35:01.711400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:33:18.612847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:18.612847+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:06.431195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"122'4714","reported_seq":4772,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.651761+0000","last_change":"2024-02-10T07:33:53.681536+0000","last_active":"2024-02-10T07:35:02.651761+0000","last_peered":"2024-02-10T07:35:02.651761+0000","last_clean":"2024-02-10T07:35:02.651761+0000","last_became_active":"2024-02-10T07:33:53.681201+0000","last_became_peered":"2024-02-10T07:33:53.681201+0000","last_unstale":"2024-02-10T07:35:02.651761+0000","last_undegraded":"2024-02-10T07:35:02.651761+0000","last_fullsized":"2024-02-10T07:35:02.651761+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:22.277713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308936704,"num_objects":4714,"num_object_clones":0,"num_object_copies":14142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4714,"num_write_kb":301696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.703244+0000","last_change":"2024-02-10T07:34:39.563811+0000","last_active":"2024-02-10T07:35:01.703244+0000","last_peered":"2024-02-10T07:35:01.703244+0000","last_clean":"2024-02-10T07:35:01.703244+0000","last_became_active":"2024-02-10T07:31:57.313854+0000","last_became_peered":"2024-02-10T07:31:57.313854+0000","last_unstale":"2024-02-10T07:35:01.703244+0000","last_undegraded":"2024-02-10T07:35:01.703244+0000","last_fullsized":"2024-02-10T07:35:01.703244+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:39.563733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:39.563733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:00.032067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":148,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.707442+0000","last_change":"2024-02-10T07:34:44.402863+0000","last_active":"2024-02-10T07:35:01.707442+0000","last_peered":"2024-02-10T07:35:01.707442+0000","last_clean":"2024-02-10T07:35:01.707442+0000","last_became_active":"2024-02-10T07:31:50.278548+0000","last_became_peered":"2024-02-10T07:31:50.278548+0000","last_unstale":"2024-02-10T07:35:01.707442+0000","last_undegraded":"2024-02-10T07:35:01.707442+0000","last_fullsized":"2024-02-10T07:35:01.707442+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:34:44.402778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:34:44.402778+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:00.957865+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001361225,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682312+0000","last_change":"2024-02-10T07:33:31.249197+0000","last_active":"2024-02-10T07:35:01.682312+0000","last_peered":"2024-02-10T07:35:01.682312+0000","last_clean":"2024-02-10T07:35:01.682312+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:35:01.682312+0000","last_undegraded":"2024-02-10T07:35:01.682312+0000","last_fullsized":"2024-02-10T07:35:01.682312+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:31.249117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:31.249117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":211,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.710640+0000","last_change":"2024-02-10T07:33:09.271524+0000","last_active":"2024-02-10T07:35:01.710640+0000","last_peered":"2024-02-10T07:35:01.710640+0000","last_clean":"2024-02-10T07:35:01.710640+0000","last_became_active":"2024-02-10T07:31:53.301863+0000","last_became_peered":"2024-02-10T07:31:53.301863+0000","last_unstale":"2024-02-10T07:35:01.710640+0000","last_undegraded":"2024-02-10T07:35:01.710640+0000","last_fullsized":"2024-02-10T07:35:01.710640+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-02-10T07:33:09.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:09.271402+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":129,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.715517+0000","last_change":"2024-02-10T07:34:45.597280+0000","last_active":"2024-02-10T07:35:01.715517+0000","last_peered":"2024-02-10T07:35:01.715517+0000","last_clean":"2024-02-10T07:35:01.715517+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:35:01.715517+0000","last_undegraded":"2024-02-10T07:35:01.715517+0000","last_fullsized":"2024-02-10T07:35:01.715517+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:45.597169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:45.597169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:01.604396+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"122'4761","reported_seq":4818,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.772186+0000","last_change":"2024-02-10T07:33:53.680679+0000","last_active":"2024-02-10T07:35:02.772186+0000","last_peered":"2024-02-10T07:35:02.772186+0000","last_clean":"2024-02-10T07:35:02.772186+0000","last_became_active":"2024-02-10T07:33:53.680304+0000","last_became_peered":"2024-02-10T07:33:53.680304+0000","last_unstale":"2024-02-10T07:35:02.772186+0000","last_undegraded":"2024-02-10T07:35:02.772186+0000","last_fullsized":"2024-02-10T07:35:02.772186+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4761,"log_dups_size":0,"ondisk_log_size":4761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:12.693018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312016896,"num_objects":4761,"num_object_clones":0,"num_object_copies":14283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4761,"num_write_kb":304704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":126,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.721621+0000","last_change":"2024-02-10T07:34:51.654337+0000","last_active":"2024-02-10T07:35:01.721621+0000","last_peered":"2024-02-10T07:35:01.721621+0000","last_clean":"2024-02-10T07:35:01.721621+0000","last_became_active":"2024-02-10T07:31:57.299958+0000","last_became_peered":"2024-02-10T07:31:57.299958+0000","last_unstale":"2024-02-10T07:35:01.721621+0000","last_undegraded":"2024-02-10T07:35:01.721621+0000","last_fullsized":"2024-02-10T07:35:01.721621+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:51.654211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:51.654211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:16.858026+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":136,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.704192+0000","last_change":"2024-02-10T07:34:48.321459+0000","last_active":"2024-02-10T07:35:01.704192+0000","last_peered":"2024-02-10T07:35:01.704192+0000","last_clean":"2024-02-10T07:35:01.704192+0000","last_became_active":"2024-02-10T07:31:51.320864+0000","last_became_peered":"2024-02-10T07:31:51.320864+0000","last_unstale":"2024-02-10T07:35:01.704192+0000","last_undegraded":"2024-02-10T07:35:01.704192+0000","last_fullsized":"2024-02-10T07:35:01.704192+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:48.321040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:48.321040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:00.679661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":146,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.712519+0000","last_change":"2024-02-10T07:34:52.140876+0000","last_active":"2024-02-10T07:35:01.712519+0000","last_peered":"2024-02-10T07:35:01.712519+0000","last_clean":"2024-02-10T07:35:01.712519+0000","last_became_active":"2024-02-10T07:31:50.281517+0000","last_became_peered":"2024-02-10T07:31:50.281517+0000","last_unstale":"2024-02-10T07:35:01.712519+0000","last_undegraded":"2024-02-10T07:35:01.712519+0000","last_fullsized":"2024-02-10T07:35:01.712519+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:52.140730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:34:52.140730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:02.004303+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708748+0000","last_change":"2024-02-10T07:34:59.267360+0000","last_active":"2024-02-10T07:35:01.708748+0000","last_peered":"2024-02-10T07:35:01.708748+0000","last_clean":"2024-02-10T07:35:01.708748+0000","last_became_active":"2024-02-10T07:33:46.500511+0000","last_became_peered":"2024-02-10T07:33:46.500511+0000","last_unstale":"2024-02-10T07:35:01.708748+0000","last_undegraded":"2024-02-10T07:35:01.708748+0000","last_fullsized":"2024-02-10T07:35:01.708748+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-10T07:34:59.267211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:59.267211+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:11.603544+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":129,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.714246+0000","last_change":"2024-02-10T07:34:41.086869+0000","last_active":"2024-02-10T07:35:01.714246+0000","last_peered":"2024-02-10T07:35:01.714246+0000","last_clean":"2024-02-10T07:35:01.714246+0000","last_became_active":"2024-02-10T07:31:55.298933+0000","last_became_peered":"2024-02-10T07:31:55.298933+0000","last_unstale":"2024-02-10T07:35:01.714246+0000","last_undegraded":"2024-02-10T07:35:01.714246+0000","last_fullsized":"2024-02-10T07:35:01.714246+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:41.086507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:41.086507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:56.970713+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"122'4789","reported_seq":4847,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.763062+0000","last_change":"2024-02-10T07:33:53.682874+0000","last_active":"2024-02-10T07:35:02.763062+0000","last_peered":"2024-02-10T07:35:02.763062+0000","last_clean":"2024-02-10T07:35:02.763062+0000","last_became_active":"2024-02-10T07:33:53.682538+0000","last_became_peered":"2024-02-10T07:33:53.682538+0000","last_unstale":"2024-02-10T07:35:02.763062+0000","last_undegraded":"2024-02-10T07:35:02.763062+0000","last_fullsized":"2024-02-10T07:35:02.763062+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4789,"log_dups_size":0,"ondisk_log_size":4789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:06.431592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313851904,"num_objects":4789,"num_object_clones":0,"num_object_copies":14367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":306496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":119,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708490+0000","last_change":"2024-02-10T07:33:41.150537+0000","last_active":"2024-02-10T07:35:01.708490+0000","last_peered":"2024-02-10T07:35:01.708490+0000","last_clean":"2024-02-10T07:35:01.708490+0000","last_became_active":"2024-02-10T07:31:57.309929+0000","last_became_peered":"2024-02-10T07:31:57.309929+0000","last_unstale":"2024-02-10T07:35:01.708490+0000","last_undegraded":"2024-02-10T07:35:01.708490+0000","last_fullsized":"2024-02-10T07:35:01.708490+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:33:41.150454+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:33:41.150454+0000","last_clean_scrub_stamp":"2024-02-10T07:33:41.150454+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":383,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.719492+0000","last_change":"2024-02-10T07:34:44.557795+0000","last_active":"2024-02-10T07:35:01.719492+0000","last_peered":"2024-02-10T07:35:01.719492+0000","last_clean":"2024-02-10T07:35:01.719492+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:35:01.719492+0000","last_undegraded":"2024-02-10T07:35:01.719492+0000","last_fullsized":"2024-02-10T07:35:01.719492+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:34:44.557700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:34:44.557700+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:07.903579+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.705969+0000","last_change":"2024-02-10T07:33:23.493888+0000","last_active":"2024-02-10T07:35:01.705969+0000","last_peered":"2024-02-10T07:35:01.705969+0000","last_clean":"2024-02-10T07:35:01.705969+0000","last_became_active":"2024-02-10T07:31:51.283332+0000","last_became_peered":"2024-02-10T07:31:51.283332+0000","last_unstale":"2024-02-10T07:35:01.705969+0000","last_undegraded":"2024-02-10T07:35:01.705969+0000","last_fullsized":"2024-02-10T07:35:01.705969+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:23.493813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:23.493813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"89'24","reported_seq":202,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708369+0000","last_change":"2024-02-10T07:34:40.809650+0000","last_active":"2024-02-10T07:35:01.708369+0000","last_peered":"2024-02-10T07:35:01.708369+0000","last_clean":"2024-02-10T07:35:01.708369+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:35:01.708369+0000","last_undegraded":"2024-02-10T07:35:01.708369+0000","last_fullsized":"2024-02-10T07:35:01.708369+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'24","last_scrub_stamp":"2024-02-10T07:34:40.809513+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:34:40.809513+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:05.666758+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":122,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.722139+0000","last_change":"2024-02-10T07:34:38.488157+0000","last_active":"2024-02-10T07:35:01.722139+0000","last_peered":"2024-02-10T07:35:01.722139+0000","last_clean":"2024-02-10T07:35:01.722139+0000","last_became_active":"2024-02-10T07:31:55.299928+0000","last_became_peered":"2024-02-10T07:31:55.299928+0000","last_unstale":"2024-02-10T07:35:01.722139+0000","last_undegraded":"2024-02-10T07:35:01.722139+0000","last_fullsized":"2024-02-10T07:35:01.722139+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:38.488082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:38.488082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:54.688121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"122'4739","reported_seq":4797,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.644303+0000","last_change":"2024-02-10T07:33:53.682991+0000","last_active":"2024-02-10T07:35:02.644303+0000","last_peered":"2024-02-10T07:35:02.644303+0000","last_clean":"2024-02-10T07:35:02.644303+0000","last_became_active":"2024-02-10T07:33:53.682606+0000","last_became_peered":"2024-02-10T07:33:53.682606+0000","last_unstale":"2024-02-10T07:35:02.644303+0000","last_undegraded":"2024-02-10T07:35:02.644303+0000","last_fullsized":"2024-02-10T07:35:02.644303+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:15.770831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310575104,"num_objects":4739,"num_object_clones":0,"num_object_copies":14217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4739,"num_write_kb":303296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":73,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.707458+0000","last_change":"2024-02-10T07:34:54.270014+0000","last_active":"2024-02-10T07:35:01.707458+0000","last_peered":"2024-02-10T07:35:01.707458+0000","last_clean":"2024-02-10T07:35:01.707458+0000","last_became_active":"2024-02-10T07:33:46.500003+0000","last_became_peered":"2024-02-10T07:33:46.500003+0000","last_unstale":"2024-02-10T07:35:01.707458+0000","last_undegraded":"2024-02-10T07:35:01.707458+0000","last_fullsized":"2024-02-10T07:35:01.707458+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:54.269886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:11.220506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":129,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.710717+0000","last_change":"2024-02-10T07:33:40.634805+0000","last_active":"2024-02-10T07:35:01.710717+0000","last_peered":"2024-02-10T07:35:01.710717+0000","last_clean":"2024-02-10T07:35:01.710717+0000","last_became_active":"2024-02-10T07:31:51.286076+0000","last_became_peered":"2024-02-10T07:31:51.286076+0000","last_unstale":"2024-02-10T07:35:01.710717+0000","last_undegraded":"2024-02-10T07:35:01.710717+0000","last_fullsized":"2024-02-10T07:35:01.710717+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:40.634728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:40.634728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:06.431256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":328,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.719417+0000","last_change":"2024-02-10T07:34:31.523532+0000","last_active":"2024-02-10T07:35:01.719417+0000","last_peered":"2024-02-10T07:35:01.719417+0000","last_clean":"2024-02-10T07:35:01.719417+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:35:01.719417+0000","last_undegraded":"2024-02-10T07:35:01.719417+0000","last_fullsized":"2024-02-10T07:35:01.719417+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:34:31.523406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:34:31.523406+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:59.908550+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":186,"num_read_kb":181,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":143,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.710129+0000","last_change":"2024-02-10T07:33:37.561833+0000","last_active":"2024-02-10T07:35:01.710129+0000","last_peered":"2024-02-10T07:35:01.710129+0000","last_clean":"2024-02-10T07:35:01.710129+0000","last_became_active":"2024-02-10T07:31:53.300687+0000","last_became_peered":"2024-02-10T07:31:53.300687+0000","last_unstale":"2024-02-10T07:35:01.710129+0000","last_undegraded":"2024-02-10T07:35:01.710129+0000","last_fullsized":"2024-02-10T07:35:01.710129+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:33:37.561741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:37.561741+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:06.431523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":121,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708841+0000","last_change":"2024-02-10T07:33:36.210648+0000","last_active":"2024-02-10T07:35:01.708841+0000","last_peered":"2024-02-10T07:35:01.708841+0000","last_clean":"2024-02-10T07:35:01.708841+0000","last_became_active":"2024-02-10T07:31:55.297178+0000","last_became_peered":"2024-02-10T07:31:55.297178+0000","last_unstale":"2024-02-10T07:35:01.708841+0000","last_undegraded":"2024-02-10T07:35:01.708841+0000","last_fullsized":"2024-02-10T07:35:01.708841+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:36.210570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:36.210570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"122'4999","reported_seq":7851,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.662544+0000","last_change":"2024-02-10T07:34:30.780875+0000","last_active":"2024-02-10T07:35:02.662544+0000","last_peered":"2024-02-10T07:35:02.662544+0000","last_clean":"2024-02-10T07:35:02.662544+0000","last_became_active":"2024-02-10T07:34:30.779160+0000","last_became_peered":"2024-02-10T07:34:30.779160+0000","last_unstale":"2024-02-10T07:35:02.662544+0000","last_undegraded":"2024-02-10T07:35:02.662544+0000","last_fullsized":"2024-02-10T07:35:02.662544+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4999,"log_dups_size":0,"ondisk_log_size":4999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:07.304396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327614464,"num_objects":4999,"num_object_clones":0,"num_object_copies":14997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4999,"num_write_kb":319936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1392,"num_bytes_recovered":91226112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":116,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.704679+0000","last_change":"2024-02-10T07:33:29.260276+0000","last_active":"2024-02-10T07:35:01.704679+0000","last_peered":"2024-02-10T07:35:01.704679+0000","last_clean":"2024-02-10T07:35:01.704679+0000","last_became_active":"2024-02-10T07:31:57.307615+0000","last_became_peered":"2024-02-10T07:31:57.307615+0000","last_unstale":"2024-02-10T07:35:01.704679+0000","last_undegraded":"2024-02-10T07:35:01.704679+0000","last_fullsized":"2024-02-10T07:35:01.704679+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:29.260196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:29.260196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":128,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.710551+0000","last_change":"2024-02-10T07:33:06.270053+0000","last_active":"2024-02-10T07:35:01.710551+0000","last_peered":"2024-02-10T07:35:01.710551+0000","last_clean":"2024-02-10T07:35:01.710551+0000","last_became_active":"2024-02-10T07:31:51.320752+0000","last_became_peered":"2024-02-10T07:31:51.320752+0000","last_unstale":"2024-02-10T07:35:01.710551+0000","last_undegraded":"2024-02-10T07:35:01.710551+0000","last_fullsized":"2024-02-10T07:35:01.710551+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.269938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.269938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":161,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.710107+0000","last_change":"2024-02-10T07:33:36.577169+0000","last_active":"2024-02-10T07:35:01.710107+0000","last_peered":"2024-02-10T07:35:01.710107+0000","last_clean":"2024-02-10T07:35:01.710107+0000","last_became_active":"2024-02-10T07:31:53.289381+0000","last_became_peered":"2024-02-10T07:31:53.289381+0000","last_unstale":"2024-02-10T07:35:01.710107+0000","last_undegraded":"2024-02-10T07:35:01.710107+0000","last_fullsized":"2024-02-10T07:35:01.710107+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-10T07:33:36.577066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:36.577066+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:06.431229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":129,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.707494+0000","last_change":"2024-02-10T07:34:46.896018+0000","last_active":"2024-02-10T07:35:01.707494+0000","last_peered":"2024-02-10T07:35:01.707494+0000","last_clean":"2024-02-10T07:35:01.707494+0000","last_became_active":"2024-02-10T07:31:55.290319+0000","last_became_peered":"2024-02-10T07:31:55.290319+0000","last_unstale":"2024-02-10T07:35:01.707494+0000","last_undegraded":"2024-02-10T07:35:01.707494+0000","last_fullsized":"2024-02-10T07:35:01.707494+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:46.895907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:46.895907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:06.712851+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"122'4745","reported_seq":4803,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.683941+0000","last_change":"2024-02-10T07:33:53.684834+0000","last_active":"2024-02-10T07:35:02.683941+0000","last_peered":"2024-02-10T07:35:02.683941+0000","last_clean":"2024-02-10T07:35:02.683941+0000","last_became_active":"2024-02-10T07:33:53.684493+0000","last_became_peered":"2024-02-10T07:33:53.684493+0000","last_unstale":"2024-02-10T07:35:02.683941+0000","last_undegraded":"2024-02-10T07:35:02.683941+0000","last_fullsized":"2024-02-10T07:35:02.683941+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4745,"log_dups_size":0,"ondisk_log_size":4745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310968320,"num_objects":4745,"num_object_clones":0,"num_object_copies":14235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4745,"num_write_kb":303680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":125,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.709850+0000","last_change":"2024-02-10T07:34:46.550492+0000","last_active":"2024-02-10T07:35:01.709850+0000","last_peered":"2024-02-10T07:35:01.709850+0000","last_clean":"2024-02-10T07:35:01.709850+0000","last_became_active":"2024-02-10T07:31:57.317976+0000","last_became_peered":"2024-02-10T07:31:57.317976+0000","last_unstale":"2024-02-10T07:35:01.709850+0000","last_undegraded":"2024-02-10T07:35:01.709850+0000","last_fullsized":"2024-02-10T07:35:01.709850+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:46.550386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:46.550386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:05.085118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":129,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708078+0000","last_change":"2024-02-10T07:33:42.126774+0000","last_active":"2024-02-10T07:35:01.708078+0000","last_peered":"2024-02-10T07:35:01.708078+0000","last_clean":"2024-02-10T07:35:01.708078+0000","last_became_active":"2024-02-10T07:31:51.320673+0000","last_became_peered":"2024-02-10T07:31:51.320673+0000","last_unstale":"2024-02-10T07:35:01.708078+0000","last_undegraded":"2024-02-10T07:35:01.708078+0000","last_fullsized":"2024-02-10T07:35:01.708078+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:42.126696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:42.126696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:07.693685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.707584+0000","last_change":"2024-02-10T07:34:40.406967+0000","last_active":"2024-02-10T07:35:01.707584+0000","last_peered":"2024-02-10T07:35:01.707584+0000","last_clean":"2024-02-10T07:35:01.707584+0000","last_became_active":"2024-02-10T07:33:46.500435+0000","last_became_peered":"2024-02-10T07:33:46.500435+0000","last_unstale":"2024-02-10T07:35:01.707584+0000","last_undegraded":"2024-02-10T07:35:01.707584+0000","last_fullsized":"2024-02-10T07:35:01.707584+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:34:40.406860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:40.406860+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:04.013861+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":168,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.711705+0000","last_change":"2024-02-10T07:33:33.610339+0000","last_active":"2024-02-10T07:35:01.711705+0000","last_peered":"2024-02-10T07:35:01.711705+0000","last_clean":"2024-02-10T07:35:01.711705+0000","last_became_active":"2024-02-10T07:31:55.295732+0000","last_became_peered":"2024-02-10T07:31:55.295732+0000","last_unstale":"2024-02-10T07:35:01.711705+0000","last_undegraded":"2024-02-10T07:35:01.711705+0000","last_fullsized":"2024-02-10T07:35:01.711705+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:33:33.610254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:33.610254+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:06.431551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"122'4952","reported_seq":7884,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.733972+0000","last_change":"2024-02-10T07:34:38.996284+0000","last_active":"2024-02-10T07:35:02.733972+0000","last_peered":"2024-02-10T07:35:02.733972+0000","last_clean":"2024-02-10T07:35:02.733972+0000","last_became_active":"2024-02-10T07:34:38.991359+0000","last_became_peered":"2024-02-10T07:34:38.991359+0000","last_unstale":"2024-02-10T07:35:02.733972+0000","last_undegraded":"2024-02-10T07:35:02.733972+0000","last_fullsized":"2024-02-10T07:35:02.733972+0000","mapping_epoch":115,"log_start":"104'400","ondisk_log_start":"104'400","created":98,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4552,"log_dups_size":0,"ondisk_log_size":4552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:06.771932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324534272,"num_objects":4952,"num_object_clones":0,"num_object_copies":14856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4952,"num_write_kb":316928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1432,"num_bytes_recovered":93847552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.719653+0000","last_change":"2024-02-10T07:33:49.128880+0000","last_active":"2024-02-10T07:35:01.719653+0000","last_peered":"2024-02-10T07:35:01.719653+0000","last_clean":"2024-02-10T07:35:01.719653+0000","last_became_active":"2024-02-10T07:31:57.305620+0000","last_became_peered":"2024-02-10T07:31:57.305620+0000","last_unstale":"2024-02-10T07:35:01.719653+0000","last_undegraded":"2024-02-10T07:35:01.719653+0000","last_fullsized":"2024-02-10T07:35:01.719653+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:49.128788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:49.128788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:02.818326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":138,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.710217+0000","last_change":"2024-02-10T07:34:41.561592+0000","last_active":"2024-02-10T07:35:01.710217+0000","last_peered":"2024-02-10T07:35:01.710217+0000","last_clean":"2024-02-10T07:35:01.710217+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:35:01.710217+0000","last_undegraded":"2024-02-10T07:35:01.710217+0000","last_fullsized":"2024-02-10T07:35:01.710217+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:34:41.561478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:41.561478+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:54.371760+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":147,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682202+0000","last_change":"2024-02-10T07:33:37.293146+0000","last_active":"2024-02-10T07:35:01.682202+0000","last_peered":"2024-02-10T07:35:01.682202+0000","last_clean":"2024-02-10T07:35:01.682202+0000","last_became_active":"2024-02-10T07:31:53.286884+0000","last_became_peered":"2024-02-10T07:31:53.286884+0000","last_unstale":"2024-02-10T07:35:01.682202+0000","last_undegraded":"2024-02-10T07:35:01.682202+0000","last_fullsized":"2024-02-10T07:35:01.682202+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-02-10T07:33:37.293058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:37.293058+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:06.976018+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":177,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.703058+0000","last_change":"2024-02-10T07:34:59.737795+0000","last_active":"2024-02-10T07:35:01.703058+0000","last_peered":"2024-02-10T07:35:01.703058+0000","last_clean":"2024-02-10T07:35:01.703058+0000","last_became_active":"2024-02-10T07:31:55.302863+0000","last_became_peered":"2024-02-10T07:31:55.302863+0000","last_unstale":"2024-02-10T07:35:01.703058+0000","last_undegraded":"2024-02-10T07:35:01.703058+0000","last_fullsized":"2024-02-10T07:35:01.703058+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:34:59.737682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:59.737682+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:01.389008+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"122'4813","reported_seq":4871,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.693141+0000","last_change":"2024-02-10T07:33:53.681778+0000","last_active":"2024-02-10T07:35:02.693141+0000","last_peered":"2024-02-10T07:35:02.693141+0000","last_clean":"2024-02-10T07:35:02.693141+0000","last_became_active":"2024-02-10T07:33:53.681493+0000","last_became_peered":"2024-02-10T07:33:53.681493+0000","last_unstale":"2024-02-10T07:35:02.693141+0000","last_undegraded":"2024-02-10T07:35:02.693141+0000","last_fullsized":"2024-02-10T07:35:02.693141+0000","mapping_epoch":98,"log_start":"104'200","ondisk_log_start":"104'200","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4613,"log_dups_size":0,"ondisk_log_size":4613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:18.243225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315424768,"num_objects":4813,"num_object_clones":0,"num_object_copies":14439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":308032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682127+0000","last_change":"2024-02-10T07:34:33.311796+0000","last_active":"2024-02-10T07:35:01.682127+0000","last_peered":"2024-02-10T07:35:01.682127+0000","last_clean":"2024-02-10T07:35:01.682127+0000","last_became_active":"2024-02-10T07:31:57.316047+0000","last_became_peered":"2024-02-10T07:31:57.316047+0000","last_unstale":"2024-02-10T07:35:01.682127+0000","last_undegraded":"2024-02-10T07:35:01.682127+0000","last_fullsized":"2024-02-10T07:35:01.682127+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:33.311699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:33.311699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:54.415243+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.705454+0000","last_change":"2024-02-10T07:33:20.491471+0000","last_active":"2024-02-10T07:35:01.705454+0000","last_peered":"2024-02-10T07:35:01.705454+0000","last_clean":"2024-02-10T07:35:01.705454+0000","last_became_active":"2024-02-10T07:31:51.285333+0000","last_became_peered":"2024-02-10T07:31:51.285333+0000","last_unstale":"2024-02-10T07:35:01.705454+0000","last_undegraded":"2024-02-10T07:35:01.705454+0000","last_fullsized":"2024-02-10T07:35:01.705454+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:20.491354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:20.491354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":158,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.702961+0000","last_change":"2024-02-10T07:33:08.541272+0000","last_active":"2024-02-10T07:35:01.702961+0000","last_peered":"2024-02-10T07:35:01.702961+0000","last_clean":"2024-02-10T07:35:01.702961+0000","last_became_active":"2024-02-10T07:31:53.290157+0000","last_became_peered":"2024-02-10T07:31:53.290157+0000","last_unstale":"2024-02-10T07:35:01.702961+0000","last_undegraded":"2024-02-10T07:35:01.702961+0000","last_fullsized":"2024-02-10T07:35:01.702961+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-10T07:33:08.541155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:08.541155+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682059+0000","last_change":"2024-02-10T07:34:42.223857+0000","last_active":"2024-02-10T07:35:01.682059+0000","last_peered":"2024-02-10T07:35:01.682059+0000","last_clean":"2024-02-10T07:35:01.682059+0000","last_became_active":"2024-02-10T07:31:55.295900+0000","last_became_peered":"2024-02-10T07:31:55.295900+0000","last_unstale":"2024-02-10T07:35:01.682059+0000","last_undegraded":"2024-02-10T07:35:01.682059+0000","last_fullsized":"2024-02-10T07:35:01.682059+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:42.223775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:42.223775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:01.442986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"122'4835","reported_seq":4893,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.676200+0000","last_change":"2024-02-10T07:33:53.676369+0000","last_active":"2024-02-10T07:35:02.676200+0000","last_peered":"2024-02-10T07:35:02.676200+0000","last_clean":"2024-02-10T07:35:02.676200+0000","last_became_active":"2024-02-10T07:33:53.676032+0000","last_became_peered":"2024-02-10T07:33:53.676032+0000","last_unstale":"2024-02-10T07:35:02.676200+0000","last_undegraded":"2024-02-10T07:35:02.676200+0000","last_fullsized":"2024-02-10T07:35:02.676200+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:21.446005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316866560,"num_objects":4835,"num_object_clones":0,"num_object_copies":14505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4835,"num_write_kb":309440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":117,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.707998+0000","last_change":"2024-02-10T07:33:46.231777+0000","last_active":"2024-02-10T07:35:01.707998+0000","last_peered":"2024-02-10T07:35:01.707998+0000","last_clean":"2024-02-10T07:35:01.707998+0000","last_became_active":"2024-02-10T07:31:57.311054+0000","last_became_peered":"2024-02-10T07:31:57.311054+0000","last_unstale":"2024-02-10T07:35:01.707998+0000","last_undegraded":"2024-02-10T07:35:01.707998+0000","last_fullsized":"2024-02-10T07:35:01.707998+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:46.231673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:46.231673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":169,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.683048+0000","last_change":"2024-02-10T07:34:28.340061+0000","last_active":"2024-02-10T07:35:01.683048+0000","last_peered":"2024-02-10T07:35:01.683048+0000","last_clean":"2024-02-10T07:35:01.683048+0000","last_became_active":"2024-02-10T07:31:55.295067+0000","last_became_peered":"2024-02-10T07:31:55.295067+0000","last_unstale":"2024-02-10T07:35:01.683048+0000","last_undegraded":"2024-02-10T07:35:01.683048+0000","last_fullsized":"2024-02-10T07:35:01.683048+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:34:28.339952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:28.339952+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:53.944683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.702868+0000","last_change":"2024-02-10T07:33:38.392661+0000","last_active":"2024-02-10T07:35:01.702868+0000","last_peered":"2024-02-10T07:35:01.702868+0000","last_clean":"2024-02-10T07:35:01.702868+0000","last_became_active":"2024-02-10T07:31:51.282137+0000","last_became_peered":"2024-02-10T07:31:51.282137+0000","last_unstale":"2024-02-10T07:35:01.702868+0000","last_undegraded":"2024-02-10T07:35:01.702868+0000","last_fullsized":"2024-02-10T07:35:01.702868+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:38.392546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:38.392546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:05.891238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":157,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.704488+0000","last_change":"2024-02-10T07:34:46.308091+0000","last_active":"2024-02-10T07:35:01.704488+0000","last_peered":"2024-02-10T07:35:01.704488+0000","last_clean":"2024-02-10T07:35:01.704488+0000","last_became_active":"2024-02-10T07:31:53.294218+0000","last_became_peered":"2024-02-10T07:31:53.294218+0000","last_unstale":"2024-02-10T07:35:01.704488+0000","last_undegraded":"2024-02-10T07:35:01.704488+0000","last_fullsized":"2024-02-10T07:35:01.704488+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:34:46.307942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:46.307942+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:52.814988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"122'4723","reported_seq":4781,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.652828+0000","last_change":"2024-02-10T07:33:53.679012+0000","last_active":"2024-02-10T07:35:02.652828+0000","last_peered":"2024-02-10T07:35:02.652828+0000","last_clean":"2024-02-10T07:35:02.652828+0000","last_became_active":"2024-02-10T07:33:53.678445+0000","last_became_peered":"2024-02-10T07:33:53.678445+0000","last_unstale":"2024-02-10T07:35:02.652828+0000","last_undegraded":"2024-02-10T07:35:02.652828+0000","last_fullsized":"2024-02-10T07:35:02.652828+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4723,"log_dups_size":0,"ondisk_log_size":4723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:04.277686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309526528,"num_objects":4723,"num_object_clones":0,"num_object_copies":14169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4723,"num_write_kb":302272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.705772+0000","last_change":"2024-02-10T07:33:04.582281+0000","last_active":"2024-02-10T07:35:01.705772+0000","last_peered":"2024-02-10T07:35:01.705772+0000","last_clean":"2024-02-10T07:35:01.705772+0000","last_became_active":"2024-02-10T07:31:57.303235+0000","last_became_peered":"2024-02-10T07:31:57.303235+0000","last_unstale":"2024-02-10T07:35:01.705772+0000","last_undegraded":"2024-02-10T07:35:01.705772+0000","last_fullsized":"2024-02-10T07:35:01.705772+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:04.582188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:04.582188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":129,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.708777+0000","last_change":"2024-02-10T07:34:43.369811+0000","last_active":"2024-02-10T07:35:01.708777+0000","last_peered":"2024-02-10T07:35:01.708777+0000","last_clean":"2024-02-10T07:35:01.708777+0000","last_became_active":"2024-02-10T07:31:55.294376+0000","last_became_peered":"2024-02-10T07:31:55.294376+0000","last_unstale":"2024-02-10T07:35:01.708777+0000","last_undegraded":"2024-02-10T07:35:01.708777+0000","last_fullsized":"2024-02-10T07:35:01.708777+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:43.369691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:43.369691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:53.842761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":138,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682082+0000","last_change":"2024-02-10T07:34:59.459857+0000","last_active":"2024-02-10T07:35:01.682082+0000","last_peered":"2024-02-10T07:35:01.682082+0000","last_clean":"2024-02-10T07:35:01.682082+0000","last_became_active":"2024-02-10T07:31:51.320715+0000","last_became_peered":"2024-02-10T07:31:51.320715+0000","last_unstale":"2024-02-10T07:35:01.682082+0000","last_undegraded":"2024-02-10T07:35:01.682082+0000","last_fullsized":"2024-02-10T07:35:01.682082+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:59.459739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:59.459739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:20.291209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"89'16","reported_seq":176,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682605+0000","last_change":"2024-02-10T07:34:56.426389+0000","last_active":"2024-02-10T07:35:01.682605+0000","last_peered":"2024-02-10T07:35:01.682605+0000","last_clean":"2024-02-10T07:35:01.682605+0000","last_became_active":"2024-02-10T07:33:46.499156+0000","last_became_peered":"2024-02-10T07:33:46.499156+0000","last_unstale":"2024-02-10T07:35:01.682605+0000","last_undegraded":"2024-02-10T07:35:01.682605+0000","last_fullsized":"2024-02-10T07:35:01.682605+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'16","last_scrub_stamp":"2024-02-10T07:34:56.426264+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:34:56.426264+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:05.963061+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"122'4866","reported_seq":4924,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.705021+0000","last_change":"2024-02-10T07:33:53.682808+0000","last_active":"2024-02-10T07:35:02.705021+0000","last_peered":"2024-02-10T07:35:02.705021+0000","last_clean":"2024-02-10T07:35:02.705021+0000","last_became_active":"2024-02-10T07:33:53.682513+0000","last_became_peered":"2024-02-10T07:33:53.682513+0000","last_unstale":"2024-02-10T07:35:02.705021+0000","last_undegraded":"2024-02-10T07:35:02.705021+0000","last_fullsized":"2024-02-10T07:35:02.705021+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:11.782016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318898176,"num_objects":4866,"num_object_clones":0,"num_object_copies":14598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4866,"num_write_kb":311424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":125,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.713807+0000","last_change":"2024-02-10T07:34:44.046519+0000","last_active":"2024-02-10T07:35:01.713807+0000","last_peered":"2024-02-10T07:35:01.713807+0000","last_clean":"2024-02-10T07:35:01.713807+0000","last_became_active":"2024-02-10T07:31:57.311427+0000","last_became_peered":"2024-02-10T07:31:57.311427+0000","last_unstale":"2024-02-10T07:35:01.713807+0000","last_undegraded":"2024-02-10T07:35:01.713807+0000","last_fullsized":"2024-02-10T07:35:01.713807+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:44.046401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:34:44.046401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:56.660979+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":120,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.710734+0000","last_change":"2024-02-10T07:33:12.259596+0000","last_active":"2024-02-10T07:35:01.710734+0000","last_peered":"2024-02-10T07:35:01.710734+0000","last_clean":"2024-02-10T07:35:01.710734+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:35:01.710734+0000","last_undegraded":"2024-02-10T07:35:01.710734+0000","last_fullsized":"2024-02-10T07:35:01.710734+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:12.259478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:12.259478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":137,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.707586+0000","last_change":"2024-02-10T07:34:35.761455+0000","last_active":"2024-02-10T07:35:01.707586+0000","last_peered":"2024-02-10T07:35:01.707586+0000","last_clean":"2024-02-10T07:35:01.707586+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:35:01.707586+0000","last_undegraded":"2024-02-10T07:35:01.707586+0000","last_fullsized":"2024-02-10T07:35:01.707586+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:35.761349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:56.900501+0000","last_clean_scrub_stamp":"2024-02-10T07:34:35.761349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:49.489772+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"89'25","reported_seq":162,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682567+0000","last_change":"2024-02-10T07:33:20.252130+0000","last_active":"2024-02-10T07:35:01.682567+0000","last_peered":"2024-02-10T07:35:01.682567+0000","last_clean":"2024-02-10T07:35:01.682567+0000","last_became_active":"2024-02-10T07:31:53.292277+0000","last_became_peered":"2024-02-10T07:31:53.292277+0000","last_unstale":"2024-02-10T07:35:01.682567+0000","last_undegraded":"2024-02-10T07:35:01.682567+0000","last_fullsized":"2024-02-10T07:35:01.682567+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'25","last_scrub_stamp":"2024-02-10T07:33:20.251980+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:33:20.251980+0000","last_clean_scrub_stamp":"2024-02-10T07:33:20.251980+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"122'4976","reported_seq":5034,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:02.766560+0000","last_change":"2024-02-10T07:33:53.681706+0000","last_active":"2024-02-10T07:35:02.766560+0000","last_peered":"2024-02-10T07:35:02.766560+0000","last_clean":"2024-02-10T07:35:02.766560+0000","last_became_active":"2024-02-10T07:33:53.681419+0000","last_became_peered":"2024-02-10T07:33:53.681419+0000","last_unstale":"2024-02-10T07:35:02.766560+0000","last_undegraded":"2024-02-10T07:35:02.766560+0000","last_fullsized":"2024-02-10T07:35:02.766560+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:22.629613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326107136,"num_objects":4976,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4976,"num_write_kb":318464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.683131+0000","last_change":"2024-02-10T07:33:25.318627+0000","last_active":"2024-02-10T07:35:01.683131+0000","last_peered":"2024-02-10T07:35:01.683131+0000","last_clean":"2024-02-10T07:35:01.683131+0000","last_became_active":"2024-02-10T07:31:57.308608+0000","last_became_peered":"2024-02-10T07:31:57.308608+0000","last_unstale":"2024-02-10T07:35:01.683131+0000","last_undegraded":"2024-02-10T07:35:01.683131+0000","last_fullsized":"2024-02-10T07:35:01.683131+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:25.318521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:25.318521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":179,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.713960+0000","last_change":"2024-02-10T07:34:59.141641+0000","last_active":"2024-02-10T07:35:01.713960+0000","last_peered":"2024-02-10T07:35:01.713960+0000","last_clean":"2024-02-10T07:35:01.713960+0000","last_became_active":"2024-02-10T07:31:55.298389+0000","last_became_peered":"2024-02-10T07:31:55.298389+0000","last_unstale":"2024-02-10T07:35:01.713960+0000","last_undegraded":"2024-02-10T07:35:01.713960+0000","last_fullsized":"2024-02-10T07:35:01.713960+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:34:59.141399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:59.141399+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:04.565877+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.703415+0000","last_change":"2024-02-10T07:33:19.515540+0000","last_active":"2024-02-10T07:35:01.703415+0000","last_peered":"2024-02-10T07:35:01.703415+0000","last_clean":"2024-02-10T07:35:01.703415+0000","last_became_active":"2024-02-10T07:31:51.288197+0000","last_became_peered":"2024-02-10T07:31:51.288197+0000","last_unstale":"2024-02-10T07:35:01.703415+0000","last_undegraded":"2024-02-10T07:35:01.703415+0000","last_fullsized":"2024-02-10T07:35:01.703415+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:19.515464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:19.515464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":141,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-10T07:35:01.682568+0000","last_change":"2024-02-10T07:33:41.311832+0000","last_active":"2024-02-10T07:35:01.682568+0000","last_peered":"2024-02-10T07:35:01.682568+0000","last_clean":"2024-02-10T07:35:01.682568+0000","last_became_active":"2024-02-10T07:31:53.287984+0000","last_became_peered":"2024-02-10T07:31:53.287984+0000","last_unstale":"2024-02-10T07:35:01.682568+0000","last_undegraded":"2024-02-10T07:35:01.682568+0000","last_fullsized":"2024-02-10T07:35:01.682568+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:33:41.311687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:41.311687+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5072551936,"num_objects":77401,"num_object_clones":0,"num_object_copies":232203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77401,"num_write_kb":4953664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2824,"num_bytes_recovered":185073664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15261892608,"data_stored":15261892608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76501,"ondisk_log_size":76501,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":186,"num_read_kb":181,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":437,"ondisk_log_size":437,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037803,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1639196,"kb_used_data":1589972,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92105956,"statfs":{"total":95995035648,"available":94316498944,"internally_reserved":0,"allocated":1628131328,"data_stored":1627635600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":3.452,"5min":1.121,"15min":0.73099999999999998},"min":{"1min":0.29099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":52.591999999999999,"5min":52.591999999999999,"15min":52.591999999999999},"last":3.7589999999999999},{"interface":"front","average":{"1min":2.8769999999999998,"5min":1.024,"15min":0.72499999999999998},"min":{"1min":0.26700000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":52.494,"5min":52.494,"15min":52.494},"last":0.52200000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53400000000000003,"15min":0.53600000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.58099999999999996,"15min":0.55700000000000005},"min":{"1min":0.32000000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.4140000000000001,"5min":3.4140000000000001,"15min":3.4140000000000001},"last":0.27500000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.621,"15min":0.57699999999999996},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":9.1600000000000001,"5min":9.1600000000000001,"15min":9.1600000000000001},"last":1.917},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.59099999999999997,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.7090000000000001,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":4.5979999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":3.0910000000000002,"5min":1.071,"15min":0.73299999999999998},"min":{"1min":0.33100000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":54.375,"5min":54.375,"15min":54.375},"last":2.823},{"interface":"front","average":{"1min":3.0990000000000002,"5min":1.091,"15min":0.755},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":54.488999999999997,"5min":54.488999999999997,"15min":54.488999999999997},"last":10.090999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.47699999999999998,"15min":0.49099999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.441},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51000000000000001,"15min":0.51400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":1.133}]},{"osd":5,"last update":"Sat Feb 10 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.497,"15min":0.497},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70999999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83399999999999996,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.39400000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57399999999999995,"15min":0.58499999999999996},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56100000000000005,"15min":0.56000000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.5}]}]},{"osd":6,"up_from":50,"seq":214748364848,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1567176,"kb_used_data":1509624,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92177976,"statfs":{"total":95995035648,"available":94390247424,"internally_reserved":0,"allocated":1545854976,"data_stored":1545386937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.38700000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.91100000000000003,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.307},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67100000000000004,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.432}]},{"osd":1,"last update":"Sat Feb 10 07:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.72599999999999998,"5min":0.752,"15min":0.752},"last":5.3410000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77800000000000002,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.36899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68200000000000005,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56499999999999995,"15min":0.56899999999999995},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.75600000000000001,"5min":0.872,"15min":0.872},"last":0.46400000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.3089999999999999,"15min":1.629},"min":{"1min":0.38800000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.2669999999999999,"5min":19.664999999999999,"15min":19.664999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.60599999999999998,"15min":0.61199999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.74399999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.38400000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47699999999999998,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48899999999999999,"15min":0.47899999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.84699999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.34699999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.496,"15min":0.48999999999999999},"min":{"1min":0.32800000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52000000000000002,"15min":0.51200000000000001},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.40699999999999997}]},{"osd":7,"last update":"Sat Feb 10 07:34:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.34699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.38100000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.57999999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1935196,"kb_used_data":1859600,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91809956,"statfs":{"total":95995035648,"available":94013394944,"internally_reserved":0,"allocated":1904230400,"data_stored":1903737748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:34:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.41099999999999998,"15min":0.35899999999999999},"min":{"1min":0.223,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.8199999999999998,"5min":2.8199999999999998,"15min":2.8199999999999998},"last":4.8730000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.41699999999999998,"15min":0.36899999999999999},"min":{"1min":0.25,"5min":0.22,"15min":0.22},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":4.9489999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.501,"15min":0.42899999999999999},"min":{"1min":0.24099999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":4.9109999999999996},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.51500000000000001,"15min":0.44700000000000001},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.2919999999999998,"5min":5.2919999999999998,"15min":5.2919999999999998},"last":1.855}]},{"osd":3,"last update":"Sat Feb 10 07:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":6.6989999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.51100000000000001,"15min":0.50900000000000001},"min":{"1min":0.33500000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":1.8859999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:34:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.82199999999999995,"15min":0.99199999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.9139999999999999,"5min":10.292,"15min":10.292},"last":2.0230000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":6.5919999999999996}]},{"osd":5,"last update":"Sat Feb 10 07:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.64600000000000002,"15min":0.63600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.952},{"interface":"front","average":{"1min":1.0389999999999999,"5min":0.68999999999999995,"15min":0.64300000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.3710000000000004,"5min":5.3710000000000004,"15min":5.3710000000000004},"last":2.0619999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:34:14 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61799999999999999,"15min":0.626},"min":{"1min":0.47599999999999998,"5min":0.436,"15min":0.436},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.5249999999999999},"last":6.673},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68500000000000005,"15min":0.68999999999999995},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":2.5710000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.9910000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:34:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.69599999999999995,"15min":0.65600000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":5.3410000000000002,"5min":5.3410000000000002,"15min":5.3410000000000002},"last":2.097},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.628,"15min":0.63},"min":{"1min":0.442,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":6.6429999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2534048,"kb_used_data":2454292,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":91211104,"statfs":{"total":95995035648,"available":93400170496,"internally_reserved":0,"allocated":2513195008,"data_stored":2512699677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34999999999999998,"15min":0.34599999999999997},"min":{"1min":0.223,"5min":0.217,"15min":0.217},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":4.1680000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.42999999999999999,"15min":0.38},"min":{"1min":0.27600000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.5640000000000001,"5min":2.5640000000000001,"15min":2.5640000000000001},"last":4.766}]},{"osd":2,"last update":"Sat Feb 10 07:34:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.497,"15min":0.42499999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":7.3330000000000002,"5min":7.3330000000000002,"15min":7.3330000000000002},"last":0.504},{"interface":"front","average":{"1min":0.874,"5min":0.50900000000000001,"15min":0.45200000000000001},"min":{"1min":0.248,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.3199999999999998,"5min":3.3199999999999998,"15min":3.3199999999999998},"last":4.7229999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:34:05 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.41799999999999998,"15min":0.438},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.52700000000000002,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":1.218},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.46300000000000002,"15min":0.46800000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.53000000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":5.1379999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:34:31 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.57399999999999995,"15min":0.55300000000000005},"min":{"1min":0.38400000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.6419999999999999,"5min":2.6419999999999999,"15min":2.6419999999999999},"last":4.2119999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.78600000000000003,"15min":0.63400000000000001},"min":{"1min":0.41899999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.9630000000000001,"5min":19.219000000000001,"15min":19.219000000000001},"last":0.54600000000000004}]},{"osd":5,"last update":"Sat Feb 10 07:34:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.63500000000000001,"15min":0.60699999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":2.427,"5min":2.427,"15min":2.427},"last":4.8559999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.64700000000000002,"15min":0.61099999999999999},"min":{"1min":0.40600000000000003,"5min":0.378,"15min":0.378},"max":{"1min":4.1959999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"last":4.2480000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.85499999999999998,"15min":0.71299999999999997},"min":{"1min":0.42799999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":2.504,"5min":19.16,"15min":19.16},"last":4.8929999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61199999999999999,"15min":0.629},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":4.274}]},{"osd":7,"last update":"Sat Feb 10 07:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":1.2110000000000001,"15min":1.429},"min":{"1min":0.42399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.2480000000000002,"5min":19.178000000000001,"15min":19.178000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.67900000000000005,"15min":0.66100000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":2.4700000000000002,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":4.9240000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1915144,"kb_used_data":1843708,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91830008,"statfs":{"total":95995035648,"available":94033928192,"internally_reserved":0,"allocated":1887956992,"data_stored":1887485233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:34:43 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.47899999999999998,"15min":0.41299999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.0739999999999998,"5min":4.0739999999999998,"15min":4.0739999999999998},"last":11.297000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":0.628,"15min":0.48399999999999999},"min":{"1min":0.27700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":9.1509999999999998,"5min":9.1509999999999998,"15min":9.1509999999999998},"last":10.281000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:34:43 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.52500000000000002,"15min":0.45300000000000001},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6840000000000002,"5min":3.6840000000000002,"15min":3.6840000000000002},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.51000000000000001,"15min":0.44600000000000001},"min":{"1min":0.26600000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.9470000000000001,"5min":3.9470000000000001,"15min":3.9470000000000001},"last":0.19500000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:34:04 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.48999999999999999,"15min":0.52500000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.624,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":11.253},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.504,"15min":0.54400000000000004},"min":{"1min":0.30099999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.995,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":10.307}]},{"osd":4,"last update":"Sat Feb 10 07:34:32 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63400000000000001,"15min":0.627},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.92400000000000004,"5min":1.212,"15min":1.212},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68700000000000006,"15min":0.65300000000000002},"min":{"1min":0.45500000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.94199999999999995,"5min":5.343,"15min":5.343},"last":0.36299999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:34:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":0.79000000000000004,"15min":0.69099999999999995},"min":{"1min":0.39400000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":9.5190000000000001,"5min":9.5190000000000001,"15min":9.5190000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.427,"5min":0.78700000000000003,"15min":0.67600000000000005},"min":{"1min":0.48499999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":5.1180000000000003,"5min":5.1180000000000003,"15min":5.1180000000000003},"last":0.41199999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.80200000000000005,"15min":0.86199999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0940000000000001,"5min":5.3230000000000004,"15min":5.3230000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.67800000000000005,"15min":0.65400000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.9750000000000001,"5min":2.9750000000000001,"15min":2.9750000000000001},"last":0.34899999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:34:39 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.70099999999999996,"15min":0.67500000000000004},"min":{"1min":0.437,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.9430000000000001,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68200000000000005,"15min":0.68600000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.436}]}]},{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2819240,"kb_used_data":2751964,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":90925912,"statfs":{"total":95995035648,"available":93108133888,"internally_reserved":0,"allocated":2818011136,"data_stored":2817506305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.35899999999999999,"15min":0.373},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.45600000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":2.5489999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.40699999999999997,"15min":0.41299999999999998},"min":{"1min":0.222,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.48199999999999998,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":2.867}]},{"osd":1,"last update":"Sat Feb 10 07:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.40400000000000003,"15min":0.41799999999999998},"min":{"1min":0.28000000000000003,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59399999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":2.9060000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.69099999999999995,"5min":1.032,"15min":1.032},"last":4.0209999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:34:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.46600000000000003,"15min":0.46899999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.57199999999999995,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":3.8370000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.41699999999999998,"15min":0.42999999999999999},"min":{"1min":0.26100000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.54700000000000004,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":2.5190000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.63500000000000001,"15min":0.59099999999999997},"min":{"1min":0.41199999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.2290000000000001,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":2.831},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.625},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":4.032,"5min":4.032,"15min":4.032},"last":3.77}]},{"osd":5,"last update":"Sat Feb 10 07:34:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.66500000000000004,"15min":0.60999999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.2519999999999998,"5min":3.2519999999999998,"15min":3.2519999999999998},"last":2.4740000000000002},{"interface":"front","average":{"1min":1.173,"5min":0.71999999999999997,"15min":0.65300000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":2.694,"5min":2.694,"15min":2.694},"last":2.9940000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.66200000000000003,"15min":0.67100000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.2410000000000001,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":3.8029999999999999},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.65100000000000002,"15min":0.61599999999999999},"min":{"1min":0.40999999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.2869999999999999,"5min":3.2869999999999999,"15min":3.2869999999999999},"last":2.7229999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.68000000000000005,"15min":0.626},"min":{"1min":0.42799999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":2.9390000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.72999999999999998,"15min":0.68999999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"last":3.7320000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953530,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1582820,"kb_used_data":1537748,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92162332,"statfs":{"total":95995035648,"available":94374227968,"internally_reserved":0,"allocated":1574653952,"data_stored":1574157315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.70099999999999996,"15min":0.625},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.827,"5min":7.5700000000000003,"15min":7.5700000000000003},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.39600000000000002,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.26200000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.57599999999999996,"15min":0.56000000000000005},"min":{"1min":0.38900000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":3.1819999999999999,"5min":3.1819999999999999,"15min":3.1819999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.56499999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Sat Feb 10 07:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.57999999999999996,"15min":0.55100000000000005},"min":{"1min":0.34899999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.6960000000000002,"5min":3.6960000000000002,"15min":3.6960000000000002},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59499999999999997,"15min":0.60399999999999998},"min":{"1min":0.36599999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.708,"5min":1.708,"15min":1.708},"last":0.318}]},{"osd":3,"last update":"Sat Feb 10 07:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57499999999999996,"15min":0.58099999999999996},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.61699999999999999,"15min":0.57399999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.78700000000000003,"5min":7.601,"15min":7.601},"last":0.42199999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.40699999999999997,"15min":0.38300000000000001},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.44800000000000001,"15min":0.40799999999999997},"min":{"1min":0.29799999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.36499999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:34:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.46700000000000003,"15min":0.438},"min":{"1min":0.308,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.436},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.46400000000000002,"15min":0.438},"min":{"1min":0.34999999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.34000000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:34:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.52800000000000002,"15min":0.51100000000000001},"min":{"1min":0.33900000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.376,"5min":0.318,"15min":0.318},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.45300000000000001}]}]},{"osd":5,"up_from":40,"seq":171798691893,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1289356,"kb_used_data":1244288,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92455796,"statfs":{"total":95995035648,"available":94674735104,"internally_reserved":0,"allocated":1274150912,"data_stored":1273675349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:34:53 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":0.77600000000000002,"15min":0.65400000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":7.3109999999999999,"5min":7.3109999999999999,"15min":7.3109999999999999},"last":6.5549999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.63500000000000001,"15min":0.57999999999999996},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":3.9980000000000002,"5min":3.9980000000000002,"15min":3.9980000000000002},"last":4.2560000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.59399999999999997,"15min":0.58899999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.2869999999999999,"5min":2.2869999999999999,"15min":2.2869999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60599999999999998,"15min":0.59499999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.1059999999999999,"5min":3.238,"15min":3.238},"last":0.44400000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.56899999999999995,"15min":0.57599999999999996},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.60199999999999998,"15min":0.61499999999999999},"min":{"1min":0.36299999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":2.4119999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.45100000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:34:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":0.70399999999999996,"15min":0.63600000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":4.5389999999999997,"5min":4.5389999999999997,"15min":4.5389999999999997},"last":4.9530000000000003},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.66500000000000004,"15min":0.60499999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":4.335,"5min":4.335,"15min":4.335},"last":4.415}]},{"osd":4,"last update":"Sat Feb 10 07:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.439,"15min":0.41099999999999998},"min":{"1min":0.26100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.372},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.44600000000000001,"15min":0.40899999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.35699999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:34:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.45000000000000001,"15min":0.42899999999999999},"min":{"1min":0.29799999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66900000000000004,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.307},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.48499999999999999,"15min":0.46000000000000002},"min":{"1min":0.29899999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.79000000000000004,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38900000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.83799999999999997,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32700000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2525954048,"data_stored":2525954048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1914830848,"data_stored":1914830848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1904476160,"data_stored":1904476160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2862481408,"data_stored":2862481408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1579614208,"data_stored":1579614208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1281032192,"data_stored":1281032192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1563426816,"data_stored":1563426816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1630076928,"data_stored":1630076928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:35:11.468 INFO:tasks.thrashosds.thrasher:Setting 6.0 to [6, 0, 4] 2024-02-10T07:35:11.468 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.0', '6', '0', '4'] 2024-02-10T07:35:11.468 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pg-upmap 6.0 6 0 4 2024-02-10T07:35:11.480 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:11.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:11.582 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:11.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:11.629 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:11.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:11.730 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:11.758 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:11.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:11.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:11.860 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:11.911 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:11.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:12.012 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:12.038 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:12.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:12.140 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:12.171 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:12.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:12.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:12.218 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 62 16 87503 87487 88.169 65.9375 0.0154744 0.0113366 2024-02-10T07:35:12.273 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:12.324 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:12.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:12.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:12.426 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:12.476 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:12.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:12.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:12.578 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:12.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:12.634 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:12.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:12.735 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:12.773 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:12.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:12.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:12.874 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:12.891 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:12.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:12.936 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:12.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:13.037 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:13.068 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:13.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:13.071 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:12 smithi008 ceph-mon[29816]: pgmap v123: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1020 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-02-10T07:35:13.071 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:12 smithi008 ceph-mon[40743]: pgmap v123: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1020 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-02-10T07:35:13.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:12 smithi170 ceph-mon[33394]: pgmap v123: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1020 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-02-10T07:35:13.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:35:12 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:35:13.170 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:13.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:13.212 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:13.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:13.218 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 63 16 88606 88590 87.8637 68.9375 0.00799376 0.0113759 2024-02-10T07:35:13.313 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:13.339 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:13.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:13.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:13.441 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:13.470 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:13.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:13.572 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:13.611 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:13.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:13.713 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:13.740 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:13.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:13.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:13.811 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:13.842 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:13.872 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:13.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:13.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:13.974 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:14.004 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:14.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:14.070 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:13 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:14.071 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:13 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:14.105 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:14.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:13 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:14.144 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:14.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:14.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:14.218 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 64 16 89713 89697 87.5719 69.1875 0.0169498 0.0114142 2024-02-10T07:35:14.246 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:14.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:14.274 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:14.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:14.375 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:14.404 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:14.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:14.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:14.506 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:14.556 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:14.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:14.658 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:14.709 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:14.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:14.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:14.811 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:14.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:14.852 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:14.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:14.953 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:14.966 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:14 smithi008 ceph-mon[29816]: pgmap v124: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-10T07:35:14.966 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:14 smithi008 ceph-mon[40743]: pgmap v124: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-10T07:35:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:14.985 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:14.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:15.086 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:15.130 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:15.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:15.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:15.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:14 smithi170 ceph-mon[33394]: pgmap v124: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-10T07:35:15.219 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 65 15 90832 90817 87.3015 70 0.0192622 0.0114497 2024-02-10T07:35:15.232 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:15.259 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:15.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:15.361 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:15.410 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:15.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:15.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:15.512 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:15.581 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:15.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:15.682 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:15.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:15.729 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:15.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:15.832 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:15.889 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:15.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:15.990 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:16.053 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:16.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:16.155 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:16.184 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:16.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:16.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:16.218 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 66 16 91845 91829 86.937 63.25 0.0050515 0.0114967 2024-02-10T07:35:16.286 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:16.316 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:16.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:16.417 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:16.446 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:16.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:16.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:16.548 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:16.577 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:16.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:16.678 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:16.709 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:16.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:16.811 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:16.856 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:16.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:16.957 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:16.996 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:16.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:16.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:17.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:16 smithi008 ceph-mon[29816]: pgmap v125: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 888 op/s 2024-02-10T07:35:17.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:16 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:35:17.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:16 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:35:17.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:16 smithi008 ceph-mon[40743]: pgmap v125: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 888 op/s 2024-02-10T07:35:17.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:16 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:35:17.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:16 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:35:17.098 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:17.125 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:17.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:17.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:16 smithi170 ceph-mon[33394]: pgmap v125: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 55 MiB/s wr, 888 op/s 2024-02-10T07:35:17.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:16 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:35:17.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:16 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:35:17.221 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 67 16 92924 92908 86.6428 67.4375 0.0165913 0.0115363 2024-02-10T07:35:17.228 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:17.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:17.257 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:17.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:17.358 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:17.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:17.386 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:17.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:17.487 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:17.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:17.533 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:17.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:17.635 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:17.676 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:17.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:17.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:17.778 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:17.806 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:17.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:17.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:17.908 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:17.960 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:17.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:18.027 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:17 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:18.027 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:17 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:18.062 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:18.111 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:18.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:18.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:17 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:18.142 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:35:18.188 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:35:18.212 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:18.221 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 68 16 93820 93804 86.1916 56 0.0130685 0.0115971 2024-02-10T07:35:18.231 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:35:18.250 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:18.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:18.272 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:35:18.316 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:35:18.352 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:18.356 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:35:18.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:18.381 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:18.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:18.399 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:35:18.443 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:35:18.455 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:35:18.456 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:35:18.456 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:35:18.456 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:35:18.456 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:35:18.456 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:35:18.456 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:35:18.456 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:35:18.481 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:18.511 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:18.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:18.613 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:18.656 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:18.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:18.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:18.758 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:18.802 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:35:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:18.810 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:18.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:18.911 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:18.941 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:18.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:18.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:19.043 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:19.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:19.074 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:19.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:18 smithi008 ceph-mon[29816]: pgmap v126: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-10T07:35:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:18 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3132848320' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:35:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:35:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:35:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:18 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1830687157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:19.083 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:35:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:35:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:35:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:18 smithi008 ceph-mon[40743]: pgmap v126: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-10T07:35:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:18 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3132848320' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:35:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:35:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:35:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:18 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1830687157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:18 smithi170 ceph-mon[33394]: pgmap v126: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-10T07:35:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:18 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3132848320' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:35:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:35:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:35:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:35:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:18 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1830687157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:19.175 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:19.202 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:19.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:19.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:19.221 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 69 16 94800 94784 85.8301 61.25 0.0288236 0.0116462 2024-02-10T07:35:19.304 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:19.346 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:19.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:19.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:19.448 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:19.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:19.489 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:19.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:19.590 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:19.629 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:19.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:19.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:19.731 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:19.777 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:19.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:19.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:19.808 INFO:teuthology.orchestra.run.smithi008.stderr:set 6.0 pg_upmap mapping to [6,0,4] 2024-02-10T07:35:19.809 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:35:19.809 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:35:19.809 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:35:19.809 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3132848320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:35:19.809 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[29816]: osdmap e123: 8 total, 8 up, 8 in 2024-02-10T07:35:19.809 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:35:19.809 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:35:19.809 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3611550421' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [6, 0, 4]}]: dispatch 2024-02-10T07:35:19.809 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [6, 0, 4]}]: dispatch 2024-02-10T07:35:19.810 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:35:19.810 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:35:19.810 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:35:19.810 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3132848320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:35:19.810 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[40743]: osdmap e123: 8 total, 8 up, 8 in 2024-02-10T07:35:19.810 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:35:19.810 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:35:19.810 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3611550421' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [6, 0, 4]}]: dispatch 2024-02-10T07:35:19.810 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:19 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [6, 0, 4]}]: dispatch 2024-02-10T07:35:19.879 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:19.912 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:19.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:19.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:20.014 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:20.061 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:20.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:20.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:19 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:35:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:19 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:35:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:19 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:35:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:19 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3132848320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:35:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:19 smithi170 ceph-mon[33394]: osdmap e123: 8 total, 8 up, 8 in 2024-02-10T07:35:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:19 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:35:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:19 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:35:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:19 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3611550421' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [6, 0, 4]}]: dispatch 2024-02-10T07:35:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:19 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [6, 0, 4]}]: dispatch 2024-02-10T07:35:20.164 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:20.194 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:20.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:20.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:20.222 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 70 16 95898 95882 85.5843 68.625 0.0189452 0.0116793 2024-02-10T07:35:20.296 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:20.331 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:20.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:20.432 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:20.467 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:20.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:20.568 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:20.615 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:20.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:20.717 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:20.748 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:20.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:20.850 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:20.879 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:20.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:20.981 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:21.009 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:21.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:21.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:21.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:20 smithi008 ceph-mon[29816]: pgmap v128: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-02-10T07:35:21.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:20 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [6, 0, 4]}]': finished 2024-02-10T07:35:21.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:20 smithi008 ceph-mon[29816]: osdmap e124: 8 total, 8 up, 8 in 2024-02-10T07:35:21.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:20 smithi008 ceph-mon[40743]: pgmap v128: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-02-10T07:35:21.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:20 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [6, 0, 4]}]': finished 2024-02-10T07:35:21.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:20 smithi008 ceph-mon[40743]: osdmap e124: 8 total, 8 up, 8 in 2024-02-10T07:35:21.111 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:21.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:20 smithi170 ceph-mon[33394]: pgmap v128: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-02-10T07:35:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:20 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.0", "id": [6, 0, 4]}]': finished 2024-02-10T07:35:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:20 smithi170 ceph-mon[33394]: osdmap e124: 8 total, 8 up, 8 in 2024-02-10T07:35:21.140 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:21.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:21.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:21.222 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 71 16 97005 96989 85.3533 69.1875 0.0208401 0.0117109 2024-02-10T07:35:21.241 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:21.270 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:21.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:21.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:21.371 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:21.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:21.418 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:21.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:21.500 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:35:21.500 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:35:21.519 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:21.564 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:21.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:21.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:21.585 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:35:21.666 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:21.694 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:21.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:21.796 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:21.827 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:21.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:21.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:21.978 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:21.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:21.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:22.080 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:22.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:21 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:35:22.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:21 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:35:22.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:21 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:35:22.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:21 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:35:22.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:21 smithi008 ceph-mon[29816]: osdmap e125: 8 total, 8 up, 8 in 2024-02-10T07:35:22.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:21 smithi008 ceph-mon[29816]: osdmap e126: 8 total, 8 up, 8 in 2024-02-10T07:35:22.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:21 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:35:22.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:21 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:35:22.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:21 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:35:22.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:21 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:35:22.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:21 smithi008 ceph-mon[40743]: osdmap e125: 8 total, 8 up, 8 in 2024-02-10T07:35:22.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:21 smithi008 ceph-mon[40743]: osdmap e126: 8 total, 8 up, 8 in 2024-02-10T07:35:22.125 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:22.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:22.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:22.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:21 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:35:22.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:21 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:35:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:21 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:35:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:21 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:35:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:21 smithi170 ceph-mon[33394]: osdmap e125: 8 total, 8 up, 8 in 2024-02-10T07:35:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:21 smithi170 ceph-mon[33394]: osdmap e126: 8 total, 8 up, 8 in 2024-02-10T07:35:22.222 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 72 16 98033 98017 85.0602 64.25 0.0138573 0.0117509 2024-02-10T07:35:22.227 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:22.236 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-10T07:35:22.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:22.285 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:22.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:22.379 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:22.387 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:22.417 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:22.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:22.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:22.519 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:22.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:22.581 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:22.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:22.593 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:35:22.682 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:22.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:22.712 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:22.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:22.813 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:22.820 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:22 smithi008 ceph-mon[29816]: pgmap v132: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 47 MiB/s wr, 753 op/s 2024-02-10T07:35:22.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:22.858 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:22.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:22.958 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:23.005 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:23.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:23.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:22 smithi008 ceph-mon[40743]: pgmap v132: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 47 MiB/s wr, 753 op/s 2024-02-10T07:35:23.107 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:23.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:35:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:35:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:22 smithi170 ceph-mon[33394]: pgmap v132: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 47 MiB/s wr, 753 op/s 2024-02-10T07:35:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:23.169 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:23.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:23.223 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 73 16 98957 98941 84.6858 57.75 0.0155665 0.0118038 2024-02-10T07:35:23.270 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:23.309 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:23.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:23.412 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:23.451 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:23.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:23.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:23.553 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:23.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:23.601 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:23.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:23.702 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:23.731 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:23.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:23.832 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:23.869 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:23.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:23.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:23.971 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:24.015 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:24.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:24.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:23 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:35:24.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:23 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:35:24.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:23 smithi008 ceph-mon[29816]: osdmap e127: 8 total, 8 up, 8 in 2024-02-10T07:35:24.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:23 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:24.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:23 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:35:24.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:23 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:35:24.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:23 smithi008 ceph-mon[40743]: osdmap e127: 8 total, 8 up, 8 in 2024-02-10T07:35:24.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:23 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:24.117 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:24.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:23 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:35:24.135 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:23 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:35:24.135 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:23 smithi170 ceph-mon[33394]: osdmap e127: 8 total, 8 up, 8 in 2024-02-10T07:35:24.135 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:23 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:24.145 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:24.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:24.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:24.222 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 74 16 99973 99957 84.3994 63.5 0.00806099 0.0118439 2024-02-10T07:35:24.247 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:24.277 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:24.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:24.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:24.379 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:24.420 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:24.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:24.522 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:24.559 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:24.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:24.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:24.661 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:24.673 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:24.690 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:24.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:24.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:24.792 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:24.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:24.840 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:24.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:24.941 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:24.971 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:24.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:24.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:24.984 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:24 smithi008 ceph-mon[40743]: pgmap v134: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-10T07:35:24.987 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:24 smithi008 ceph-mon[29816]: pgmap v134: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-10T07:35:25.073 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:25.122 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:25.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:25.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:24 smithi170 ceph-mon[33394]: pgmap v134: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-10T07:35:25.222 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 75 16 101000 100984 84.1299 64.1875 0.00924099 0.0118813 2024-02-10T07:35:25.224 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:25.255 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:25.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:25.359 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:25.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:25.391 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:25.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:25.493 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:25.517 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:25.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:25.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:25.619 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:25.667 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:25.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:25.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:25.769 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:25.810 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:25.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:25.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:25.907 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T07:35:25.907 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:35:25.908 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-02-10T07:35:25.908 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:35:25.912 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:25.939 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:25.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:26.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:26.071 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:26.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:26.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:26.176 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:26.188 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:26.201 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:26.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:26.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:26.224 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 76 16 101893 101877 83.7568 55.8125 0.0185969 0.0119336 2024-02-10T07:35:26.302 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:26.364 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:26.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:26.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:26.466 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:26.497 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:26.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:26.599 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:26.629 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:26.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:26.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:26.705 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:35:26.731 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:26.758 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:26.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:26.860 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:26.890 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:26.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:26.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:26.992 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:27.037 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:27.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:27.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:26 smithi008 ceph-mon[29816]: pgmap v135: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 863 B/s rd, 54 MiB/s wr, 862 op/s 2024-02-10T07:35:27.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:26 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3558527281' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:35:27.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:26 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:35:27.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:26 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3558527281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:35:27.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:26 smithi008 ceph-mon[29816]: osdmap e128: 8 total, 8 up, 8 in 2024-02-10T07:35:27.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:26 smithi008 ceph-mon[40743]: pgmap v135: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 863 B/s rd, 54 MiB/s wr, 862 op/s 2024-02-10T07:35:27.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:26 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3558527281' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:35:27.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:26 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:35:27.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:26 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3558527281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:35:27.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:26 smithi008 ceph-mon[40743]: osdmap e128: 8 total, 8 up, 8 in 2024-02-10T07:35:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:26 smithi170 ceph-mon[33394]: pgmap v135: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 863 B/s rd, 54 MiB/s wr, 862 op/s 2024-02-10T07:35:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:26 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3558527281' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:35:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:26 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:35:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:26 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3558527281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:35:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:26 smithi170 ceph-mon[33394]: osdmap e128: 8 total, 8 up, 8 in 2024-02-10T07:35:27.139 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:27.171 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:27.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:27.223 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 77 16 102960 102944 83.5351 66.6875 0.0164042 0.0119663 2024-02-10T07:35:27.272 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:27.313 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:27.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:27.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:27.415 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:27.455 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:27.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:27.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:27.557 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:27.598 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:27.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:27.700 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:27.730 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:27.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:27.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:27.832 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:27.871 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:27.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:27.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:27.973 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:28.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:28.014 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:28.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:28.115 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:28.144 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:28.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:28.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:28.223 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 78 16 104064 104048 83.3487 69 0.0111032 0.0119922 2024-02-10T07:35:28.245 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:28.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:28.281 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:28.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:28.382 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:28.422 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:28.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:28.524 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:28.554 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:28.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:28.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:28.656 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:28.685 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:28.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:28.786 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:28.833 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:28.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:28.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:28.880 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:35:28.921 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:35:28.935 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:28.962 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:35:28.965 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:28.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:29.005 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:35:29.044 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:35:29.045 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:35:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:35:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:35:29.045 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:28 smithi008 ceph-mon[29816]: pgmap v137: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-10T07:35:29.045 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:28 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3882174379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:29.045 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:28 smithi008 ceph-mon[40743]: pgmap v137: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-10T07:35:29.045 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:28 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3882174379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:29.067 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:29.091 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:35:29.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:29.114 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:29.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:29.128 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:35:29.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:28 smithi170 ceph-mon[33394]: pgmap v137: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-10T07:35:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:28 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3882174379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:29.177 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:35:29.197 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:35:29.197 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:35:29.197 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:35:29.197 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:35:29.197 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:35:29.197 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:35:29.197 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:35:29.198 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:35:29.216 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:29.228 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 79 16 104989 104973 83.0242 57.8125 0.0291893 0.0120393 2024-02-10T07:35:29.245 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:29.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:29.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:29.347 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:29.379 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:29.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:29.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:29.481 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:29.527 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:29.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:29.629 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:29.684 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:29.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:29.785 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:29.829 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:29.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:29.930 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:29.963 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:29.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:29.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:30.065 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:29 smithi008 ceph-mon[29816]: 7.5 scrub starts 2024-02-10T07:35:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:29 smithi008 ceph-mon[29816]: 7.d scrub starts 2024-02-10T07:35:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:29 smithi008 ceph-mon[29816]: 4.15 scrub starts 2024-02-10T07:35:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:29 smithi008 ceph-mon[29816]: 4.15 scrub ok 2024-02-10T07:35:30.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:29 smithi008 ceph-mon[40743]: 7.5 scrub starts 2024-02-10T07:35:30.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:29 smithi008 ceph-mon[40743]: 7.d scrub starts 2024-02-10T07:35:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:29 smithi008 ceph-mon[40743]: 4.15 scrub starts 2024-02-10T07:35:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:29 smithi008 ceph-mon[40743]: 4.15 scrub ok 2024-02-10T07:35:30.094 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:30.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:30.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:29 smithi170 ceph-mon[33394]: 7.5 scrub starts 2024-02-10T07:35:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:29 smithi170 ceph-mon[33394]: 7.d scrub starts 2024-02-10T07:35:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:29 smithi170 ceph-mon[33394]: 4.15 scrub starts 2024-02-10T07:35:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:29 smithi170 ceph-mon[33394]: 4.15 scrub ok 2024-02-10T07:35:30.197 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:30.225 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:35:30.225635+0000 min lat: 0.00141219 max lat: 2.36731 avg lat: 0.0120936 2024-02-10T07:35:30.225 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:35:30.225 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 80 16 105842 105826 82.6528 53.3125 0.0228988 0.0120936 2024-02-10T07:35:30.229 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:30.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:30.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:30.262 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:35:30.331 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:30.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:30.373 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:30.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:30.474 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:30.495 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:30.505 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:30.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:30.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:30.524 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:35:30.524 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:35:30.524 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-02-10T07:35:30.525 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-02-10T07:35:30.525 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:35:30.606 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:30.637 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:30.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:30.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:30.740 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:30.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:30.801 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:30.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:30.902 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:30.936 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:30.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:31.038 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:31.083 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:31.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:31.124 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:30 smithi008 ceph-mon[29816]: 4.18 scrub starts 2024-02-10T07:35:31.124 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:30 smithi008 ceph-mon[29816]: 4.18 scrub ok 2024-02-10T07:35:31.124 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:30 smithi008 ceph-mon[29816]: 4.b scrub starts 2024-02-10T07:35:31.124 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:30 smithi008 ceph-mon[29816]: 4.b scrub ok 2024-02-10T07:35:31.124 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:30 smithi008 ceph-mon[29816]: pgmap v138: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-02-10T07:35:31.124 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:30 smithi008 ceph-mon[29816]: 7.5 scrub ok 2024-02-10T07:35:31.126 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:30 smithi008 ceph-mon[40743]: 4.18 scrub starts 2024-02-10T07:35:31.126 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:30 smithi008 ceph-mon[40743]: 4.18 scrub ok 2024-02-10T07:35:31.126 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:30 smithi008 ceph-mon[40743]: 4.b scrub starts 2024-02-10T07:35:31.126 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:30 smithi008 ceph-mon[40743]: 4.b scrub ok 2024-02-10T07:35:31.126 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:30 smithi008 ceph-mon[40743]: pgmap v138: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-02-10T07:35:31.126 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:30 smithi008 ceph-mon[40743]: 7.5 scrub ok 2024-02-10T07:35:31.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:30 smithi170 ceph-mon[33394]: 4.18 scrub starts 2024-02-10T07:35:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:30 smithi170 ceph-mon[33394]: 4.18 scrub ok 2024-02-10T07:35:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:30 smithi170 ceph-mon[33394]: 4.b scrub starts 2024-02-10T07:35:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:30 smithi170 ceph-mon[33394]: 4.b scrub ok 2024-02-10T07:35:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:30 smithi170 ceph-mon[33394]: pgmap v138: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-02-10T07:35:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:30 smithi170 ceph-mon[33394]: 7.5 scrub ok 2024-02-10T07:35:31.185 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:31.225 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 81 16 106694 106678 82.2896 53.25 0.0140276 0.012147 2024-02-10T07:35:31.225 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:31.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:31.327 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:31.357 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:31.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:31.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:31.459 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:31.501 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:31.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:31.602 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:31.638 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:31.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:31.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:31.741 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:31.783 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:31.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:31.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:31.885 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:31.923 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:31.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:31.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:32.025 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:32.060 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:32.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:31 smithi170 ceph-mon[33394]: 3.f scrub starts 2024-02-10T07:35:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:31 smithi170 ceph-mon[33394]: 3.f scrub ok 2024-02-10T07:35:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:31 smithi170 ceph-mon[33394]: 6.1b scrub starts 2024-02-10T07:35:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:31 smithi170 ceph-mon[33394]: 6.1b scrub ok 2024-02-10T07:35:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:31 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]: dispatch 2024-02-10T07:35:32.161 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:32.185 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:32.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:32.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:32.215 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:35:32.225 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 82 16 107680 107664 82.0376 61.625 0.0117332 0.0121841 2024-02-10T07:35:32.266 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:31 smithi008 ceph-mon[29816]: 3.f scrub starts 2024-02-10T07:35:32.266 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:31 smithi008 ceph-mon[29816]: 3.f scrub ok 2024-02-10T07:35:32.266 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:31 smithi008 ceph-mon[29816]: 6.1b scrub starts 2024-02-10T07:35:32.266 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:31 smithi008 ceph-mon[29816]: 6.1b scrub ok 2024-02-10T07:35:32.266 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:31 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]: dispatch 2024-02-10T07:35:32.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:31 smithi008 ceph-mon[40743]: 3.f scrub starts 2024-02-10T07:35:32.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:31 smithi008 ceph-mon[40743]: 3.f scrub ok 2024-02-10T07:35:32.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:31 smithi008 ceph-mon[40743]: 6.1b scrub starts 2024-02-10T07:35:32.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:31 smithi008 ceph-mon[40743]: 6.1b scrub ok 2024-02-10T07:35:32.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:31 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]: dispatch 2024-02-10T07:35:32.287 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:32.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:32.337 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:32.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:32.436 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-10T07:35:32.438 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:32.469 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:32.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:32.571 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:32.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:32.613 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:32.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:32.714 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:32.745 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:32.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:32.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:32.847 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:32.922 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:32.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:33.023 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:33.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:33.066 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:33.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:32 smithi170 ceph-mon[33394]: 7.d scrub ok 2024-02-10T07:35:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:32 smithi170 ceph-mon[33394]: 4.3 scrub starts 2024-02-10T07:35:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:32 smithi170 ceph-mon[33394]: 4.3 scrub ok 2024-02-10T07:35:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:32 smithi170 ceph-mon[33394]: 4.9 scrub starts 2024-02-10T07:35:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:32 smithi170 ceph-mon[33394]: 4.9 scrub ok 2024-02-10T07:35:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:32 smithi170 ceph-mon[33394]: pgmap v139: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 574 B/s rd, 35 MiB/s wr, 566 op/s 2024-02-10T07:35:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:32 smithi170 ceph-mon[33394]: 7.c scrub starts 2024-02-10T07:35:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]': finished 2024-02-10T07:35:33.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:32 smithi170 ceph-mon[33394]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T07:35:33.134 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:35:32 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:35:33.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:33.191 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:33.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:33.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:33.225 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 83 16 108552 108536 81.7058 54.5 0.0891472 0.0122326 2024-02-10T07:35:33.293 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:33.326 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:33.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[29816]: 7.d scrub ok 2024-02-10T07:35:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[29816]: 4.3 scrub starts 2024-02-10T07:35:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[29816]: 4.3 scrub ok 2024-02-10T07:35:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[29816]: 4.9 scrub starts 2024-02-10T07:35:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[29816]: 4.9 scrub ok 2024-02-10T07:35:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[29816]: pgmap v139: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 574 B/s rd, 35 MiB/s wr, 566 op/s 2024-02-10T07:35:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[29816]: 7.c scrub starts 2024-02-10T07:35:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]': finished 2024-02-10T07:35:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[29816]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T07:35:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[40743]: 7.d scrub ok 2024-02-10T07:35:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[40743]: 4.3 scrub starts 2024-02-10T07:35:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[40743]: 4.3 scrub ok 2024-02-10T07:35:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[40743]: 4.9 scrub starts 2024-02-10T07:35:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[40743]: 4.9 scrub ok 2024-02-10T07:35:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[40743]: pgmap v139: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 574 B/s rd, 35 MiB/s wr, 566 op/s 2024-02-10T07:35:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[40743]: 7.c scrub starts 2024-02-10T07:35:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [3, 5]}]': finished 2024-02-10T07:35:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:32 smithi008 ceph-mon[40743]: osdmap e129: 8 total, 8 up, 8 in 2024-02-10T07:35:33.428 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:33.457 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:33.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:33.559 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:33.599 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:33.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:33.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:33.701 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:33.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:33.743 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:33.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:33.844 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:33.872 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:33.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:33.974 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:34.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:34.018 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:34.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:34.119 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:34.143 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:34.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:34.225 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 84 16 109541 109525 81.469 61.8125 0.00752833 0.0122697 2024-02-10T07:35:34.245 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:34.263 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[29816]: 6.1d scrub starts 2024-02-10T07:35:34.263 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[29816]: 6.1d scrub ok 2024-02-10T07:35:34.263 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[29816]: 5.9 scrub starts 2024-02-10T07:35:34.263 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[29816]: 5.9 scrub ok 2024-02-10T07:35:34.263 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[29816]: 3.1 scrub starts 2024-02-10T07:35:34.263 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[29816]: 3.1 scrub ok 2024-02-10T07:35:34.263 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[29816]: 3.18 scrub starts 2024-02-10T07:35:34.263 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[29816]: 3.18 scrub ok 2024-02-10T07:35:34.263 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:34.264 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[29816]: 5.7 scrub starts 2024-02-10T07:35:34.264 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[29816]: 5.7 scrub ok 2024-02-10T07:35:34.264 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[29816]: osdmap e130: 8 total, 8 up, 8 in 2024-02-10T07:35:34.264 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[40743]: 6.1d scrub starts 2024-02-10T07:35:34.264 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[40743]: 6.1d scrub ok 2024-02-10T07:35:34.264 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[40743]: 5.9 scrub starts 2024-02-10T07:35:34.264 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[40743]: 5.9 scrub ok 2024-02-10T07:35:34.264 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[40743]: 3.1 scrub starts 2024-02-10T07:35:34.264 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[40743]: 3.1 scrub ok 2024-02-10T07:35:34.264 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[40743]: 3.18 scrub starts 2024-02-10T07:35:34.264 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[40743]: 3.18 scrub ok 2024-02-10T07:35:34.264 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:34.265 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[40743]: 5.7 scrub starts 2024-02-10T07:35:34.265 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[40743]: 5.7 scrub ok 2024-02-10T07:35:34.265 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:33 smithi008 ceph-mon[40743]: osdmap e130: 8 total, 8 up, 8 in 2024-02-10T07:35:34.275 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:34.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:34.378 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:33 smithi170 ceph-mon[33394]: 6.1d scrub starts 2024-02-10T07:35:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:33 smithi170 ceph-mon[33394]: 6.1d scrub ok 2024-02-10T07:35:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:33 smithi170 ceph-mon[33394]: 5.9 scrub starts 2024-02-10T07:35:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:33 smithi170 ceph-mon[33394]: 5.9 scrub ok 2024-02-10T07:35:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:33 smithi170 ceph-mon[33394]: 3.1 scrub starts 2024-02-10T07:35:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:33 smithi170 ceph-mon[33394]: 3.1 scrub ok 2024-02-10T07:35:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:33 smithi170 ceph-mon[33394]: 3.18 scrub starts 2024-02-10T07:35:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:33 smithi170 ceph-mon[33394]: 3.18 scrub ok 2024-02-10T07:35:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:33 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:33 smithi170 ceph-mon[33394]: 5.7 scrub starts 2024-02-10T07:35:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:33 smithi170 ceph-mon[33394]: 5.7 scrub ok 2024-02-10T07:35:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:33 smithi170 ceph-mon[33394]: osdmap e130: 8 total, 8 up, 8 in 2024-02-10T07:35:34.411 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:34.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:34.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:34.513 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:34.552 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:34.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:34.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:34.654 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:34.680 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:34.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:34.782 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:34.813 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:34.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:34.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:34.915 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:34.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:34.961 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:34.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:35.062 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:35.113 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:35.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:35.149 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:35.151 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:35.189 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:35.214 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:35.227 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 85 16 110548 110532 81.2494 62.9375 0.0189434 0.0123016 2024-02-10T07:35:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:35.242 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:35.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:35.297 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:34 smithi008 ceph-mon[29816]: 7.c scrub ok 2024-02-10T07:35:35.297 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:34 smithi008 ceph-mon[29816]: 3.3 scrub starts 2024-02-10T07:35:35.297 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:34 smithi008 ceph-mon[29816]: 3.3 scrub ok 2024-02-10T07:35:35.297 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:34 smithi008 ceph-mon[29816]: pgmap v141: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-02-10T07:35:35.297 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:34 smithi008 ceph-mon[40743]: 7.c scrub ok 2024-02-10T07:35:35.297 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:34 smithi008 ceph-mon[40743]: 3.3 scrub starts 2024-02-10T07:35:35.297 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:34 smithi008 ceph-mon[40743]: 3.3 scrub ok 2024-02-10T07:35:35.297 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:34 smithi008 ceph-mon[40743]: pgmap v141: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-02-10T07:35:35.343 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:35.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:35.379 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:35.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:35.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:34 smithi170 ceph-mon[33394]: 7.c scrub ok 2024-02-10T07:35:35.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:34 smithi170 ceph-mon[33394]: 3.3 scrub starts 2024-02-10T07:35:35.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:34 smithi170 ceph-mon[33394]: 3.3 scrub ok 2024-02-10T07:35:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:34 smithi170 ceph-mon[33394]: pgmap v141: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-02-10T07:35:35.480 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:35.536 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:35.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:35.638 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:35.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:35.685 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:35.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:35.786 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:35.831 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:35.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:35.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:35.933 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:35.968 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:35.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:35.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:36.070 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:36.106 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:36.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:36.208 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:36.227 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 86 16 111574 111558 81.0503 64.125 0.00780954 0.0123325 2024-02-10T07:35:36.255 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:36.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:36.281 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[29816]: 4.d scrub starts 2024-02-10T07:35:36.281 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[29816]: 4.d scrub ok 2024-02-10T07:35:36.282 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[29816]: 3.1f scrub starts 2024-02-10T07:35:36.282 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[29816]: 3.1f scrub ok 2024-02-10T07:35:36.282 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[29816]: 4.1d scrub starts 2024-02-10T07:35:36.282 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[29816]: 4.1d scrub ok 2024-02-10T07:35:36.282 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[29816]: 7.7 scrub starts 2024-02-10T07:35:36.282 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[29816]: 3.c scrub starts 2024-02-10T07:35:36.282 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[29816]: 3.c scrub ok 2024-02-10T07:35:36.282 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[40743]: 4.d scrub starts 2024-02-10T07:35:36.282 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[40743]: 4.d scrub ok 2024-02-10T07:35:36.282 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[40743]: 3.1f scrub starts 2024-02-10T07:35:36.282 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[40743]: 3.1f scrub ok 2024-02-10T07:35:36.282 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[40743]: 4.1d scrub starts 2024-02-10T07:35:36.282 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[40743]: 4.1d scrub ok 2024-02-10T07:35:36.283 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[40743]: 7.7 scrub starts 2024-02-10T07:35:36.283 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[40743]: 3.c scrub starts 2024-02-10T07:35:36.283 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:35 smithi008 ceph-mon[40743]: 3.c scrub ok 2024-02-10T07:35:36.357 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:35 smithi170 ceph-mon[33394]: 4.d scrub starts 2024-02-10T07:35:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:35 smithi170 ceph-mon[33394]: 4.d scrub ok 2024-02-10T07:35:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:35 smithi170 ceph-mon[33394]: 3.1f scrub starts 2024-02-10T07:35:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:35 smithi170 ceph-mon[33394]: 3.1f scrub ok 2024-02-10T07:35:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:35 smithi170 ceph-mon[33394]: 4.1d scrub starts 2024-02-10T07:35:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:35 smithi170 ceph-mon[33394]: 4.1d scrub ok 2024-02-10T07:35:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:35 smithi170 ceph-mon[33394]: 7.7 scrub starts 2024-02-10T07:35:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:35 smithi170 ceph-mon[33394]: 3.c scrub starts 2024-02-10T07:35:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:35 smithi170 ceph-mon[33394]: 3.c scrub ok 2024-02-10T07:35:36.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:36.404 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:36.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:36.505 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:36.551 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:36.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:36.651 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:36.680 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:36.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:36.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:36.782 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:36.812 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:36.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:36.914 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:36.971 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:36.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:37.074 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:37.089 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:36 smithi008 ceph-mon[29816]: 4.4 scrub starts 2024-02-10T07:35:37.090 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:36 smithi008 ceph-mon[29816]: 4.4 scrub ok 2024-02-10T07:35:37.090 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:36 smithi008 ceph-mon[29816]: 5.10 scrub starts 2024-02-10T07:35:37.090 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:36 smithi008 ceph-mon[29816]: 5.10 scrub ok 2024-02-10T07:35:37.090 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:36 smithi008 ceph-mon[29816]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 638 B/s rd, 58 MiB/s wr, 927 op/s 2024-02-10T07:35:37.090 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:36 smithi008 ceph-mon[40743]: 4.4 scrub starts 2024-02-10T07:35:37.090 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:36 smithi008 ceph-mon[40743]: 4.4 scrub ok 2024-02-10T07:35:37.090 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:36 smithi008 ceph-mon[40743]: 5.10 scrub starts 2024-02-10T07:35:37.090 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:36 smithi008 ceph-mon[40743]: 5.10 scrub ok 2024-02-10T07:35:37.090 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:36 smithi008 ceph-mon[40743]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 638 B/s rd, 58 MiB/s wr, 927 op/s 2024-02-10T07:35:37.103 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:37.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:37.205 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:37.227 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 87 16 112586 112570 80.8456 63.25 0.00716067 0.0123638 2024-02-10T07:35:37.234 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:37.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:37.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:37.336 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:37.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:37.365 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:37.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:36 smithi170 ceph-mon[33394]: 4.4 scrub starts 2024-02-10T07:35:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:36 smithi170 ceph-mon[33394]: 4.4 scrub ok 2024-02-10T07:35:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:36 smithi170 ceph-mon[33394]: 5.10 scrub starts 2024-02-10T07:35:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:36 smithi170 ceph-mon[33394]: 5.10 scrub ok 2024-02-10T07:35:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:36 smithi170 ceph-mon[33394]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 638 B/s rd, 58 MiB/s wr, 927 op/s 2024-02-10T07:35:37.467 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:37.494 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:37.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:37.596 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:37.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:37.644 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:37.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:37.718 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:35:37.731 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:35:37.745 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:37.773 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:37.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:37.875 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:37.927 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:37.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:38.028 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:38.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:38.058 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:38.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:38.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:37 smithi008 ceph-mon[29816]: 3.4 scrub starts 2024-02-10T07:35:38.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:37 smithi008 ceph-mon[29816]: 3.4 scrub ok 2024-02-10T07:35:38.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:37 smithi008 ceph-mon[29816]: 5.13 scrub starts 2024-02-10T07:35:38.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:37 smithi008 ceph-mon[29816]: 5.13 scrub ok 2024-02-10T07:35:38.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:37 smithi008 ceph-mon[29816]: 7.e scrub starts 2024-02-10T07:35:38.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:37 smithi008 ceph-mon[40743]: 3.4 scrub starts 2024-02-10T07:35:38.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:37 smithi008 ceph-mon[40743]: 3.4 scrub ok 2024-02-10T07:35:38.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:37 smithi008 ceph-mon[40743]: 5.13 scrub starts 2024-02-10T07:35:38.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:37 smithi008 ceph-mon[40743]: 5.13 scrub ok 2024-02-10T07:35:38.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:37 smithi008 ceph-mon[40743]: 7.e scrub starts 2024-02-10T07:35:38.164 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:38.199 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:38.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:38.227 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 88 16 113494 113478 80.5718 56.75 0.0204141 0.012406 2024-02-10T07:35:38.302 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:38.332 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:38.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:38.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:37 smithi170 ceph-mon[33394]: 3.4 scrub starts 2024-02-10T07:35:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:37 smithi170 ceph-mon[33394]: 3.4 scrub ok 2024-02-10T07:35:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:37 smithi170 ceph-mon[33394]: 5.13 scrub starts 2024-02-10T07:35:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:37 smithi170 ceph-mon[33394]: 5.13 scrub ok 2024-02-10T07:35:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:37 smithi170 ceph-mon[33394]: 7.e scrub starts 2024-02-10T07:35:38.434 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:38.472 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:38.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:38.572 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:38.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:38.618 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:38.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:38.719 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:38.746 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:38.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:38.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:38.769 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-02-10T07:35:35.741174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7088773468,"num_objects":108355,"num_object_clones":0,"num_object_copies":325065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108355,"num_whiteouts":0,"num_read":1147,"num_read_kb":993,"num_write":108772,"num_write_kb":6924509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2869,"num_bytes_recovered":186845648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89077,"ondisk_log_size":89077,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21323228,"kb_used_data":20718428,"kb_used_omap":12,"kb_used_meta":604723,"kb_avail":728637988,"statfs":{"total":767960285184,"available":746125299712,"internally_reserved":0,"allocated":21215670272,"data_stored":21211569668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":619236913},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,3,7,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":486932480,"num_objects":7430,"num_object_clones":0,"num_object_copies":22290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7430,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7430,"num_write_kb":475520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.019933"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":130,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.695444+0000","last_change":"2024-02-10T07:33:40.321091+0000","last_active":"2024-02-10T07:35:26.695444+0000","last_peered":"2024-02-10T07:35:26.695444+0000","last_clean":"2024-02-10T07:35:26.695444+0000","last_became_active":"2024-02-10T07:31:57.311360+0000","last_became_peered":"2024-02-10T07:31:57.311360+0000","last_unstale":"2024-02-10T07:35:26.695444+0000","last_undegraded":"2024-02-10T07:35:26.695444+0000","last_fullsized":"2024-02-10T07:35:26.695444+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:40.320971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:40.320971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:29.503800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":132,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.705179+0000","last_change":"2024-02-10T07:33:14.320677+0000","last_active":"2024-02-10T07:35:26.705179+0000","last_peered":"2024-02-10T07:35:26.705179+0000","last_clean":"2024-02-10T07:35:26.705179+0000","last_became_active":"2024-02-10T07:31:55.299946+0000","last_became_peered":"2024-02-10T07:31:55.299946+0000","last_unstale":"2024-02-10T07:35:26.705179+0000","last_undegraded":"2024-02-10T07:35:26.705179+0000","last_fullsized":"2024-02-10T07:35:26.705179+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:14.320588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:14.320588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:29.269932+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":149,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.738526+0000","last_change":"2024-02-10T07:33:29.609321+0000","last_active":"2024-02-10T07:35:32.738526+0000","last_peered":"2024-02-10T07:35:32.738526+0000","last_clean":"2024-02-10T07:35:32.738526+0000","last_became_active":"2024-02-10T07:31:51.320873+0000","last_became_peered":"2024-02-10T07:31:51.320873+0000","last_unstale":"2024-02-10T07:35:32.738526+0000","last_undegraded":"2024-02-10T07:35:32.738526+0000","last_fullsized":"2024-02-10T07:35:32.738526+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:33:29.609159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:29.609159+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":162,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:29.694657+0000","last_change":"2024-02-10T07:35:29.694708+0000","last_active":"2024-02-10T07:35:29.694657+0000","last_peered":"2024-02-10T07:35:29.694657+0000","last_clean":"2024-02-10T07:35:29.694657+0000","last_became_active":"2024-02-10T07:31:53.300846+0000","last_became_peered":"2024-02-10T07:31:53.300846+0000","last_unstale":"2024-02-10T07:35:29.694657+0000","last_undegraded":"2024-02-10T07:35:29.694657+0000","last_fullsized":"2024-02-10T07:35:29.694657+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:35:29.694556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:35:29.694556+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:34.389703+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":140,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.742266+0000","last_change":"2024-02-10T07:35:30.772554+0000","last_active":"2024-02-10T07:35:32.742266+0000","last_peered":"2024-02-10T07:35:32.742266+0000","last_clean":"2024-02-10T07:35:32.742266+0000","last_became_active":"2024-02-10T07:31:57.300056+0000","last_became_peered":"2024-02-10T07:31:57.300056+0000","last_unstale":"2024-02-10T07:35:32.742266+0000","last_undegraded":"2024-02-10T07:35:32.742266+0000","last_fullsized":"2024-02-10T07:35:32.742266+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:35:30.772428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:35:30.772428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:37.224815+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":142,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.699144+0000","last_change":"2024-02-10T07:34:38.285132+0000","last_active":"2024-02-10T07:35:26.699144+0000","last_peered":"2024-02-10T07:35:26.699144+0000","last_clean":"2024-02-10T07:35:26.699144+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:35:26.699144+0000","last_undegraded":"2024-02-10T07:35:26.699144+0000","last_fullsized":"2024-02-10T07:35:26.699144+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:38.285054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:38.285054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:07.608477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.746463+0000","last_change":"2024-02-10T07:33:24.458907+0000","last_active":"2024-02-10T07:35:32.746463+0000","last_peered":"2024-02-10T07:35:32.746463+0000","last_clean":"2024-02-10T07:35:32.746463+0000","last_became_active":"2024-02-10T07:31:51.323331+0000","last_became_peered":"2024-02-10T07:31:51.323331+0000","last_unstale":"2024-02-10T07:35:32.746463+0000","last_undegraded":"2024-02-10T07:35:32.746463+0000","last_fullsized":"2024-02-10T07:35:32.746463+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:33:24.458825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":173,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.739702+0000","last_change":"2024-02-10T07:34:46.545956+0000","last_active":"2024-02-10T07:35:32.739702+0000","last_peered":"2024-02-10T07:35:32.739702+0000","last_clean":"2024-02-10T07:35:32.739702+0000","last_became_active":"2024-02-10T07:31:53.286957+0000","last_became_peered":"2024-02-10T07:31:53.286957+0000","last_unstale":"2024-02-10T07:35:32.739702+0000","last_undegraded":"2024-02-10T07:35:32.739702+0000","last_fullsized":"2024-02-10T07:35:32.739702+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:34:46.545852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:46.545852+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:55.758249+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.736647+0000","last_change":"2024-02-10T07:33:39.604426+0000","last_active":"2024-02-10T07:35:32.736647+0000","last_peered":"2024-02-10T07:35:32.736647+0000","last_clean":"2024-02-10T07:35:32.736647+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:35:32.736647+0000","last_undegraded":"2024-02-10T07:35:32.736647+0000","last_fullsized":"2024-02-10T07:35:32.736647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:39.604346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:39.604346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":133,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698690+0000","last_change":"2024-02-10T07:33:43.115599+0000","last_active":"2024-02-10T07:35:26.698690+0000","last_peered":"2024-02-10T07:35:26.698690+0000","last_clean":"2024-02-10T07:35:26.698690+0000","last_became_active":"2024-02-10T07:31:55.297715+0000","last_became_peered":"2024-02-10T07:31:55.297715+0000","last_unstale":"2024-02-10T07:35:26.698690+0000","last_undegraded":"2024-02-10T07:35:26.698690+0000","last_fullsized":"2024-02-10T07:35:26.698690+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:43.115523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:43.115523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":156,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698270+0000","last_change":"2024-02-10T07:34:49.349481+0000","last_active":"2024-02-10T07:35:26.698270+0000","last_peered":"2024-02-10T07:35:26.698270+0000","last_clean":"2024-02-10T07:35:26.698270+0000","last_became_active":"2024-02-10T07:31:51.283229+0000","last_became_peered":"2024-02-10T07:31:51.283229+0000","last_unstale":"2024-02-10T07:35:26.698270+0000","last_undegraded":"2024-02-10T07:35:26.698270+0000","last_fullsized":"2024-02-10T07:35:26.698270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:34:49.349360+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:34:49.349360+0000","last_clean_scrub_stamp":"2024-02-10T07:34:49.349360+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:05.734497+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":154,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.695689+0000","last_change":"2024-02-10T07:33:48.378628+0000","last_active":"2024-02-10T07:35:26.695689+0000","last_peered":"2024-02-10T07:35:26.695689+0000","last_clean":"2024-02-10T07:35:26.695689+0000","last_became_active":"2024-02-10T07:31:53.287858+0000","last_became_peered":"2024-02-10T07:31:53.287858+0000","last_unstale":"2024-02-10T07:35:26.695689+0000","last_undegraded":"2024-02-10T07:35:26.695689+0000","last_fullsized":"2024-02-10T07:35:26.695689+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:33:48.378524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:48.378524+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:29.503782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":129,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698514+0000","last_change":"2024-02-10T07:33:47.227552+0000","last_active":"2024-02-10T07:35:26.698514+0000","last_peered":"2024-02-10T07:35:26.698514+0000","last_clean":"2024-02-10T07:35:26.698514+0000","last_became_active":"2024-02-10T07:31:57.310088+0000","last_became_peered":"2024-02-10T07:31:57.310088+0000","last_unstale":"2024-02-10T07:35:26.698514+0000","last_undegraded":"2024-02-10T07:35:26.698514+0000","last_fullsized":"2024-02-10T07:35:26.698514+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:47.227448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:47.227448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":134,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.712986+0000","last_change":"2024-02-10T07:33:33.199150+0000","last_active":"2024-02-10T07:35:26.712986+0000","last_peered":"2024-02-10T07:35:26.712986+0000","last_clean":"2024-02-10T07:35:26.712986+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:35:26.712986+0000","last_undegraded":"2024-02-10T07:35:26.712986+0000","last_fullsized":"2024-02-10T07:35:26.712986+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:33.199031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:33.199031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:30.304553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":141,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698143+0000","last_change":"2024-02-10T07:33:28.316764+0000","last_active":"2024-02-10T07:35:26.698143+0000","last_peered":"2024-02-10T07:35:26.698143+0000","last_clean":"2024-02-10T07:35:26.698143+0000","last_became_active":"2024-02-10T07:31:51.276350+0000","last_became_peered":"2024-02-10T07:31:51.276350+0000","last_unstale":"2024-02-10T07:35:26.698143+0000","last_undegraded":"2024-02-10T07:35:26.698143+0000","last_fullsized":"2024-02-10T07:35:26.698143+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:28.316679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:28.316679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":148,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.695865+0000","last_change":"2024-02-10T07:34:27.386460+0000","last_active":"2024-02-10T07:35:26.695865+0000","last_peered":"2024-02-10T07:35:26.695865+0000","last_clean":"2024-02-10T07:35:26.695865+0000","last_became_active":"2024-02-10T07:31:53.292417+0000","last_became_peered":"2024-02-10T07:31:53.292417+0000","last_unstale":"2024-02-10T07:35:26.695865+0000","last_undegraded":"2024-02-10T07:35:26.695865+0000","last_fullsized":"2024-02-10T07:35:26.695865+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-10T07:34:27.386341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:27.386341+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:38.466835+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":138,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.695975+0000","last_change":"2024-02-10T07:34:51.300380+0000","last_active":"2024-02-10T07:35:26.695975+0000","last_peered":"2024-02-10T07:35:26.695975+0000","last_clean":"2024-02-10T07:35:26.695975+0000","last_became_active":"2024-02-10T07:31:57.305613+0000","last_became_peered":"2024-02-10T07:31:57.305613+0000","last_unstale":"2024-02-10T07:35:26.695975+0000","last_undegraded":"2024-02-10T07:35:26.695975+0000","last_fullsized":"2024-02-10T07:35:26.695975+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:51.300266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:51.300266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:04.129482+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":132,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.701757+0000","last_change":"2024-02-10T07:33:19.252470+0000","last_active":"2024-02-10T07:35:26.701757+0000","last_peered":"2024-02-10T07:35:26.701757+0000","last_clean":"2024-02-10T07:35:26.701757+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:35:26.701757+0000","last_undegraded":"2024-02-10T07:35:26.701757+0000","last_fullsized":"2024-02-10T07:35:26.701757+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:19.252386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:19.252386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:29.269983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.736154+0000","last_change":"2024-02-10T07:33:47.723413+0000","last_active":"2024-02-10T07:35:32.736154+0000","last_peered":"2024-02-10T07:35:32.736154+0000","last_clean":"2024-02-10T07:35:32.736154+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:35:32.736154+0000","last_undegraded":"2024-02-10T07:35:32.736154+0000","last_fullsized":"2024-02-10T07:35:32.736154+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:33:47.723328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"89'22","reported_seq":179,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.744465+0000","last_change":"2024-02-10T07:34:46.082140+0000","last_active":"2024-02-10T07:35:32.744465+0000","last_peered":"2024-02-10T07:35:32.744465+0000","last_clean":"2024-02-10T07:35:32.744465+0000","last_became_active":"2024-02-10T07:31:53.295527+0000","last_became_peered":"2024-02-10T07:31:53.295527+0000","last_unstale":"2024-02-10T07:35:32.744465+0000","last_undegraded":"2024-02-10T07:35:32.744465+0000","last_fullsized":"2024-02-10T07:35:32.744465+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2024-02-10T07:34:46.081979+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:34:46.081979+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:06.591014+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":132,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.740369+0000","last_change":"2024-02-10T07:34:44.596132+0000","last_active":"2024-02-10T07:35:32.740369+0000","last_peered":"2024-02-10T07:35:32.740369+0000","last_clean":"2024-02-10T07:35:32.740369+0000","last_became_active":"2024-02-10T07:31:57.306315+0000","last_became_peered":"2024-02-10T07:31:57.306315+0000","last_unstale":"2024-02-10T07:35:32.740369+0000","last_undegraded":"2024-02-10T07:35:32.740369+0000","last_fullsized":"2024-02-10T07:35:32.740369+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:44.595328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:44.595328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:57.623977+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":142,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.696607+0000","last_change":"2024-02-10T07:34:58.502489+0000","last_active":"2024-02-10T07:35:26.696607+0000","last_peered":"2024-02-10T07:35:26.696607+0000","last_clean":"2024-02-10T07:35:26.696607+0000","last_became_active":"2024-02-10T07:31:55.298474+0000","last_became_peered":"2024-02-10T07:31:55.298474+0000","last_unstale":"2024-02-10T07:35:26.696607+0000","last_undegraded":"2024-02-10T07:35:26.696607+0000","last_fullsized":"2024-02-10T07:35:26.696607+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:58.502372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:27.368799+0000","last_clean_scrub_stamp":"2024-02-10T07:34:58.502372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:08.046058+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":150,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.695782+0000","last_change":"2024-02-10T07:34:37.329912+0000","last_active":"2024-02-10T07:35:26.695782+0000","last_peered":"2024-02-10T07:35:26.695782+0000","last_clean":"2024-02-10T07:35:26.695782+0000","last_became_active":"2024-02-10T07:31:51.321237+0000","last_became_peered":"2024-02-10T07:31:51.321237+0000","last_unstale":"2024-02-10T07:35:26.695782+0000","last_undegraded":"2024-02-10T07:35:26.695782+0000","last_fullsized":"2024-02-10T07:35:26.695782+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:37.329817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:37.329817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:39.334428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":163,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.739896+0000","last_change":"2024-02-10T07:33:13.556374+0000","last_active":"2024-02-10T07:35:32.739896+0000","last_peered":"2024-02-10T07:35:32.739896+0000","last_clean":"2024-02-10T07:35:32.739896+0000","last_became_active":"2024-02-10T07:31:53.294914+0000","last_became_peered":"2024-02-10T07:31:53.294914+0000","last_unstale":"2024-02-10T07:35:32.739896+0000","last_undegraded":"2024-02-10T07:35:32.739896+0000","last_fullsized":"2024-02-10T07:35:32.739896+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:33:13.556271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:13.556271+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":131,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.714557+0000","last_change":"2024-02-10T07:34:43.532210+0000","last_active":"2024-02-10T07:35:26.714557+0000","last_peered":"2024-02-10T07:35:26.714557+0000","last_clean":"2024-02-10T07:35:26.714557+0000","last_became_active":"2024-02-10T07:31:57.306892+0000","last_became_peered":"2024-02-10T07:31:57.306892+0000","last_unstale":"2024-02-10T07:35:26.714557+0000","last_undegraded":"2024-02-10T07:35:26.714557+0000","last_fullsized":"2024-02-10T07:35:26.714557+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:34:43.531966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:43.531966+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:59.235526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":197,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.703235+0000","last_change":"2024-02-10T07:34:38.826446+0000","last_active":"2024-02-10T07:35:26.703235+0000","last_peered":"2024-02-10T07:35:26.703235+0000","last_clean":"2024-02-10T07:35:26.703235+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:35:26.703235+0000","last_undegraded":"2024-02-10T07:35:26.703235+0000","last_fullsized":"2024-02-10T07:35:26.703235+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:34:38.826344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:38.826344+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:47.498990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":149,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.705256+0000","last_change":"2024-02-10T07:34:54.328873+0000","last_active":"2024-02-10T07:35:26.705256+0000","last_peered":"2024-02-10T07:35:26.705256+0000","last_clean":"2024-02-10T07:35:26.705256+0000","last_became_active":"2024-02-10T07:31:51.282077+0000","last_became_peered":"2024-02-10T07:31:51.282077+0000","last_unstale":"2024-02-10T07:35:26.705256+0000","last_undegraded":"2024-02-10T07:35:26.705256+0000","last_fullsized":"2024-02-10T07:35:26.705256+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:34:54.328677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:54.328677+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:12.122414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":162,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.744651+0000","last_change":"2024-02-10T07:34:51.520784+0000","last_active":"2024-02-10T07:35:32.744651+0000","last_peered":"2024-02-10T07:35:32.744651+0000","last_clean":"2024-02-10T07:35:32.744651+0000","last_became_active":"2024-02-10T07:31:53.292936+0000","last_became_peered":"2024-02-10T07:31:53.292936+0000","last_unstale":"2024-02-10T07:35:32.744651+0000","last_undegraded":"2024-02-10T07:35:32.744651+0000","last_fullsized":"2024-02-10T07:35:32.744651+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:34:51.520532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:51.520532+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:59.847057+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":136,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.303322+0000","last_change":"2024-02-10T07:35:32.303361+0000","last_active":"2024-02-10T07:35:32.303322+0000","last_peered":"2024-02-10T07:35:32.303322+0000","last_clean":"2024-02-10T07:35:32.303322+0000","last_became_active":"2024-02-10T07:31:57.308820+0000","last_became_peered":"2024-02-10T07:31:57.308820+0000","last_unstale":"2024-02-10T07:35:32.303322+0000","last_undegraded":"2024-02-10T07:35:32.303322+0000","last_fullsized":"2024-02-10T07:35:32.303322+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:35:32.303238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:35:32.303238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:56.162051+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.744913+0000","last_change":"2024-02-10T07:33:48.707506+0000","last_active":"2024-02-10T07:35:32.744913+0000","last_peered":"2024-02-10T07:35:32.744913+0000","last_clean":"2024-02-10T07:35:32.744913+0000","last_became_active":"2024-02-10T07:31:55.297595+0000","last_became_peered":"2024-02-10T07:31:55.297595+0000","last_unstale":"2024-02-10T07:35:32.744913+0000","last_undegraded":"2024-02-10T07:35:32.744913+0000","last_fullsized":"2024-02-10T07:35:32.744913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:48.707394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:48.707394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":152,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.736481+0000","last_change":"2024-02-10T07:35:32.732289+0000","last_active":"2024-02-10T07:35:32.736481+0000","last_peered":"2024-02-10T07:35:32.736481+0000","last_clean":"2024-02-10T07:35:32.736481+0000","last_became_active":"2024-02-10T07:31:51.285405+0000","last_became_peered":"2024-02-10T07:31:51.285405+0000","last_unstale":"2024-02-10T07:35:32.736481+0000","last_undegraded":"2024-02-10T07:35:32.736481+0000","last_fullsized":"2024-02-10T07:35:32.736481+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:35:32.732039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:35:32.732039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:47.995637+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":164,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.703158+0000","last_change":"2024-02-10T07:34:34.728508+0000","last_active":"2024-02-10T07:35:26.703158+0000","last_peered":"2024-02-10T07:35:26.703158+0000","last_clean":"2024-02-10T07:35:26.703158+0000","last_became_active":"2024-02-10T07:31:53.300871+0000","last_became_peered":"2024-02-10T07:31:53.300871+0000","last_unstale":"2024-02-10T07:35:26.703158+0000","last_undegraded":"2024-02-10T07:35:26.703158+0000","last_fullsized":"2024-02-10T07:35:26.703158+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:34:34.728407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:34.728407+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:00.562652+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":130,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.712922+0000","last_change":"2024-02-10T07:33:44.251598+0000","last_active":"2024-02-10T07:35:26.712922+0000","last_peered":"2024-02-10T07:35:26.712922+0000","last_clean":"2024-02-10T07:35:26.712922+0000","last_became_active":"2024-02-10T07:31:57.306776+0000","last_became_peered":"2024-02-10T07:31:57.306776+0000","last_unstale":"2024-02-10T07:35:26.712922+0000","last_undegraded":"2024-02-10T07:35:26.712922+0000","last_fullsized":"2024-02-10T07:35:26.712922+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:44.251492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:44.251492+0000","last_clean_scrub_stamp":"2024-02-10T07:33:44.251492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:30.304581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":141,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.702603+0000","last_change":"2024-02-10T07:34:41.840128+0000","last_active":"2024-02-10T07:35:26.702603+0000","last_peered":"2024-02-10T07:35:26.702603+0000","last_clean":"2024-02-10T07:35:26.702603+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:35:26.702603+0000","last_undegraded":"2024-02-10T07:35:26.702603+0000","last_fullsized":"2024-02-10T07:35:26.702603+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:41.839991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:41.839991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:06.871399+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":151,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.738730+0000","last_change":"2024-02-10T07:34:56.494219+0000","last_active":"2024-02-10T07:35:32.738730+0000","last_peered":"2024-02-10T07:35:32.738730+0000","last_clean":"2024-02-10T07:35:32.738730+0000","last_became_active":"2024-02-10T07:31:51.284022+0000","last_became_peered":"2024-02-10T07:31:51.284022+0000","last_unstale":"2024-02-10T07:35:32.738730+0000","last_undegraded":"2024-02-10T07:35:32.738730+0000","last_fullsized":"2024-02-10T07:35:32.738730+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:56.494106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:56.494106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:02.762479+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"95'6","reported_seq":148,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.736514+0000","last_change":"2024-02-10T07:33:46.438241+0000","last_active":"2024-02-10T07:35:32.736514+0000","last_peered":"2024-02-10T07:35:32.736514+0000","last_clean":"2024-02-10T07:35:32.736514+0000","last_became_active":"2024-02-10T07:31:53.295520+0000","last_became_peered":"2024-02-10T07:31:53.295520+0000","last_unstale":"2024-02-10T07:35:32.736514+0000","last_undegraded":"2024-02-10T07:35:32.736514+0000","last_fullsized":"2024-02-10T07:35:32.736514+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:33:46.438117+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":132,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.745996+0000","last_change":"2024-02-10T07:33:26.457321+0000","last_active":"2024-02-10T07:35:32.745996+0000","last_peered":"2024-02-10T07:35:32.745996+0000","last_clean":"2024-02-10T07:35:32.745996+0000","last_became_active":"2024-02-10T07:31:57.315738+0000","last_became_peered":"2024-02-10T07:31:57.315738+0000","last_unstale":"2024-02-10T07:35:32.745996+0000","last_undegraded":"2024-02-10T07:35:32.745996+0000","last_fullsized":"2024-02-10T07:35:32.745996+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:26.457244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:26.457244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":134,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.712999+0000","last_change":"2024-02-10T07:33:29.198369+0000","last_active":"2024-02-10T07:35:26.712999+0000","last_peered":"2024-02-10T07:35:26.712999+0000","last_clean":"2024-02-10T07:35:26.712999+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:35:26.712999+0000","last_undegraded":"2024-02-10T07:35:26.712999+0000","last_fullsized":"2024-02-10T07:35:26.712999+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:29.198265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:29.198265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:30.304496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":149,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.701049+0000","last_change":"2024-02-10T07:34:37.328219+0000","last_active":"2024-02-10T07:35:26.701049+0000","last_peered":"2024-02-10T07:35:26.701049+0000","last_clean":"2024-02-10T07:35:26.701049+0000","last_became_active":"2024-02-10T07:31:51.279438+0000","last_became_peered":"2024-02-10T07:31:51.279438+0000","last_unstale":"2024-02-10T07:35:26.701049+0000","last_undegraded":"2024-02-10T07:35:26.701049+0000","last_fullsized":"2024-02-10T07:35:26.701049+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:37.328147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:34:37.328147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:02.438373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":167,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.739588+0000","last_change":"2024-02-10T07:34:47.537455+0000","last_active":"2024-02-10T07:35:32.739588+0000","last_peered":"2024-02-10T07:35:32.739588+0000","last_clean":"2024-02-10T07:35:32.739588+0000","last_became_active":"2024-02-10T07:31:53.287823+0000","last_became_peered":"2024-02-10T07:31:53.287823+0000","last_unstale":"2024-02-10T07:35:32.739588+0000","last_undegraded":"2024-02-10T07:35:32.739588+0000","last_fullsized":"2024-02-10T07:35:32.739588+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:34:47.537324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:47.537324+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:00.222329+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.734997+0000","last_change":"2024-02-10T07:34:32.584922+0000","last_active":"2024-02-10T07:35:32.734997+0000","last_peered":"2024-02-10T07:35:32.734997+0000","last_clean":"2024-02-10T07:35:32.734997+0000","last_became_active":"2024-02-10T07:31:57.313529+0000","last_became_peered":"2024-02-10T07:31:57.313529+0000","last_unstale":"2024-02-10T07:35:32.734997+0000","last_undegraded":"2024-02-10T07:35:32.734997+0000","last_fullsized":"2024-02-10T07:35:32.734997+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:32.584777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:32.584777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:54.431843+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":136,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.742092+0000","last_change":"2024-02-10T07:32:59.584373+0000","last_active":"2024-02-10T07:35:32.742092+0000","last_peered":"2024-02-10T07:35:32.742092+0000","last_clean":"2024-02-10T07:35:32.742092+0000","last_became_active":"2024-02-10T07:31:55.296670+0000","last_became_peered":"2024-02-10T07:31:55.296670+0000","last_unstale":"2024-02-10T07:35:32.742092+0000","last_undegraded":"2024-02-10T07:35:32.742092+0000","last_fullsized":"2024-02-10T07:35:32.742092+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:32:59.584219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:32:59.584219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":150,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.713606+0000","last_change":"2024-02-10T07:34:56.606543+0000","last_active":"2024-02-10T07:35:26.713606+0000","last_peered":"2024-02-10T07:35:26.713606+0000","last_clean":"2024-02-10T07:35:26.713606+0000","last_became_active":"2024-02-10T07:31:51.288165+0000","last_became_peered":"2024-02-10T07:31:51.288165+0000","last_unstale":"2024-02-10T07:35:26.713606+0000","last_undegraded":"2024-02-10T07:35:26.713606+0000","last_fullsized":"2024-02-10T07:35:26.713606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:56.606253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:56.606253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:05.787229+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":129,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.702806+0000","last_change":"2024-02-10T07:34:47.925231+0000","last_active":"2024-02-10T07:35:26.702806+0000","last_peered":"2024-02-10T07:35:26.702806+0000","last_clean":"2024-02-10T07:35:26.702806+0000","last_became_active":"2024-02-10T07:33:46.501312+0000","last_became_peered":"2024-02-10T07:33:46.501312+0000","last_unstale":"2024-02-10T07:35:26.702806+0000","last_undegraded":"2024-02-10T07:35:26.702806+0000","last_fullsized":"2024-02-10T07:35:26.702806+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:34:47.925100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:47.925100+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:16.951245+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":132,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.739339+0000","last_change":"2024-02-10T07:33:31.451285+0000","last_active":"2024-02-10T07:35:32.739339+0000","last_peered":"2024-02-10T07:35:32.739339+0000","last_clean":"2024-02-10T07:35:32.739339+0000","last_became_active":"2024-02-10T07:31:57.315781+0000","last_became_peered":"2024-02-10T07:31:57.315781+0000","last_unstale":"2024-02-10T07:35:32.739339+0000","last_undegraded":"2024-02-10T07:35:32.739339+0000","last_fullsized":"2024-02-10T07:35:32.739339+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:31.451210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:31.451210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":140,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.703116+0000","last_change":"2024-02-10T07:34:59.399183+0000","last_active":"2024-02-10T07:35:26.703116+0000","last_peered":"2024-02-10T07:35:26.703116+0000","last_clean":"2024-02-10T07:35:26.703116+0000","last_became_active":"2024-02-10T07:31:55.301260+0000","last_became_peered":"2024-02-10T07:31:55.301260+0000","last_unstale":"2024-02-10T07:35:26.703116+0000","last_undegraded":"2024-02-10T07:35:26.703116+0000","last_fullsized":"2024-02-10T07:35:26.703116+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:59.399046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:59.399046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:25.056822+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":150,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.696391+0000","last_change":"2024-02-10T07:34:54.340934+0000","last_active":"2024-02-10T07:35:26.696391+0000","last_peered":"2024-02-10T07:35:26.696391+0000","last_clean":"2024-02-10T07:35:26.696391+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:35:26.696391+0000","last_undegraded":"2024-02-10T07:35:26.696391+0000","last_fullsized":"2024-02-10T07:35:26.696391+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:54.340863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:54.340863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:24.123274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"120'16","reported_seq":188,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.695655+0000","last_change":"2024-02-10T07:34:52.295223+0000","last_active":"2024-02-10T07:35:26.695655+0000","last_peered":"2024-02-10T07:35:26.695655+0000","last_clean":"2024-02-10T07:35:26.695655+0000","last_became_active":"2024-02-10T07:32:18.389564+0000","last_became_peered":"2024-02-10T07:32:18.389564+0000","last_unstale":"2024-02-10T07:35:26.695655+0000","last_undegraded":"2024-02-10T07:35:26.695655+0000","last_fullsized":"2024-02-10T07:35:26.695655+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'16","last_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:34:52.295064+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:55.073746+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.744313+0000","last_change":"2024-02-10T07:34:58.488060+0000","last_active":"2024-02-10T07:35:32.744313+0000","last_peered":"2024-02-10T07:35:32.744313+0000","last_clean":"2024-02-10T07:35:32.744313+0000","last_became_active":"2024-02-10T07:31:57.318009+0000","last_became_peered":"2024-02-10T07:31:57.318009+0000","last_unstale":"2024-02-10T07:35:32.744313+0000","last_undegraded":"2024-02-10T07:35:32.744313+0000","last_fullsized":"2024-02-10T07:35:32.744313+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:34:58.487937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:01.955191+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":194,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698406+0000","last_change":"2024-02-10T07:34:58.311787+0000","last_active":"2024-02-10T07:35:26.698406+0000","last_peered":"2024-02-10T07:35:26.698406+0000","last_clean":"2024-02-10T07:35:26.698406+0000","last_became_active":"2024-02-10T07:31:55.297670+0000","last_became_peered":"2024-02-10T07:31:55.297670+0000","last_unstale":"2024-02-10T07:35:26.698406+0000","last_undegraded":"2024-02-10T07:35:26.698406+0000","last_fullsized":"2024-02-10T07:35:26.698406+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:34:58.311646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:58.311646+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:23.772344+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":150,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.713226+0000","last_change":"2024-02-10T07:34:46.575053+0000","last_active":"2024-02-10T07:35:26.713226+0000","last_peered":"2024-02-10T07:35:26.713226+0000","last_clean":"2024-02-10T07:35:26.713226+0000","last_became_active":"2024-02-10T07:31:51.320644+0000","last_became_peered":"2024-02-10T07:31:51.320644+0000","last_unstale":"2024-02-10T07:35:26.713226+0000","last_undegraded":"2024-02-10T07:35:26.713226+0000","last_fullsized":"2024-02-10T07:35:26.713226+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:46.574930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:34:46.574930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:58.089207+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"96'15","reported_seq":194,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.718226+0000","last_change":"2024-02-10T07:34:18.901515+0000","last_active":"2024-02-10T07:35:26.718226+0000","last_peered":"2024-02-10T07:35:26.718226+0000","last_clean":"2024-02-10T07:35:26.718226+0000","last_became_active":"2024-02-10T07:34:18.894619+0000","last_became_peered":"2024-02-10T07:34:18.894619+0000","last_unstale":"2024-02-10T07:35:26.718226+0000","last_undegraded":"2024-02-10T07:35:26.718226+0000","last_fullsized":"2024-02-10T07:35:26.718226+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'15","last_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:33:48.458885+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:30.652462+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":130,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.696331+0000","last_change":"2024-02-10T07:34:30.335810+0000","last_active":"2024-02-10T07:35:26.696331+0000","last_peered":"2024-02-10T07:35:26.696331+0000","last_clean":"2024-02-10T07:35:26.696331+0000","last_became_active":"2024-02-10T07:31:57.307583+0000","last_became_peered":"2024-02-10T07:31:57.307583+0000","last_unstale":"2024-02-10T07:35:26.696331+0000","last_undegraded":"2024-02-10T07:35:26.696331+0000","last_fullsized":"2024-02-10T07:35:26.696331+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:30.335721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:30.335721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:51.756306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":189,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.749487+0000","last_change":"2024-02-10T07:33:28.459581+0000","last_active":"2024-02-10T07:35:32.749487+0000","last_peered":"2024-02-10T07:35:32.749487+0000","last_clean":"2024-02-10T07:35:32.749487+0000","last_became_active":"2024-02-10T07:31:55.294594+0000","last_became_peered":"2024-02-10T07:31:55.294594+0000","last_unstale":"2024-02-10T07:35:32.749487+0000","last_undegraded":"2024-02-10T07:35:32.749487+0000","last_fullsized":"2024-02-10T07:35:32.749487+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:33:28.459467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:28.459467+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":151,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.735631+0000","last_change":"2024-02-10T07:34:33.603966+0000","last_active":"2024-02-10T07:35:32.735631+0000","last_peered":"2024-02-10T07:35:32.735631+0000","last_clean":"2024-02-10T07:35:32.735631+0000","last_became_active":"2024-02-10T07:31:51.282353+0000","last_became_peered":"2024-02-10T07:31:51.282353+0000","last_unstale":"2024-02-10T07:35:32.735631+0000","last_undegraded":"2024-02-10T07:35:32.735631+0000","last_fullsized":"2024-02-10T07:35:32.735631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:33.603844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:33.603844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:00.993478+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":161,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:29.435176+0000","last_change":"2024-02-10T07:35:29.435226+0000","last_active":"2024-02-10T07:35:29.435176+0000","last_peered":"2024-02-10T07:35:29.435176+0000","last_clean":"2024-02-10T07:35:29.435176+0000","last_became_active":"2024-02-10T07:31:53.285777+0000","last_became_peered":"2024-02-10T07:31:53.285777+0000","last_unstale":"2024-02-10T07:35:29.435176+0000","last_undegraded":"2024-02-10T07:35:29.435176+0000","last_fullsized":"2024-02-10T07:35:29.435176+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:35:29.435087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:35:29.435087+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:43.326764+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.744784+0000","last_change":"2024-02-10T07:34:42.090173+0000","last_active":"2024-02-10T07:35:32.744784+0000","last_peered":"2024-02-10T07:35:32.744784+0000","last_clean":"2024-02-10T07:35:32.744784+0000","last_became_active":"2024-02-10T07:31:57.308615+0000","last_became_peered":"2024-02-10T07:31:57.308615+0000","last_unstale":"2024-02-10T07:35:32.744784+0000","last_undegraded":"2024-02-10T07:35:32.744784+0000","last_fullsized":"2024-02-10T07:35:32.744784+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:42.090037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:34:42.090037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:50.668927+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":136,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.740905+0000","last_change":"2024-02-10T07:33:17.587557+0000","last_active":"2024-02-10T07:35:32.740905+0000","last_peered":"2024-02-10T07:35:32.740905+0000","last_clean":"2024-02-10T07:35:32.740905+0000","last_became_active":"2024-02-10T07:31:55.297431+0000","last_became_peered":"2024-02-10T07:31:55.297431+0000","last_unstale":"2024-02-10T07:35:32.740905+0000","last_undegraded":"2024-02-10T07:35:32.740905+0000","last_fullsized":"2024-02-10T07:35:32.740905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:33:17.587433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":150,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.713224+0000","last_change":"2024-02-10T07:34:52.665968+0000","last_active":"2024-02-10T07:35:26.713224+0000","last_peered":"2024-02-10T07:35:26.713224+0000","last_clean":"2024-02-10T07:35:26.713224+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:35:26.713224+0000","last_undegraded":"2024-02-10T07:35:26.713224+0000","last_fullsized":"2024-02-10T07:35:26.713224+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:52.665846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:52.665846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:52.800299+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"89'13","reported_seq":167,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.744778+0000","last_change":"2024-02-10T07:34:35.582398+0000","last_active":"2024-02-10T07:35:32.744778+0000","last_peered":"2024-02-10T07:35:32.744778+0000","last_clean":"2024-02-10T07:35:32.744778+0000","last_became_active":"2024-02-10T07:31:53.295353+0000","last_became_peered":"2024-02-10T07:31:53.295353+0000","last_unstale":"2024-02-10T07:35:32.744778+0000","last_undegraded":"2024-02-10T07:35:32.744778+0000","last_fullsized":"2024-02-10T07:35:32.744778+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-10T07:34:35.582276+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:34:35.582276+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:47.622072+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":130,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.713520+0000","last_change":"2024-02-10T07:34:32.515334+0000","last_active":"2024-02-10T07:35:26.713520+0000","last_peered":"2024-02-10T07:35:26.713520+0000","last_clean":"2024-02-10T07:35:26.713520+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:35:26.713520+0000","last_undegraded":"2024-02-10T07:35:26.713520+0000","last_fullsized":"2024-02-10T07:35:26.713520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:32.515225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:32.515225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:59.065744+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":133,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.701055+0000","last_change":"2024-02-10T07:33:32.294625+0000","last_active":"2024-02-10T07:35:26.701055+0000","last_peered":"2024-02-10T07:35:26.701055+0000","last_clean":"2024-02-10T07:35:26.701055+0000","last_became_active":"2024-02-10T07:31:55.296369+0000","last_became_peered":"2024-02-10T07:31:55.296369+0000","last_unstale":"2024-02-10T07:35:26.701055+0000","last_undegraded":"2024-02-10T07:35:26.701055+0000","last_fullsized":"2024-02-10T07:35:26.701055+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:32.294504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:32.294504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":149,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.702372+0000","last_change":"2024-02-10T07:34:42.822759+0000","last_active":"2024-02-10T07:35:26.702372+0000","last_peered":"2024-02-10T07:35:26.702372+0000","last_clean":"2024-02-10T07:35:26.702372+0000","last_became_active":"2024-02-10T07:31:51.285159+0000","last_became_peered":"2024-02-10T07:31:51.285159+0000","last_unstale":"2024-02-10T07:35:26.702372+0000","last_undegraded":"2024-02-10T07:35:26.702372+0000","last_fullsized":"2024-02-10T07:35:26.702372+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:42.822615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:42.822615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:54.821259+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":149,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.740673+0000","last_change":"2024-02-10T07:34:40.532359+0000","last_active":"2024-02-10T07:35:32.740673+0000","last_peered":"2024-02-10T07:35:32.740673+0000","last_clean":"2024-02-10T07:35:32.740673+0000","last_became_active":"2024-02-10T07:31:53.296831+0000","last_became_peered":"2024-02-10T07:31:53.296831+0000","last_unstale":"2024-02-10T07:35:32.740673+0000","last_undegraded":"2024-02-10T07:35:32.740673+0000","last_fullsized":"2024-02-10T07:35:32.740673+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-10T07:34:40.531579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:40.531579+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:48.219679+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"128'6796","reported_seq":6866,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.691956+0000","last_change":"2024-02-10T07:33:53.679248+0000","last_active":"2024-02-10T07:35:32.691956+0000","last_peered":"2024-02-10T07:35:32.691956+0000","last_clean":"2024-02-10T07:35:32.691956+0000","last_became_active":"2024-02-10T07:33:53.678919+0000","last_became_peered":"2024-02-10T07:33:53.678919+0000","last_unstale":"2024-02-10T07:35:32.691956+0000","last_undegraded":"2024-02-10T07:35:32.691956+0000","last_fullsized":"2024-02-10T07:35:32.691956+0000","mapping_epoch":98,"log_start":"108'1400","ondisk_log_start":"108'1400","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445382656,"num_objects":6796,"num_object_clones":0,"num_object_copies":20388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6796,"num_write_kb":434944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":129,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698117+0000","last_change":"2024-02-10T07:33:48.258791+0000","last_active":"2024-02-10T07:35:26.698117+0000","last_peered":"2024-02-10T07:35:26.698117+0000","last_clean":"2024-02-10T07:35:26.698117+0000","last_became_active":"2024-02-10T07:31:57.311110+0000","last_became_peered":"2024-02-10T07:31:57.311110+0000","last_unstale":"2024-02-10T07:35:26.698117+0000","last_undegraded":"2024-02-10T07:35:26.698117+0000","last_fullsized":"2024-02-10T07:35:26.698117+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:48.258671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:48.258671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":162,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:30.395483+0000","last_change":"2024-02-10T07:35:30.395532+0000","last_active":"2024-02-10T07:35:30.395483+0000","last_peered":"2024-02-10T07:35:30.395483+0000","last_clean":"2024-02-10T07:35:30.395483+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:35:30.395483+0000","last_undegraded":"2024-02-10T07:35:30.395483+0000","last_fullsized":"2024-02-10T07:35:30.395483+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-10T07:35:30.395386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:35:30.395386+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:38.024652+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":170,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.697773+0000","last_change":"2024-02-10T07:34:31.437585+0000","last_active":"2024-02-10T07:35:26.697773+0000","last_peered":"2024-02-10T07:35:26.697773+0000","last_clean":"2024-02-10T07:35:26.697773+0000","last_became_active":"2024-02-10T07:31:53.284817+0000","last_became_peered":"2024-02-10T07:31:53.284817+0000","last_unstale":"2024-02-10T07:35:26.697773+0000","last_undegraded":"2024-02-10T07:35:26.697773+0000","last_fullsized":"2024-02-10T07:35:26.697773+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:34:31.437421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:31.437421+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:37.912389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":195,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.452296+0000","last_change":"2024-02-10T07:35:32.452350+0000","last_active":"2024-02-10T07:35:32.452296+0000","last_peered":"2024-02-10T07:35:32.452296+0000","last_clean":"2024-02-10T07:35:32.452296+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:35:32.452296+0000","last_undegraded":"2024-02-10T07:35:32.452296+0000","last_fullsized":"2024-02-10T07:35:32.452296+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:35:32.452220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:35:32.452220+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:43.372893+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"129'6806","reported_seq":6877,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.822754+0000","last_change":"2024-02-10T07:33:53.679379+0000","last_active":"2024-02-10T07:35:32.822754+0000","last_peered":"2024-02-10T07:35:32.822754+0000","last_clean":"2024-02-10T07:35:32.822754+0000","last_became_active":"2024-02-10T07:33:53.679099+0000","last_became_peered":"2024-02-10T07:33:53.679099+0000","last_unstale":"2024-02-10T07:35:32.822754+0000","last_undegraded":"2024-02-10T07:35:32.822754+0000","last_fullsized":"2024-02-10T07:35:32.822754+0000","mapping_epoch":98,"log_start":"110'2100","ondisk_log_start":"110'2100","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4706,"log_dups_size":0,"ondisk_log_size":4706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446038016,"num_objects":6806,"num_object_clones":0,"num_object_copies":20418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6806,"num_write_kb":435584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":132,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.745827+0000","last_change":"2024-02-10T07:33:34.504958+0000","last_active":"2024-02-10T07:35:32.745827+0000","last_peered":"2024-02-10T07:35:32.745827+0000","last_clean":"2024-02-10T07:35:32.745827+0000","last_became_active":"2024-02-10T07:31:57.314872+0000","last_became_peered":"2024-02-10T07:31:57.314872+0000","last_unstale":"2024-02-10T07:35:32.745827+0000","last_undegraded":"2024-02-10T07:35:32.745827+0000","last_fullsized":"2024-02-10T07:35:32.745827+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:34.504852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:34.504852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":142,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.713875+0000","last_change":"2024-02-10T07:33:31.170623+0000","last_active":"2024-02-10T07:35:26.713875+0000","last_peered":"2024-02-10T07:35:26.713875+0000","last_clean":"2024-02-10T07:35:26.713875+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:35:26.713875+0000","last_undegraded":"2024-02-10T07:35:26.713875+0000","last_fullsized":"2024-02-10T07:35:26.713875+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:33:31.170544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:30.304525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":187,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:31.662064+0000","last_change":"2024-02-10T07:35:31.662108+0000","last_active":"2024-02-10T07:35:31.662064+0000","last_peered":"2024-02-10T07:35:31.662064+0000","last_clean":"2024-02-10T07:35:31.662064+0000","last_became_active":"2024-02-10T07:32:18.389409+0000","last_became_peered":"2024-02-10T07:32:18.389409+0000","last_unstale":"2024-02-10T07:35:31.662064+0000","last_undegraded":"2024-02-10T07:35:31.662064+0000","last_fullsized":"2024-02-10T07:35:31.662064+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:35:31.661986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:35:31.661986+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:45.111224+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.755290+0000","last_change":"2024-02-10T07:34:40.067429+0000","last_active":"2024-02-10T07:35:32.755290+0000","last_peered":"2024-02-10T07:35:32.755290+0000","last_clean":"2024-02-10T07:35:32.755290+0000","last_became_active":"2024-02-10T07:31:55.303059+0000","last_became_peered":"2024-02-10T07:31:55.303059+0000","last_unstale":"2024-02-10T07:35:32.755290+0000","last_undegraded":"2024-02-10T07:35:32.755290+0000","last_fullsized":"2024-02-10T07:35:32.755290+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:40.067288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:40.067288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:55.717664+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"129'6912","reported_seq":6983,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.805270+0000","last_change":"2024-02-10T07:33:53.680345+0000","last_active":"2024-02-10T07:35:32.805270+0000","last_peered":"2024-02-10T07:35:32.805270+0000","last_clean":"2024-02-10T07:35:32.805270+0000","last_became_active":"2024-02-10T07:33:53.680042+0000","last_became_peered":"2024-02-10T07:33:53.680042+0000","last_unstale":"2024-02-10T07:35:32.805270+0000","last_undegraded":"2024-02-10T07:35:32.805270+0000","last_fullsized":"2024-02-10T07:35:32.805270+0000","mapping_epoch":98,"log_start":"108'1500","ondisk_log_start":"108'1500","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":5412,"log_dups_size":0,"ondisk_log_size":5412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452984832,"num_objects":6912,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6912,"num_write_kb":442368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":138,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.712706+0000","last_change":"2024-02-10T07:34:30.506461+0000","last_active":"2024-02-10T07:35:26.712706+0000","last_peered":"2024-02-10T07:35:26.712706+0000","last_clean":"2024-02-10T07:35:26.712706+0000","last_became_active":"2024-02-10T07:31:57.304971+0000","last_became_peered":"2024-02-10T07:31:57.304971+0000","last_unstale":"2024-02-10T07:35:26.712706+0000","last_undegraded":"2024-02-10T07:35:26.712706+0000","last_fullsized":"2024-02-10T07:35:26.712706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:30.506386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:30.506386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:57.021606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.751367+0000","last_change":"2024-02-10T07:34:50.188777+0000","last_active":"2024-02-10T07:35:32.751367+0000","last_peered":"2024-02-10T07:35:32.751367+0000","last_clean":"2024-02-10T07:35:32.751367+0000","last_became_active":"2024-02-10T07:31:55.298322+0000","last_became_peered":"2024-02-10T07:31:55.298322+0000","last_unstale":"2024-02-10T07:35:32.751367+0000","last_undegraded":"2024-02-10T07:35:32.751367+0000","last_fullsized":"2024-02-10T07:35:32.751367+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:50.188510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:50.188510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:09.879905+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":150,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.713798+0000","last_change":"2024-02-10T07:34:57.552605+0000","last_active":"2024-02-10T07:35:26.713798+0000","last_peered":"2024-02-10T07:35:26.713798+0000","last_clean":"2024-02-10T07:35:26.713798+0000","last_became_active":"2024-02-10T07:31:51.283433+0000","last_became_peered":"2024-02-10T07:31:51.283433+0000","last_unstale":"2024-02-10T07:35:26.713798+0000","last_undegraded":"2024-02-10T07:35:26.713798+0000","last_fullsized":"2024-02-10T07:35:26.713798+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:57.552495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:57.552495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:59.008097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":176,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.702569+0000","last_change":"2024-02-10T07:34:37.811083+0000","last_active":"2024-02-10T07:35:26.702569+0000","last_peered":"2024-02-10T07:35:26.702569+0000","last_clean":"2024-02-10T07:35:26.702569+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:35:26.702569+0000","last_undegraded":"2024-02-10T07:35:26.702569+0000","last_fullsized":"2024-02-10T07:35:26.702569+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-10T07:34:37.810931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:37.810931+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:59.912539+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"128'6729","reported_seq":6801,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.755873+0000","last_change":"2024-02-10T07:33:53.680187+0000","last_active":"2024-02-10T07:35:32.755873+0000","last_peered":"2024-02-10T07:35:32.755873+0000","last_clean":"2024-02-10T07:35:32.755873+0000","last_became_active":"2024-02-10T07:33:53.679911+0000","last_became_peered":"2024-02-10T07:33:53.679911+0000","last_unstale":"2024-02-10T07:35:32.755873+0000","last_undegraded":"2024-02-10T07:35:32.755873+0000","last_fullsized":"2024-02-10T07:35:32.755873+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":6729,"log_dups_size":0,"ondisk_log_size":6729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440991744,"num_objects":6729,"num_object_clones":0,"num_object_copies":20187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6729,"num_write_kb":430656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.735332+0000","last_change":"2024-02-10T07:34:39.652304+0000","last_active":"2024-02-10T07:35:32.735332+0000","last_peered":"2024-02-10T07:35:32.735332+0000","last_clean":"2024-02-10T07:35:32.735332+0000","last_became_active":"2024-02-10T07:31:57.313394+0000","last_became_peered":"2024-02-10T07:31:57.313394+0000","last_unstale":"2024-02-10T07:35:32.735332+0000","last_undegraded":"2024-02-10T07:35:32.735332+0000","last_fullsized":"2024-02-10T07:35:32.735332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:39.652060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:39.652060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:01.416562+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.745151+0000","last_change":"2024-02-10T07:34:53.160632+0000","last_active":"2024-02-10T07:35:32.745151+0000","last_peered":"2024-02-10T07:35:32.745151+0000","last_clean":"2024-02-10T07:35:32.745151+0000","last_became_active":"2024-02-10T07:31:55.298622+0000","last_became_peered":"2024-02-10T07:31:55.298622+0000","last_unstale":"2024-02-10T07:35:32.745151+0000","last_undegraded":"2024-02-10T07:35:32.745151+0000","last_fullsized":"2024-02-10T07:35:32.745151+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:53.160525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:53.160525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:12.352277+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":141,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698190+0000","last_change":"2024-02-10T07:33:19.291637+0000","last_active":"2024-02-10T07:35:26.698190+0000","last_peered":"2024-02-10T07:35:26.698190+0000","last_clean":"2024-02-10T07:35:26.698190+0000","last_became_active":"2024-02-10T07:31:51.283208+0000","last_became_peered":"2024-02-10T07:31:51.283208+0000","last_unstale":"2024-02-10T07:35:26.698190+0000","last_undegraded":"2024-02-10T07:35:26.698190+0000","last_fullsized":"2024-02-10T07:35:26.698190+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:19.291556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:19.291556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":163,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.736288+0000","last_change":"2024-02-10T07:35:29.716412+0000","last_active":"2024-02-10T07:35:32.736288+0000","last_peered":"2024-02-10T07:35:32.736288+0000","last_clean":"2024-02-10T07:35:32.736288+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:35:32.736288+0000","last_undegraded":"2024-02-10T07:35:32.736288+0000","last_fullsized":"2024-02-10T07:35:32.736288+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:35:29.716214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:35:29.716214+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:36.485212+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"128'6649","reported_seq":6719,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.653353+0000","last_change":"2024-02-10T07:33:53.681536+0000","last_active":"2024-02-10T07:35:32.653353+0000","last_peered":"2024-02-10T07:35:32.653353+0000","last_clean":"2024-02-10T07:35:32.653353+0000","last_became_active":"2024-02-10T07:33:53.681201+0000","last_became_peered":"2024-02-10T07:33:53.681201+0000","last_unstale":"2024-02-10T07:35:32.653353+0000","last_undegraded":"2024-02-10T07:35:32.653353+0000","last_fullsized":"2024-02-10T07:35:32.653353+0000","mapping_epoch":98,"log_start":"104'600","ondisk_log_start":"104'600","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":6049,"log_dups_size":0,"ondisk_log_size":6049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435748864,"num_objects":6649,"num_object_clones":0,"num_object_copies":19947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6649,"num_write_kb":425536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":132,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.746837+0000","last_change":"2024-02-10T07:34:39.563811+0000","last_active":"2024-02-10T07:35:32.746837+0000","last_peered":"2024-02-10T07:35:32.746837+0000","last_clean":"2024-02-10T07:35:32.746837+0000","last_became_active":"2024-02-10T07:31:57.313854+0000","last_became_peered":"2024-02-10T07:31:57.313854+0000","last_unstale":"2024-02-10T07:35:32.746837+0000","last_undegraded":"2024-02-10T07:35:32.746837+0000","last_fullsized":"2024-02-10T07:35:32.746837+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:39.563733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:39.563733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:00.032067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":160,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698264+0000","last_change":"2024-02-10T07:34:44.402863+0000","last_active":"2024-02-10T07:35:26.698264+0000","last_peered":"2024-02-10T07:35:26.698264+0000","last_clean":"2024-02-10T07:35:26.698264+0000","last_became_active":"2024-02-10T07:31:50.278548+0000","last_became_peered":"2024-02-10T07:31:50.278548+0000","last_unstale":"2024-02-10T07:35:26.698264+0000","last_undegraded":"2024-02-10T07:35:26.698264+0000","last_fullsized":"2024-02-10T07:35:26.698264+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:34:44.402778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:34:44.402778+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:00.957865+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001361225,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":142,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.695541+0000","last_change":"2024-02-10T07:33:31.249197+0000","last_active":"2024-02-10T07:35:26.695541+0000","last_peered":"2024-02-10T07:35:26.695541+0000","last_clean":"2024-02-10T07:35:26.695541+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:35:26.695541+0000","last_undegraded":"2024-02-10T07:35:26.695541+0000","last_fullsized":"2024-02-10T07:35:26.695541+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:31.249117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:31.249117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:29.503689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":229,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.701299+0000","last_change":"2024-02-10T07:33:09.271524+0000","last_active":"2024-02-10T07:35:26.701299+0000","last_peered":"2024-02-10T07:35:26.701299+0000","last_clean":"2024-02-10T07:35:26.701299+0000","last_became_active":"2024-02-10T07:31:53.301863+0000","last_became_peered":"2024-02-10T07:31:53.301863+0000","last_unstale":"2024-02-10T07:35:26.701299+0000","last_undegraded":"2024-02-10T07:35:26.701299+0000","last_fullsized":"2024-02-10T07:35:26.701299+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-02-10T07:33:09.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:09.271402+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.736224+0000","last_change":"2024-02-10T07:34:45.597280+0000","last_active":"2024-02-10T07:35:32.736224+0000","last_peered":"2024-02-10T07:35:32.736224+0000","last_clean":"2024-02-10T07:35:32.736224+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:35:32.736224+0000","last_undegraded":"2024-02-10T07:35:32.736224+0000","last_fullsized":"2024-02-10T07:35:32.736224+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:45.597169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:45.597169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:01.604396+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"129'6661","reported_seq":6733,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.797229+0000","last_change":"2024-02-10T07:33:53.680679+0000","last_active":"2024-02-10T07:35:32.797229+0000","last_peered":"2024-02-10T07:35:32.797229+0000","last_clean":"2024-02-10T07:35:32.797229+0000","last_became_active":"2024-02-10T07:33:53.680304+0000","last_became_peered":"2024-02-10T07:33:53.680304+0000","last_unstale":"2024-02-10T07:35:32.797229+0000","last_undegraded":"2024-02-10T07:35:32.797229+0000","last_fullsized":"2024-02-10T07:35:32.797229+0000","mapping_epoch":98,"log_start":"108'1300","ondisk_log_start":"108'1300","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":5361,"log_dups_size":0,"ondisk_log_size":5361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436535296,"num_objects":6661,"num_object_clones":0,"num_object_copies":19983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6661,"num_write_kb":426304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":138,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.713927+0000","last_change":"2024-02-10T07:34:51.654337+0000","last_active":"2024-02-10T07:35:26.713927+0000","last_peered":"2024-02-10T07:35:26.713927+0000","last_clean":"2024-02-10T07:35:26.713927+0000","last_became_active":"2024-02-10T07:31:57.299958+0000","last_became_peered":"2024-02-10T07:31:57.299958+0000","last_unstale":"2024-02-10T07:35:26.713927+0000","last_undegraded":"2024-02-10T07:35:26.713927+0000","last_fullsized":"2024-02-10T07:35:26.713927+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:51.654211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:51.654211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:16.858026+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":148,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.703753+0000","last_change":"2024-02-10T07:34:48.321459+0000","last_active":"2024-02-10T07:35:26.703753+0000","last_peered":"2024-02-10T07:35:26.703753+0000","last_clean":"2024-02-10T07:35:26.703753+0000","last_became_active":"2024-02-10T07:31:51.320864+0000","last_became_peered":"2024-02-10T07:31:51.320864+0000","last_unstale":"2024-02-10T07:35:26.703753+0000","last_undegraded":"2024-02-10T07:35:26.703753+0000","last_fullsized":"2024-02-10T07:35:26.703753+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:48.321040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:48.321040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:00.679661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":160,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.745320+0000","last_change":"2024-02-10T07:34:52.140876+0000","last_active":"2024-02-10T07:35:32.745320+0000","last_peered":"2024-02-10T07:35:32.745320+0000","last_clean":"2024-02-10T07:35:32.745320+0000","last_became_active":"2024-02-10T07:31:50.281517+0000","last_became_peered":"2024-02-10T07:31:50.281517+0000","last_unstale":"2024-02-10T07:35:32.745320+0000","last_undegraded":"2024-02-10T07:35:32.745320+0000","last_fullsized":"2024-02-10T07:35:32.745320+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:52.140730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:34:52.140730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:02.004303+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":142,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698315+0000","last_change":"2024-02-10T07:34:59.267360+0000","last_active":"2024-02-10T07:35:26.698315+0000","last_peered":"2024-02-10T07:35:26.698315+0000","last_clean":"2024-02-10T07:35:26.698315+0000","last_became_active":"2024-02-10T07:33:46.500511+0000","last_became_peered":"2024-02-10T07:33:46.500511+0000","last_unstale":"2024-02-10T07:35:26.698315+0000","last_undegraded":"2024-02-10T07:35:26.698315+0000","last_fullsized":"2024-02-10T07:35:26.698315+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-10T07:34:59.267211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:59.267211+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:11.603544+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.745541+0000","last_change":"2024-02-10T07:34:41.086869+0000","last_active":"2024-02-10T07:35:32.745541+0000","last_peered":"2024-02-10T07:35:32.745541+0000","last_clean":"2024-02-10T07:35:32.745541+0000","last_became_active":"2024-02-10T07:31:55.298933+0000","last_became_peered":"2024-02-10T07:31:55.298933+0000","last_unstale":"2024-02-10T07:35:32.745541+0000","last_undegraded":"2024-02-10T07:35:32.745541+0000","last_fullsized":"2024-02-10T07:35:32.745541+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:41.086507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:41.086507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:56.970713+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"129'6684","reported_seq":6756,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.774714+0000","last_change":"2024-02-10T07:33:53.682874+0000","last_active":"2024-02-10T07:35:32.774714+0000","last_peered":"2024-02-10T07:35:32.774714+0000","last_clean":"2024-02-10T07:35:32.774714+0000","last_became_active":"2024-02-10T07:33:53.682538+0000","last_became_peered":"2024-02-10T07:33:53.682538+0000","last_unstale":"2024-02-10T07:35:32.774714+0000","last_undegraded":"2024-02-10T07:35:32.774714+0000","last_fullsized":"2024-02-10T07:35:32.774714+0000","mapping_epoch":98,"log_start":"108'1300","ondisk_log_start":"108'1300","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":5384,"log_dups_size":0,"ondisk_log_size":5384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438042624,"num_objects":6684,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6684,"num_write_kb":427776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":131,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698091+0000","last_change":"2024-02-10T07:33:41.150537+0000","last_active":"2024-02-10T07:35:26.698091+0000","last_peered":"2024-02-10T07:35:26.698091+0000","last_clean":"2024-02-10T07:35:26.698091+0000","last_became_active":"2024-02-10T07:31:57.309929+0000","last_became_peered":"2024-02-10T07:31:57.309929+0000","last_unstale":"2024-02-10T07:35:26.698091+0000","last_undegraded":"2024-02-10T07:35:26.698091+0000","last_fullsized":"2024-02-10T07:35:26.698091+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:33:41.150454+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:33:41.150454+0000","last_clean_scrub_stamp":"2024-02-10T07:33:41.150454+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":395,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.712921+0000","last_change":"2024-02-10T07:34:44.557795+0000","last_active":"2024-02-10T07:35:26.712921+0000","last_peered":"2024-02-10T07:35:26.712921+0000","last_clean":"2024-02-10T07:35:26.712921+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:35:26.712921+0000","last_undegraded":"2024-02-10T07:35:26.712921+0000","last_fullsized":"2024-02-10T07:35:26.712921+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:34:44.557700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:34:44.557700+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:07.903579+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.739424+0000","last_change":"2024-02-10T07:33:23.493888+0000","last_active":"2024-02-10T07:35:32.739424+0000","last_peered":"2024-02-10T07:35:32.739424+0000","last_clean":"2024-02-10T07:35:32.739424+0000","last_became_active":"2024-02-10T07:31:51.283332+0000","last_became_peered":"2024-02-10T07:31:51.283332+0000","last_unstale":"2024-02-10T07:35:32.739424+0000","last_undegraded":"2024-02-10T07:35:32.739424+0000","last_fullsized":"2024-02-10T07:35:32.739424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:23.493813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:23.493813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"89'24","reported_seq":220,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.703077+0000","last_change":"2024-02-10T07:34:40.809650+0000","last_active":"2024-02-10T07:35:26.703077+0000","last_peered":"2024-02-10T07:35:26.703077+0000","last_clean":"2024-02-10T07:35:26.703077+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:35:26.703077+0000","last_undegraded":"2024-02-10T07:35:26.703077+0000","last_fullsized":"2024-02-10T07:35:26.703077+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'24","last_scrub_stamp":"2024-02-10T07:34:40.809513+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:34:40.809513+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:05.666758+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":134,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.714559+0000","last_change":"2024-02-10T07:34:38.488157+0000","last_active":"2024-02-10T07:35:26.714559+0000","last_peered":"2024-02-10T07:35:26.714559+0000","last_clean":"2024-02-10T07:35:26.714559+0000","last_became_active":"2024-02-10T07:31:55.299928+0000","last_became_peered":"2024-02-10T07:31:55.299928+0000","last_unstale":"2024-02-10T07:35:26.714559+0000","last_undegraded":"2024-02-10T07:35:26.714559+0000","last_fullsized":"2024-02-10T07:35:26.714559+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:38.488082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:38.488082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:54.688121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"128'6642","reported_seq":6785,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.642300+0000","last_change":"2024-02-10T07:35:30.258558+0000","last_active":"2024-02-10T07:35:32.642300+0000","last_peered":"2024-02-10T07:35:32.642300+0000","last_clean":"2024-02-10T07:35:32.642300+0000","last_became_active":"2024-02-10T07:33:53.682606+0000","last_became_peered":"2024-02-10T07:33:53.682606+0000","last_unstale":"2024-02-10T07:35:32.642300+0000","last_undegraded":"2024-02-10T07:35:32.642300+0000","last_fullsized":"2024-02-10T07:35:32.642300+0000","mapping_epoch":98,"log_start":"108'1500","ondisk_log_start":"108'1500","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'6502","last_scrub_stamp":"2024-02-10T07:35:30.258355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:35:30.258355+0000","objects_scrubbed":6441,"log_size":5142,"log_dups_size":0,"ondisk_log_size":5142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:45.136610+0000","scrub_duration":1931,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435290112,"num_objects":6642,"num_object_clones":0,"num_object_copies":19926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6642,"num_write_kb":425088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":85,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.697868+0000","last_change":"2024-02-10T07:34:54.270014+0000","last_active":"2024-02-10T07:35:26.697868+0000","last_peered":"2024-02-10T07:35:26.697868+0000","last_clean":"2024-02-10T07:35:26.697868+0000","last_became_active":"2024-02-10T07:33:46.500003+0000","last_became_peered":"2024-02-10T07:33:46.500003+0000","last_unstale":"2024-02-10T07:35:26.697868+0000","last_undegraded":"2024-02-10T07:35:26.697868+0000","last_fullsized":"2024-02-10T07:35:26.697868+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:54.269886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:11.220506+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":151,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.735547+0000","last_change":"2024-02-10T07:35:32.672415+0000","last_active":"2024-02-10T07:35:32.735547+0000","last_peered":"2024-02-10T07:35:32.735547+0000","last_clean":"2024-02-10T07:35:32.735547+0000","last_became_active":"2024-02-10T07:31:51.286076+0000","last_became_peered":"2024-02-10T07:31:51.286076+0000","last_unstale":"2024-02-10T07:35:32.735547+0000","last_undegraded":"2024-02-10T07:35:32.735547+0000","last_fullsized":"2024-02-10T07:35:32.735547+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:35:32.672121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:35:32.672121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:37:01.205620+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":370,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:31.478018+0000","last_change":"2024-02-10T07:34:31.523532+0000","last_active":"2024-02-10T07:35:31.478018+0000","last_peered":"2024-02-10T07:35:31.478018+0000","last_clean":"2024-02-10T07:35:31.478018+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:35:31.478018+0000","last_undegraded":"2024-02-10T07:35:31.478018+0000","last_fullsized":"2024-02-10T07:35:31.478018+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:34:31.523406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:34:31.523406+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:59.908550+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":216,"num_read_kb":211,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":157,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.744431+0000","last_change":"2024-02-10T07:33:37.561833+0000","last_active":"2024-02-10T07:35:32.744431+0000","last_peered":"2024-02-10T07:35:32.744431+0000","last_clean":"2024-02-10T07:35:32.744431+0000","last_became_active":"2024-02-10T07:31:53.300687+0000","last_became_peered":"2024-02-10T07:31:53.300687+0000","last_unstale":"2024-02-10T07:35:32.744431+0000","last_undegraded":"2024-02-10T07:35:32.744431+0000","last_fullsized":"2024-02-10T07:35:32.744431+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:33:37.561741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:37.561741+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":133,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698410+0000","last_change":"2024-02-10T07:33:36.210648+0000","last_active":"2024-02-10T07:35:26.698410+0000","last_peered":"2024-02-10T07:35:26.698410+0000","last_clean":"2024-02-10T07:35:26.698410+0000","last_became_active":"2024-02-10T07:31:55.297178+0000","last_became_peered":"2024-02-10T07:31:55.297178+0000","last_unstale":"2024-02-10T07:35:26.698410+0000","last_undegraded":"2024-02-10T07:35:26.698410+0000","last_fullsized":"2024-02-10T07:35:26.698410+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:36.210570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:36.210570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"128'6923","reported_seq":9787,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.629633+0000","last_change":"2024-02-10T07:34:30.780875+0000","last_active":"2024-02-10T07:35:32.629633+0000","last_peered":"2024-02-10T07:35:32.629633+0000","last_clean":"2024-02-10T07:35:32.629633+0000","last_became_active":"2024-02-10T07:34:30.779160+0000","last_became_peered":"2024-02-10T07:34:30.779160+0000","last_unstale":"2024-02-10T07:35:32.629633+0000","last_undegraded":"2024-02-10T07:35:32.629633+0000","last_fullsized":"2024-02-10T07:35:32.629633+0000","mapping_epoch":112,"log_start":"105'1000","ondisk_log_start":"105'1000","created":98,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":5923,"log_dups_size":0,"ondisk_log_size":5923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453705728,"num_objects":6923,"num_object_clones":0,"num_object_copies":20769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6923,"num_write_kb":443072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1392,"num_bytes_recovered":91226112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":128,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.705083+0000","last_change":"2024-02-10T07:33:29.260276+0000","last_active":"2024-02-10T07:35:26.705083+0000","last_peered":"2024-02-10T07:35:26.705083+0000","last_clean":"2024-02-10T07:35:26.705083+0000","last_became_active":"2024-02-10T07:31:57.307615+0000","last_became_peered":"2024-02-10T07:31:57.307615+0000","last_unstale":"2024-02-10T07:35:26.705083+0000","last_undegraded":"2024-02-10T07:35:26.705083+0000","last_fullsized":"2024-02-10T07:35:26.705083+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:29.260196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:29.260196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:29.270008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":140,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.701181+0000","last_change":"2024-02-10T07:33:06.270053+0000","last_active":"2024-02-10T07:35:26.701181+0000","last_peered":"2024-02-10T07:35:26.701181+0000","last_clean":"2024-02-10T07:35:26.701181+0000","last_became_active":"2024-02-10T07:31:51.320752+0000","last_became_peered":"2024-02-10T07:31:51.320752+0000","last_unstale":"2024-02-10T07:35:26.701181+0000","last_undegraded":"2024-02-10T07:35:26.701181+0000","last_fullsized":"2024-02-10T07:35:26.701181+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:06.269938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:06.269938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:29.269906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":183,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.736049+0000","last_change":"2024-02-10T07:35:31.654332+0000","last_active":"2024-02-10T07:35:32.736049+0000","last_peered":"2024-02-10T07:35:32.736049+0000","last_clean":"2024-02-10T07:35:32.736049+0000","last_became_active":"2024-02-10T07:31:53.289381+0000","last_became_peered":"2024-02-10T07:31:53.289381+0000","last_unstale":"2024-02-10T07:35:32.736049+0000","last_undegraded":"2024-02-10T07:35:32.736049+0000","last_fullsized":"2024-02-10T07:35:32.736049+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-10T07:35:31.654199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:35:31.654199+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:53.655833+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":141,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.702649+0000","last_change":"2024-02-10T07:34:46.896018+0000","last_active":"2024-02-10T07:35:26.702649+0000","last_peered":"2024-02-10T07:35:26.702649+0000","last_clean":"2024-02-10T07:35:26.702649+0000","last_became_active":"2024-02-10T07:31:55.290319+0000","last_became_peered":"2024-02-10T07:31:55.290319+0000","last_unstale":"2024-02-10T07:35:26.702649+0000","last_undegraded":"2024-02-10T07:35:26.702649+0000","last_fullsized":"2024-02-10T07:35:26.702649+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:46.895907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:46.895907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:06.712851+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"128'6640","reported_seq":6710,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.680796+0000","last_change":"2024-02-10T07:33:53.684834+0000","last_active":"2024-02-10T07:35:32.680796+0000","last_peered":"2024-02-10T07:35:32.680796+0000","last_clean":"2024-02-10T07:35:32.680796+0000","last_became_active":"2024-02-10T07:33:53.684493+0000","last_became_peered":"2024-02-10T07:33:53.684493+0000","last_unstale":"2024-02-10T07:35:32.680796+0000","last_undegraded":"2024-02-10T07:35:32.680796+0000","last_fullsized":"2024-02-10T07:35:32.680796+0000","mapping_epoch":98,"log_start":"110'1926","ondisk_log_start":"110'1926","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435159040,"num_objects":6640,"num_object_clones":0,"num_object_copies":19920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6640,"num_write_kb":424960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":17,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.702306+0000","last_change":"2024-02-10T07:35:20.820757+0000","last_active":"2024-02-10T07:35:26.702306+0000","last_peered":"2024-02-10T07:35:26.702306+0000","last_clean":"2024-02-10T07:35:26.702306+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:35:26.702306+0000","last_undegraded":"2024-02-10T07:35:26.702306+0000","last_fullsized":"2024-02-10T07:35:26.702306+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:46.550386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:46.550386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:54.896859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":141,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698202+0000","last_change":"2024-02-10T07:33:42.126774+0000","last_active":"2024-02-10T07:35:26.698202+0000","last_peered":"2024-02-10T07:35:26.698202+0000","last_clean":"2024-02-10T07:35:26.698202+0000","last_became_active":"2024-02-10T07:31:51.320673+0000","last_became_peered":"2024-02-10T07:31:51.320673+0000","last_unstale":"2024-02-10T07:35:26.698202+0000","last_undegraded":"2024-02-10T07:35:26.698202+0000","last_fullsized":"2024-02-10T07:35:26.698202+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:42.126696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:42.126696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":130,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698416+0000","last_change":"2024-02-10T07:34:40.406967+0000","last_active":"2024-02-10T07:35:26.698416+0000","last_peered":"2024-02-10T07:35:26.698416+0000","last_clean":"2024-02-10T07:35:26.698416+0000","last_became_active":"2024-02-10T07:33:46.500435+0000","last_became_peered":"2024-02-10T07:33:46.500435+0000","last_unstale":"2024-02-10T07:35:26.698416+0000","last_undegraded":"2024-02-10T07:35:26.698416+0000","last_fullsized":"2024-02-10T07:35:26.698416+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:34:40.406860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:40.406860+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:04.013861+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":188,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.736377+0000","last_change":"2024-02-10T07:33:33.610339+0000","last_active":"2024-02-10T07:35:32.736377+0000","last_peered":"2024-02-10T07:35:32.736377+0000","last_clean":"2024-02-10T07:35:32.736377+0000","last_became_active":"2024-02-10T07:31:55.295732+0000","last_became_peered":"2024-02-10T07:31:55.295732+0000","last_unstale":"2024-02-10T07:35:32.736377+0000","last_undegraded":"2024-02-10T07:35:32.736377+0000","last_fullsized":"2024-02-10T07:35:32.736377+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:33:33.610254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:33.610254+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"129'6874","reported_seq":9820,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.797453+0000","last_change":"2024-02-10T07:34:38.996284+0000","last_active":"2024-02-10T07:35:32.797453+0000","last_peered":"2024-02-10T07:35:32.797453+0000","last_clean":"2024-02-10T07:35:32.797453+0000","last_became_active":"2024-02-10T07:34:38.991359+0000","last_became_peered":"2024-02-10T07:34:38.991359+0000","last_unstale":"2024-02-10T07:35:32.797453+0000","last_undegraded":"2024-02-10T07:35:32.797453+0000","last_fullsized":"2024-02-10T07:35:32.797453+0000","mapping_epoch":115,"log_start":"111'2200","ondisk_log_start":"111'2200","created":98,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450494464,"num_objects":6874,"num_object_clones":0,"num_object_copies":20622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6874,"num_write_kb":439936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1432,"num_bytes_recovered":93847552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":130,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.713383+0000","last_change":"2024-02-10T07:33:49.128880+0000","last_active":"2024-02-10T07:35:26.713383+0000","last_peered":"2024-02-10T07:35:26.713383+0000","last_clean":"2024-02-10T07:35:26.713383+0000","last_became_active":"2024-02-10T07:31:57.305620+0000","last_became_peered":"2024-02-10T07:31:57.305620+0000","last_unstale":"2024-02-10T07:35:26.713383+0000","last_undegraded":"2024-02-10T07:35:26.713383+0000","last_fullsized":"2024-02-10T07:35:26.713383+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:49.128788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:49.128788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:30.304610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":152,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.744516+0000","last_change":"2024-02-10T07:34:41.561592+0000","last_active":"2024-02-10T07:35:32.744516+0000","last_peered":"2024-02-10T07:35:32.744516+0000","last_clean":"2024-02-10T07:35:32.744516+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:35:32.744516+0000","last_undegraded":"2024-02-10T07:35:32.744516+0000","last_fullsized":"2024-02-10T07:35:32.744516+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:34:41.561478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:41.561478+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:54.371760+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":159,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.696400+0000","last_change":"2024-02-10T07:33:37.293146+0000","last_active":"2024-02-10T07:35:26.696400+0000","last_peered":"2024-02-10T07:35:26.696400+0000","last_clean":"2024-02-10T07:35:26.696400+0000","last_became_active":"2024-02-10T07:31:53.286884+0000","last_became_peered":"2024-02-10T07:31:53.286884+0000","last_unstale":"2024-02-10T07:35:26.696400+0000","last_undegraded":"2024-02-10T07:35:26.696400+0000","last_fullsized":"2024-02-10T07:35:26.696400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-02-10T07:33:37.293058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:37.293058+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:29.503818+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":197,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.736734+0000","last_change":"2024-02-10T07:34:59.737795+0000","last_active":"2024-02-10T07:35:32.736734+0000","last_peered":"2024-02-10T07:35:32.736734+0000","last_clean":"2024-02-10T07:35:32.736734+0000","last_became_active":"2024-02-10T07:31:55.302863+0000","last_became_peered":"2024-02-10T07:31:55.302863+0000","last_unstale":"2024-02-10T07:35:32.736734+0000","last_undegraded":"2024-02-10T07:35:32.736734+0000","last_fullsized":"2024-02-10T07:35:32.736734+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:34:59.737682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:59.737682+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:01.389008+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"129'6685","reported_seq":6757,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.808835+0000","last_change":"2024-02-10T07:33:53.681778+0000","last_active":"2024-02-10T07:35:32.808835+0000","last_peered":"2024-02-10T07:35:32.808835+0000","last_clean":"2024-02-10T07:35:32.808835+0000","last_became_active":"2024-02-10T07:33:53.681493+0000","last_became_peered":"2024-02-10T07:33:53.681493+0000","last_unstale":"2024-02-10T07:35:32.808835+0000","last_undegraded":"2024-02-10T07:35:32.808835+0000","last_fullsized":"2024-02-10T07:35:32.808835+0000","mapping_epoch":98,"log_start":"110'2000","ondisk_log_start":"110'2000","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":4685,"log_dups_size":0,"ondisk_log_size":4685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438108160,"num_objects":6685,"num_object_clones":0,"num_object_copies":20055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6685,"num_write_kb":427840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":130,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.695551+0000","last_change":"2024-02-10T07:34:33.311796+0000","last_active":"2024-02-10T07:35:26.695551+0000","last_peered":"2024-02-10T07:35:26.695551+0000","last_clean":"2024-02-10T07:35:26.695551+0000","last_became_active":"2024-02-10T07:31:57.316047+0000","last_became_peered":"2024-02-10T07:31:57.316047+0000","last_unstale":"2024-02-10T07:35:26.695551+0000","last_undegraded":"2024-02-10T07:35:26.695551+0000","last_fullsized":"2024-02-10T07:35:26.695551+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:33.311699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:34:33.311699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:54.415243+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.740514+0000","last_change":"2024-02-10T07:33:20.491471+0000","last_active":"2024-02-10T07:35:32.740514+0000","last_peered":"2024-02-10T07:35:32.740514+0000","last_clean":"2024-02-10T07:35:32.740514+0000","last_became_active":"2024-02-10T07:31:51.285333+0000","last_became_peered":"2024-02-10T07:31:51.285333+0000","last_unstale":"2024-02-10T07:35:32.740514+0000","last_undegraded":"2024-02-10T07:35:32.740514+0000","last_fullsized":"2024-02-10T07:35:32.740514+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:20.491354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:20.491354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":172,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.736620+0000","last_change":"2024-02-10T07:33:08.541272+0000","last_active":"2024-02-10T07:35:32.736620+0000","last_peered":"2024-02-10T07:35:32.736620+0000","last_clean":"2024-02-10T07:35:32.736620+0000","last_became_active":"2024-02-10T07:31:53.290157+0000","last_became_peered":"2024-02-10T07:31:53.290157+0000","last_unstale":"2024-02-10T07:35:32.736620+0000","last_undegraded":"2024-02-10T07:35:32.736620+0000","last_fullsized":"2024-02-10T07:35:32.736620+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-10T07:33:08.541155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:08.541155+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":142,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.696200+0000","last_change":"2024-02-10T07:34:42.223857+0000","last_active":"2024-02-10T07:35:26.696200+0000","last_peered":"2024-02-10T07:35:26.696200+0000","last_clean":"2024-02-10T07:35:26.696200+0000","last_became_active":"2024-02-10T07:31:55.295900+0000","last_became_peered":"2024-02-10T07:31:55.295900+0000","last_unstale":"2024-02-10T07:35:26.696200+0000","last_undegraded":"2024-02-10T07:35:26.696200+0000","last_fullsized":"2024-02-10T07:35:26.696200+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:42.223775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:42.223775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:01.442986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"128'6823","reported_seq":6927,"reported_epoch":128,"state":"active+clean+scrubbing","last_fresh":"2024-02-10T07:35:32.678600+0000","last_change":"2024-02-10T07:35:31.982338+0000","last_active":"2024-02-10T07:35:32.678600+0000","last_peered":"2024-02-10T07:35:32.678600+0000","last_clean":"2024-02-10T07:35:32.678600+0000","last_became_active":"2024-02-10T07:33:53.676032+0000","last_became_peered":"2024-02-10T07:33:53.676032+0000","last_unstale":"2024-02-10T07:35:32.678600+0000","last_undegraded":"2024-02-10T07:35:32.678600+0000","last_fullsized":"2024-02-10T07:35:32.678600+0000","mapping_epoch":98,"log_start":"104'800","ondisk_log_start":"104'800","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":3299,"log_size":6023,"log_dups_size":0,"ondisk_log_size":6023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447152128,"num_objects":6823,"num_object_clones":0,"num_object_copies":20469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6823,"num_write_kb":436672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":129,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698117+0000","last_change":"2024-02-10T07:33:46.231777+0000","last_active":"2024-02-10T07:35:26.698117+0000","last_peered":"2024-02-10T07:35:26.698117+0000","last_clean":"2024-02-10T07:35:26.698117+0000","last_became_active":"2024-02-10T07:31:57.311054+0000","last_became_peered":"2024-02-10T07:31:57.311054+0000","last_unstale":"2024-02-10T07:35:26.698117+0000","last_undegraded":"2024-02-10T07:35:26.698117+0000","last_fullsized":"2024-02-10T07:35:26.698117+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:46.231673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:46.231673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":187,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.696143+0000","last_change":"2024-02-10T07:34:28.340061+0000","last_active":"2024-02-10T07:35:26.696143+0000","last_peered":"2024-02-10T07:35:26.696143+0000","last_clean":"2024-02-10T07:35:26.696143+0000","last_became_active":"2024-02-10T07:31:55.295067+0000","last_became_peered":"2024-02-10T07:31:55.295067+0000","last_unstale":"2024-02-10T07:35:26.696143+0000","last_undegraded":"2024-02-10T07:35:26.696143+0000","last_fullsized":"2024-02-10T07:35:26.696143+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:34:28.339952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:28.339952+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:53.944683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.736567+0000","last_change":"2024-02-10T07:33:38.392661+0000","last_active":"2024-02-10T07:35:32.736567+0000","last_peered":"2024-02-10T07:35:32.736567+0000","last_clean":"2024-02-10T07:35:32.736567+0000","last_became_active":"2024-02-10T07:31:51.282137+0000","last_became_peered":"2024-02-10T07:31:51.282137+0000","last_unstale":"2024-02-10T07:35:32.736567+0000","last_undegraded":"2024-02-10T07:35:32.736567+0000","last_fullsized":"2024-02-10T07:35:32.736567+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:38.392546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:38.392546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":169,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.702110+0000","last_change":"2024-02-10T07:34:46.308091+0000","last_active":"2024-02-10T07:35:26.702110+0000","last_peered":"2024-02-10T07:35:26.702110+0000","last_clean":"2024-02-10T07:35:26.702110+0000","last_became_active":"2024-02-10T07:31:53.294218+0000","last_became_peered":"2024-02-10T07:31:53.294218+0000","last_unstale":"2024-02-10T07:35:26.702110+0000","last_undegraded":"2024-02-10T07:35:26.702110+0000","last_fullsized":"2024-02-10T07:35:26.702110+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:34:46.307942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:34:46.307942+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:52.814988+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"128'6687","reported_seq":6830,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.632026+0000","last_change":"2024-02-10T07:35:30.993226+0000","last_active":"2024-02-10T07:35:32.632026+0000","last_peered":"2024-02-10T07:35:32.632026+0000","last_clean":"2024-02-10T07:35:32.632026+0000","last_became_active":"2024-02-10T07:33:53.678445+0000","last_became_peered":"2024-02-10T07:33:53.678445+0000","last_unstale":"2024-02-10T07:35:32.632026+0000","last_undegraded":"2024-02-10T07:35:32.632026+0000","last_fullsized":"2024-02-10T07:35:32.632026+0000","mapping_epoch":98,"log_start":"104'600","ondisk_log_start":"104'600","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'6580","last_scrub_stamp":"2024-02-10T07:35:30.993050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:35:30.993050+0000","objects_scrubbed":6518,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:36.699519+0000","scrub_duration":2071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438239232,"num_objects":6687,"num_object_clones":0,"num_object_copies":20061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":427968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":132,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.748381+0000","last_change":"2024-02-10T07:33:04.582281+0000","last_active":"2024-02-10T07:35:32.748381+0000","last_peered":"2024-02-10T07:35:32.748381+0000","last_clean":"2024-02-10T07:35:32.748381+0000","last_became_active":"2024-02-10T07:31:57.303235+0000","last_became_peered":"2024-02-10T07:31:57.303235+0000","last_unstale":"2024-02-10T07:35:32.748381+0000","last_undegraded":"2024-02-10T07:35:32.748381+0000","last_fullsized":"2024-02-10T07:35:32.748381+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:04.582188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:04.582188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":141,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.698383+0000","last_change":"2024-02-10T07:34:43.369811+0000","last_active":"2024-02-10T07:35:26.698383+0000","last_peered":"2024-02-10T07:35:26.698383+0000","last_clean":"2024-02-10T07:35:26.698383+0000","last_became_active":"2024-02-10T07:31:55.294376+0000","last_became_peered":"2024-02-10T07:31:55.294376+0000","last_unstale":"2024-02-10T07:35:26.698383+0000","last_undegraded":"2024-02-10T07:35:26.698383+0000","last_fullsized":"2024-02-10T07:35:26.698383+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:43.369691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:43.369691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:53.842761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":150,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.696273+0000","last_change":"2024-02-10T07:34:59.459857+0000","last_active":"2024-02-10T07:35:26.696273+0000","last_peered":"2024-02-10T07:35:26.696273+0000","last_clean":"2024-02-10T07:35:26.696273+0000","last_became_active":"2024-02-10T07:31:51.320715+0000","last_became_peered":"2024-02-10T07:31:51.320715+0000","last_unstale":"2024-02-10T07:35:26.696273+0000","last_undegraded":"2024-02-10T07:35:26.696273+0000","last_fullsized":"2024-02-10T07:35:26.696273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:59.459739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:34:59.459739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:20.291209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"89'16","reported_seq":188,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.695881+0000","last_change":"2024-02-10T07:34:56.426389+0000","last_active":"2024-02-10T07:35:26.695881+0000","last_peered":"2024-02-10T07:35:26.695881+0000","last_clean":"2024-02-10T07:35:26.695881+0000","last_became_active":"2024-02-10T07:33:46.499156+0000","last_became_peered":"2024-02-10T07:33:46.499156+0000","last_unstale":"2024-02-10T07:35:26.695881+0000","last_undegraded":"2024-02-10T07:35:26.695881+0000","last_fullsized":"2024-02-10T07:35:26.695881+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'16","last_scrub_stamp":"2024-02-10T07:34:56.426264+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:34:56.426264+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:05.963061+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"128'6740","reported_seq":6812,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.751201+0000","last_change":"2024-02-10T07:33:53.682808+0000","last_active":"2024-02-10T07:35:32.751201+0000","last_peered":"2024-02-10T07:35:32.751201+0000","last_clean":"2024-02-10T07:35:32.751201+0000","last_became_active":"2024-02-10T07:33:53.682513+0000","last_became_peered":"2024-02-10T07:33:53.682513+0000","last_unstale":"2024-02-10T07:35:32.751201+0000","last_undegraded":"2024-02-10T07:35:32.751201+0000","last_fullsized":"2024-02-10T07:35:32.751201+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":6740,"log_dups_size":0,"ondisk_log_size":6740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441712640,"num_objects":6740,"num_object_clones":0,"num_object_copies":20220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6740,"num_write_kb":431360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.745055+0000","last_change":"2024-02-10T07:34:44.046519+0000","last_active":"2024-02-10T07:35:32.745055+0000","last_peered":"2024-02-10T07:35:32.745055+0000","last_clean":"2024-02-10T07:35:32.745055+0000","last_became_active":"2024-02-10T07:31:57.311427+0000","last_became_peered":"2024-02-10T07:31:57.311427+0000","last_unstale":"2024-02-10T07:35:32.745055+0000","last_undegraded":"2024-02-10T07:35:32.745055+0000","last_fullsized":"2024-02-10T07:35:32.745055+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:44.046401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:34:44.046401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:56.660979+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":132,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.701300+0000","last_change":"2024-02-10T07:33:12.259596+0000","last_active":"2024-02-10T07:35:26.701300+0000","last_peered":"2024-02-10T07:35:26.701300+0000","last_clean":"2024-02-10T07:35:26.701300+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:35:26.701300+0000","last_undegraded":"2024-02-10T07:35:26.701300+0000","last_fullsized":"2024-02-10T07:35:26.701300+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:12.259478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:33:12.259478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:29.269958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":149,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.702670+0000","last_change":"2024-02-10T07:34:35.761455+0000","last_active":"2024-02-10T07:35:26.702670+0000","last_peered":"2024-02-10T07:35:26.702670+0000","last_clean":"2024-02-10T07:35:26.702670+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:35:26.702670+0000","last_undegraded":"2024-02-10T07:35:26.702670+0000","last_fullsized":"2024-02-10T07:35:26.702670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:34:35.761349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:56.900501+0000","last_clean_scrub_stamp":"2024-02-10T07:34:35.761349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:49.489772+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"89'25","reported_seq":174,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.695938+0000","last_change":"2024-02-10T07:33:20.252130+0000","last_active":"2024-02-10T07:35:26.695938+0000","last_peered":"2024-02-10T07:35:26.695938+0000","last_clean":"2024-02-10T07:35:26.695938+0000","last_became_active":"2024-02-10T07:31:53.292277+0000","last_became_peered":"2024-02-10T07:31:53.292277+0000","last_unstale":"2024-02-10T07:35:26.695938+0000","last_undegraded":"2024-02-10T07:35:26.695938+0000","last_fullsized":"2024-02-10T07:35:26.695938+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'25","last_scrub_stamp":"2024-02-10T07:33:20.251980+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:33:20.251980+0000","last_clean_scrub_stamp":"2024-02-10T07:33:20.251980+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:29.503707+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"129'6906","reported_seq":6976,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.788357+0000","last_change":"2024-02-10T07:33:53.681706+0000","last_active":"2024-02-10T07:35:32.788357+0000","last_peered":"2024-02-10T07:35:32.788357+0000","last_clean":"2024-02-10T07:35:32.788357+0000","last_became_active":"2024-02-10T07:33:53.681419+0000","last_became_peered":"2024-02-10T07:33:53.681419+0000","last_unstale":"2024-02-10T07:35:32.788357+0000","last_undegraded":"2024-02-10T07:35:32.788357+0000","last_fullsized":"2024-02-10T07:35:32.788357+0000","mapping_epoch":98,"log_start":"108'1500","ondisk_log_start":"108'1500","created":98,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:52.661748+0000","last_clean_scrub_stamp":"2024-02-10T07:33:52.661748+0000","objects_scrubbed":0,"log_size":5406,"log_dups_size":0,"ondisk_log_size":5406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452591616,"num_objects":6906,"num_object_clones":0,"num_object_copies":20718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6906,"num_write_kb":441984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":130,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.696189+0000","last_change":"2024-02-10T07:33:25.318627+0000","last_active":"2024-02-10T07:35:26.696189+0000","last_peered":"2024-02-10T07:35:26.696189+0000","last_clean":"2024-02-10T07:35:26.696189+0000","last_became_active":"2024-02-10T07:31:57.308608+0000","last_became_peered":"2024-02-10T07:31:57.308608+0000","last_unstale":"2024-02-10T07:35:26.696189+0000","last_undegraded":"2024-02-10T07:35:26.696189+0000","last_fullsized":"2024-02-10T07:35:26.696189+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:25.318521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:33:25.318521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:29.503723+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":199,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.745101+0000","last_change":"2024-02-10T07:34:59.141641+0000","last_active":"2024-02-10T07:35:32.745101+0000","last_peered":"2024-02-10T07:35:32.745101+0000","last_clean":"2024-02-10T07:35:32.745101+0000","last_became_active":"2024-02-10T07:31:55.298389+0000","last_became_peered":"2024-02-10T07:31:55.298389+0000","last_unstale":"2024-02-10T07:35:32.745101+0000","last_undegraded":"2024-02-10T07:35:32.745101+0000","last_fullsized":"2024-02-10T07:35:32.745101+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:34:59.141399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:34:59.141399+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:36:04.565877+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":144,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-02-10T07:35:32.742004+0000","last_change":"2024-02-10T07:33:19.515540+0000","last_active":"2024-02-10T07:35:32.742004+0000","last_peered":"2024-02-10T07:35:32.742004+0000","last_clean":"2024-02-10T07:35:32.742004+0000","last_became_active":"2024-02-10T07:31:51.288197+0000","last_became_peered":"2024-02-10T07:31:51.288197+0000","last_unstale":"2024-02-10T07:35:32.742004+0000","last_undegraded":"2024-02-10T07:35:32.742004+0000","last_fullsized":"2024-02-10T07:35:32.742004+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:33:19.515464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:33:19.515464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":153,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-02-10T07:35:26.696517+0000","last_change":"2024-02-10T07:33:41.311832+0000","last_active":"2024-02-10T07:35:26.696517+0000","last_peered":"2024-02-10T07:35:26.696517+0000","last_clean":"2024-02-10T07:35:26.696517+0000","last_became_active":"2024-02-10T07:31:53.287984+0000","last_became_peered":"2024-02-10T07:31:53.287984+0000","last_unstale":"2024-02-10T07:35:26.696517+0000","last_undegraded":"2024-02-10T07:35:26.696517+0000","last_fullsized":"2024-02-10T07:35:26.696517+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:33:41.311687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:33:41.311687+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:35:29.503764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":7088177152,"num_objects":108157,"num_object_clones":0,"num_object_copies":324471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108157,"num_write_kb":6922048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2824,"num_bytes_recovered":185073664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21264531456,"data_stored":21264531456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88431,"ondisk_log_size":88431,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":216,"num_read_kb":211,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":437,"ondisk_log_size":437,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037809,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2228708,"kb_used_data":2157276,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91516444,"statfs":{"total":95995035648,"available":93712838656,"internally_reserved":0,"allocated":2209050624,"data_stored":2208529888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":3.452,"5min":1.121,"15min":0.73099999999999998},"min":{"1min":0.29099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":52.591999999999999,"5min":52.591999999999999,"15min":52.591999999999999},"last":1.9359999999999999},{"interface":"front","average":{"1min":2.8769999999999998,"5min":1.024,"15min":0.72499999999999998},"min":{"1min":0.26700000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":52.494,"5min":52.494,"15min":52.494},"last":2.1920000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53400000000000003,"15min":0.53600000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.58099999999999996,"15min":0.55700000000000005},"min":{"1min":0.32000000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":3.4140000000000001,"5min":3.4140000000000001,"15min":3.4140000000000001},"last":0.59399999999999997}]},{"osd":2,"last update":"Sat Feb 10 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.621,"15min":0.57699999999999996},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":9.1600000000000001,"5min":9.1600000000000001,"15min":9.1600000000000001},"last":0.315},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.59099999999999997,"15min":0.57999999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.7090000000000001,"5min":2.7090000000000001,"15min":2.7090000000000001},"last":1.909}]},{"osd":3,"last update":"Sat Feb 10 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":3.0910000000000002,"5min":1.071,"15min":0.73299999999999998},"min":{"1min":0.33100000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":54.375,"5min":54.375,"15min":54.375},"last":0.36399999999999999},{"interface":"front","average":{"1min":3.0990000000000002,"5min":1.091,"15min":0.755},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":54.488999999999997,"5min":54.488999999999997,"15min":54.488999999999997},"last":0.67700000000000005}]},{"osd":4,"last update":"Sat Feb 10 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.47699999999999998,"15min":0.49099999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.498},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51000000000000001,"15min":0.51400000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.39000000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.497,"15min":0.497},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70999999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83399999999999996,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56799999999999995}]},{"osd":6,"last update":"Sat Feb 10 07:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57399999999999995,"15min":0.58499999999999996},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.56100000000000005,"15min":0.56000000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.61099999999999999}]}]},{"osd":6,"up_from":50,"seq":214748364854,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2203144,"kb_used_data":2140032,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91542008,"statfs":{"total":95995035648,"available":93739016192,"internally_reserved":0,"allocated":2191392768,"data_stored":2190899721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:35:21 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.64500000000000002,"15min":0.58599999999999997},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.5839999999999996,"5min":4.5839999999999996,"15min":4.5839999999999996},"last":4.1210000000000004},{"interface":"front","average":{"1min":2.0259999999999998,"5min":0.83599999999999997,"15min":0.64600000000000002},"min":{"1min":0.32600000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":9.423,"5min":9.423,"15min":9.423},"last":0.42999999999999999}]},{"osd":1,"last update":"Sat Feb 10 07:35:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":0.66600000000000004,"15min":0.59399999999999997},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":5.3410000000000002,"5min":5.3410000000000002,"15min":5.3410000000000002},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.169,"5min":0.68300000000000005,"15min":0.60899999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":9.6120000000000001,"5min":9.6120000000000001,"15min":9.6120000000000001},"last":0.499}]},{"osd":2,"last update":"Sat Feb 10 07:35:21 2024","interfaces":[{"interface":"back","average":{"1min":4.0819999999999999,"5min":1.274,"15min":0.81399999999999995},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":65.278999999999996,"5min":65.278999999999996,"15min":65.278999999999996},"last":0.45100000000000001},{"interface":"front","average":{"1min":4.9260000000000002,"5min":1.4359999999999999,"15min":0.85999999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":65.265000000000001,"5min":65.265000000000001,"15min":65.265000000000001},"last":0.54300000000000004}]},{"osd":3,"last update":"Sat Feb 10 07:35:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.2629999999999999,"15min":1.613},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":4.0510000000000002,"5min":19.664999999999999,"15min":19.664999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.68899999999999995,"15min":0.64000000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":4.077,"5min":4.077,"15min":4.077},"last":0.58699999999999997}]},{"osd":4,"last update":"Sat Feb 10 07:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.49199999999999999,"15min":0.47599999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.50900000000000001,"15min":0.48599999999999999},"min":{"1min":0.33200000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.3199999999999998,"5min":2.3199999999999998,"15min":2.3199999999999998},"last":0.46600000000000003}]},{"osd":5,"last update":"Sat Feb 10 07:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.54200000000000004,"15min":0.52000000000000002},"min":{"1min":0.32800000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.286,"5min":2.286,"15min":2.286},"last":0.70299999999999996}]},{"osd":7,"last update":"Sat Feb 10 07:34:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.34699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.38100000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.76600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2691552,"kb_used_data":2606232,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":91053600,"statfs":{"total":95995035648,"available":93238886400,"internally_reserved":0,"allocated":2668781568,"data_stored":2668263908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87357894},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:35:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":0.625,"15min":0.42999999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":4.8730000000000002,"5min":4.8730000000000002,"15min":4.8730000000000002},"last":1.4119999999999999},{"interface":"front","average":{"1min":2.331,"5min":0.81399999999999995,"15min":0.501},"min":{"1min":0.32700000000000001,"5min":0.22900000000000001,"15min":0.22},"max":{"1min":6.7220000000000004,"5min":6.7220000000000004,"15min":6.7220000000000004},"last":2.012}]},{"osd":2,"last update":"Sat Feb 10 07:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.501,"15min":0.42899999999999999},"min":{"1min":0.24099999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":2.508,"5min":2.508,"15min":2.508},"last":1.571},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.51500000000000001,"15min":0.44700000000000001},"min":{"1min":0.25,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.2919999999999998,"5min":5.2919999999999998,"15min":5.2919999999999998},"last":1.657}]},{"osd":3,"last update":"Sat Feb 10 07:35:12 2024","interfaces":[{"interface":"back","average":{"1min":2.2530000000000001,"5min":0.78900000000000003,"15min":0.54500000000000004},"min":{"1min":0.34599999999999997,"5min":0.246,"15min":0.246},"max":{"1min":6.9370000000000003,"5min":6.9370000000000003,"15min":6.9370000000000003},"last":3.5699999999999998},{"interface":"front","average":{"1min":1.637,"5min":0.73699999999999999,"15min":0.58399999999999996},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.1589999999999998,"5min":5.1589999999999998,"15min":5.1589999999999998},"last":1.607}]},{"osd":4,"last update":"Sat Feb 10 07:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.80300000000000005,"15min":0.98499999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":3.3010000000000002,"5min":10.292,"15min":10.292},"last":1.488},{"interface":"front","average":{"1min":1.4219999999999999,"5min":0.74399999999999999,"15min":0.63600000000000001},"min":{"1min":0.34100000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":3.4809999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.64600000000000002,"15min":0.63600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":1.53},{"interface":"front","average":{"1min":1.0389999999999999,"5min":0.68999999999999995,"15min":0.64300000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.3710000000000004,"5min":5.3710000000000004,"15min":5.3710000000000004},"last":1.944}]},{"osd":6,"last update":"Sat Feb 10 07:35:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":0.80600000000000005,"15min":0.68899999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":6.673,"5min":6.673,"15min":6.673},"last":3.5179999999999998},{"interface":"front","average":{"1min":1.0489999999999999,"5min":0.75600000000000001,"15min":0.71399999999999997},"min":{"1min":0.48699999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.2599999999999998,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":1.6379999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:34:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":0.69599999999999995,"15min":0.65600000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":5.3410000000000002,"5min":5.3410000000000002,"15min":5.3410000000000002},"last":1.9810000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.628,"15min":0.63},"min":{"1min":0.442,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":3.5449999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3528036,"kb_used_data":3446876,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":90217116,"statfs":{"total":95995035648,"available":92382326784,"internally_reserved":0,"allocated":3529601024,"data_stored":3529080685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:35:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":0.56999999999999995,"15min":0.41899999999999998},"min":{"1min":0.27400000000000002,"5min":0.217,"15min":0.217},"max":{"1min":4.1680000000000001,"5min":4.1680000000000001,"15min":4.1680000000000001},"last":7.0389999999999997},{"interface":"front","average":{"1min":1.4690000000000001,"5min":0.65300000000000002,"15min":0.45400000000000001},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.766,"5min":4.766,"15min":4.766},"last":1.554}]},{"osd":2,"last update":"Sat Feb 10 07:34:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.497,"15min":0.42499999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":7.3330000000000002,"5min":7.3330000000000002,"15min":7.3330000000000002},"last":0.497},{"interface":"front","average":{"1min":0.874,"5min":0.50900000000000001,"15min":0.45200000000000001},"min":{"1min":0.248,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.3199999999999998,"5min":3.3199999999999998,"15min":3.3199999999999998},"last":1.4359999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:35:07 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":0.749,"15min":0.54900000000000004},"min":{"1min":0.39500000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.7430000000000003,"5min":4.7430000000000003,"15min":4.7430000000000003},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.8380000000000001,"5min":0.73599999999999999,"15min":0.55900000000000005},"min":{"1min":0.32600000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.1379999999999999,"5min":5.1379999999999999,"15min":5.1379999999999999},"last":12.885}]},{"osd":4,"last update":"Sat Feb 10 07:34:31 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.57399999999999995,"15min":0.55300000000000005},"min":{"1min":0.38400000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.6419999999999999,"5min":2.6419999999999999,"15min":2.6419999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.78600000000000003,"15min":0.63400000000000001},"min":{"1min":0.41899999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.9630000000000001,"5min":19.219000000000001,"15min":19.219000000000001},"last":0.56699999999999995}]},{"osd":5,"last update":"Sat Feb 10 07:34:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.63500000000000001,"15min":0.60699999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":2.427,"5min":2.427,"15min":2.427},"last":1.5189999999999999},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.64700000000000002,"15min":0.61099999999999999},"min":{"1min":0.40600000000000003,"5min":0.378,"15min":0.378},"max":{"1min":4.1959999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"last":0.64600000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:35:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.0229999999999999,"15min":0.76900000000000002},"min":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":4.8929999999999998,"5min":19.16,"15min":19.16},"last":1.482},{"interface":"front","average":{"1min":1.407,"5min":0.76500000000000001,"15min":0.68000000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":4.274,"5min":4.274,"15min":4.274},"last":0.60499999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":1.2110000000000001,"15min":1.429},"min":{"1min":0.42399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.2480000000000002,"5min":19.178000000000001,"15min":19.178000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.67900000000000005,"15min":0.66100000000000003},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":2.4700000000000002,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":1.6020000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2634892,"kb_used_data":2571780,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91110260,"statfs":{"total":95995035648,"available":93296906240,"internally_reserved":0,"allocated":2633502720,"data_stored":2633005953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:34:43 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.47899999999999998,"15min":0.41299999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.0739999999999998,"5min":4.0739999999999998,"15min":4.0739999999999998},"last":3.5720000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":0.628,"15min":0.48399999999999999},"min":{"1min":0.27700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":9.1509999999999998,"5min":9.1509999999999998,"15min":9.1509999999999998},"last":4.2720000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:34:43 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.52500000000000002,"15min":0.45300000000000001},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6840000000000002,"5min":3.6840000000000002,"15min":3.6840000000000002},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.51000000000000001,"15min":0.44600000000000001},"min":{"1min":0.26600000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.9470000000000001,"5min":3.9470000000000001,"15min":3.9470000000000001},"last":0.93899999999999995}]},{"osd":3,"last update":"Sat Feb 10 07:35:07 2024","interfaces":[{"interface":"back","average":{"1min":2.4710000000000001,"5min":0.875,"15min":0.65300000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.253,"5min":11.253,"15min":11.253},"last":1.1220000000000001},{"interface":"front","average":{"1min":2.948,"5min":0.97999999999999998,"15min":0.70199999999999996},"min":{"1min":0.34200000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":10.307,"5min":10.307,"15min":10.307},"last":4.3449999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:34:32 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.63400000000000001,"15min":0.627},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.92400000000000004,"5min":1.212,"15min":1.212},"last":1.216},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68700000000000006,"15min":0.65300000000000002},"min":{"1min":0.45500000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.94199999999999995,"5min":5.343,"15min":5.343},"last":3.1030000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:34:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":0.79000000000000004,"15min":0.69099999999999995},"min":{"1min":0.39400000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":9.5190000000000001,"5min":9.5190000000000001,"15min":9.5190000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.427,"5min":0.78700000000000003,"15min":0.67600000000000005},"min":{"1min":0.48499999999999999,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":5.1180000000000003,"5min":5.1180000000000003,"15min":5.1180000000000003},"last":0.34999999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:35:17 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":0.91800000000000004,"15min":0.90100000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":3.9289999999999998,"5min":5.3230000000000004,"15min":5.3230000000000004},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.97,"5min":0.94399999999999995,"15min":0.74199999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":8.1899999999999995,"5min":8.1899999999999995,"15min":8.1899999999999995},"last":0.315}]},{"osd":7,"last update":"Sat Feb 10 07:34:39 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.70099999999999996,"15min":0.67500000000000004},"min":{"1min":0.437,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.9430000000000001,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":0.376},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.68200000000000005,"15min":0.68600000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":1.1559999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4010476,"kb_used_data":3902948,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":89734676,"statfs":{"total":95995035648,"available":91888308224,"internally_reserved":0,"allocated":3996618752,"data_stored":3996088913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:35:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":0.59299999999999997,"15min":0.45100000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":4.4080000000000004,"5min":4.4080000000000004,"15min":4.4080000000000004},"last":5.165},{"interface":"front","average":{"1min":1.966,"5min":0.71599999999999997,"15min":0.51600000000000001},"min":{"1min":0.28299999999999997,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":7.3680000000000003,"5min":7.3680000000000003,"15min":7.3680000000000003},"last":0.496}]},{"osd":1,"last update":"Sat Feb 10 07:35:12 2024","interfaces":[{"interface":"back","average":{"1min":2.2189999999999999,"5min":0.76300000000000001,"15min":0.53700000000000003},"min":{"1min":0.26800000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":13.076000000000001,"5min":13.076000000000001,"15min":13.076000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.536,"5min":0.65100000000000002,"15min":0.50700000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":6.891,"5min":6.891,"15min":6.891},"last":8.6690000000000005}]},{"osd":2,"last update":"Sat Feb 10 07:35:12 2024","interfaces":[{"interface":"back","average":{"1min":8.4260000000000002,"5min":2.0569999999999999,"15min":0.999},"min":{"1min":0.315,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":163.441,"5min":163.441,"15min":163.441},"last":1.9350000000000001},{"interface":"front","average":{"1min":9.157,"5min":2.161,"15min":1.0109999999999999},"min":{"1min":0.26200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":166.82400000000001,"5min":166.82400000000001,"15min":166.82400000000001},"last":4.4199999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:34:35 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.63500000000000001,"15min":0.59099999999999997},"min":{"1min":0.41199999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.2290000000000001,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.64800000000000002,"15min":0.625},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":4.032,"5min":4.032,"15min":4.032},"last":1.8420000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:34:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.66500000000000004,"15min":0.60999999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.2519999999999998,"5min":3.2519999999999998,"15min":3.2519999999999998},"last":4.3499999999999996},{"interface":"front","average":{"1min":1.173,"5min":0.71999999999999997,"15min":0.65300000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":2.694,"5min":2.694,"15min":2.694},"last":0.437}]},{"osd":6,"last update":"Sat Feb 10 07:35:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":0.81299999999999994,"15min":0.72099999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":4.0789999999999997,"5min":4.0789999999999997,"15min":4.0789999999999997},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":0.80600000000000005,"15min":0.66800000000000004},"min":{"1min":0.374,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.9809999999999999,"5min":3.9809999999999999,"15min":3.9809999999999999},"last":4.3099999999999996}]},{"osd":7,"last update":"Sat Feb 10 07:34:45 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.68000000000000005,"15min":0.626},"min":{"1min":0.42799999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.72999999999999998,"15min":0.68999999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":4.1159999999999997,"5min":4.1159999999999997,"15min":4.1159999999999997},"last":1.8049999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953536,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2222884,"kb_used_data":2151452,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91522268,"statfs":{"total":95995035648,"available":93718802432,"internally_reserved":0,"allocated":2203086848,"data_stored":2202565203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:35:27 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":0.84399999999999997,"15min":0.67200000000000004},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":4.2009999999999996,"5min":7.5700000000000003,"15min":7.5700000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.621,"15min":0.57499999999999996},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.9689999999999999,"5min":2.9689999999999999,"15min":2.9689999999999999},"last":0.32100000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:35:27 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.64900000000000002,"15min":0.58399999999999996},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":4.0250000000000004,"5min":4.0250000000000004,"15min":4.0250000000000004},"last":5.0899999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":0.70899999999999996,"15min":0.61799999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":8.0370000000000008,"5min":8.0370000000000008,"15min":8.0370000000000008},"last":0.40699999999999997}]},{"osd":2,"last update":"Sat Feb 10 07:35:27 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":0.79500000000000004,"15min":0.623},"min":{"1min":0.40300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":8.1980000000000004,"5min":8.1980000000000004,"15min":8.1980000000000004},"last":2.694},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.66400000000000003,"15min":0.627},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.7040000000000002,"5min":2.7040000000000002,"15min":2.7040000000000002},"last":7.6779999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:35:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":0.71699999999999997,"15min":0.629},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":5.4050000000000002,"5min":5.4050000000000002,"15min":5.4050000000000002},"last":7.2610000000000001},{"interface":"front","average":{"1min":1.135,"5min":0.73299999999999998,"15min":0.61299999999999999},"min":{"1min":0.34999999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.9889999999999999,"5min":7.601,"15min":7.601},"last":4.9020000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:34:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.40699999999999997,"15min":0.38300000000000001},"min":{"1min":0.27800000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.504,"5min":0.44800000000000001,"15min":0.40799999999999997},"min":{"1min":0.29799999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.496,"15min":0.44800000000000001},"min":{"1min":0.33400000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.48699999999999999,"15min":0.44500000000000001},"min":{"1min":0.313,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.307}]},{"osd":7,"last update":"Sat Feb 10 07:34:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.52800000000000002,"15min":0.51100000000000001},"min":{"1min":0.33900000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.376,"5min":0.318,"15min":0.318},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.56899999999999995}]}]},{"osd":5,"up_from":40,"seq":171798691899,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1803536,"kb_used_data":1741832,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":91941616,"statfs":{"total":95995035648,"available":94148214784,"internally_reserved":0,"allocated":1783635968,"data_stored":1783135397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:34:53 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":0.77600000000000002,"15min":0.65400000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":7.3109999999999999,"5min":7.3109999999999999,"15min":7.3109999999999999},"last":6.5270000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.63500000000000001,"15min":0.57999999999999996},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":3.9980000000000002,"5min":3.9980000000000002,"15min":3.9980000000000002},"last":1.6000000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.59399999999999997,"15min":0.58899999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.2869999999999999,"5min":2.2869999999999999,"15min":2.2869999999999999},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.60599999999999998,"15min":0.59499999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.1059999999999999,"5min":3.238,"15min":3.238},"last":0.41999999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.56899999999999995,"15min":0.57599999999999996},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.4279999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.60199999999999998,"15min":0.61499999999999999},"min":{"1min":0.36299999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":2.4119999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":1.139}]},{"osd":3,"last update":"Sat Feb 10 07:34:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":0.70399999999999996,"15min":0.63600000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":4.5389999999999997,"5min":4.5389999999999997,"15min":4.5389999999999997},"last":3.2730000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.66500000000000004,"15min":0.60499999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":4.335,"5min":4.335,"15min":4.335},"last":0.83799999999999997}]},{"osd":4,"last update":"Sat Feb 10 07:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.439,"15min":0.41099999999999998},"min":{"1min":0.26100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.44600000000000001,"15min":0.40899999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.33100000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.47099999999999997,"15min":0.436},"min":{"1min":0.307,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.52500000000000002,"15min":0.47299999999999998},"min":{"1min":0.33200000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.1110000000000002,"5min":2.1110000000000002,"15min":2.1110000000000002},"last":0.501}]},{"osd":7,"last update":"Sat Feb 10 07:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.33200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.83799999999999997,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32700000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.54900000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3532914688,"data_stored":3532914688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2676490240,"data_stored":2676490240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2652504064,"data_stored":2652504064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3999793152,"data_stored":3999793152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2202009600,"data_stored":2202009600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1786118144,"data_stored":1786118144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2201812992,"data_stored":2201812992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2212888576,"data_stored":2212888576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:35:38.777 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-10T07:35:38.777 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool set unique_pool_0 pg_num 26 2024-02-10T07:35:38.848 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:38.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:38.877 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:38.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:38.978 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:38.994 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:38 smithi008 ceph-mon[29816]: 7.7 scrub ok 2024-02-10T07:35:38.994 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:38 smithi008 ceph-mon[29816]: 3.e scrub starts 2024-02-10T07:35:38.994 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:38 smithi008 ceph-mon[29816]: 3.e scrub ok 2024-02-10T07:35:38.994 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:38 smithi008 ceph-mon[29816]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:35:38.994 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:38 smithi008 ceph-mon[29816]: pgmap v144: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 847 op/s 2024-02-10T07:35:38.994 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:38 smithi008 ceph-mon[29816]: 7.8 scrub starts 2024-02-10T07:35:38.995 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:35:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:35:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:35:38.995 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:38 smithi008 ceph-mon[40743]: 7.7 scrub ok 2024-02-10T07:35:38.995 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:38 smithi008 ceph-mon[40743]: 3.e scrub starts 2024-02-10T07:35:38.995 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:38 smithi008 ceph-mon[40743]: 3.e scrub ok 2024-02-10T07:35:38.995 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:38 smithi008 ceph-mon[40743]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:35:38.995 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:38 smithi008 ceph-mon[40743]: pgmap v144: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 847 op/s 2024-02-10T07:35:38.995 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:38 smithi008 ceph-mon[40743]: 7.8 scrub starts 2024-02-10T07:35:39.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:39.007 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:39.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:39.108 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:39.138 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:39.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:39.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:39.227 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 89 15 114389 114374 80.2954 56 0.0237015 0.0124488 2024-02-10T07:35:39.240 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:39.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:39.299 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:39.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:38 smithi170 ceph-mon[33394]: 7.7 scrub ok 2024-02-10T07:35:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:38 smithi170 ceph-mon[33394]: 3.e scrub starts 2024-02-10T07:35:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:38 smithi170 ceph-mon[33394]: 3.e scrub ok 2024-02-10T07:35:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:38 smithi170 ceph-mon[33394]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:35:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:38 smithi170 ceph-mon[33394]: pgmap v144: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 847 op/s 2024-02-10T07:35:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:38 smithi170 ceph-mon[33394]: 7.8 scrub starts 2024-02-10T07:35:39.401 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:39.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:39.460 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:39.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:39.561 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:39.607 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:39.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:39.633 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:35:39.680 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:35:39.709 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:39.729 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:35:39.738 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:39.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:39.774 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:35:39.821 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:35:39.840 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:39.865 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:35:39.867 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:39.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:39.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:39.915 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:35:39.962 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:35:39.968 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:39.974 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:35:39.974 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:35:39.974 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:35:39.974 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:35:39.974 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:35:39.974 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:35:39.974 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:35:39.974 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:35:39.995 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:39.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:40.097 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:40.140 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:40.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:40.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:40.228 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Total time run: 90.0078 2024-02-10T07:35:40.228 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Total writes made: 115135 2024-02-10T07:35:40.228 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Write size: 65536 2024-02-10T07:35:40.228 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Object size: 65536 2024-02-10T07:35:40.228 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Bandwidth (MB/sec): 79.9479 2024-02-10T07:35:40.228 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev Bandwidth: 35.5088 2024-02-10T07:35:40.228 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max bandwidth (MB/sec): 217.688 2024-02-10T07:35:40.228 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min bandwidth (MB/sec): 0 2024-02-10T07:35:40.228 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Average IOPS: 1279 2024-02-10T07:35:40.228 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev IOPS: 568.145 2024-02-10T07:35:40.228 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max IOPS: 3483 2024-02-10T07:35:40.228 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min IOPS: 0 2024-02-10T07:35:40.228 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Average Latency(s): 0.012504 2024-02-10T07:35:40.229 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev Latency(s): 0.0330671 2024-02-10T07:35:40.229 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max latency(s): 2.36731 2024-02-10T07:35:40.229 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min latency(s): 0.00141219 2024-02-10T07:35:40.232 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Cleaning up (deleting benchmark objects) 2024-02-10T07:35:40.242 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:40.271 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:40.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:40.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:40.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:39 smithi008 ceph-mon[29816]: 5.19 scrub starts 2024-02-10T07:35:40.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:39 smithi008 ceph-mon[29816]: 5.19 scrub ok 2024-02-10T07:35:40.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:39 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2036002882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:40.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:39 smithi008 ceph-mon[40743]: 5.19 scrub starts 2024-02-10T07:35:40.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:39 smithi008 ceph-mon[40743]: 5.19 scrub ok 2024-02-10T07:35:40.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:39 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2036002882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:40.372 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:40.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:39 smithi170 ceph-mon[33394]: 5.19 scrub starts 2024-02-10T07:35:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:39 smithi170 ceph-mon[33394]: 5.19 scrub ok 2024-02-10T07:35:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:39 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2036002882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:40.402 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:40.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:40.504 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:40.542 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:40.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:40.643 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:40.676 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:40.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:40.777 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:40.809 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:40.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:40.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:40.911 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:40.941 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:40.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:40.969 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:35:41.044 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:41.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:41.068 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:41.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:41.169 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:41.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:41.211 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:41.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:41.312 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:41.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[29816]: 6.18 scrub starts 2024-02-10T07:35:41.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[29816]: 6.18 scrub ok 2024-02-10T07:35:41.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[29816]: 6.e scrub starts 2024-02-10T07:35:41.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[29816]: 6.e scrub ok 2024-02-10T07:35:41.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[29816]: pgmap v145: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-10T07:35:41.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[29816]: 7.e scrub ok 2024-02-10T07:35:41.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1604765001' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:35:41.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:35:41.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[29816]: 5.16 scrub starts 2024-02-10T07:35:41.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[29816]: 5.16 scrub ok 2024-02-10T07:35:41.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[40743]: 6.18 scrub starts 2024-02-10T07:35:41.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[40743]: 6.18 scrub ok 2024-02-10T07:35:41.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[40743]: 6.e scrub starts 2024-02-10T07:35:41.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[40743]: 6.e scrub ok 2024-02-10T07:35:41.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[40743]: pgmap v145: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-10T07:35:41.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[40743]: 7.e scrub ok 2024-02-10T07:35:41.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1604765001' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:35:41.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:35:41.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[40743]: 5.16 scrub starts 2024-02-10T07:35:41.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:40 smithi008 ceph-mon[40743]: 5.16 scrub ok 2024-02-10T07:35:41.351 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:41.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:41.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:40 smithi170 ceph-mon[33394]: 6.18 scrub starts 2024-02-10T07:35:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:40 smithi170 ceph-mon[33394]: 6.18 scrub ok 2024-02-10T07:35:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:40 smithi170 ceph-mon[33394]: 6.e scrub starts 2024-02-10T07:35:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:40 smithi170 ceph-mon[33394]: 6.e scrub ok 2024-02-10T07:35:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:40 smithi170 ceph-mon[33394]: pgmap v145: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-10T07:35:41.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:40 smithi170 ceph-mon[33394]: 7.e scrub ok 2024-02-10T07:35:41.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:40 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1604765001' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:35:41.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:40 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:35:41.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:40 smithi170 ceph-mon[33394]: 5.16 scrub starts 2024-02-10T07:35:41.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:40 smithi170 ceph-mon[33394]: 5.16 scrub ok 2024-02-10T07:35:41.453 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:41.491 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:41.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:41.593 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:41.625 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:41.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:41.726 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:41.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:41.769 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:41.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:41.870 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:41.903 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:41.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:42.005 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:42.036 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:42.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:42.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[29816]: 7.8 scrub ok 2024-02-10T07:35:42.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[29816]: 4.6 scrub starts 2024-02-10T07:35:42.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[29816]: 4.6 scrub ok 2024-02-10T07:35:42.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[29816]: 7.1 scrub starts 2024-02-10T07:35:42.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:35:42.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:35:42.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:35:42.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[29816]: osdmap e131: 8 total, 8 up, 8 in 2024-02-10T07:35:42.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[40743]: 7.8 scrub ok 2024-02-10T07:35:42.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[40743]: 4.6 scrub starts 2024-02-10T07:35:42.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[40743]: 4.6 scrub ok 2024-02-10T07:35:42.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[40743]: 7.1 scrub starts 2024-02-10T07:35:42.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:35:42.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:35:42.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:35:42.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:41 smithi008 ceph-mon[40743]: osdmap e131: 8 total, 8 up, 8 in 2024-02-10T07:35:42.138 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:42.173 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:42.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:42.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:42.275 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:42.307 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:42.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:42.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:41 smithi170 ceph-mon[33394]: 7.8 scrub ok 2024-02-10T07:35:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:41 smithi170 ceph-mon[33394]: 4.6 scrub starts 2024-02-10T07:35:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:41 smithi170 ceph-mon[33394]: 4.6 scrub ok 2024-02-10T07:35:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:41 smithi170 ceph-mon[33394]: 7.1 scrub starts 2024-02-10T07:35:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:41 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:35:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:41 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:35:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:41 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:35:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:41 smithi170 ceph-mon[33394]: osdmap e131: 8 total, 8 up, 8 in 2024-02-10T07:35:42.409 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:42.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:42.442 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:42.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:42.544 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:42.590 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:42.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:42.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:42.692 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:42.736 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:42.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:42.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:42.838 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:42.869 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:42.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:42.970 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:43.010 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:43.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:43.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:43.111 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:43.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:35:42 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:35:43.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:42 smithi170 ceph-mon[33394]: 5.c scrub starts 2024-02-10T07:35:43.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:42 smithi170 ceph-mon[33394]: 5.c scrub ok 2024-02-10T07:35:43.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:42 smithi170 ceph-mon[33394]: 5.3 scrub starts 2024-02-10T07:35:43.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:42 smithi170 ceph-mon[33394]: 5.3 scrub ok 2024-02-10T07:35:43.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:42 smithi170 ceph-mon[33394]: pgmap v147: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 638 B/s rd, 38 MiB/s wr, 613 op/s 2024-02-10T07:35:43.141 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:43.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:43.243 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:43.276 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:43.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:43.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:43.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:42 smithi008 ceph-mon[29816]: 5.c scrub starts 2024-02-10T07:35:43.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:42 smithi008 ceph-mon[29816]: 5.c scrub ok 2024-02-10T07:35:43.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:42 smithi008 ceph-mon[29816]: 5.3 scrub starts 2024-02-10T07:35:43.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:42 smithi008 ceph-mon[29816]: 5.3 scrub ok 2024-02-10T07:35:43.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:42 smithi008 ceph-mon[29816]: pgmap v147: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 638 B/s rd, 38 MiB/s wr, 613 op/s 2024-02-10T07:35:43.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:42 smithi008 ceph-mon[40743]: 5.c scrub starts 2024-02-10T07:35:43.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:42 smithi008 ceph-mon[40743]: 5.c scrub ok 2024-02-10T07:35:43.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:42 smithi008 ceph-mon[40743]: 5.3 scrub starts 2024-02-10T07:35:43.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:42 smithi008 ceph-mon[40743]: 5.3 scrub ok 2024-02-10T07:35:43.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:42 smithi008 ceph-mon[40743]: pgmap v147: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 638 B/s rd, 38 MiB/s wr, 613 op/s 2024-02-10T07:35:43.378 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:43.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:43.409 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:43.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:43.510 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:43.539 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:43.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:43.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:43.641 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:43.690 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:43.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:43.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:43.792 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:43.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:43.828 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:43.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:43.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:43.968 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:43.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:43.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:44.070 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:44.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:44.111 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:44.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:44.212 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:44.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:44.236 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:44.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:44.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[29816]: 7.1 scrub ok 2024-02-10T07:35:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[29816]: 4.c scrub starts 2024-02-10T07:35:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[29816]: 4.c scrub ok 2024-02-10T07:35:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:35:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:35:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[29816]: 6.d scrub starts 2024-02-10T07:35:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[29816]: 6.d scrub ok 2024-02-10T07:35:44.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[40743]: 7.1 scrub ok 2024-02-10T07:35:44.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[40743]: 4.c scrub starts 2024-02-10T07:35:44.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[40743]: 4.c scrub ok 2024-02-10T07:35:44.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:35:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:35:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[40743]: 6.d scrub starts 2024-02-10T07:35:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:43 smithi008 ceph-mon[40743]: 6.d scrub ok 2024-02-10T07:35:44.337 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:44.372 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:44.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:43 smithi170 ceph-mon[33394]: 7.1 scrub ok 2024-02-10T07:35:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:43 smithi170 ceph-mon[33394]: 4.c scrub starts 2024-02-10T07:35:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:43 smithi170 ceph-mon[33394]: 4.c scrub ok 2024-02-10T07:35:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:43 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:43 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:35:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:43 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:35:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:43 smithi170 ceph-mon[33394]: 6.d scrub starts 2024-02-10T07:35:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:43 smithi170 ceph-mon[33394]: 6.d scrub ok 2024-02-10T07:35:44.474 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:44.503 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:44.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:44.605 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:44.645 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:44.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:44.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:44.747 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:44.774 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:44.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:44.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:44.876 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:44.916 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:44.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:44.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:45.018 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:45.057 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:45.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:45.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[29816]: 5.1d scrub starts 2024-02-10T07:35:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[29816]: 5.1d scrub ok 2024-02-10T07:35:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[29816]: 5.1a scrub starts 2024-02-10T07:35:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[29816]: 5.1a scrub ok 2024-02-10T07:35:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[29816]: pgmap v148: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 965 op/s 2024-02-10T07:35:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[29816]: osdmap e132: 8 total, 8 up, 8 in 2024-02-10T07:35:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[29816]: 3.1e scrub starts 2024-02-10T07:35:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[29816]: 3.1e scrub ok 2024-02-10T07:35:45.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[40743]: 5.1d scrub starts 2024-02-10T07:35:45.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[40743]: 5.1d scrub ok 2024-02-10T07:35:45.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[40743]: 5.1a scrub starts 2024-02-10T07:35:45.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[40743]: 5.1a scrub ok 2024-02-10T07:35:45.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[40743]: pgmap v148: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 965 op/s 2024-02-10T07:35:45.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[40743]: osdmap e132: 8 total, 8 up, 8 in 2024-02-10T07:35:45.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[40743]: 3.1e scrub starts 2024-02-10T07:35:45.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:44 smithi008 ceph-mon[40743]: 3.1e scrub ok 2024-02-10T07:35:45.159 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:45.194 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:45.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:45.204 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:45.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:45.306 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:45.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:45.337 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:45.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:45.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:44 smithi170 ceph-mon[33394]: 5.1d scrub starts 2024-02-10T07:35:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:44 smithi170 ceph-mon[33394]: 5.1d scrub ok 2024-02-10T07:35:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:44 smithi170 ceph-mon[33394]: 5.1a scrub starts 2024-02-10T07:35:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:44 smithi170 ceph-mon[33394]: 5.1a scrub ok 2024-02-10T07:35:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:44 smithi170 ceph-mon[33394]: pgmap v148: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 44 MiB/s wr, 965 op/s 2024-02-10T07:35:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:44 smithi170 ceph-mon[33394]: osdmap e132: 8 total, 8 up, 8 in 2024-02-10T07:35:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:44 smithi170 ceph-mon[33394]: 3.1e scrub starts 2024-02-10T07:35:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:44 smithi170 ceph-mon[33394]: 3.1e scrub ok 2024-02-10T07:35:45.438 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:45.465 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:45.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:45.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:45.568 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:45.602 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:45.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:45.703 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:45.748 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:45.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:45.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:45.849 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:45.879 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:45.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:45.980 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:46.008 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:46.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:46.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:45 smithi008 ceph-mon[29816]: 6.1 scrub starts 2024-02-10T07:35:46.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:45 smithi008 ceph-mon[29816]: 6.1 scrub ok 2024-02-10T07:35:46.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:45 smithi008 ceph-mon[29816]: 7.6 scrub starts 2024-02-10T07:35:46.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:45 smithi008 ceph-mon[40743]: 6.1 scrub starts 2024-02-10T07:35:46.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:45 smithi008 ceph-mon[40743]: 6.1 scrub ok 2024-02-10T07:35:46.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:45 smithi008 ceph-mon[40743]: 7.6 scrub starts 2024-02-10T07:35:46.110 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:46.148 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:46.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:46.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:46.158 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:35:46.158 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:35:46.250 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:46.306 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:46.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:46.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:46.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:45 smithi170 ceph-mon[33394]: 6.1 scrub starts 2024-02-10T07:35:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:45 smithi170 ceph-mon[33394]: 6.1 scrub ok 2024-02-10T07:35:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:45 smithi170 ceph-mon[33394]: 7.6 scrub starts 2024-02-10T07:35:46.408 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:46.437 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:46.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:46.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:46.539 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:46.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:46.568 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:46.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:46.615 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:35:46.669 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:46.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:46.696 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:46.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:46.728 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:35:46.797 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:46.823 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:46.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:46.925 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:46.952 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:46.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:46.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:47.054 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:47.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:46 smithi008 ceph-mon[29816]: pgmap v150: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 767 B/s rd, 38 MiB/s wr, 936 op/s 2024-02-10T07:35:47.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:46 smithi008 ceph-mon[29816]: 6.c scrub starts 2024-02-10T07:35:47.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:46 smithi008 ceph-mon[29816]: 6.c scrub ok 2024-02-10T07:35:47.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:46 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:35:47.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:46 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:35:47.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:46 smithi008 ceph-mon[40743]: pgmap v150: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 767 B/s rd, 38 MiB/s wr, 936 op/s 2024-02-10T07:35:47.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:46 smithi008 ceph-mon[40743]: 6.c scrub starts 2024-02-10T07:35:47.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:46 smithi008 ceph-mon[40743]: 6.c scrub ok 2024-02-10T07:35:47.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:46 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:35:47.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:46 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:35:47.108 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:47.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:47.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:47.210 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:47.237 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:47.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:47.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:47.339 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:47.371 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:47.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:47.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:47.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:46 smithi170 ceph-mon[33394]: pgmap v150: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 767 B/s rd, 38 MiB/s wr, 936 op/s 2024-02-10T07:35:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:46 smithi170 ceph-mon[33394]: 6.c scrub starts 2024-02-10T07:35:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:46 smithi170 ceph-mon[33394]: 6.c scrub ok 2024-02-10T07:35:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:46 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:35:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:46 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:35:47.473 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:47.517 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:47.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:47.619 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:47.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:47.645 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:47.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:47.668 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-10T07:35:47.746 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:47.793 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:47.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:47.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:47.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:47.921 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:47.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:48.022 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:48.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:48.073 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:48.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:48.161 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:47 smithi008 ceph-mon[29816]: 6.12 scrub starts 2024-02-10T07:35:48.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:47 smithi008 ceph-mon[29816]: 6.12 scrub ok 2024-02-10T07:35:48.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:47 smithi008 ceph-mon[29816]: 3.1b scrub starts 2024-02-10T07:35:48.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:47 smithi008 ceph-mon[29816]: 3.1b scrub ok 2024-02-10T07:35:48.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:47 smithi008 ceph-mon[29816]: osdmap e133: 8 total, 8 up, 8 in 2024-02-10T07:35:48.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:47 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:35:48.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:47 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:48.162 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:47 smithi008 ceph-mon[40743]: 6.12 scrub starts 2024-02-10T07:35:48.162 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:47 smithi008 ceph-mon[40743]: 6.12 scrub ok 2024-02-10T07:35:48.162 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:47 smithi008 ceph-mon[40743]: 3.1b scrub starts 2024-02-10T07:35:48.162 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:47 smithi008 ceph-mon[40743]: 3.1b scrub ok 2024-02-10T07:35:48.163 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:47 smithi008 ceph-mon[40743]: osdmap e133: 8 total, 8 up, 8 in 2024-02-10T07:35:48.163 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:47 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:35:48.163 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:47 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:48.174 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:48.175 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:48.253 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:48.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:48.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:48.355 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:47 smithi170 ceph-mon[33394]: 6.12 scrub starts 2024-02-10T07:35:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:47 smithi170 ceph-mon[33394]: 6.12 scrub ok 2024-02-10T07:35:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:47 smithi170 ceph-mon[33394]: 3.1b scrub starts 2024-02-10T07:35:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:47 smithi170 ceph-mon[33394]: 3.1b scrub ok 2024-02-10T07:35:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:47 smithi170 ceph-mon[33394]: osdmap e133: 8 total, 8 up, 8 in 2024-02-10T07:35:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:47 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:35:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:47 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:35:48.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:48.384 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:48.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:48.485 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:48.554 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:48.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:48.655 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:48.702 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:48.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:48.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:48.803 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:48.840 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:48.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:48.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:48.942 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:48.992 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:48.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:48.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:49.047 INFO:teuthology.orchestra.run.smithi008.stderr:set pool 7 pg_num to 26 2024-02-10T07:35:49.048 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:35:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:35:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:35:49.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[29816]: 7.6 scrub ok 2024-02-10T07:35:49.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[29816]: 7.4 scrub starts 2024-02-10T07:35:49.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[29816]: 4.1a scrub starts 2024-02-10T07:35:49.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[29816]: 4.1a scrub ok 2024-02-10T07:35:49.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[29816]: pgmap v152: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.6 KiB/s rd, 19 MiB/s wr, 970 op/s 2024-02-10T07:35:49.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[29816]: 6.7 scrub starts 2024-02-10T07:35:49.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[29816]: 6.7 scrub ok 2024-02-10T07:35:49.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/293239581' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-02-10T07:35:49.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-02-10T07:35:49.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[29816]: 5.1f scrub starts 2024-02-10T07:35:49.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[29816]: 5.1f scrub ok 2024-02-10T07:35:49.049 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[40743]: 7.6 scrub ok 2024-02-10T07:35:49.049 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[40743]: 7.4 scrub starts 2024-02-10T07:35:49.049 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[40743]: 4.1a scrub starts 2024-02-10T07:35:49.049 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[40743]: 4.1a scrub ok 2024-02-10T07:35:49.049 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[40743]: pgmap v152: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.6 KiB/s rd, 19 MiB/s wr, 970 op/s 2024-02-10T07:35:49.049 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[40743]: 6.7 scrub starts 2024-02-10T07:35:49.049 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[40743]: 6.7 scrub ok 2024-02-10T07:35:49.050 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/293239581' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-02-10T07:35:49.050 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-02-10T07:35:49.050 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[40743]: 5.1f scrub starts 2024-02-10T07:35:49.050 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:49 smithi008 ceph-mon[40743]: 5.1f scrub ok 2024-02-10T07:35:49.093 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:49.133 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:49.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:49.235 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:49.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:49.260 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:49.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:49.362 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:49 smithi170 ceph-mon[33394]: 7.6 scrub ok 2024-02-10T07:35:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:49 smithi170 ceph-mon[33394]: 7.4 scrub starts 2024-02-10T07:35:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:49 smithi170 ceph-mon[33394]: 4.1a scrub starts 2024-02-10T07:35:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:49 smithi170 ceph-mon[33394]: 4.1a scrub ok 2024-02-10T07:35:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:49 smithi170 ceph-mon[33394]: pgmap v152: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.6 KiB/s rd, 19 MiB/s wr, 970 op/s 2024-02-10T07:35:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:49 smithi170 ceph-mon[33394]: 6.7 scrub starts 2024-02-10T07:35:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:49 smithi170 ceph-mon[33394]: 6.7 scrub ok 2024-02-10T07:35:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:49 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/293239581' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-02-10T07:35:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:49 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-02-10T07:35:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:49 smithi170 ceph-mon[33394]: 5.1f scrub starts 2024-02-10T07:35:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:49 smithi170 ceph-mon[33394]: 5.1f scrub ok 2024-02-10T07:35:49.401 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:49.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:49.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:49.503 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:49.540 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:49.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:49.641 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:49.667 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:49.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:49.769 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:35:49.799 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:49.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:35:49.893 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:49.894 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:49.901 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:49.964 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:49.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:50.065 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:50.107 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:50.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:50.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:50.209 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:50.261 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:50.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:50.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:50.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:50 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-02-10T07:35:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:50 smithi008 ceph-mon[40743]: osdmap e134: 8 total, 8 up, 8 in 2024-02-10T07:35:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:50 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-10T07:35:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:50 smithi008 ceph-mon[40743]: 3.a scrub starts 2024-02-10T07:35:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:50 smithi008 ceph-mon[40743]: 3.a scrub ok 2024-02-10T07:35:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:50 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-02-10T07:35:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:50 smithi008 ceph-mon[29816]: osdmap e134: 8 total, 8 up, 8 in 2024-02-10T07:35:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:50 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-10T07:35:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:50 smithi008 ceph-mon[29816]: 3.a scrub starts 2024-02-10T07:35:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:50 smithi008 ceph-mon[29816]: 3.a scrub ok 2024-02-10T07:35:50.364 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:50 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-02-10T07:35:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:50 smithi170 ceph-mon[33394]: osdmap e134: 8 total, 8 up, 8 in 2024-02-10T07:35:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:50 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-10T07:35:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:50 smithi170 ceph-mon[33394]: 3.a scrub starts 2024-02-10T07:35:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:50 smithi170 ceph-mon[33394]: 3.a scrub ok 2024-02-10T07:35:50.389 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:50.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:50.491 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:50.514 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:50.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:50.616 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:50.640 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:50.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:50.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:50.741 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:50.766 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:50.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:50.867 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:50.890 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:50.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:50.992 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:51.020 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:51.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:51.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:51.122 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:51.152 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:51.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:51.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:51.254 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:51.285 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:51.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:51.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:51.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:51 smithi008 ceph-mon[29816]: pgmap v154: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 833 op/s 2024-02-10T07:35:51.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:51 smithi008 ceph-mon[29816]: 7.4 scrub ok 2024-02-10T07:35:51.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:51 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-10T07:35:51.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:51 smithi008 ceph-mon[29816]: osdmap e135: 8 total, 8 up, 8 in 2024-02-10T07:35:51.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:51 smithi008 ceph-mon[40743]: pgmap v154: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 833 op/s 2024-02-10T07:35:51.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:51 smithi008 ceph-mon[40743]: 7.4 scrub ok 2024-02-10T07:35:51.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:51 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-10T07:35:51.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:51 smithi008 ceph-mon[40743]: osdmap e135: 8 total, 8 up, 8 in 2024-02-10T07:35:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:51 smithi170 ceph-mon[33394]: pgmap v154: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 833 op/s 2024-02-10T07:35:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:51 smithi170 ceph-mon[33394]: 7.4 scrub ok 2024-02-10T07:35:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:51 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-10T07:35:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:51 smithi170 ceph-mon[33394]: osdmap e135: 8 total, 8 up, 8 in 2024-02-10T07:35:51.388 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:51.414 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:51.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:51.515 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:51.555 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:51.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:51.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:51.657 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:51.699 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:51.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:51.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:51.801 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:51.835 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:51.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:51.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:51.938 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:51.989 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:51.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:51.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:52.092 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:52.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:52 smithi008 ceph-mon[29816]: 3.7 scrub starts 2024-02-10T07:35:52.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:52 smithi008 ceph-mon[29816]: 3.7 scrub ok 2024-02-10T07:35:52.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:52 smithi008 ceph-mon[29816]: osdmap e136: 8 total, 8 up, 8 in 2024-02-10T07:35:52.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:52 smithi008 ceph-mon[29816]: 7.0 scrub starts 2024-02-10T07:35:52.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:52 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-10T07:35:52.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:52 smithi008 ceph-mon[29816]: 7.7 scrub starts 2024-02-10T07:35:52.116 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:52 smithi008 ceph-mon[40743]: 3.7 scrub starts 2024-02-10T07:35:52.116 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:52 smithi008 ceph-mon[40743]: 3.7 scrub ok 2024-02-10T07:35:52.116 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:52 smithi008 ceph-mon[40743]: osdmap e136: 8 total, 8 up, 8 in 2024-02-10T07:35:52.117 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:52 smithi008 ceph-mon[40743]: 7.0 scrub starts 2024-02-10T07:35:52.117 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:52 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-10T07:35:52.117 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:52 smithi008 ceph-mon[40743]: 7.7 scrub starts 2024-02-10T07:35:52.128 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:52.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:52.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:52.230 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:52.275 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:52.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:52.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:52.377 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:52 smithi170 ceph-mon[33394]: 3.7 scrub starts 2024-02-10T07:35:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:52 smithi170 ceph-mon[33394]: 3.7 scrub ok 2024-02-10T07:35:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:52 smithi170 ceph-mon[33394]: osdmap e136: 8 total, 8 up, 8 in 2024-02-10T07:35:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:52 smithi170 ceph-mon[33394]: 7.0 scrub starts 2024-02-10T07:35:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:52 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-10T07:35:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:52 smithi170 ceph-mon[33394]: 7.7 scrub starts 2024-02-10T07:35:52.414 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:52.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:52.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:52.516 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:52.560 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:52.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:52.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:52.662 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:52.714 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:52.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:52.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:52.816 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:52.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:52.843 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:52.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:52.944 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:52.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:52.972 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:52.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:53.073 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:53.099 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:53.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:53.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:53.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:35:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:35:53.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:53 smithi170 ceph-mon[33394]: 7.5 scrub starts 2024-02-10T07:35:53.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:53 smithi170 ceph-mon[33394]: 7.9 deep-scrub starts 2024-02-10T07:35:53.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:53 smithi170 ceph-mon[33394]: 7.3 deep-scrub starts 2024-02-10T07:35:53.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:53 smithi170 ceph-mon[33394]: pgmap v157: 158 pgs: 10 unknown, 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-10T07:35:53.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:53 smithi170 ceph-mon[33394]: 7.1 scrub starts 2024-02-10T07:35:53.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:53 smithi170 ceph-mon[33394]: 7.8 deep-scrub starts 2024-02-10T07:35:53.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:53 smithi170 ceph-mon[33394]: 7.5 scrub ok 2024-02-10T07:35:53.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:53 smithi170 ceph-mon[33394]: 7.0 scrub ok 2024-02-10T07:35:53.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:53 smithi170 ceph-mon[33394]: 7.10 scrub starts 2024-02-10T07:35:53.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:53 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-10T07:35:53.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:53 smithi170 ceph-mon[33394]: osdmap e137: 8 total, 8 up, 8 in 2024-02-10T07:35:53.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:53 smithi170 ceph-mon[33394]: 7.7 scrub ok 2024-02-10T07:35:53.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:53.228 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:53.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:53.330 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:53.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[29816]: 7.5 scrub starts 2024-02-10T07:35:53.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[29816]: 7.9 deep-scrub starts 2024-02-10T07:35:53.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[29816]: 7.3 deep-scrub starts 2024-02-10T07:35:53.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[29816]: pgmap v157: 158 pgs: 10 unknown, 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-10T07:35:53.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[29816]: 7.1 scrub starts 2024-02-10T07:35:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[29816]: 7.8 deep-scrub starts 2024-02-10T07:35:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[29816]: 7.5 scrub ok 2024-02-10T07:35:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[29816]: 7.0 scrub ok 2024-02-10T07:35:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[29816]: 7.10 scrub starts 2024-02-10T07:35:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-10T07:35:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[29816]: osdmap e137: 8 total, 8 up, 8 in 2024-02-10T07:35:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[29816]: 7.7 scrub ok 2024-02-10T07:35:53.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[40743]: 7.5 scrub starts 2024-02-10T07:35:53.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[40743]: 7.9 deep-scrub starts 2024-02-10T07:35:53.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[40743]: 7.3 deep-scrub starts 2024-02-10T07:35:53.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[40743]: pgmap v157: 158 pgs: 10 unknown, 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-10T07:35:53.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[40743]: 7.1 scrub starts 2024-02-10T07:35:53.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[40743]: 7.8 deep-scrub starts 2024-02-10T07:35:53.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[40743]: 7.5 scrub ok 2024-02-10T07:35:53.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[40743]: 7.0 scrub ok 2024-02-10T07:35:53.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[40743]: 7.10 scrub starts 2024-02-10T07:35:53.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-10T07:35:53.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[40743]: osdmap e137: 8 total, 8 up, 8 in 2024-02-10T07:35:53.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:53 smithi008 ceph-mon[40743]: 7.7 scrub ok 2024-02-10T07:35:53.359 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:53.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:53.461 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:53.491 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:53.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:53.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:53.593 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:53.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:53.622 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:53.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:53.723 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:53.724 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:35:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:53.764 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:53.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:53.809 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:35:53.843 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:35:53.865 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:53.879 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:35:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:53.901 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:53.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:53.911 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:35:53.945 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:35:53.976 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:35:54.002 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:54.012 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:35:54.029 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:54.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:54.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:54.047 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:35:54.049 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:35:54.050 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:35:54.050 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:35:54.050 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:35:54.050 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:35:54.050 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:35:54.050 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:35:54.050 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:35:54.130 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:54.159 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:54.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:54.261 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:54.288 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:54.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:54.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[29816]: 7.15 deep-scrub starts 2024-02-10T07:35:54.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[29816]: 7.1 scrub ok 2024-02-10T07:35:54.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[29816]: 7.6 scrub starts 2024-02-10T07:35:54.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[29816]: 7.2 scrub starts 2024-02-10T07:35:54.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:54.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2505095108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:35:54.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:35:54.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[29816]: 7.2 scrub ok 2024-02-10T07:35:54.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:35:54.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:35:54.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[29816]: osdmap e138: 8 total, 8 up, 8 in 2024-02-10T07:35:54.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2351343724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:54.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[40743]: 7.15 deep-scrub starts 2024-02-10T07:35:54.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[40743]: 7.1 scrub ok 2024-02-10T07:35:54.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[40743]: 7.6 scrub starts 2024-02-10T07:35:54.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[40743]: 7.2 scrub starts 2024-02-10T07:35:54.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:54.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2505095108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:35:54.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:35:54.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[40743]: 7.2 scrub ok 2024-02-10T07:35:54.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:35:54.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:35:54.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[40743]: osdmap e138: 8 total, 8 up, 8 in 2024-02-10T07:35:54.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:54 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2351343724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:54 smithi170 ceph-mon[33394]: 7.15 deep-scrub starts 2024-02-10T07:35:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:54 smithi170 ceph-mon[33394]: 7.1 scrub ok 2024-02-10T07:35:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:54 smithi170 ceph-mon[33394]: 7.6 scrub starts 2024-02-10T07:35:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:54 smithi170 ceph-mon[33394]: 7.2 scrub starts 2024-02-10T07:35:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:54 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:35:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:54 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2505095108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:35:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:54 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:35:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:54 smithi170 ceph-mon[33394]: 7.2 scrub ok 2024-02-10T07:35:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:54 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:35:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:54 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:35:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:54 smithi170 ceph-mon[33394]: osdmap e138: 8 total, 8 up, 8 in 2024-02-10T07:35:54.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:54 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2351343724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:35:54.390 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:54.418 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:54.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:54.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:54.520 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:54.556 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:54.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:54.657 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:54.690 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:54.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:54.791 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:54.841 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:54.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:54.943 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:54.982 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:54.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:54.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:55.083 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:55.095 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:55 smithi008 ceph-mon[40743]: 7.6 scrub ok 2024-02-10T07:35:55.096 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:55 smithi008 ceph-mon[29816]: 7.6 scrub ok 2024-02-10T07:35:55.096 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:55 smithi008 ceph-mon[29816]: pgmap v160: 158 pgs: 4 active+clean+scrubbing+deep, 7 unknown, 3 active+clean+scrubbing, 144 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:55.096 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:55 smithi008 ceph-mon[29816]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T07:35:55.121 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:55.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:55.223 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:55.261 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:55.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:55.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:55.363 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:55.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:55 smithi170 ceph-mon[33394]: 7.6 scrub ok 2024-02-10T07:35:55.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:55 smithi170 ceph-mon[33394]: pgmap v160: 158 pgs: 4 active+clean+scrubbing+deep, 7 unknown, 3 active+clean+scrubbing, 144 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:55.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:55 smithi170 ceph-mon[33394]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T07:35:55.389 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:55.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:55.397 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T07:35:55.397 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:35:55.397 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T07:35:55.397 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-10T07:35:55.397 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.5 2024-02-10T07:35:55.491 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:55.528 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:55.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:55.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:55.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:55 smithi008 ceph-mon[40743]: pgmap v160: 158 pgs: 4 active+clean+scrubbing+deep, 7 unknown, 3 active+clean+scrubbing, 144 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:35:55.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:55 smithi008 ceph-mon[40743]: osdmap e139: 8 total, 8 up, 8 in 2024-02-10T07:35:55.630 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:55.662 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:55.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:55.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:55.764 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:55.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:55.794 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:55.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:55.853 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:35:55 smithi170 systemd[1]: Stopping Ceph osd.5 for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:35:55.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:55.924 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:55.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:55.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:56.026 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:56.069 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:56.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:56.133 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:35:55 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[44249]: 2024-02-10T07:35:55.853+0000 7f8ec420b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:35:56.133 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:35:55 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[44249]: 2024-02-10T07:35:55.853+0000 7f8ec420b700 -1 osd.5 139 *** Got signal Terminated *** 2024-02-10T07:35:56.133 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:35:55 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[44249]: 2024-02-10T07:35:55.853+0000 7f8ec420b700 -1 osd.5 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:35:56.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 136'8077 2024-02-10T07:35:56.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 136'7743 2024-02-10T07:35:56.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 136'8032 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 136'7742 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 136'7754 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 136'8013 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 136'7855 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 136'7726 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 136'8032 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.18 starting backfill to osd.6 from (0'0,0'0] MAX to 136'7855 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 136'8013 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.19 starting backfill to osd.7 from (0'0,0'0] MAX to 136'8013 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 136'7726 2024-02-10T07:35:56.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.17 starting backfill to osd.7 from (0'0,0'0] MAX to 136'7726 2024-02-10T07:35:56.135 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.135 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:56 smithi170 ceph-mon[33394]: osd.5 marked itself down and dead 2024-02-10T07:35:56.171 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:56.197 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:56.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:56.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:56.299 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:56.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:56.336 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:56.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:56.437 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:56.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:56.463 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:56.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:56.564 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 136'8077 2024-02-10T07:35:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 136'7743 2024-02-10T07:35:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 136'8032 2024-02-10T07:35:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 136'7742 2024-02-10T07:35:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 136'7754 2024-02-10T07:35:56.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 136'8013 2024-02-10T07:35:56.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 136'7855 2024-02-10T07:35:56.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 136'7726 2024-02-10T07:35:56.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 136'8032 2024-02-10T07:35:56.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.18 starting backfill to osd.6 from (0'0,0'0] MAX to 136'7855 2024-02-10T07:35:56.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 136'8013 2024-02-10T07:35:56.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.19 starting backfill to osd.7 from (0'0,0'0] MAX to 136'8013 2024-02-10T07:35:56.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 136'7726 2024-02-10T07:35:56.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.17 starting backfill to osd.7 from (0'0,0'0] MAX to 136'7726 2024-02-10T07:35:56.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.585 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[29816]: osd.5 marked itself down and dead 2024-02-10T07:35:56.586 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 136'8077 2024-02-10T07:35:56.586 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 136'7743 2024-02-10T07:35:56.586 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 136'8032 2024-02-10T07:35:56.586 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 136'7742 2024-02-10T07:35:56.587 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 136'7754 2024-02-10T07:35:56.587 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 136'8013 2024-02-10T07:35:56.587 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 136'7855 2024-02-10T07:35:56.587 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 136'7726 2024-02-10T07:35:56.587 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.587 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 136'8032 2024-02-10T07:35:56.587 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.587 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.18 starting backfill to osd.6 from (0'0,0'0] MAX to 136'7855 2024-02-10T07:35:56.587 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.587 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 136'8013 2024-02-10T07:35:56.587 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.587 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.19 starting backfill to osd.7 from (0'0,0'0] MAX to 136'8013 2024-02-10T07:35:56.587 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 136'7726 2024-02-10T07:35:56.588 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.17 starting backfill to osd.7 from (0'0,0'0] MAX to 136'7726 2024-02-10T07:35:56.588 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 136'7707 2024-02-10T07:35:56.588 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:56 smithi008 ceph-mon[40743]: osd.5 marked itself down and dead 2024-02-10T07:35:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:56.606 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:56.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:56.707 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:56.752 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:56.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:56.853 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:56.878 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:56.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:56.883 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:35:56 smithi170 podman[75283]: 2024-02-10 07:35:56.628853361 +0000 UTC m=+0.918493855 container died 3d9970f9a97b7df2d6091324f07f8dfbdf6149a2d2fd2635706980673be38889 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-10T07:35:56.898 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:35:56.980 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:57.028 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:57.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:57.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:57.130 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:35:57.193 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:57.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:57.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:35:57.295 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:57.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:57.329 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:57.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:57.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:57 smithi008 ceph-mon[29816]: pgmap v162: 158 pgs: 4 active+clean+scrubbing+deep, 7 unknown, 3 active+clean+scrubbing, 144 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:35:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:57 smithi008 ceph-mon[29816]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:35:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:57 smithi008 ceph-mon[29816]: osdmap e140: 8 total, 7 up, 8 in 2024-02-10T07:35:57.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:57 smithi008 ceph-mon[40743]: pgmap v162: 158 pgs: 4 active+clean+scrubbing+deep, 7 unknown, 3 active+clean+scrubbing, 144 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:35:57.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:57 smithi008 ceph-mon[40743]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:35:57.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:57 smithi008 ceph-mon[40743]: osdmap e140: 8 total, 7 up, 8 in 2024-02-10T07:35:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:57 smithi170 ceph-mon[33394]: pgmap v162: 158 pgs: 4 active+clean+scrubbing+deep, 7 unknown, 3 active+clean+scrubbing, 144 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:35:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:57 smithi170 ceph-mon[33394]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:35:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:57 smithi170 ceph-mon[33394]: osdmap e140: 8 total, 7 up, 8 in 2024-02-10T07:35:57.430 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:57.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:57.469 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:57.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:57.570 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:57.599 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:57.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:57.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:57.701 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:57.746 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:57.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:57.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:57.848 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:57.877 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:57.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:57.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:57.882 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:35:57 smithi170 podman[75283]: 2024-02-10 07:35:57.62957875 +0000 UTC m=+1.919219236 container cleanup 3d9970f9a97b7df2d6091324f07f8dfbdf6149a2d2fd2635706980673be38889 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-10T07:35:57.882 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:35:57 smithi170 bash[75283]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5 2024-02-10T07:35:57.979 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:58.012 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:58.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:58.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:58.114 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:58.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:58.164 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:58.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:35:58.265 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:58.297 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:58.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:58.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:58.349 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:35:57 smithi170 podman[75333]: 2024-02-10 07:35:57.915304275 +0000 UTC m=+1.288283407 container remove 3d9970f9a97b7df2d6091324f07f8dfbdf6149a2d2fd2635706980673be38889 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2) 2024-02-10T07:35:58.349 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:58 smithi170 ceph-mon[33394]: osdmap e141: 8 total, 7 up, 8 in 2024-02-10T07:35:58.349 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:58 smithi170 ceph-mon[33394]: 7.14 continuing backfill to osd.2 from (112'2200,139'7730] MIN to 139'7730 2024-02-10T07:35:58.349 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:58 smithi170 ceph-mon[33394]: 7.14 continuing backfill to osd.3 from (112'2200,139'7730] MIN to 139'7730 2024-02-10T07:35:58.349 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:58 smithi170 ceph-mon[33394]: 7.17 continuing backfill to osd.2 from (109'1600,139'7756] MIN to 139'7756 2024-02-10T07:35:58.349 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:58 smithi170 ceph-mon[33394]: 7.17 continuing backfill to osd.7 from (109'1600,139'7756] MIN to 139'7756 2024-02-10T07:35:58.349 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:58 smithi170 ceph-mon[33394]: 7.10 continuing backfill to osd.2 from (111'2255,139'8107] 7:08083ed8:::benchmark_data_smithi008_82046_object54438:head to 139'8107 2024-02-10T07:35:58.399 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:58.434 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:58.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:58.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:58.535 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:58.536 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:58 smithi008 ceph-mon[29816]: osdmap e141: 8 total, 7 up, 8 in 2024-02-10T07:35:58.536 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:58 smithi008 ceph-mon[29816]: 7.14 continuing backfill to osd.2 from (112'2200,139'7730] MIN to 139'7730 2024-02-10T07:35:58.537 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:58 smithi008 ceph-mon[29816]: 7.14 continuing backfill to osd.3 from (112'2200,139'7730] MIN to 139'7730 2024-02-10T07:35:58.537 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:58 smithi008 ceph-mon[29816]: 7.17 continuing backfill to osd.2 from (109'1600,139'7756] MIN to 139'7756 2024-02-10T07:35:58.537 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:58 smithi008 ceph-mon[29816]: 7.17 continuing backfill to osd.7 from (109'1600,139'7756] MIN to 139'7756 2024-02-10T07:35:58.537 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:58 smithi008 ceph-mon[29816]: 7.10 continuing backfill to osd.2 from (111'2255,139'8107] 7:08083ed8:::benchmark_data_smithi008_82046_object54438:head to 139'8107 2024-02-10T07:35:58.537 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:58 smithi008 ceph-mon[40743]: osdmap e141: 8 total, 7 up, 8 in 2024-02-10T07:35:58.537 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:58 smithi008 ceph-mon[40743]: 7.14 continuing backfill to osd.2 from (112'2200,139'7730] MIN to 139'7730 2024-02-10T07:35:58.537 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:58 smithi008 ceph-mon[40743]: 7.14 continuing backfill to osd.3 from (112'2200,139'7730] MIN to 139'7730 2024-02-10T07:35:58.537 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:58 smithi008 ceph-mon[40743]: 7.17 continuing backfill to osd.2 from (109'1600,139'7756] MIN to 139'7756 2024-02-10T07:35:58.537 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:58 smithi008 ceph-mon[40743]: 7.17 continuing backfill to osd.7 from (109'1600,139'7756] MIN to 139'7756 2024-02-10T07:35:58.537 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:58 smithi008 ceph-mon[40743]: 7.10 continuing backfill to osd.2 from (111'2255,139'8107] 7:08083ed8:::benchmark_data_smithi008_82046_object54438:head to 139'8107 2024-02-10T07:35:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:58.570 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:58.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:58.633 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:35:58 smithi170 podman[75481]: 2024-02-10 07:35:58.252090439 +0000 UTC m=+0.018239319 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:35:58.633 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:35:58 smithi170 podman[75481]: 2024-02-10 07:35:58.400552974 +0000 UTC m=+0.166701841 container create cbed3a71e59474c219bf205f1dcd7e1ec3d569a9e2c4428e60bb7e6e73ac0718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, RELEASE=main-290212a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-10T07:35:58.671 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:58.702 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:58.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:58.804 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:35:58.833 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:58.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:58.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:35:58.935 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:58.967 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:58.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:59.069 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:59.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:35:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:35:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:35:59.101 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:35:58 smithi170 podman[75481]: 2024-02-10 07:35:58.696935629 +0000 UTC m=+0.463084516 container init cbed3a71e59474c219bf205f1dcd7e1ec3d569a9e2c4428e60bb7e6e73ac0718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:35:59.102 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:35:58 smithi170 podman[75481]: 2024-02-10 07:35:58.707845316 +0000 UTC m=+0.473994192 container start cbed3a71e59474c219bf205f1dcd7e1ec3d569a9e2c4428e60bb7e6e73ac0718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-290212a, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-02-10T07:35:59.102 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:35:58 smithi170 podman[75481]: 2024-02-10 07:35:58.803754157 +0000 UTC m=+0.569903028 container attach cbed3a71e59474c219bf205f1dcd7e1ec3d569a9e2c4428e60bb7e6e73ac0718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a) 2024-02-10T07:35:59.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:59.132 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:59.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:59.177 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:35:59.223 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:35:59.223 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:35:59.223 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.223 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "rep_scrubmap(7.3 e135 from shard 2)", 2024-02-10T07:35:59.223 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:59.217140+0000", 2024-02-10T07:35:59.223 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0030920209999999999, 2024-02-10T07:35:59.223 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0039088700000000001, 2024-02-10T07:35:59.223 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:35:59.223 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:35:59.223 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.217140+0000", 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.217140+0000", 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.217141+0000", 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5199999999999995e-07 2024-02-10T07:35:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.217166+0000", 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5350999999999998e-05 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.217176+0000", 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0098999999999999e-05 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.217183+0000", 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.759e-06 2024-02-10T07:35:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:35:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:35:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:35:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:128306 7.3 7:c2d78dee:::benchmark_data_smithi008_82046_object13168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:35:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:59.077987+0000", 2024-02-10T07:35:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.142244856, 2024-02-10T07:35:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.14314043400000001, 2024-02-10T07:35:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:35:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:35:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:35:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:35:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:35:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 128306 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.077987+0000", 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.077987+0000", 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:35:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.077989+0000", 2024-02-10T07:35:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5600000000000001e-06 2024-02-10T07:35:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:35:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.077993+0000", 2024-02-10T07:35:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0879999999999997e-06 2024-02-10T07:35:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:35:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.078004+0000", 2024-02-10T07:35:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1165999999999999e-05 2024-02-10T07:35:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:35:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.078011+0000", 2024-02-10T07:35:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7800000000000003e-06 2024-02-10T07:35:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:35:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.191777+0000", 2024-02-10T07:35:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.113765963 2024-02-10T07:35:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:35:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.191903+0000", 2024-02-10T07:35:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012637099999999999 2024-02-10T07:35:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.192333+0000", 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00042961200000000001 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.199431+0000", 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.007098226 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.203397+0000", 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0039662539999999998 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.220107+0000", 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016709699000000001 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.220296+0000", 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00018874500000000001 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop_reply(client.24740.0:128306 7.3 e141/135 ondisk, result = 0)", 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:59.213358+0000", 2024-02-10T07:35:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.006874308, 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0078706090000000006, 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.213358+0000", 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.213358+0000", 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.213358+0000", 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1600000000000002e-07 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.213361+0000", 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7489999999999999e-06 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.213369+0000", 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8469999999999996e-06 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:35:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.213374+0000", 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6879999999999996e-06 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.220093+0000", 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0067191270000000001 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.220101+0000", 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.994e-06 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:128360 7.7 e141/135 7:e5d78ebd:::benchmark_data_smithi008_82046_object13222:head v 141'7780, mlcod=141'7778)", 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:59.213405+0000", 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0068272819999999996, 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0078862689999999996, 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.213405+0000", 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:35:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.213405+0000", 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.213405+0000", 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3000000000000001e-07 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.213410+0000", 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2050000000000001e-06 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:35:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.213419+0000", 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6440000000000006e-06 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.213425+0000", 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.139e-06 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.215281+0000", 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001855853 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.215305+0000", 2024-02-10T07:35:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3983999999999999e-05 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:128359 7.3 7.cfa091c3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:59.214034+0000", 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0061975520000000003, 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0073184219999999998, 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 128359 2024-02-10T07:35:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.214034+0000", 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.214034+0000", 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.214036+0000", 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2669999999999999e-06 2024-02-10T07:35:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.214039+0000", 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4869999999999998e-06 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.214049+0000", 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0128e-05 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.214077+0000", 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8008999999999999e-05 2024-02-10T07:35:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "rep_scrubmap(7.3 e135 from shard 4)", 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:59.214103+0000", 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0061291499999999999, 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0073019570000000004, 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.214103+0000", 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:35:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.214103+0000", 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.214103+0000", 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2600000000000002e-07 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.214115+0000", 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1824999999999999e-05 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:35:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.214139+0000", 2024-02-10T07:35:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3574000000000002e-05 2024-02-10T07:35:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:35:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:35:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:35:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:59.214145+0000", 2024-02-10T07:35:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.156e-06 2024-02-10T07:35:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:35:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:35:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:35:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:35:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:35:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 6 2024-02-10T07:35:59.243 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:35:59.244 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:35:59.272 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:59.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:59.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:35:59.374 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:59.382 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:35:59 smithi170 podman[75481]: 2024-02-10 07:35:59.110537955 +0000 UTC m=+0.876686831 container died cbed3a71e59474c219bf205f1dcd7e1ec3d569a9e2c4428e60bb7e6e73ac0718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, ceph=True) 2024-02-10T07:35:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:59 smithi170 ceph-mon[33394]: pgmap v165: 158 pgs: 1 active+remapped+backfilling, 112 active+clean, 11 active+undersized, 2 active+clean+scrubbing, 14 stale+active+clean, 7 unknown, 8 active+undersized+degraded, 3 active+clean+scrubbing+deep; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 759 B/s rd, 0 op/s; 31/252543 objects degraded (0.012%); 2567/252543 objects misplaced (1.016%); 9.1 MiB/s, 144 objects/s recovering 2024-02-10T07:35:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:35:59 smithi170 ceph-mon[33394]: Health check failed: Degraded data redundancy: 31/252543 objects degraded (0.012%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T07:35:59.414 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:59.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:59.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:59.516 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:35:59.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:59.560 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:59.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:35:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:59 smithi008 ceph-mon[29816]: pgmap v165: 158 pgs: 1 active+remapped+backfilling, 112 active+clean, 11 active+undersized, 2 active+clean+scrubbing, 14 stale+active+clean, 7 unknown, 8 active+undersized+degraded, 3 active+clean+scrubbing+deep; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 759 B/s rd, 0 op/s; 31/252543 objects degraded (0.012%); 2567/252543 objects misplaced (1.016%); 9.1 MiB/s, 144 objects/s recovering 2024-02-10T07:35:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:35:59 smithi008 ceph-mon[29816]: Health check failed: Degraded data redundancy: 31/252543 objects degraded (0.012%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T07:35:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:59 smithi008 ceph-mon[40743]: pgmap v165: 158 pgs: 1 active+remapped+backfilling, 112 active+clean, 11 active+undersized, 2 active+clean+scrubbing, 14 stale+active+clean, 7 unknown, 8 active+undersized+degraded, 3 active+clean+scrubbing+deep; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 759 B/s rd, 0 op/s; 31/252543 objects degraded (0.012%); 2567/252543 objects misplaced (1.016%); 9.1 MiB/s, 144 objects/s recovering 2024-02-10T07:35:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:35:59 smithi008 ceph-mon[40743]: Health check failed: Degraded data redundancy: 31/252543 objects degraded (0.012%), 8 pgs degraded (PG_DEGRADED) 2024-02-10T07:35:59.661 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:35:59.686 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:35:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:59.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:35:59.788 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:35:59.832 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:59.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:59.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:35:59.934 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:35:59.977 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:35:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:35:59.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:00.079 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:00.113 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:00.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:00.215 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:00.262 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:00.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:00.364 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:00.419 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:00.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:00.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:00.476 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:00 smithi008 ceph-mon[29816]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:36:00.476 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:00 smithi008 ceph-mon[40743]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:36:00.521 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:00.551 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:00.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:00.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:00.584 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:36:00.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:00 smithi170 ceph-mon[33394]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:36:00.653 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:00.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:00.719 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:00.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:00.820 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:00.835 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-10T07:36:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:00.894 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:00.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:00.995 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:01.021 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:01.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:01.123 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:01.150 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:01.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:01.252 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.5.service' 2024-02-10T07:36:01.252 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:01.306 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:36:01.306 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-10T07:36:01.307 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-02-10T07:36:01.307 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi170.front.sepia.ceph.com 2024-02-10T07:36:01.307 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -n osd.5 --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-02-10T07:36:01.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:01.318 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:01.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:01.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:01 smithi008 ceph-mon[29816]: pgmap v166: 158 pgs: 4 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 1 active+clean+scrubbing, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 511 B/s rd, 113 MiB/s wr, 2.17k op/s; 25849/307923 objects degraded (8.395%); 55205/307923 objects misplaced (17.928%); 21 MiB/s, 335 objects/s recovering 2024-02-10T07:36:01.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:01 smithi008 ceph-mon[40743]: pgmap v166: 158 pgs: 4 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 1 active+clean+scrubbing, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 511 B/s rd, 113 MiB/s wr, 2.17k op/s; 25849/307923 objects degraded (8.395%); 55205/307923 objects misplaced (17.928%); 21 MiB/s, 335 objects/s recovering 2024-02-10T07:36:01.420 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:01.471 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:01.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:01.569 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/osd.5/config 2024-02-10T07:36:01.573 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:01.620 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:01.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:01.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:01 smithi170 ceph-mon[33394]: pgmap v166: 158 pgs: 4 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 1 active+clean+scrubbing, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 511 B/s rd, 113 MiB/s wr, 2.17k op/s; 25849/307923 objects degraded (8.395%); 55205/307923 objects misplaced (17.928%); 21 MiB/s, 335 objects/s recovering 2024-02-10T07:36:01.721 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:01.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:01.765 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:01.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:01.833 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:36:01.867 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:01.878 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:36:01.895 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:01.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:01.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:01.997 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:02.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:02.032 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:02.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:02.133 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:02.164 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:02.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:02.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:02.266 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:02.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:02.312 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:02.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:02.413 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:02.457 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:02.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:02.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:02.559 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:02.607 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:02.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:02.709 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:02.755 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:02.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:02.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:02.857 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:02.886 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:02.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:02.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:02.988 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:03.032 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:03.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:03.053 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:36:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:36:03.134 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:03.163 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:03.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:03.265 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:03.310 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:03.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:03.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:03.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:03 smithi008 ceph-mon[29816]: pgmap v167: 158 pgs: 4 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 1 active+clean+scrubbing, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 436 B/s rd, 96 MiB/s wr, 1.85k op/s; 25849/307923 objects degraded (8.395%); 55205/307923 objects misplaced (17.928%); 18 MiB/s, 286 objects/s recovering 2024-02-10T07:36:03.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:03.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:03.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:03 smithi008 ceph-mon[40743]: pgmap v167: 158 pgs: 4 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 1 active+clean+scrubbing, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 436 B/s rd, 96 MiB/s wr, 1.85k op/s; 25849/307923 objects degraded (8.395%); 55205/307923 objects misplaced (17.928%); 18 MiB/s, 286 objects/s recovering 2024-02-10T07:36:03.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:03.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:03.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:03 smithi170 ceph-mon[33394]: pgmap v167: 158 pgs: 4 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 1 active+clean+scrubbing, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 436 B/s rd, 96 MiB/s wr, 1.85k op/s; 25849/307923 objects degraded (8.395%); 55205/307923 objects misplaced (17.928%); 18 MiB/s, 286 objects/s recovering 2024-02-10T07:36:03.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:03.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:03.412 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:03.442 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:03.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:03.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:03.544 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:03.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:03.575 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:03.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:03.676 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:03.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:03.717 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:03.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:03.818 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:03.852 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:03.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:03.890 INFO:teuthology.orchestra.run.smithi170.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-10T07:36:03.954 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:04.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:04.003 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:04.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:04.104 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:04.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:04 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:04.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:04.148 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:04.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:04.250 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:04.283 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:04.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:04.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:04.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:04 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:04.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:04 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:04.385 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:04.419 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:04.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:04.521 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:04.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:04.569 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:04.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:04.671 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:04.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:04.716 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:04.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:04.817 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:04.859 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:04.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:04.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:04.961 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:05.001 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:05.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:05.103 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:05.135 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:05.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:05.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:05.158 INFO:teuthology.orchestra.run.smithi170.stdout:7.e 2024-02-10T07:36:05.159 INFO:teuthology.orchestra.run.smithi170.stdout:7.b 2024-02-10T07:36:05.159 INFO:teuthology.orchestra.run.smithi170.stdout:6.d 2024-02-10T07:36:05.159 INFO:teuthology.orchestra.run.smithi170.stdout:6.c 2024-02-10T07:36:05.159 INFO:teuthology.orchestra.run.smithi170.stdout:6.4 2024-02-10T07:36:05.159 INFO:teuthology.orchestra.run.smithi170.stdout:6.1e 2024-02-10T07:36:05.159 INFO:teuthology.orchestra.run.smithi170.stdout:6.1c 2024-02-10T07:36:05.159 INFO:teuthology.orchestra.run.smithi170.stdout:7.17 2024-02-10T07:36:05.159 INFO:teuthology.orchestra.run.smithi170.stdout:6.1a 2024-02-10T07:36:05.159 INFO:teuthology.orchestra.run.smithi170.stdout:4.1f 2024-02-10T07:36:05.159 INFO:teuthology.orchestra.run.smithi170.stdout:4.1 2024-02-10T07:36:05.159 INFO:teuthology.orchestra.run.smithi170.stdout:6.1d 2024-02-10T07:36:05.159 INFO:teuthology.orchestra.run.smithi170.stdout:5.d 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:4.14 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:3.d 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:6.1f 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:3.4 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:7.a 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:4.15 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:3.1c 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:3.c 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:4.8 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:3.2 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:5.16 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:3.1f 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:5.7 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:3.5 2024-02-10T07:36:05.160 INFO:teuthology.orchestra.run.smithi170.stdout:3.10 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:7.14 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:2.2 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:3.16 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:3.17 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:7.10 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:5.1b 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:3.1d 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:6.a 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:5.2 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:4.3 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:7.0 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:4.7 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:5.12 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:5.15 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:5.3 2024-02-10T07:36:05.161 INFO:teuthology.orchestra.run.smithi170.stdout:5.19 2024-02-10T07:36:05.162 INFO:teuthology.orchestra.run.smithi170.stdout:6.19 2024-02-10T07:36:05.162 INFO:teuthology.orchestra.run.smithi170.stdout:5.4 2024-02-10T07:36:05.162 INFO:teuthology.orchestra.run.smithi170.stdout:4.2 2024-02-10T07:36:05.162 INFO:teuthology.orchestra.run.smithi170.stdout:5.6 2024-02-10T07:36:05.162 INFO:teuthology.orchestra.run.smithi170.stdout:5.b 2024-02-10T07:36:05.162 INFO:teuthology.orchestra.run.smithi170.stdout:5.f 2024-02-10T07:36:05.162 INFO:teuthology.orchestra.run.smithi170.stdout:5.e 2024-02-10T07:36:05.162 INFO:teuthology.orchestra.run.smithi170.stdout:6.17 2024-02-10T07:36:05.162 INFO:teuthology.orchestra.run.smithi170.stdout:6.10 2024-02-10T07:36:05.162 INFO:teuthology.orchestra.run.smithi170.stdout:6.7 2024-02-10T07:36:05.163 INFO:teuthology.orchestra.run.smithi170.stdout:6.11 2024-02-10T07:36:05.236 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:05.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:05.280 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:05.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:05.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:05 smithi008 ceph-mon[29816]: pgmap v168: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.96k op/s; 25132/299988 objects degraded (8.378%); 50011/299988 objects misplaced (16.671%); 41 MiB/s, 650 objects/s recovering 2024-02-10T07:36:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:05 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 25132/299988 objects degraded (8.378%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:05.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:05 smithi008 ceph-mon[40743]: pgmap v168: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.96k op/s; 25132/299988 objects degraded (8.378%); 50011/299988 objects misplaced (16.671%); 41 MiB/s, 650 objects/s recovering 2024-02-10T07:36:05.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:05 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 25132/299988 objects degraded (8.378%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:05.381 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:05 smithi170 ceph-mon[33394]: pgmap v168: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.96k op/s; 25132/299988 objects degraded (8.378%); 50011/299988 objects misplaced (16.671%); 41 MiB/s, 650 objects/s recovering 2024-02-10T07:36:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:05 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 25132/299988 objects degraded (8.378%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:05.427 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:05.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:05.528 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:05.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:05.575 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:05.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:05.676 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:05.701 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:05.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:05.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:05.803 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:05.844 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:05.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:05.946 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:05.984 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:05.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:06.085 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:06.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:06.132 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:06.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:06.233 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:06.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:06.266 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:06.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:06.367 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:06.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:06 smithi170 ceph-mon[33394]: 7.3 deep-scrub ok 2024-02-10T07:36:06.397 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:06.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:06.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:06.486 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -n osd.5 --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 5.7 --file /var/log/ceph/exp.5.7.5 2024-02-10T07:36:06.500 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:06.542 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:06.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:06.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:06 smithi008 ceph-mon[29816]: 7.3 deep-scrub ok 2024-02-10T07:36:06.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:06 smithi008 ceph-mon[40743]: 7.3 deep-scrub ok 2024-02-10T07:36:06.643 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:06.686 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:06.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:06.727 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/osd.5/config 2024-02-10T07:36:06.788 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:06.817 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:06.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:06.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:06.919 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:06.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:06.943 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:06.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:07.044 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:07.079 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:07.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:07.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:07.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:07 smithi170 ceph-mon[33394]: 7.4 scrub starts 2024-02-10T07:36:07.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:07 smithi170 ceph-mon[33394]: pgmap v169: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 847 B/s rd, 70 MiB/s wr, 1.62k op/s; 25132/299988 objects degraded (8.378%); 50011/299988 objects misplaced (16.671%); 34 MiB/s, 537 objects/s recovering 2024-02-10T07:36:07.180 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:07.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:07.211 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:07.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:07.312 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:07.355 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:07.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:07.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:07.485 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:07.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:07.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:07.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:07 smithi008 ceph-mon[29816]: 7.4 scrub starts 2024-02-10T07:36:07.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:07 smithi008 ceph-mon[29816]: pgmap v169: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 847 B/s rd, 70 MiB/s wr, 1.62k op/s; 25132/299988 objects degraded (8.378%); 50011/299988 objects misplaced (16.671%); 34 MiB/s, 537 objects/s recovering 2024-02-10T07:36:07.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:07 smithi008 ceph-mon[40743]: 7.4 scrub starts 2024-02-10T07:36:07.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:07 smithi008 ceph-mon[40743]: pgmap v169: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 847 B/s rd, 70 MiB/s wr, 1.62k op/s; 25132/299988 objects degraded (8.378%); 50011/299988 objects misplaced (16.671%); 34 MiB/s, 537 objects/s recovering 2024-02-10T07:36:07.587 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:07.611 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:07.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:07.713 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:07.743 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:07.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:07.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:07.845 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:07.873 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:07.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:07.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:07.976 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:08.002 INFO:teuthology.orchestra.run.smithi170.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-10T07:36:08.013 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:08.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:08.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:08.115 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:08.140 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:08.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:08.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:08 smithi170 ceph-mon[33394]: 7.8 deep-scrub ok 2024-02-10T07:36:08.231 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:08 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:36:08.231 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:08 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:36:08.240 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:08.272 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:08.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:08.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:08.374 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:08.403 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:08.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:08.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:08.506 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:08.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:08.545 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:08.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:08.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:08 smithi008 ceph-mon[29816]: 7.8 deep-scrub ok 2024-02-10T07:36:08.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:08 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:36:08.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:08 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:36:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:08 smithi008 ceph-mon[40743]: 7.8 deep-scrub ok 2024-02-10T07:36:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:08 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:36:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:08 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:36:08.646 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:08.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:08.679 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:08.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:08.696 INFO:teuthology.orchestra.run.smithi170.stderr:Exporting 5.7 info 5.7( empty local-lis/les=70/71 n=0 ec=70/70 lis/c=70/70 les/c/f=71/71/0 sis=70) 2024-02-10T07:36:08.696 INFO:teuthology.orchestra.run.smithi170.stderr:Export successful 2024-02-10T07:36:08.779 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:08.829 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:08.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:08.930 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:08.961 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:08.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:08.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:09.063 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:09.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:36:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:36:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:36:09.090 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:09.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:09.193 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:09.238 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:09.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:09.339 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:09.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:09.368 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:09.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:09 smithi170 ceph-mon[33394]: pgmap v170: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.53k op/s; 25132/297855 objects degraded (8.438%); 49085/297855 objects misplaced (16.479%); 36 MiB/s, 570 objects/s recovering 2024-02-10T07:36:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:09 smithi170 ceph-mon[33394]: 7.9 deep-scrub ok 2024-02-10T07:36:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:09 smithi170 ceph-mon[33394]: osdmap e142: 8 total, 7 up, 8 in 2024-02-10T07:36:09.470 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:09.506 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:09.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:09.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:09.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:09 smithi008 ceph-mon[29816]: pgmap v170: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.53k op/s; 25132/297855 objects degraded (8.438%); 49085/297855 objects misplaced (16.479%); 36 MiB/s, 570 objects/s recovering 2024-02-10T07:36:09.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:09 smithi008 ceph-mon[29816]: 7.9 deep-scrub ok 2024-02-10T07:36:09.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:09 smithi008 ceph-mon[29816]: osdmap e142: 8 total, 7 up, 8 in 2024-02-10T07:36:09.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:09 smithi008 ceph-mon[40743]: pgmap v170: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+clean+scrubbing, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 92 active+clean, 31 active+undersized, 21 active+undersized+degraded, 3 active+clean+scrubbing+deep; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.53k op/s; 25132/297855 objects degraded (8.438%); 49085/297855 objects misplaced (16.479%); 36 MiB/s, 570 objects/s recovering 2024-02-10T07:36:09.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:09 smithi008 ceph-mon[40743]: 7.9 deep-scrub ok 2024-02-10T07:36:09.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:09 smithi008 ceph-mon[40743]: osdmap e142: 8 total, 7 up, 8 in 2024-02-10T07:36:09.608 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:09.649 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:09.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:09.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:09.751 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:09.790 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:09.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:09.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:09.892 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:09.922 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:09.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:09.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:10.024 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:10.050 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:10.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:10.151 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:10.182 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:10.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:10.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:10.284 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:10.328 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:10.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:10.347 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -n osd.5 --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 5.7 2024-02-10T07:36:10.430 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:10.455 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:10 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 24305/291750 objects degraded (8.331%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:10.457 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:10 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 24305/291750 objects degraded (8.331%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:10.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:10.481 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:10.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:10.581 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/osd.5/config 2024-02-10T07:36:10.581 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:10.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:10.626 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:10.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:10.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:10 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 24305/291750 objects degraded (8.331%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:10.727 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:10.755 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:10.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:10.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:10.857 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:10.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:10.883 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:10.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:10.984 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:11.026 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:11.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:11.127 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:11.157 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:11.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:11.259 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:11.306 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:11.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:11.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:11 smithi170 ceph-mon[33394]: pgmap v172: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 96 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 24305/291750 objects degraded (8.331%); 44350/291750 objects misplaced (15.201%); 42 MiB/s, 679 objects/s recovering 2024-02-10T07:36:11.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:11 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1960884384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:36:11.407 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:11.447 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:11.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:11.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:11.502 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:36:11.502 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:36:11.502 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:36:11.503 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:36:11.503 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:36:11.503 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:36:11.503 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:36:11.503 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:36:11.503 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:36:11.503 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:36:11.503 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:36:11.503 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:36:11.503 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-02-10T07:36:11.503 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:36:11.503 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:36:11.504 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:11 smithi008 ceph-mon[29816]: pgmap v172: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 96 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 24305/291750 objects degraded (8.331%); 44350/291750 objects misplaced (15.201%); 42 MiB/s, 679 objects/s recovering 2024-02-10T07:36:11.504 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:11 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1960884384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:36:11.504 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:11 smithi008 ceph-mon[40743]: pgmap v172: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 96 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 24305/291750 objects degraded (8.331%); 44350/291750 objects misplaced (15.201%); 42 MiB/s, 679 objects/s recovering 2024-02-10T07:36:11.504 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:11 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1960884384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:36:11.549 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:11.574 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:11.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:11.675 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:11.708 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:11.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:11.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:11.810 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:11.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:11.851 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:11.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:11.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:11.979 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:11.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:11.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:12.081 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:12.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:12.117 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:12.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:12.157 INFO:teuthology.orchestra.run.smithi170.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-10T07:36:12.218 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:12.256 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:12.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:12.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:12.358 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:12.392 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:12.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:12.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:12.494 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:12.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:12.522 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:12.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:12.623 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:12.656 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:12.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:12.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:12.758 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:12.789 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:12.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:12.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:12.871 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:12 smithi170 ceph-mon[33394]: osdmap e143: 8 total, 7 up, 8 in 2024-02-10T07:36:12.871 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:12 smithi170 ceph-mon[33394]: pgmap v174: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 96 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 24305/291750 objects degraded (8.331%); 44350/291750 objects misplaced (15.201%); 28 MiB/s, 451 objects/s recovering 2024-02-10T07:36:12.891 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:12.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:12.935 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:12.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:13.036 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:13.056 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:12 smithi008 ceph-mon[40743]: osdmap e143: 8 total, 7 up, 8 in 2024-02-10T07:36:13.056 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:12 smithi008 ceph-mon[40743]: pgmap v174: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 96 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 24305/291750 objects degraded (8.331%); 44350/291750 objects misplaced (15.201%); 28 MiB/s, 451 objects/s recovering 2024-02-10T07:36:13.056 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:12 smithi008 ceph-mon[29816]: osdmap e143: 8 total, 7 up, 8 in 2024-02-10T07:36:13.056 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:12 smithi008 ceph-mon[29816]: pgmap v174: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 96 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 24305/291750 objects degraded (8.331%); 44350/291750 objects misplaced (15.201%); 28 MiB/s, 451 objects/s recovering 2024-02-10T07:36:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:13.082 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:13.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:13.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:36:12 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:36:13.182 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:13.211 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:13.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:13.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:13.313 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:13.317 INFO:teuthology.orchestra.run.smithi170.stdout: marking collection for removal 2024-02-10T07:36:13.318 INFO:teuthology.orchestra.run.smithi170.stdout:setting '_remove' omap key 2024-02-10T07:36:13.318 INFO:teuthology.orchestra.run.smithi170.stdout:finish_remove_pgs 5.7_head removing 5.7 2024-02-10T07:36:13.318 INFO:teuthology.orchestra.run.smithi170.stdout:Remove successful 2024-02-10T07:36:13.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:13.342 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:13.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:13.443 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:13.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:13.469 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:13.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:36:13.566 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:36:13.570 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:13.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:13.602 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:13.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:13.703 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:13.734 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:13.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:13.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:13 smithi008 ceph-mon[29816]: osdmap e144: 8 total, 7 up, 8 in 2024-02-10T07:36:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:13 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:13.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:13 smithi008 ceph-mon[40743]: osdmap e144: 8 total, 7 up, 8 in 2024-02-10T07:36:13.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:13 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:13.835 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:13.873 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:13.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:13.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:13.975 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:14.016 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:14.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:14.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:14.118 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:14.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:13 smithi170 ceph-mon[33394]: osdmap e144: 8 total, 7 up, 8 in 2024-02-10T07:36:14.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:13 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:14.157 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:14.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:14.258 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:14.293 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:14.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:14.395 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:14.430 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:14.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:14.532 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:14.557 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:14.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:14.609 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -n osd.5 --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.7.5 2024-02-10T07:36:14.659 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:14.684 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:14.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:14.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:14.778 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:36:14.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:14.818 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:14.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:14.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:14.820 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/osd.5/config 2024-02-10T07:36:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:14 smithi170 ceph-mon[33394]: pgmap v176: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 96 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 22638/280644 objects degraded (8.066%); 38607/280644 objects misplaced (13.757%); 67 MiB/s, 1.07k objects/s recovering 2024-02-10T07:36:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:14 smithi170 ceph-mon[33394]: osdmap e145: 8 total, 7 up, 8 in 2024-02-10T07:36:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:14 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3420649079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:36:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:14 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:36:14.920 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:14.948 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:14.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:15.050 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:15.076 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:15.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:15.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:15.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:14 smithi008 ceph-mon[29816]: pgmap v176: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 96 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 22638/280644 objects degraded (8.066%); 38607/280644 objects misplaced (13.757%); 67 MiB/s, 1.07k objects/s recovering 2024-02-10T07:36:15.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:14 smithi008 ceph-mon[29816]: osdmap e145: 8 total, 7 up, 8 in 2024-02-10T07:36:15.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:14 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3420649079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:36:15.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:14 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:36:15.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:14 smithi008 ceph-mon[40743]: pgmap v176: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 96 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 22638/280644 objects degraded (8.066%); 38607/280644 objects misplaced (13.757%); 67 MiB/s, 1.07k objects/s recovering 2024-02-10T07:36:15.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:14 smithi008 ceph-mon[40743]: osdmap e145: 8 total, 7 up, 8 in 2024-02-10T07:36:15.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:14 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3420649079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:36:15.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:14 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:36:15.178 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:15.201 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:15.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:15.303 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:15.349 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:15.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:15.450 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:15.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:15.493 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:15.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:15.567 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:36:15.594 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:15.671 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:15.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:15.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:15.773 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:15.801 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:15.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:15.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:15.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:15 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:36:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:15 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:36:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:15 smithi170 ceph-mon[33394]: osdmap e146: 8 total, 7 up, 8 in 2024-02-10T07:36:15.904 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:15.932 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:15.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:15.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:16.034 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:16.062 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:16.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:16.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:16.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:15 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:36:16.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:15 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:36:16.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:15 smithi008 ceph-mon[29816]: osdmap e146: 8 total, 7 up, 8 in 2024-02-10T07:36:16.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:15 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:36:16.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:15 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:36:16.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:15 smithi008 ceph-mon[40743]: osdmap e146: 8 total, 7 up, 8 in 2024-02-10T07:36:16.165 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:16.193 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:16.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:16.295 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:16.323 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:16.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:16.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:16.426 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:16.469 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:16.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:16.570 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:16.596 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:16.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:16.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:16.697 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:16.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:16.751 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:16.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:16.852 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:16.898 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:16.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:16.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:17.000 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:17.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:17.041 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:17.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:17.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:16 smithi008 ceph-mon[29816]: pgmap v179: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 96 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 914 op/s; 22638/280644 objects degraded (8.066%); 38607/280644 objects misplaced (13.757%); 57 MiB/s, 910 objects/s recovering 2024-02-10T07:36:17.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:16 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 22638/280644 objects degraded (8.066%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:17.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:16 smithi008 ceph-mon[29816]: osdmap e147: 8 total, 7 up, 8 in 2024-02-10T07:36:17.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:16 smithi008 ceph-mon[40743]: pgmap v179: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 96 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 914 op/s; 22638/280644 objects degraded (8.066%); 38607/280644 objects misplaced (13.757%); 57 MiB/s, 910 objects/s recovering 2024-02-10T07:36:17.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:16 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 22638/280644 objects degraded (8.066%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:17.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:16 smithi008 ceph-mon[40743]: osdmap e147: 8 total, 7 up, 8 in 2024-02-10T07:36:17.099 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:16 smithi170 ceph-mon[33394]: pgmap v179: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 96 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 914 op/s; 22638/280644 objects degraded (8.066%); 38607/280644 objects misplaced (13.757%); 57 MiB/s, 910 objects/s recovering 2024-02-10T07:36:17.100 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:16 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 22638/280644 objects degraded (8.066%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:17.100 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:16 smithi170 ceph-mon[33394]: osdmap e147: 8 total, 7 up, 8 in 2024-02-10T07:36:17.141 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:17.149 INFO:teuthology.orchestra.run.smithi170.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-10T07:36:17.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:17.184 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:17.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:17.285 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:17.338 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:17.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:17.440 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:17.486 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:17.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:17.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:17.588 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:17.594 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:36:17.594 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:36:17.620 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:17.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:17.647 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:36:17.722 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:17.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:17.777 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:17.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:17.878 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:17.909 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:17.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:17.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:17.946 INFO:teuthology.orchestra.run.smithi170.stdout:get_pg_num_history pg_num_history pg_num_history(e139 pg_nums {1={19=1},2={65=3},3={66=32},4={68=32},5={70=32},6={72=32},7={98=16,135=26}} deleted_pools ) 2024-02-10T07:36:17.946 INFO:teuthology.orchestra.run.smithi170.stdout:Importing pgid 5.7 2024-02-10T07:36:17.946 INFO:teuthology.orchestra.run.smithi170.stdout:write_pg epoch 129 info 5.7( empty local-lis/les=70/71 n=0 ec=70/70 lis/c=70/70 les/c/f=71/71/0 sis=70) 2024-02-10T07:36:17.946 INFO:teuthology.orchestra.run.smithi170.stdout:Import successful 2024-02-10T07:36:18.012 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:18.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:18.069 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:18.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:18.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:17 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:18.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:17 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:18.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:17 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:18.170 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:18.217 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:18.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:18.319 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:18.366 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:18.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:18.468 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:18.492 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:18.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:18.593 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:18.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:18.626 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:18.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:18.727 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:18.778 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:18.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:18.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:18.880 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:18.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:18.937 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:18.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:19.018 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:18 smithi008 ceph-mon[29816]: pgmap v181: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1 KiB/s rd, 0 B/s wr, 826 op/s; 22638/276510 objects degraded (8.187%); 37765/276510 objects misplaced (13.658%); 47 MiB/s, 753 objects/s recovering 2024-02-10T07:36:19.018 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:36:19.019 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:36:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:36:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:36:19.019 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:18 smithi008 ceph-mon[40743]: pgmap v181: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1 KiB/s rd, 0 B/s wr, 826 op/s; 22638/276510 objects degraded (8.187%); 37765/276510 objects misplaced (13.658%); 47 MiB/s, 753 objects/s recovering 2024-02-10T07:36:19.019 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:36:19.038 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:19.080 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-10T07:36:19.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:19.111 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:19.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:19.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:18 smithi170 ceph-mon[33394]: pgmap v181: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1 KiB/s rd, 0 B/s wr, 826 op/s; 22638/276510 objects degraded (8.187%); 37765/276510 objects misplaced (13.658%); 47 MiB/s, 753 objects/s recovering 2024-02-10T07:36:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:36:19.152 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:36:19.165 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:36:19.212 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:19.285 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/exp.5.7.5 2024-02-10T07:36:19.301 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:19.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:19.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:19.402 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:19.459 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:19.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:19.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:19.561 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:19.610 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:19.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:19.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:19.711 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:19.770 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:19.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:19.871 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:19.923 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:19.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:20.025 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:20.065 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:20.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:20.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:20.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:20.198 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:20.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:20.299 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:20.333 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:20.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:20.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:20.436 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:20.470 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:20.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:20.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:20.572 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:20.616 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:20.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:20.719 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:20.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:20.745 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:20.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:20.846 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:20.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:20.886 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:20.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:20.987 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:21.030 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:21.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:21.087 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:20 smithi008 ceph-mon[29816]: pgmap v182: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 695 op/s; 20696/268134 objects degraded (7.719%); 32799/268134 objects misplaced (12.232%); 37 MiB/s, 593 objects/s recovering 2024-02-10T07:36:21.088 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:20 smithi008 ceph-mon[40743]: pgmap v182: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 695 op/s; 20696/268134 objects degraded (7.719%); 32799/268134 objects misplaced (12.232%); 37 MiB/s, 593 objects/s recovering 2024-02-10T07:36:21.131 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:21.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:20 smithi170 ceph-mon[33394]: pgmap v182: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 695 op/s; 20696/268134 objects degraded (7.719%); 32799/268134 objects misplaced (12.232%); 37 MiB/s, 593 objects/s recovering 2024-02-10T07:36:21.161 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:21.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:21.263 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:21.290 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:21.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:21.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:21.392 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:21.420 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:21.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:21.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:21.522 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:21.571 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:21.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:21.673 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:21.720 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:21.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:21.821 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:21.860 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:21.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:21.962 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:21.963 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:21 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 20696/268134 objects degraded (7.719%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:21.963 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:21 smithi008 ceph-mon[29816]: osdmap e148: 8 total, 7 up, 8 in 2024-02-10T07:36:21.964 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:21 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 20696/268134 objects degraded (7.719%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:21.964 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:21 smithi008 ceph-mon[40743]: osdmap e148: 8 total, 7 up, 8 in 2024-02-10T07:36:22.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:22.021 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:22.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:22.122 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:22.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:21 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 20696/268134 objects degraded (7.719%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:21 smithi170 ceph-mon[33394]: osdmap e148: 8 total, 7 up, 8 in 2024-02-10T07:36:22.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:22.166 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:22.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:22.267 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:22.293 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:22.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:22.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:22.312 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:36:22.395 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:22.451 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:22.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:22.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:22.553 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:22.597 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:22.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:22.698 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:22.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:22.727 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:22.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:22.828 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:22.866 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:22 smithi170 ceph-mon[33394]: pgmap v184: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 597 op/s; 20696/268134 objects degraded (7.719%); 32799/268134 objects misplaced (12.232%); 32 MiB/s, 509 objects/s recovering 2024-02-10T07:36:22.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:22.872 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:22.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:22.976 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:23.007 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:22 smithi008 ceph-mon[29816]: pgmap v184: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 597 op/s; 20696/268134 objects degraded (7.719%); 32799/268134 objects misplaced (12.232%); 32 MiB/s, 509 objects/s recovering 2024-02-10T07:36:23.007 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:22 smithi008 ceph-mon[40743]: pgmap v184: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 732 B/s rd, 0 B/s wr, 597 op/s; 20696/268134 objects degraded (7.719%); 32799/268134 objects misplaced (12.232%); 32 MiB/s, 509 objects/s recovering 2024-02-10T07:36:23.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:23.024 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:23.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:23.125 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:23.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:36:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:36:23.155 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:23.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:23.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:23.257 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:23.295 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:23.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:23.397 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:23.427 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:23.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:23.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:23.529 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:23.553 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:23.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:23.655 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:23.682 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:23.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:23.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:23.784 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:23.809 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:23.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:23.910 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:23.954 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:23.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:24.056 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:24.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:23 smithi008 ceph-mon[29816]: osdmap e149: 8 total, 7 up, 8 in 2024-02-10T07:36:24.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:23 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:24.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:23 smithi008 ceph-mon[40743]: osdmap e149: 8 total, 7 up, 8 in 2024-02-10T07:36:24.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:23 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:24.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:24.093 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:24.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:24.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:23 smithi170 ceph-mon[33394]: osdmap e149: 8 total, 7 up, 8 in 2024-02-10T07:36:24.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:23 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:24.193 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:24.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:24.223 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:24.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:24.323 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-02-10T07:36:24.323 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-02-10T07:36:24.323 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-02-10T07:36:24.324 DEBUG:teuthology.orchestra.run.smithi170:osd.5> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.5.service 2024-02-10T07:36:24.324 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:24.326 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl start ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.5 2024-02-10T07:36:24.351 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:24.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:24.452 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:24.478 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:24.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:24.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:24.501 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:24 smithi170 systemd[1]: Starting Ceph osd.5 for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:36:24.580 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:24.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:24.606 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:24.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:24.707 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:24.731 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:24.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:24.833 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:24.844 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:36:24.865 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:24.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:24.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:24.967 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:24.997 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:24.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:25.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:25.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:25.134 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:25.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:25.135 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:24 smithi170 podman[79514]: 2024-02-10 07:36:24.874449303 +0000 UTC m=+0.026331085 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:36:25.135 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 podman[79514]: 2024-02-10 07:36:25.003633639 +0000 UTC m=+0.155515387 container create cbbd2a12be70e8d4abc7b5523f58746e613963f60a90803b0ae58d0fddd178ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-10T07:36:25.136 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:24 smithi170 ceph-mon[33394]: pgmap v186: 158 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.35k op/s; 18942/252093 objects degraded (7.514%); 29126/252093 objects misplaced (11.554%); 44 MiB/s, 699 objects/s recovering 2024-02-10T07:36:25.136 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:24 smithi170 ceph-mon[33394]: osdmap e150: 8 total, 7 up, 8 in 2024-02-10T07:36:25.136 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:24 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:36:25.136 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:24 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:36:25.136 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:24 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2996036713' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:36:25.137 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:24 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:36:25.137 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:24 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:25.137 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:24 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:25.236 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:25.271 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:25.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:25.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:25.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[29816]: pgmap v186: 158 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.35k op/s; 18942/252093 objects degraded (7.514%); 29126/252093 objects misplaced (11.554%); 44 MiB/s, 699 objects/s recovering 2024-02-10T07:36:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[29816]: osdmap e150: 8 total, 7 up, 8 in 2024-02-10T07:36:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:36:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:36:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2996036713' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:36:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:36:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[40743]: pgmap v186: 158 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.35k op/s; 18942/252093 objects degraded (7.514%); 29126/252093 objects misplaced (11.554%); 44 MiB/s, 699 objects/s recovering 2024-02-10T07:36:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[40743]: osdmap e150: 8 total, 7 up, 8 in 2024-02-10T07:36:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:36:25.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:36:25.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2996036713' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:36:25.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:36:25.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:25.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:24 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:25.372 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:25.399 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:25.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:25.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:25.502 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:25.527 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:25.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:25.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:25.629 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:25.633 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 podman[79514]: 2024-02-10 07:36:25.273874453 +0000 UTC m=+0.425756221 container init cbbd2a12be70e8d4abc7b5523f58746e613963f60a90803b0ae58d0fddd178ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-activate, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131) 2024-02-10T07:36:25.633 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 podman[79514]: 2024-02-10 07:36:25.278412305 +0000 UTC m=+0.430294061 container start cbbd2a12be70e8d4abc7b5523f58746e613963f60a90803b0ae58d0fddd178ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-10T07:36:25.633 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 podman[79514]: 2024-02-10 07:36:25.390661466 +0000 UTC m=+0.542543236 container attach cbbd2a12be70e8d4abc7b5523f58746e613963f60a90803b0ae58d0fddd178ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-activate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:36:25.655 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:25.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:25.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:25.757 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:25.786 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:25.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:25.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:25.872 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:36:25.872 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:36:25.872 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[29816]: osdmap e151: 8 total, 7 up, 8 in 2024-02-10T07:36:25.872 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[29816]: 6.13 scrub starts 2024-02-10T07:36:25.872 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[29816]: 6.13 scrub ok 2024-02-10T07:36:25.873 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[29816]: 5.1e scrub starts 2024-02-10T07:36:25.873 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[29816]: 5.1e scrub ok 2024-02-10T07:36:25.873 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[29816]: osdmap e152: 8 total, 7 up, 8 in 2024-02-10T07:36:25.873 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:36:25.873 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:36:25.873 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[40743]: osdmap e151: 8 total, 7 up, 8 in 2024-02-10T07:36:25.873 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[40743]: 6.13 scrub starts 2024-02-10T07:36:25.873 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[40743]: 6.13 scrub ok 2024-02-10T07:36:25.873 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[40743]: 5.1e scrub starts 2024-02-10T07:36:25.873 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[40743]: 5.1e scrub ok 2024-02-10T07:36:25.873 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:25 smithi008 ceph-mon[40743]: osdmap e152: 8 total, 7 up, 8 in 2024-02-10T07:36:25.888 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:25.945 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:25.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:25.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:26.047 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:26.077 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:26.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:26.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:36:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:36:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-mon[33394]: osdmap e151: 8 total, 7 up, 8 in 2024-02-10T07:36:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-mon[33394]: 6.13 scrub starts 2024-02-10T07:36:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-mon[33394]: 6.13 scrub ok 2024-02-10T07:36:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-mon[33394]: 5.1e scrub starts 2024-02-10T07:36:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-mon[33394]: 5.1e scrub ok 2024-02-10T07:36:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-mon[33394]: osdmap e152: 8 total, 7 up, 8 in 2024-02-10T07:36:26.134 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-activate[79561]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T07:36:26.134 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-activate[79561]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T07:36:26.134 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 bash[79514]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T07:36:26.134 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 bash[79514]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-10T07:36:26.134 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 bash[79514]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T07:36:26.134 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 bash[79514]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T07:36:26.134 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 bash[79514]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-10T07:36:26.134 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 bash[79514]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T07:36:26.134 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 bash[79514]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-10T07:36:26.134 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-activate[79561]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-10T07:36:26.134 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-activate[79561]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-10T07:36:26.134 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-activate[79561]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-10T07:36:26.135 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-activate[79561]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-10T07:36:26.135 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-activate[79561]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-10T07:36:26.135 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:25 smithi170 podman[79514]: 2024-02-10 07:36:25.820719738 +0000 UTC m=+0.972601501 container died cbbd2a12be70e8d4abc7b5523f58746e613963f60a90803b0ae58d0fddd178ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-activate, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-10T07:36:26.178 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:26.222 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:26.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:26.275 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:36:26.319 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:36:26.323 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:26.349 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:26.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:26.363 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:36:26.451 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:26.476 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:26.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:26.492 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:36:26.530 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:36:26.572 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:36:26.577 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:26.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:26.608 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:26.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:26.616 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:36:26.617 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:36:26.617 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:36:26.617 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:36:26.617 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:36:26.617 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:36:26.617 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-02-10T07:36:26.617 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:36:26.617 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:36:26.710 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:26.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:26.757 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:26.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:26.851 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:26 smithi170 podman[79745]: 2024-02-10 07:36:26.588260592 +0000 UTC m=+0.755233123 container remove cbbd2a12be70e8d4abc7b5523f58746e613963f60a90803b0ae58d0fddd178ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , RELEASE=main-290212a) 2024-02-10T07:36:26.851 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:26 smithi170 podman[79853]: 2024-02-10 07:36:26.736002132 +0000 UTC m=+0.027379420 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:36:26.857 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:26.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:26.882 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:26.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:26.983 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:27.029 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:27.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:27.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:27.102 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:26 smithi170 podman[79853]: 2024-02-10 07:36:26.872954191 +0000 UTC m=+0.164331460 container create f398c377f85d658abc6c0040e1b95c12700080ea3a4f7a675b374d41d5b0e155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-10T07:36:27.102 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:27 smithi170 podman[79853]: 2024-02-10 07:36:27.092881331 +0000 UTC m=+0.384258605 container init f398c377f85d658abc6c0040e1b95c12700080ea3a4f7a675b374d41d5b0e155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2) 2024-02-10T07:36:27.102 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:26 smithi170 ceph-mon[33394]: 7.11 scrub starts 2024-02-10T07:36:27.102 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:26 smithi170 ceph-mon[33394]: pgmap v189: 158 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 18942/252093 objects degraded (7.514%); 29126/252093 objects misplaced (11.554%); 21 MiB/s, 338 objects/s recovering 2024-02-10T07:36:27.103 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:26 smithi170 ceph-mon[33394]: 7.11 scrub ok 2024-02-10T07:36:27.103 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:26 smithi170 ceph-mon[33394]: 7.10 scrub starts 2024-02-10T07:36:27.103 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:26 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3824742835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:36:27.103 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:26 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 18942/252093 objects degraded (7.514%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:27.103 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:26 smithi170 ceph-mon[33394]: osdmap e153: 8 total, 7 up, 8 in 2024-02-10T07:36:27.131 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:27.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:27.157 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:27.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:27.258 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:27.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:27.297 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:27.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:27.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:26 smithi008 ceph-mon[29816]: 7.11 scrub starts 2024-02-10T07:36:27.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:26 smithi008 ceph-mon[29816]: pgmap v189: 158 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 18942/252093 objects degraded (7.514%); 29126/252093 objects misplaced (11.554%); 21 MiB/s, 338 objects/s recovering 2024-02-10T07:36:27.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:26 smithi008 ceph-mon[29816]: 7.11 scrub ok 2024-02-10T07:36:27.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:26 smithi008 ceph-mon[29816]: 7.10 scrub starts 2024-02-10T07:36:27.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:26 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3824742835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:36:27.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:26 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 18942/252093 objects degraded (7.514%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:27.323 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:26 smithi008 ceph-mon[29816]: osdmap e153: 8 total, 7 up, 8 in 2024-02-10T07:36:27.323 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:26 smithi008 ceph-mon[40743]: 7.11 scrub starts 2024-02-10T07:36:27.323 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:26 smithi008 ceph-mon[40743]: pgmap v189: 158 pgs: 1 active+remapped+backfilling, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 18942/252093 objects degraded (7.514%); 29126/252093 objects misplaced (11.554%); 21 MiB/s, 338 objects/s recovering 2024-02-10T07:36:27.323 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:26 smithi008 ceph-mon[40743]: 7.11 scrub ok 2024-02-10T07:36:27.323 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:26 smithi008 ceph-mon[40743]: 7.10 scrub starts 2024-02-10T07:36:27.323 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:26 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3824742835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:36:27.323 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:26 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 18942/252093 objects degraded (7.514%), 22 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:27.323 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:26 smithi008 ceph-mon[40743]: osdmap e153: 8 total, 7 up, 8 in 2024-02-10T07:36:27.324 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T07:36:27.355 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:27 smithi170 podman[79853]: 2024-02-10 07:36:27.102378365 +0000 UTC m=+0.393755636 container start f398c377f85d658abc6c0040e1b95c12700080ea3a4f7a675b374d41d5b0e155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131) 2024-02-10T07:36:27.356 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:27 smithi170 bash[79853]: f398c377f85d658abc6c0040e1b95c12700080ea3a4f7a675b374d41d5b0e155 2024-02-10T07:36:27.356 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:27 smithi170 systemd[1]: Started Ceph osd.5 for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:36:27.398 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:27.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:27.441 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:27.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:27.542 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:27.577 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:27.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:27.627 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:36:27.678 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:27.701 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:27.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:27.803 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:27.847 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:27.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:27.910 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:27 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:36:27.652+0000 7f6fae7c37c0 -1 Falling back to public interface 2024-02-10T07:36:27.910 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:27 smithi170 ceph-mon[33394]: 7.16 scrub starts 2024-02-10T07:36:27.911 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:27 smithi170 ceph-mon[33394]: 7.17 scrub starts 2024-02-10T07:36:27.911 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:27 smithi170 ceph-mon[33394]: 7.4 scrub starts 2024-02-10T07:36:27.911 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:27 smithi170 ceph-mon[33394]: 7.10 scrub ok 2024-02-10T07:36:27.911 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:27 smithi170 ceph-mon[33394]: 7.16 scrub ok 2024-02-10T07:36:27.921 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:36:27.948 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:27.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:27.992 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:27.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:28.093 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:28.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:28.120 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:28.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:28.221 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:28.250 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:28.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:28.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:27 smithi008 ceph-mon[29816]: 7.16 scrub starts 2024-02-10T07:36:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:27 smithi008 ceph-mon[29816]: 7.17 scrub starts 2024-02-10T07:36:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:27 smithi008 ceph-mon[29816]: 7.4 scrub starts 2024-02-10T07:36:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:27 smithi008 ceph-mon[29816]: 7.10 scrub ok 2024-02-10T07:36:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:27 smithi008 ceph-mon[29816]: 7.16 scrub ok 2024-02-10T07:36:28.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:27 smithi008 ceph-mon[40743]: 7.16 scrub starts 2024-02-10T07:36:28.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:27 smithi008 ceph-mon[40743]: 7.17 scrub starts 2024-02-10T07:36:28.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:27 smithi008 ceph-mon[40743]: 7.4 scrub starts 2024-02-10T07:36:28.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:27 smithi008 ceph-mon[40743]: 7.10 scrub ok 2024-02-10T07:36:28.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:27 smithi008 ceph-mon[40743]: 7.16 scrub ok 2024-02-10T07:36:28.352 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:28.394 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:28.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:28.495 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:28.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:28.527 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:28.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:28.628 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:28.656 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:28.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:28.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:28.757 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:28.801 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:28.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:28.903 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:28.958 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:28.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:29.059 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-mon[29816]: 7.17 scrub ok 2024-02-10T07:36:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-mon[29816]: 6.0 scrub starts 2024-02-10T07:36:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-mon[29816]: 6.0 scrub ok 2024-02-10T07:36:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-mon[29816]: pgmap v192: 158 pgs: 1 active+clean+remapped, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18942/248706 objects degraded (7.616%); 28952/248706 objects misplaced (11.641%) 2024-02-10T07:36:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-mon[29816]: 4.0 scrub starts 2024-02-10T07:36:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-mon[29816]: 4.0 scrub ok 2024-02-10T07:36:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-mon[29816]: 7.4 scrub ok 2024-02-10T07:36:29.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:36:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:36:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-mon[40743]: 7.17 scrub ok 2024-02-10T07:36:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-mon[40743]: 6.0 scrub starts 2024-02-10T07:36:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-mon[40743]: 6.0 scrub ok 2024-02-10T07:36:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-mon[40743]: pgmap v192: 158 pgs: 1 active+clean+remapped, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18942/248706 objects degraded (7.616%); 28952/248706 objects misplaced (11.641%) 2024-02-10T07:36:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-mon[40743]: 4.0 scrub starts 2024-02-10T07:36:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-mon[40743]: 4.0 scrub ok 2024-02-10T07:36:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:28 smithi008 ceph-mon[40743]: 7.4 scrub ok 2024-02-10T07:36:29.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:29.103 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:29.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:28 smithi170 ceph-mon[33394]: 7.17 scrub ok 2024-02-10T07:36:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:28 smithi170 ceph-mon[33394]: 6.0 scrub starts 2024-02-10T07:36:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:28 smithi170 ceph-mon[33394]: 6.0 scrub ok 2024-02-10T07:36:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:28 smithi170 ceph-mon[33394]: pgmap v192: 158 pgs: 1 active+clean+remapped, 1 peering, 1 active+undersized+degraded+remapped+backfilling, 4 active+remapped+backfill_toofull, 99 active+clean, 31 active+undersized, 21 active+undersized+degraded; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 18942/248706 objects degraded (7.616%); 28952/248706 objects misplaced (11.641%) 2024-02-10T07:36:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:28 smithi170 ceph-mon[33394]: 4.0 scrub starts 2024-02-10T07:36:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:28 smithi170 ceph-mon[33394]: 4.0 scrub ok 2024-02-10T07:36:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:28 smithi170 ceph-mon[33394]: 7.4 scrub ok 2024-02-10T07:36:29.204 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:29.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:29.260 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:29.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:29.362 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:29.401 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:29.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:29.503 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:29.542 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:29.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:29.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:29.621 INFO:teuthology.orchestra.run.smithi170.stderr:no valid command found; 10 closest matches: 2024-02-10T07:36:29.621 INFO:teuthology.orchestra.run.smithi170.stderr:0 2024-02-10T07:36:29.622 INFO:teuthology.orchestra.run.smithi170.stderr:1 2024-02-10T07:36:29.622 INFO:teuthology.orchestra.run.smithi170.stderr:2 2024-02-10T07:36:29.622 INFO:teuthology.orchestra.run.smithi170.stderr:abort 2024-02-10T07:36:29.622 INFO:teuthology.orchestra.run.smithi170.stderr:assert 2024-02-10T07:36:29.622 INFO:teuthology.orchestra.run.smithi170.stderr:config diff 2024-02-10T07:36:29.622 INFO:teuthology.orchestra.run.smithi170.stderr:config diff get 2024-02-10T07:36:29.622 INFO:teuthology.orchestra.run.smithi170.stderr:config get 2024-02-10T07:36:29.622 INFO:teuthology.orchestra.run.smithi170.stderr:config help [] 2024-02-10T07:36:29.622 INFO:teuthology.orchestra.run.smithi170.stderr:config set ... 2024-02-10T07:36:29.622 INFO:teuthology.orchestra.run.smithi170.stderr:admin_socket: invalid command 2024-02-10T07:36:29.645 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:29.660 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:36:29.661 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:36:29.686 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:29.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:29.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:29.789 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:29.820 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:29.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:29.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:29.922 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:29.951 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:29.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:30.053 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:30.102 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:30.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:30.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:29 smithi170 ceph-mon[33394]: 7.15 scrub starts 2024-02-10T07:36:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:29 smithi170 ceph-mon[33394]: 7.15 scrub ok 2024-02-10T07:36:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:29 smithi170 ceph-mon[33394]: 7.13 scrub starts 2024-02-10T07:36:30.203 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:30.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:30.237 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:30.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:29 smithi008 ceph-mon[29816]: 7.15 scrub starts 2024-02-10T07:36:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:29 smithi008 ceph-mon[29816]: 7.15 scrub ok 2024-02-10T07:36:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:29 smithi008 ceph-mon[29816]: 7.13 scrub starts 2024-02-10T07:36:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:29 smithi008 ceph-mon[40743]: 7.15 scrub starts 2024-02-10T07:36:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:29 smithi008 ceph-mon[40743]: 7.15 scrub ok 2024-02-10T07:36:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:29 smithi008 ceph-mon[40743]: 7.13 scrub starts 2024-02-10T07:36:30.338 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:30.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:30.386 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:30.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:30.484 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-10T07:36:30.485 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-02-10T07:36:30.487 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:30.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:30.528 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:30.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:30.606 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:36:30.607 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-02-10T07:36:30.607 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-02-10T07:36:30.607 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:36:30.607 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.607 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47293 7.2 7:44192736:::benchmark_data_smithi008_82046_object47292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-02-10T07:36:30.607 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.923555+0000", 2024-02-10T07:36:30.607 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.67907408400001, 2024-02-10T07:36:30.607 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0745729319999999, 2024-02-10T07:36:30.607 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.607 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.607 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.607 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.607 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.607 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47293 2024-02-10T07:36:30.608 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.608 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.608 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.608 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.608 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923555+0000", 2024-02-10T07:36:30.608 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.608 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.608 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.608 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.608 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923555+0000", 2024-02-10T07:36:30.608 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.608 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.608 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.608 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.609 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923557+0000", 2024-02-10T07:36:30.609 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7439999999999999e-06 2024-02-10T07:36:30.609 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.609 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.609 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.609 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923579+0000", 2024-02-10T07:36:30.609 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2155e-05 2024-02-10T07:36:30.609 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.609 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.609 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.609 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923589+0000", 2024-02-10T07:36:30.609 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9410000000000002e-06 2024-02-10T07:36:30.609 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.609 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.610 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.610 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923597+0000", 2024-02-10T07:36:30.610 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7409999999999992e-06 2024-02-10T07:36:30.610 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.610 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.610 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.610 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.927637+0000", 2024-02-10T07:36:30.610 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0040407170000000001 2024-02-10T07:36:30.610 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.610 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.610 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.610 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.927647+0000", 2024-02-10T07:36:30.610 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7249999999999992e-06 2024-02-10T07:36:30.610 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.994086+0000", 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0664391659999999 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.994230+0000", 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014405200000000001 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.994425+0000", 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019504200000000001 2024-02-10T07:36:30.611 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.612 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.612 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.612 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.997950+0000", 2024-02-10T07:36:30.612 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035248499999999999 2024-02-10T07:36:30.612 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.612 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.612 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.612 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998016+0000", 2024-02-10T07:36:30.612 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5520999999999998e-05 2024-02-10T07:36:30.612 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.612 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.612 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.612 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998034+0000", 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8629000000000001e-05 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998103+0000", 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8366000000000002e-05 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998128+0000", 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5282999999999998e-05 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.613 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47301 7.2 7:4dfdce57:::benchmark_data_smithi008_82046_object47300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.930507+0000", 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.672121803, 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.071251151, 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47301 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.614 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930507+0000", 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930507+0000", 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930509+0000", 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9690000000000001e-06 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.615 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930534+0000", 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4635999999999999e-05 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930545+0000", 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0684e-05 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930553+0000", 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4270000000000008e-06 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.616 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930839+0000", 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000285665 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930848+0000", 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8269999999999993e-06 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.996538+0000", 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0656891589999999 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.617 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.996677+0000", 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013986200000000001 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.996838+0000", 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016012900000000001 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001568+0000", 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0047307110000000003 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001587+0000", 2024-02-10T07:36:30.618 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8393000000000002e-05 2024-02-10T07:36:30.619 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.619 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.619 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.619 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001655+0000", 2024-02-10T07:36:30.619 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8763999999999994e-05 2024-02-10T07:36:30.619 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.619 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.619 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.619 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001715+0000", 2024-02-10T07:36:30.620 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9156000000000001e-05 2024-02-10T07:36:30.620 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.620 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.620 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.620 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001758+0000", 2024-02-10T07:36:30.620 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3769000000000001e-05 2024-02-10T07:36:30.620 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.620 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.620 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.620 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.620 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47312 7.2 7:441ce8f4:::benchmark_data_smithi008_82046_object47311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.938921+0000", 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.663708403, 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0674384029999999, 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47312 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938921+0000", 2024-02-10T07:36:30.621 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938921+0000", 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938922+0000", 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.539e-06 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938949+0000", 2024-02-10T07:36:30.622 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6395999999999999e-05 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938965+0000", 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6665999999999999e-05 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938974+0000", 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9069999999999996e-06 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939045+0000", 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1165000000000001e-05 2024-02-10T07:36:30.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939058+0000", 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3026e-05 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998227+0000", 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.05916901 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998357+0000", 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012962100000000001 2024-02-10T07:36:30.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998522+0000", 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000165365 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001821+0000", 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032990379999999998 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001868+0000", 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7039999999999997e-05 2024-02-10T07:36:30.625 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006259+0000", 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0043904510000000001 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006317+0000", 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8346e-05 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006359+0000", 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1832999999999997e-05 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.626 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47317 7.2 7:477da457:::benchmark_data_smithi008_82046_object47316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.939750+0000", 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.66287876, 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.068997754, 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47317 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.627 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.628 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.628 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.628 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939750+0000", 2024-02-10T07:36:30.628 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.628 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.628 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.628 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.628 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939750+0000", 2024-02-10T07:36:30.628 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.628 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.628 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.629 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.629 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939751+0000", 2024-02-10T07:36:30.629 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.034e-06 2024-02-10T07:36:30.629 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.629 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.629 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.629 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939785+0000", 2024-02-10T07:36:30.629 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3855e-05 2024-02-10T07:36:30.629 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.629 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.629 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.629 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939801+0000", 2024-02-10T07:36:30.629 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5654999999999998e-05 2024-02-10T07:36:30.629 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939808+0000", 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6599999999999995e-06 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939884+0000", 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5796000000000004e-05 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939893+0000", 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9460000000000004e-06 2024-02-10T07:36:30.630 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.999976+0000", 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.060082902 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.000113+0000", 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013713899999999999 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.000289+0000", 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017616999999999999 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.631 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.003850+0000", 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035609219999999998 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.003902+0000", 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1848000000000002e-05 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008648+0000", 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0047453490000000003 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.632 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008704+0000", 2024-02-10T07:36:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6620999999999999e-05 2024-02-10T07:36:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008748+0000", 2024-02-10T07:36:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3856999999999997e-05 2024-02-10T07:36:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.633 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47376 7.2 7:4df203ae:::benchmark_data_smithi008_82046_object47375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.973270+0000", 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.62935925399999, 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.035779212, 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47376 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973270+0000", 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.634 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973270+0000", 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973272+0000", 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2000000000000001e-06 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973297+0000", 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4769999999999998e-05 2024-02-10T07:36:30.635 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973313+0000", 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5959000000000001e-05 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973321+0000", 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7810000000000005e-06 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973489+0000", 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016799299999999999 2024-02-10T07:36:30.636 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973499+0000", 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0176e-05 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001910+0000", 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0284110420000001 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.002044+0000", 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013467899999999999 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.637 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.002181+0000", 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013709900000000001 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008762+0000", 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0065806520000000002 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008834+0000", 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2015999999999996e-05 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.638 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008896+0000", 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2131000000000005e-05 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009004+0000", 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000107298 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009049+0000", 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5416e-05 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.639 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47408 7.2 7:48404cbe:::benchmark_data_smithi008_82046_object47407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.992841+0000", 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.60978854, 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.021168692, 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47408 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.640 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992841+0000", 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992841+0000", 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992841+0000", 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.13e-07 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.641 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992864+0000", 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2410000000000001e-05 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992873+0000", 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9679999999999995e-06 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992879+0000", 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7769999999999997e-06 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.642 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992957+0000", 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7385000000000005e-05 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992967+0000", 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9499999999999996e-06 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.003963+0000", 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.010996497 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.643 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.004110+0000", 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000147136 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.004280+0000", 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000170192 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009080+0000", 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0047996489999999996 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009138+0000", 2024-02-10T07:36:30.644 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7642999999999997e-05 2024-02-10T07:36:30.645 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.645 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.645 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:30.646 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.646 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.013897+0000", 2024-02-10T07:36:30.646 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0047593119999999999 2024-02-10T07:36:30.646 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.646 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.646 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.646 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.013966+0000", 2024-02-10T07:36:30.646 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9319000000000005e-05 2024-02-10T07:36:30.646 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.646 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.646 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014009+0000", 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2840999999999999e-05 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47416 7.2 7:4733a3ba:::benchmark_data_smithi008_82046_object47415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.996764+0000", 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.60586477, 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0174102730000001, 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.647 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47416 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996764+0000", 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996764+0000", 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.648 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996766+0000", 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4500000000000001e-06 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996788+0000", 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1861e-05 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996799+0000", 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1829e-05 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.649 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.650 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.650 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996807+0000", 2024-02-10T07:36:30.650 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5830000000000001e-06 2024-02-10T07:36:30.650 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.650 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.650 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.650 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996922+0000", 2024-02-10T07:36:30.650 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011469 2024-02-10T07:36:30.650 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.650 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.650 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.650 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996931+0000", 2024-02-10T07:36:30.650 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9910000000000008e-06 2024-02-10T07:36:30.650 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.651 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.651 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.651 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006393+0000", 2024-02-10T07:36:30.651 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.009462222 2024-02-10T07:36:30.651 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.651 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.651 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.651 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006520+0000", 2024-02-10T07:36:30.651 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012698599999999999 2024-02-10T07:36:30.651 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.651 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.651 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006667+0000", 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014698399999999999 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009091+0000", 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024244480000000001 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009194+0000", 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010273299999999999 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.652 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014084+0000", 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0048898680000000003 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014141+0000", 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6919999999999997e-05 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014175+0000", 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3708000000000003e-05 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.653 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47454 7.2 7:45bf9bfc:::benchmark_data_smithi008_82046_object47453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.019922+0000", 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.58270677900001, 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.99457532599999998, 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47454 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.654 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019922+0000", 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019922+0000", 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019924+0000", 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.979e-06 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.655 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019948+0000", 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3913999999999998e-05 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019965+0000", 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7254000000000001e-05 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019975+0000", 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1290000000000002e-06 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.656 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.020044+0000", 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9432999999999997e-05 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.020055+0000", 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0963e-05 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009296+0000", 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.98924138500000003 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009424+0000", 2024-02-10T07:36:30.657 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000128059 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009572+0000", 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000147128 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014029+0000", 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0044571560000000003 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014231+0000", 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00020206999999999999 2024-02-10T07:36:30.658 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014335+0000", 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010406799999999999 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014395+0000", 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0081000000000003e-05 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014498+0000", 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000102707 2024-02-10T07:36:30.659 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.660 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.660 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.660 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.660 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.660 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47461 7.2 7:41d2193b:::benchmark_data_smithi008_82046_object47460:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.660 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.026190+0000", 2024-02-10T07:36:30.660 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.57643923099999, 2024-02-10T07:36:30.660 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.99060737499999996, 2024-02-10T07:36:30.660 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.660 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.660 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.660 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.660 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.660 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47461 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026190+0000", 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026190+0000", 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026191+0000", 2024-02-10T07:36:30.661 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4020000000000001e-06 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026215+0000", 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3498000000000001e-05 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026228+0000", 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.293e-05 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026236+0000", 2024-02-10T07:36:30.662 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1389999999999995e-06 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.027093+0000", 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000857371 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.027104+0000", 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1285e-05 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.011669+0000", 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.98456444099999996 2024-02-10T07:36:30.663 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.011816+0000", 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014664100000000001 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.011969+0000", 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015315700000000001 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014192+0000", 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022234389999999998 2024-02-10T07:36:30.664 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016650+0000", 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002457636 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016693+0000", 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3699999999999998e-05 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016755+0000", 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1278000000000005e-05 2024-02-10T07:36:30.665 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016797+0000", 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2457999999999999e-05 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47486 7.2 7:46e23e1a:::benchmark_data_smithi008_82046_object47485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.041230+0000", 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.561399177, 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.99476295699999995, 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.666 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47486 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041230+0000", 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041230+0000", 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.667 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041231+0000", 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.471e-06 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041254+0000", 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2226e-05 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041270+0000", 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6858e-05 2024-02-10T07:36:30.668 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041279+0000", 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8249999999999994e-06 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041346+0000", 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6929999999999998e-05 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041357+0000", 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0429e-05 2024-02-10T07:36:30.669 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014562+0000", 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97320494599999996 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014756+0000", 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019477900000000001 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014928+0000", 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017193500000000001 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.670 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.671 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.672 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021669+0000", 2024-02-10T07:36:30.672 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0067405290000000003 2024-02-10T07:36:30.672 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.672 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.672 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.672 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021707+0000", 2024-02-10T07:36:30.672 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8451999999999997e-05 2024-02-10T07:36:30.672 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.672 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.672 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.672 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.035895+0000", 2024-02-10T07:36:30.672 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.01418823 2024-02-10T07:36:30.672 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.035953+0000", 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7839e-05 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.035993+0000", 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9508000000000002e-05 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.673 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47497 7.2 7:492ac9a6:::benchmark_data_smithi008_82046_object47496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.047446+0000", 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.555182856, 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.98990700600000003, 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47497 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047446+0000", 2024-02-10T07:36:30.674 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047446+0000", 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047448+0000", 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4330000000000001e-06 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047469+0000", 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1447999999999999e-05 2024-02-10T07:36:30.675 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047481+0000", 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1843e-05 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047540+0000", 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8989000000000001e-05 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047603+0000", 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2974999999999999e-05 2024-02-10T07:36:30.676 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047613+0000", 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8439999999999993e-06 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016852+0000", 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96923946999999999 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016993+0000", 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014067999999999999 2024-02-10T07:36:30.677 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.678 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.678 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.678 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.017152+0000", 2024-02-10T07:36:30.678 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015878999999999999 2024-02-10T07:36:30.678 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.678 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.678 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.678 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.026100+0000", 2024-02-10T07:36:30.678 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0089479090000000004 2024-02-10T07:36:30.678 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.678 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.678 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.678 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032492+0000", 2024-02-10T07:36:30.679 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0063921489999999997 2024-02-10T07:36:30.679 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.679 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.679 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.679 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.036002+0000", 2024-02-10T07:36:30.679 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035099559999999998 2024-02-10T07:36:30.679 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.679 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.679 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.679 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.036042+0000", 2024-02-10T07:36:30.679 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0071999999999999e-05 2024-02-10T07:36:30.679 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.679 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.680 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.680 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037353+0000", 2024-02-10T07:36:30.680 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001311448 2024-02-10T07:36:30.680 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.680 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.680 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.680 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.680 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.680 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47509 7.2 7:46639bc8:::benchmark_data_smithi008_82046_object47508:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.680 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.056564+0000", 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.54606484999999, 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.98093845899999998, 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47509 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056564+0000", 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.681 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056564+0000", 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056566+0000", 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.516e-06 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056588+0000", 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2405e-05 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.682 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056600+0000", 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.236e-05 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056609+0000", 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3380000000000005e-06 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056741+0000", 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013242899999999999 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.683 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056751+0000", 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8460000000000003e-06 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.019111+0000", 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96235988699999997 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.019240+0000", 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000128565 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.684 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.019393+0000", 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000153294 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032535+0000", 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013142513999999999 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032697+0000", 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016114300000000001 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.685 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037374+0000", 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046779819999999998 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037429+0000", 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4669999999999997e-05 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037503+0000", 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3510000000000006e-05 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.686 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47513 7.2 7:4ccef1d2:::benchmark_data_smithi008_82046_object47512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.059397+0000", 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.543231621, 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97837375900000001, 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47513 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.687 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059397+0000", 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059397+0000", 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059400+0000", 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1749999999999999e-06 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.688 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059425+0000", 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5701e-05 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059443+0000", 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7710000000000002e-05 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059453+0000", 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0029e-05 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.689 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059555+0000", 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000102403 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059567+0000", 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1172000000000001e-05 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021735+0000", 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96216885299999999 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.690 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021846+0000", 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000110096 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021980+0000", 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013462200000000001 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032576+0000", 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010596202000000001 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.691 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032865+0000", 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00028853 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037514+0000", 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046494279999999997 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037580+0000", 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5399000000000002e-05 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.692 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037771+0000", 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019143900000000001 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47519 7.2 7:474b3507:::benchmark_data_smithi008_82046_object47518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.063509+0000", 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.539119637, 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97463434199999999, 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.693 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47519 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063509+0000", 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063509+0000", 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.694 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063511+0000", 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3659999999999999e-06 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063535+0000", 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3765e-05 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063546+0000", 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1199000000000001e-05 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063553+0000", 2024-02-10T07:36:30.695 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7920000000000004e-06 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063622+0000", 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9577000000000001e-05 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063633+0000", 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.134e-05 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.023833+0000", 2024-02-10T07:36:30.696 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96019993999999997 2024-02-10T07:36:30.697 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.697 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.697 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.697 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.024002+0000", 2024-02-10T07:36:30.697 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016875700000000001 2024-02-10T07:36:30.698 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.698 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.698 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.698 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.024184+0000", 2024-02-10T07:36:30.698 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000182227 2024-02-10T07:36:30.698 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.698 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032615+0000", 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0084308600000000001 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037780+0000", 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0051646779999999998 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.038029+0000", 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00024897100000000001 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.699 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.038106+0000", 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6736000000000005e-05 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.038144+0000", 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8133999999999998e-05 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47530 7.2 7:44bc3d2a:::benchmark_data_smithi008_82046_object47529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.700 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.076595+0000", 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.526033587, 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96853736000000001, 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47530 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076595+0000", 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.701 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076595+0000", 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076597+0000", 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0899999999999999e-06 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076613+0000", 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.683e-05 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.702 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076623+0000", 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9070000000000002e-06 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076629+0000", 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9689999999999999e-06 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076694+0000", 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4873e-05 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.703 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076707+0000", 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2424e-05 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.026136+0000", 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.94942972400000003 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.026302+0000", 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000165744 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.704 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.026474+0000", 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017199699999999999 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032657+0000", 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0061833010000000004 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037785+0000", 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0051276610000000004 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.705 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.044963+0000", 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0071781509999999998 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045018+0000", 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4934e-05 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045133+0000", 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011475499999999999 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.706 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47545 7.2 7:41492b52:::benchmark_data_smithi008_82046_object47544:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.108305+0000", 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 112.494323804, 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.93696623499999998, 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47545 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.707 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108305+0000", 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108305+0000", 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108306+0000", 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2100000000000001e-06 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108325+0000", 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8094999999999999e-05 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108335+0000", 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0301e-05 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108340+0000", 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6420000000000004e-06 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108392+0000", 2024-02-10T07:36:30.709 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1669999999999998e-05 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108399+0000", 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8279999999999997e-06 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.030442+0000", 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.92204303600000004 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.030575+0000", 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000133089 2024-02-10T07:36:30.710 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.030728+0000", 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015333400000000001 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037828+0000", 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.007100014 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037841+0000", 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2738999999999999e-05 2024-02-10T07:36:30.711 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045181+0000", 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0073395769999999999 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045235+0000", 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4469000000000003e-05 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045271+0000", 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6230999999999999e-05 2024-02-10T07:36:30.712 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127258 7.a 7:583c5e32:::benchmark_data_smithi008_82046_object12120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.252605+0000", 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 34.350024339000001, 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.90179492999999999, 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127258 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.713 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.252605+0000", 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.252605+0000", 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.252607+0000", 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.857e-06 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.714 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.252612+0000", 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2140000000000004e-06 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.252629+0000", 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7657000000000001e-05 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.252638+0000", 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.799e-06 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.715 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.252702+0000", 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4113000000000001e-05 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.252711+0000", 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0399999999999998e-06 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.135085+0000", 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.88237310800000002 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.716 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.135198+0000", 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011309100000000001 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2", 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.135311+0000", 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000113719 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.151879+0000", 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016567536000000001 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.717 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.154221+0000", 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0023423939999999998 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.154278+0000", 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6725999999999999e-05 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.154400+0000", 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012167600000000001 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.718 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127262 7.a 7:56aa624f:::benchmark_data_smithi008_82046_object12124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.265611+0000", 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 34.337017934000002, 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.89170804100000001, 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127262 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.719 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.265611+0000", 2024-02-10T07:36:30.720 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.720 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.720 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.720 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.720 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.265611+0000", 2024-02-10T07:36:30.720 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.720 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.720 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.720 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.720 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.265613+0000", 2024-02-10T07:36:30.720 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5519999999999999e-06 2024-02-10T07:36:30.720 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.720 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.720 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.265617+0000", 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6110000000000004e-06 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.265629+0000", 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1493e-05 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.265636+0000", 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7020000000000001e-06 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.721 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.265698+0000", 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1517000000000001e-05 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.265708+0000", 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7000000000000003e-06 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.142622+0000", 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.87691436899999997 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.722 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.148857+0000", 2024-02-10T07:36:30.724 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.006234524 2024-02-10T07:36:30.724 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.724 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.724 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2", 2024-02-10T07:36:30.724 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.148973+0000", 2024-02-10T07:36:30.724 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000116279 2024-02-10T07:36:30.724 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.724 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.724 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.724 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.151942+0000", 2024-02-10T07:36:30.724 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0029694349999999999 2024-02-10T07:36:30.724 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.724 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.157178+0000", 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.005235763 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.157235+0000", 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6511999999999997e-05 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.157319+0000", 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4584000000000002e-05 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.725 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127282 7.10 7:083f4428:::benchmark_data_smithi008_82046_object12144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.334211+0000", 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 34.268417808000002, 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.91253578899999999, 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127282 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.726 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.334211+0000", 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.334211+0000", 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.334213+0000", 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3850000000000001e-06 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.727 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.334216+0000", 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4029999999999999e-06 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.334224+0000", 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4300000000000006e-06 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.334230+0000", 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6380000000000001e-06 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.728 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.334283+0000", 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3239000000000003e-05 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.334293+0000", 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4199999999999996e-06 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.130207+0000", 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.79591468200000004 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.729 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.130326+0000", 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011857 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.130587+0000", 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026057599999999999 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.140257+0000", 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0096706269999999993 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.730 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.151851+0000", 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011593592 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.246575+0000", 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.094724482999999998 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.246639+0000", 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4086999999999998e-05 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.731 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.246747+0000", 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000107657 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127291 7.10 7:0b95925a:::benchmark_data_smithi008_82046_object12153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.373993+0000", 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 34.228636453999997, 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.87299162299999999, 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:36:30.732 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127291 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.373993+0000", 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.373993+0000", 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.733 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.373994+0000", 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7549999999999999e-06 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.373999+0000", 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2139999999999998e-06 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.374012+0000", 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3076e-05 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:36:30.734 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.374020+0000", 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3189999999999995e-06 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.374093+0000", 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3083999999999993e-05 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.374102+0000", 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0059999999999998e-06 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.132012+0000", 2024-02-10T07:36:30.735 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.75791003099999998 2024-02-10T07:36:30.736 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.736 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.736 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:36:30.736 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.132133+0000", 2024-02-10T07:36:30.736 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012081199999999999 2024-02-10T07:36:30.736 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.736 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.736 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:36:30.736 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.134053+0000", 2024-02-10T07:36:30.736 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001919977 2024-02-10T07:36:30.736 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.736 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.736 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:36:30.736 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.140264+0000", 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0062115130000000001 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.151896+0000", 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011631689000000001 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.246794+0000", 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.094897811999999998 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.246884+0000", 2024-02-10T07:36:30.737 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0136000000000006e-05 2024-02-10T07:36:30.738 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:36:30.738 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:36:30.738 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:36:30.738 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.246984+0000", 2024-02-10T07:36:30.738 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010019899999999999 2024-02-10T07:36:30.738 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.738 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.738 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.738 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:36:30.738 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:36:30.738 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:36:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:30.739 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:30.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:30.840 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:30.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:30.873 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:30.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:30.975 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:31.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:31.002 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:31.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:31.103 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:31.128 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:31.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:31.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:31.132 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:36:30.972+0000 7f6fae7c37c0 -1 osd.5 139 log_to_monitors true 2024-02-10T07:36:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:30 smithi170 ceph-mon[33394]: 4.a scrub starts 2024-02-10T07:36:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:30 smithi170 ceph-mon[33394]: 4.a scrub ok 2024-02-10T07:36:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:30 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:36:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:30 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:36:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:30 smithi170 ceph-mon[33394]: 6.14 scrub starts 2024-02-10T07:36:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:30 smithi170 ceph-mon[33394]: 6.14 scrub ok 2024-02-10T07:36:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:30 smithi170 ceph-mon[33394]: pgmap v193: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 100 active+clean, 31 active+undersized, 21 active+undersized+degraded; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 862 B/s rd, 0 op/s; 17673/241836 objects degraded (7.308%); 27819/241836 objects misplaced (11.503%) 2024-02-10T07:36:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:30 smithi170 ceph-mon[33394]: osdmap e154: 8 total, 7 up, 8 in 2024-02-10T07:36:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:30 smithi170 ceph-mon[33394]: 7.13 scrub ok 2024-02-10T07:36:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:30 smithi170 ceph-mon[33394]: 5.14 scrub starts 2024-02-10T07:36:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:30 smithi170 ceph-mon[33394]: 5.14 scrub ok 2024-02-10T07:36:31.229 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:31.250 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[29816]: 4.a scrub starts 2024-02-10T07:36:31.250 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[29816]: 4.a scrub ok 2024-02-10T07:36:31.250 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:36:31.250 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:36:31.250 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[29816]: 6.14 scrub starts 2024-02-10T07:36:31.250 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[29816]: 6.14 scrub ok 2024-02-10T07:36:31.250 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[29816]: pgmap v193: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 100 active+clean, 31 active+undersized, 21 active+undersized+degraded; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 862 B/s rd, 0 op/s; 17673/241836 objects degraded (7.308%); 27819/241836 objects misplaced (11.503%) 2024-02-10T07:36:31.250 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[29816]: osdmap e154: 8 total, 7 up, 8 in 2024-02-10T07:36:31.250 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[29816]: 7.13 scrub ok 2024-02-10T07:36:31.250 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[29816]: 5.14 scrub starts 2024-02-10T07:36:31.250 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[29816]: 5.14 scrub ok 2024-02-10T07:36:31.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[40743]: 4.a scrub starts 2024-02-10T07:36:31.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[40743]: 4.a scrub ok 2024-02-10T07:36:31.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:36:31.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:36:31.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[40743]: 6.14 scrub starts 2024-02-10T07:36:31.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[40743]: 6.14 scrub ok 2024-02-10T07:36:31.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[40743]: pgmap v193: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 100 active+clean, 31 active+undersized, 21 active+undersized+degraded; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 862 B/s rd, 0 op/s; 17673/241836 objects degraded (7.308%); 27819/241836 objects misplaced (11.503%) 2024-02-10T07:36:31.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[40743]: osdmap e154: 8 total, 7 up, 8 in 2024-02-10T07:36:31.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[40743]: 7.13 scrub ok 2024-02-10T07:36:31.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[40743]: 5.14 scrub starts 2024-02-10T07:36:31.251 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:30 smithi008 ceph-mon[40743]: 5.14 scrub ok 2024-02-10T07:36:31.258 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:31.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:31.360 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:31.403 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:31.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:31.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:31.505 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:31.554 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:31.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:31.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:31.656 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:31.661 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:36:31.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:31.701 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:31.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:31.802 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:31.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:31.864 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:31.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:31.965 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:31.991 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:31.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:31.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:32.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[29816]: from='osd.5 [v2:172.21.15.170:6808/3225334381,v1:172.21.15.170:6809/3225334381]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:36:32.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[29816]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:36:32.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[29816]: 7.f scrub starts 2024-02-10T07:36:32.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 17673/241836 objects degraded (7.308%), 21 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:32.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[29816]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T07:36:32.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[29816]: from='osd.5 [v2:172.21.15.170:6808/3225334381,v1:172.21.15.170:6809/3225334381]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:36:32.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[29816]: osdmap e155: 8 total, 7 up, 8 in 2024-02-10T07:36:32.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[29816]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:36:32.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[40743]: from='osd.5 [v2:172.21.15.170:6808/3225334381,v1:172.21.15.170:6809/3225334381]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:36:32.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[40743]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:36:32.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[40743]: 7.f scrub starts 2024-02-10T07:36:32.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 17673/241836 objects degraded (7.308%), 21 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:32.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[40743]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T07:36:32.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[40743]: from='osd.5 [v2:172.21.15.170:6808/3225334381,v1:172.21.15.170:6809/3225334381]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:36:32.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[40743]: osdmap e155: 8 total, 7 up, 8 in 2024-02-10T07:36:32.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:31 smithi008 ceph-mon[40743]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:36:32.094 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:32.133 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:36:31 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:36:31.746+0000 7f6f9fc1b700 -1 osd.5 139 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:36:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:31 smithi170 ceph-mon[33394]: from='osd.5 [v2:172.21.15.170:6808/3225334381,v1:172.21.15.170:6809/3225334381]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:36:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:31 smithi170 ceph-mon[33394]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T07:36:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:31 smithi170 ceph-mon[33394]: 7.f scrub starts 2024-02-10T07:36:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:31 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 17673/241836 objects degraded (7.308%), 21 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:31 smithi170 ceph-mon[33394]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T07:36:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:31 smithi170 ceph-mon[33394]: from='osd.5 [v2:172.21.15.170:6808/3225334381,v1:172.21.15.170:6809/3225334381]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:36:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:31 smithi170 ceph-mon[33394]: osdmap e155: 8 total, 7 up, 8 in 2024-02-10T07:36:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:31 smithi170 ceph-mon[33394]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-02-10T07:36:32.134 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:32.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:32.236 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:32.263 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:32.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:32.274 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-10T07:36:32.364 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:32.391 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:32.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:32.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:32.493 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:32.524 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:32.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:32.610 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:36:32.625 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:32.691 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:32.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:32.793 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:32.818 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:32.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:32.919 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:32.944 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:32.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:33.045 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:33.071 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:33.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:33.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[40743]: 4.5 scrub starts 2024-02-10T07:36:33.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[40743]: 4.5 scrub ok 2024-02-10T07:36:33.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[40743]: 4.1b scrub starts 2024-02-10T07:36:33.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[40743]: 4.1b scrub ok 2024-02-10T07:36:33.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[40743]: 5.8 scrub starts 2024-02-10T07:36:33.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[40743]: 5.8 scrub ok 2024-02-10T07:36:33.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[40743]: pgmap v196: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 100 active+clean, 31 active+undersized, 21 active+undersized+degraded; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 863 B/s rd, 0 op/s; 17673/241836 objects degraded (7.308%); 27819/241836 objects misplaced (11.503%) 2024-02-10T07:36:33.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[40743]: 5.17 scrub starts 2024-02-10T07:36:33.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[40743]: 5.17 scrub ok 2024-02-10T07:36:33.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:33.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[40743]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:36:33.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[40743]: osd.5 [v2:172.21.15.170:6808/3225334381,v1:172.21.15.170:6809/3225334381] boot 2024-02-10T07:36:33.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[40743]: osdmap e156: 8 total, 8 up, 8 in 2024-02-10T07:36:33.085 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:36:33.085 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[29816]: 4.5 scrub starts 2024-02-10T07:36:33.085 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[29816]: 4.5 scrub ok 2024-02-10T07:36:33.085 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[29816]: 4.1b scrub starts 2024-02-10T07:36:33.086 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[29816]: 4.1b scrub ok 2024-02-10T07:36:33.086 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[29816]: 5.8 scrub starts 2024-02-10T07:36:33.086 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[29816]: 5.8 scrub ok 2024-02-10T07:36:33.086 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[29816]: pgmap v196: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 100 active+clean, 31 active+undersized, 21 active+undersized+degraded; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 863 B/s rd, 0 op/s; 17673/241836 objects degraded (7.308%); 27819/241836 objects misplaced (11.503%) 2024-02-10T07:36:33.086 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[29816]: 5.17 scrub starts 2024-02-10T07:36:33.086 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[29816]: 5.17 scrub ok 2024-02-10T07:36:33.087 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:33.087 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[29816]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:36:33.087 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[29816]: osd.5 [v2:172.21.15.170:6808/3225334381,v1:172.21.15.170:6809/3225334381] boot 2024-02-10T07:36:33.087 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[29816]: osdmap e156: 8 total, 8 up, 8 in 2024-02-10T07:36:33.087 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:36:33.108 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:36:33.121 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:36:33.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:36:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-mon[33394]: 4.5 scrub starts 2024-02-10T07:36:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-mon[33394]: 4.5 scrub ok 2024-02-10T07:36:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-mon[33394]: 4.1b scrub starts 2024-02-10T07:36:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-mon[33394]: 4.1b scrub ok 2024-02-10T07:36:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-mon[33394]: 5.8 scrub starts 2024-02-10T07:36:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-mon[33394]: 5.8 scrub ok 2024-02-10T07:36:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-mon[33394]: pgmap v196: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 100 active+clean, 31 active+undersized, 21 active+undersized+degraded; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 863 B/s rd, 0 op/s; 17673/241836 objects degraded (7.308%); 27819/241836 objects misplaced (11.503%) 2024-02-10T07:36:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-mon[33394]: 5.17 scrub starts 2024-02-10T07:36:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-mon[33394]: 5.17 scrub ok 2024-02-10T07:36:33.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:33.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-mon[33394]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:36:33.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-mon[33394]: osd.5 [v2:172.21.15.170:6808/3225334381,v1:172.21.15.170:6809/3225334381] boot 2024-02-10T07:36:33.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-mon[33394]: osdmap e156: 8 total, 8 up, 8 in 2024-02-10T07:36:33.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T07:36:33.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:33.199 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:33.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:33.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:33.301 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:33.328 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:33.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:33.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:33.430 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:33.456 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:33.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:33.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:33.558 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:33.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:33.585 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:33.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:33.686 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:33.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:33.711 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:33.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:33.812 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:33.841 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:33.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:33.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:33.943 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:33.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:33.968 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:33.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:34.069 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:34.096 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:34.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:34.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:34.163 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[29816]: 3.1a scrub starts 2024-02-10T07:36:34.163 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[40743]: 3.1a scrub starts 2024-02-10T07:36:34.199 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:34.228 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:34.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:34.330 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:34.360 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:34.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:34 smithi170 ceph-mon[33394]: 3.1a scrub starts 2024-02-10T07:36:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:34 smithi170 ceph-mon[33394]: 3.1a scrub ok 2024-02-10T07:36:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:34 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:36:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:34 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:36:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:34 smithi170 ceph-mon[33394]: 6.3 scrub starts 2024-02-10T07:36:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:34 smithi170 ceph-mon[33394]: 6.3 scrub ok 2024-02-10T07:36:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:34 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:34 smithi170 ceph-mon[33394]: 7.f scrub ok 2024-02-10T07:36:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:34 smithi170 ceph-mon[33394]: osdmap e157: 8 total, 8 up, 8 in 2024-02-10T07:36:34.461 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:34.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:34.487 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:34.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[40743]: 3.1a scrub ok 2024-02-10T07:36:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:36:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:36:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[40743]: 6.3 scrub starts 2024-02-10T07:36:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[40743]: 6.3 scrub ok 2024-02-10T07:36:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[40743]: 7.f scrub ok 2024-02-10T07:36:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:34.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[40743]: osdmap e157: 8 total, 8 up, 8 in 2024-02-10T07:36:34.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[29816]: 3.1a scrub ok 2024-02-10T07:36:34.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:36:34.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:36:34.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[29816]: 6.3 scrub starts 2024-02-10T07:36:34.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[29816]: 6.3 scrub ok 2024-02-10T07:36:34.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:34.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[29816]: 7.f scrub ok 2024-02-10T07:36:34.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:34.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:34.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:34 smithi008 ceph-mon[29816]: osdmap e157: 8 total, 8 up, 8 in 2024-02-10T07:36:34.588 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:34.612 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:34.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:34.713 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:34.744 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:34.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:34.845 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:34.884 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:34.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:34.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:34.986 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:35.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:35.027 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:35.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:35.128 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:35.167 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:35.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:35.268 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:35 smithi170 ceph-mon[33394]: 7.17 starting backfill to osd.5 from (109'1600,139'7756] MIN to 155'8612 2024-02-10T07:36:35.268 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:35 smithi170 ceph-mon[33394]: 7.14 continuing backfill to osd.2 from (112'2200,155'8538] MIN to 155'8538 2024-02-10T07:36:35.268 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:35 smithi170 ceph-mon[33394]: 7.14 continuing backfill to osd.3 from (112'2200,155'8538] MIN to 155'8538 2024-02-10T07:36:35.268 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:35 smithi170 ceph-mon[33394]: 7.14 starting backfill to osd.5 from (112'2200,139'7730] MIN to 155'8538 2024-02-10T07:36:35.268 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:35 smithi170 ceph-mon[33394]: pgmap v199: 158 pgs: 1 remapped+peering, 19 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 101 active+clean, 19 active+undersized, 14 active+undersized+degraded; 4.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 2614/227475 objects degraded (1.149%); 18907/227475 objects misplaced (8.312%) 2024-02-10T07:36:35.268 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:35 smithi170 ceph-mon[33394]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:36:35.268 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:35 smithi170 ceph-mon[33394]: osdmap e158: 8 total, 8 up, 8 in 2024-02-10T07:36:35.270 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:35.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:35.305 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:35.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:35.406 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:35.445 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:35.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:35.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:35.456 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:35 smithi008 ceph-mon[29816]: 7.17 starting backfill to osd.5 from (109'1600,139'7756] MIN to 155'8612 2024-02-10T07:36:35.456 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:35 smithi008 ceph-mon[29816]: 7.14 continuing backfill to osd.2 from (112'2200,155'8538] MIN to 155'8538 2024-02-10T07:36:35.456 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:35 smithi008 ceph-mon[29816]: 7.14 continuing backfill to osd.3 from (112'2200,155'8538] MIN to 155'8538 2024-02-10T07:36:35.456 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:35 smithi008 ceph-mon[29816]: 7.14 starting backfill to osd.5 from (112'2200,139'7730] MIN to 155'8538 2024-02-10T07:36:35.456 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:35 smithi008 ceph-mon[29816]: pgmap v199: 158 pgs: 1 remapped+peering, 19 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 101 active+clean, 19 active+undersized, 14 active+undersized+degraded; 4.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 2614/227475 objects degraded (1.149%); 18907/227475 objects misplaced (8.312%) 2024-02-10T07:36:35.456 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:35 smithi008 ceph-mon[29816]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:36:35.456 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:35 smithi008 ceph-mon[29816]: osdmap e158: 8 total, 8 up, 8 in 2024-02-10T07:36:35.457 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:35 smithi008 ceph-mon[40743]: 7.17 starting backfill to osd.5 from (109'1600,139'7756] MIN to 155'8612 2024-02-10T07:36:35.457 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:35 smithi008 ceph-mon[40743]: 7.14 continuing backfill to osd.2 from (112'2200,155'8538] MIN to 155'8538 2024-02-10T07:36:35.457 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:35 smithi008 ceph-mon[40743]: 7.14 continuing backfill to osd.3 from (112'2200,155'8538] MIN to 155'8538 2024-02-10T07:36:35.457 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:35 smithi008 ceph-mon[40743]: 7.14 starting backfill to osd.5 from (112'2200,139'7730] MIN to 155'8538 2024-02-10T07:36:35.457 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:35 smithi008 ceph-mon[40743]: pgmap v199: 158 pgs: 1 remapped+peering, 19 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 101 active+clean, 19 active+undersized, 14 active+undersized+degraded; 4.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 2614/227475 objects degraded (1.149%); 18907/227475 objects misplaced (8.312%) 2024-02-10T07:36:35.457 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:35 smithi008 ceph-mon[40743]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:36:35.457 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:35 smithi008 ceph-mon[40743]: osdmap e158: 8 total, 8 up, 8 in 2024-02-10T07:36:35.486 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T07:36:35.547 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:35.592 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:35.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:35.622 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:36:35.622 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2024-02-10T07:36:35.622 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2024-02-10T07:36:35.623 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:36:35.694 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:35.728 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:35.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:35.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:35.830 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:35.870 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:35.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:35.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:35.972 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:36.004 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:36.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:36.106 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:36.130 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:36.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:36.231 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:36.265 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:36.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:36.367 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:36.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:36 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:36:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:36 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:36:36.393 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:36.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:36.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:36.495 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:36.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:36.540 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:36.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:36.562 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:36 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:36:36.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:36 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:36:36.563 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:36 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:36:36.563 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:36 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:36:36.642 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:36.680 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:36.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:36.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:36.761 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:36:36.773 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-10T07:36:36.783 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:36.819 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:36.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:36.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:36.924 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:36.977 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:36.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:37.079 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:37.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:37.108 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:37.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:37.209 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:37.246 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:37.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:37.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:37.318 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:37 smithi170 ceph-mon[33394]: 6.1a scrub starts 2024-02-10T07:36:37.318 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:37 smithi170 ceph-mon[33394]: 6.1a scrub ok 2024-02-10T07:36:37.318 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:37 smithi170 ceph-mon[33394]: pgmap v201: 158 pgs: 1 remapped+peering, 19 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 101 active+clean, 19 active+undersized, 14 active+undersized+degraded; 4.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 2614/227475 objects degraded (1.149%); 18907/227475 objects misplaced (8.312%) 2024-02-10T07:36:37.318 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:37 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 2614/227475 objects degraded (1.149%), 14 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:37.318 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:37 smithi170 ceph-mon[33394]: osdmap e159: 8 total, 8 up, 8 in 2024-02-10T07:36:37.348 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:37.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:37.379 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:37.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:37.480 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:37.525 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:37.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:37.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:37 smithi008 ceph-mon[29816]: 6.1a scrub starts 2024-02-10T07:36:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:37 smithi008 ceph-mon[29816]: 6.1a scrub ok 2024-02-10T07:36:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:37 smithi008 ceph-mon[29816]: pgmap v201: 158 pgs: 1 remapped+peering, 19 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 101 active+clean, 19 active+undersized, 14 active+undersized+degraded; 4.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 2614/227475 objects degraded (1.149%); 18907/227475 objects misplaced (8.312%) 2024-02-10T07:36:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:37 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 2614/227475 objects degraded (1.149%), 14 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:37 smithi008 ceph-mon[29816]: osdmap e159: 8 total, 8 up, 8 in 2024-02-10T07:36:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:37 smithi008 ceph-mon[40743]: 6.1a scrub starts 2024-02-10T07:36:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:37 smithi008 ceph-mon[40743]: 6.1a scrub ok 2024-02-10T07:36:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:37 smithi008 ceph-mon[40743]: pgmap v201: 158 pgs: 1 remapped+peering, 19 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 101 active+clean, 19 active+undersized, 14 active+undersized+degraded; 4.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 2614/227475 objects degraded (1.149%); 18907/227475 objects misplaced (8.312%) 2024-02-10T07:36:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:37 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 2614/227475 objects degraded (1.149%), 14 pgs degraded (PG_DEGRADED) 2024-02-10T07:36:37.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:37 smithi008 ceph-mon[40743]: osdmap e159: 8 total, 8 up, 8 in 2024-02-10T07:36:37.627 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:37.670 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:37.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:37.771 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:37.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:37.813 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:37.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:37.914 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:37.943 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:37.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:37.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:38.049 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:38.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:38.091 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:38.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:38.174 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:36:38.192 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:38.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:38.255 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:38.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:38.315 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:38 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:38.315 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:38 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:36:38.315 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:38 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:38.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:38 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:38.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:38 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:36:38.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:38 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:38 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:36:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:38 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:36:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:38 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:38.356 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:38.388 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:38.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:38.490 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:38.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:38.533 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:38.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:38.634 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:38.665 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:38.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:38.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:38.767 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:38.818 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:38.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:38.919 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:38.965 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:38.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:39.067 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:39.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:36:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:36:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:36:39.114 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:39.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:39.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:39.215 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:39.256 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:39.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:39.357 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:39.388 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:39.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:39.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:39.440 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:39 smithi008 ceph-mon[29816]: pgmap v203: 158 pgs: 1 remapped+peering, 19 peering, 4 active+remapped+backfill_toofull, 133 active+clean, 1 active+undersized+degraded; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s; 2570/224391 objects degraded (1.145%); 18807/224391 objects misplaced (8.381%) 2024-02-10T07:36:39.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:39 smithi008 ceph-mon[40743]: pgmap v203: 158 pgs: 1 remapped+peering, 19 peering, 4 active+remapped+backfill_toofull, 133 active+clean, 1 active+undersized+degraded; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s; 2570/224391 objects degraded (1.145%); 18807/224391 objects misplaced (8.381%) 2024-02-10T07:36:39.490 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:39.539 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:39.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:39 smithi170 ceph-mon[33394]: pgmap v203: 158 pgs: 1 remapped+peering, 19 peering, 4 active+remapped+backfill_toofull, 133 active+clean, 1 active+undersized+degraded; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1017 B/s rd, 0 op/s; 2570/224391 objects degraded (1.145%); 18807/224391 objects misplaced (8.381%) 2024-02-10T07:36:39.640 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:39.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:39.684 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:39.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:39.785 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:39.826 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:39.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:39.927 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:39.954 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:36:39.965 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:39.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:39.967 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:36:40.021 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:36:40.067 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:40.073 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:36:40.106 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:40.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:40.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:40.124 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:36:40.177 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:36:40.208 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:40.233 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:36:40.249 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:40.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:40.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:40.279 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:36:40.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:40 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2790080666' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:36:40.280 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:40 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2790080666' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:36:40.285 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:36:40.285 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:36:40.285 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:36:40.286 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:36:40.286 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:36:40.286 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:36:40.286 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:36:40.286 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:36:40.352 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:40.385 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:40.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:40.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:40.487 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:40.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:40.523 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:40.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:40.523 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:40 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2790080666' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:36:40.624 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:40.658 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:40.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:40.714 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:36:40.714 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:36:40.760 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:40.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:40.802 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:40.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:40.903 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:40.950 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:40.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:41.052 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:41.075 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:41.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:41.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:41.177 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:41.197 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:36:41.202 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:41 smithi008 ceph-mon[29816]: pgmap v204: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 149 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 848 B/s rd, 0 B/s wr, 548 op/s; 1739/217557 objects degraded (0.799%); 23768/217557 objects misplaced (10.925%); 4.3 MiB/s, 187 objects/s recovering 2024-02-10T07:36:41.202 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:41 smithi008 ceph-mon[29816]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:36:41.202 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:41 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2341096189' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:36:41.202 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:41 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:36:41.202 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:41 smithi008 ceph-mon[40743]: pgmap v204: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 149 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 848 B/s rd, 0 B/s wr, 548 op/s; 1739/217557 objects degraded (0.799%); 23768/217557 objects misplaced (10.925%); 4.3 MiB/s, 187 objects/s recovering 2024-02-10T07:36:41.202 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:41 smithi008 ceph-mon[40743]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:36:41.203 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:41 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2341096189' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:36:41.203 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:41 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:36:41.214 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:41.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:41.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:41.315 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:41.347 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:41.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:41.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:41.449 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:41.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:41.491 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:41.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:41.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:41 smithi170 ceph-mon[33394]: pgmap v204: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 149 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 848 B/s rd, 0 B/s wr, 548 op/s; 1739/217557 objects degraded (0.799%); 23768/217557 objects misplaced (10.925%); 4.3 MiB/s, 187 objects/s recovering 2024-02-10T07:36:41.526 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:41 smithi170 ceph-mon[33394]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:36:41.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:41 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2341096189' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:36:41.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:41 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:36:41.592 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 0 1 2024-02-10T07:36:41.592 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:41.621 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:41.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:41.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:41.722 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:41.762 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:41.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:41.864 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:41.898 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:41.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:41.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:41.920 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:36:41.920 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:36:42.000 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:42.034 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:42.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:42.135 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:42.176 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:42.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:42.277 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:42.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:42.319 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:42.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:42.420 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:42.433 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:42 smithi008 ceph-mon[29816]: 4.1 scrub starts 2024-02-10T07:36:42.433 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:42 smithi008 ceph-mon[29816]: 4.1 scrub ok 2024-02-10T07:36:42.433 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:42 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:36:42.434 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:42 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:36:42.434 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:42 smithi008 ceph-mon[29816]: osdmap e160: 8 total, 8 up, 8 in 2024-02-10T07:36:42.434 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:42 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 1739/217557 objects degraded (0.799%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:36:42.434 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:42 smithi008 ceph-mon[29816]: osdmap e161: 8 total, 8 up, 8 in 2024-02-10T07:36:42.434 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:42 smithi008 ceph-mon[40743]: 4.1 scrub starts 2024-02-10T07:36:42.434 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:42 smithi008 ceph-mon[40743]: 4.1 scrub ok 2024-02-10T07:36:42.434 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:42 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:36:42.434 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:42 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:36:42.434 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:42 smithi008 ceph-mon[40743]: osdmap e160: 8 total, 8 up, 8 in 2024-02-10T07:36:42.434 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:42 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 1739/217557 objects degraded (0.799%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:36:42.434 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:42 smithi008 ceph-mon[40743]: osdmap e161: 8 total, 8 up, 8 in 2024-02-10T07:36:42.466 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:42.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:42.568 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:42.598 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:42.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:42.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:42 smithi170 ceph-mon[33394]: 4.1 scrub starts 2024-02-10T07:36:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:42 smithi170 ceph-mon[33394]: 4.1 scrub ok 2024-02-10T07:36:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:42 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:36:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:42 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:36:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:42 smithi170 ceph-mon[33394]: osdmap e160: 8 total, 8 up, 8 in 2024-02-10T07:36:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:42 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 1739/217557 objects degraded (0.799%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:36:42.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:42 smithi170 ceph-mon[33394]: osdmap e161: 8 total, 8 up, 8 in 2024-02-10T07:36:42.700 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:42.730 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:42.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:42.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:42.832 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:42.860 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:42.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:42.964 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:42.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:42.996 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:42.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:43.097 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:43.126 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:43.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:43.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:43.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:36:42 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:36:43.227 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:43.256 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:43.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:43.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:43.357 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:43.383 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:43.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:43.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:43.485 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:43.510 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:43.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:43.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:43.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:43 smithi008 ceph-mon[29816]: pgmap v207: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 149 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 551 op/s; 1739/217557 objects degraded (0.799%); 23768/217557 objects misplaced (10.925%); 4.3 MiB/s, 188 objects/s recovering 2024-02-10T07:36:43.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:43 smithi008 ceph-mon[29816]: 5.4 scrub starts 2024-02-10T07:36:43.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:43 smithi008 ceph-mon[29816]: 5.4 scrub ok 2024-02-10T07:36:43.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:43 smithi008 ceph-mon[29816]: osdmap e162: 8 total, 8 up, 8 in 2024-02-10T07:36:43.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:43 smithi008 ceph-mon[40743]: pgmap v207: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 149 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 551 op/s; 1739/217557 objects degraded (0.799%); 23768/217557 objects misplaced (10.925%); 4.3 MiB/s, 188 objects/s recovering 2024-02-10T07:36:43.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:43 smithi008 ceph-mon[40743]: 5.4 scrub starts 2024-02-10T07:36:43.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:43 smithi008 ceph-mon[40743]: 5.4 scrub ok 2024-02-10T07:36:43.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:43 smithi008 ceph-mon[40743]: osdmap e162: 8 total, 8 up, 8 in 2024-02-10T07:36:43.612 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:43 smithi170 ceph-mon[33394]: pgmap v207: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 149 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 551 op/s; 1739/217557 objects degraded (0.799%); 23768/217557 objects misplaced (10.925%); 4.3 MiB/s, 188 objects/s recovering 2024-02-10T07:36:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:43 smithi170 ceph-mon[33394]: 5.4 scrub starts 2024-02-10T07:36:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:43 smithi170 ceph-mon[33394]: 5.4 scrub ok 2024-02-10T07:36:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:43 smithi170 ceph-mon[33394]: osdmap e162: 8 total, 8 up, 8 in 2024-02-10T07:36:43.637 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:43.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:43.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:43.739 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:43.767 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:43.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:43.869 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:43.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:43.897 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:43.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:43.998 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:44.025 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:44.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:44.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:44.127 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:44.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:44.150 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:44.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:44.251 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:44.294 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:44.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:44.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:44.396 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:44.433 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:44.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:44.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:44.535 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:44.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:44 smithi008 ceph-mon[29816]: 6.17 scrub starts 2024-02-10T07:36:44.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:44 smithi008 ceph-mon[29816]: 6.17 scrub ok 2024-02-10T07:36:44.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:44 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:44.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:44 smithi008 ceph-mon[29816]: osdmap e163: 8 total, 8 up, 8 in 2024-02-10T07:36:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:44 smithi008 ceph-mon[40743]: 6.17 scrub starts 2024-02-10T07:36:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:44 smithi008 ceph-mon[40743]: 6.17 scrub ok 2024-02-10T07:36:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:44 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:44 smithi008 ceph-mon[40743]: osdmap e163: 8 total, 8 up, 8 in 2024-02-10T07:36:44.584 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:44.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:44.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:44.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:44 smithi170 ceph-mon[33394]: 6.17 scrub starts 2024-02-10T07:36:44.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:44 smithi170 ceph-mon[33394]: 6.17 scrub ok 2024-02-10T07:36:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:44 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:44 smithi170 ceph-mon[33394]: osdmap e163: 8 total, 8 up, 8 in 2024-02-10T07:36:44.686 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:44.730 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:44.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:44.805 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-10T07:36:44.831 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:44.873 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:44.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:44.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:44.975 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:45.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:45.015 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:45.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:45.116 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:45.122 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:36:45.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:45.140 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:45.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:45.241 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:45.288 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:45.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:45.389 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:45.431 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:45.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:45.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:45 smithi170 ceph-mon[33394]: pgmap v210: 158 pgs: 4 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 149 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 22246/203223 objects misplaced (10.947%); 0 B/s, 221 objects/s recovering 2024-02-10T07:36:45.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:45 smithi170 ceph-mon[33394]: 5.e scrub starts 2024-02-10T07:36:45.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:45 smithi170 ceph-mon[33394]: 5.e scrub ok 2024-02-10T07:36:45.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:45 smithi170 ceph-mon[33394]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1739/217557 objects degraded (0.799%), 1 pg degraded) 2024-02-10T07:36:45.533 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:45.578 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:45.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:45.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:45 smithi008 ceph-mon[29816]: pgmap v210: 158 pgs: 4 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 149 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 22246/203223 objects misplaced (10.947%); 0 B/s, 221 objects/s recovering 2024-02-10T07:36:45.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:45 smithi008 ceph-mon[29816]: 5.e scrub starts 2024-02-10T07:36:45.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:45 smithi008 ceph-mon[29816]: 5.e scrub ok 2024-02-10T07:36:45.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:45 smithi008 ceph-mon[29816]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1739/217557 objects degraded (0.799%), 1 pg degraded) 2024-02-10T07:36:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:45 smithi008 ceph-mon[40743]: pgmap v210: 158 pgs: 4 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 149 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 22246/203223 objects misplaced (10.947%); 0 B/s, 221 objects/s recovering 2024-02-10T07:36:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:45 smithi008 ceph-mon[40743]: 5.e scrub starts 2024-02-10T07:36:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:45 smithi008 ceph-mon[40743]: 5.e scrub ok 2024-02-10T07:36:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:45 smithi008 ceph-mon[40743]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1739/217557 objects degraded (0.799%), 1 pg degraded) 2024-02-10T07:36:45.680 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:45.711 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:45.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:45.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:45.813 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:45.844 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:45.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:45.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:45.946 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:45.995 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:45.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:46.096 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:46.133 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:46.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:46.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:46.235 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:46.279 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:46.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:46 smithi008 ceph-mon[29816]: 3.11 scrub starts 2024-02-10T07:36:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:46 smithi008 ceph-mon[29816]: 3.11 scrub ok 2024-02-10T07:36:46.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:46 smithi008 ceph-mon[40743]: 3.11 scrub starts 2024-02-10T07:36:46.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:46 smithi008 ceph-mon[40743]: 3.11 scrub ok 2024-02-10T07:36:46.380 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:46.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:46.414 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:46.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:46.515 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:46.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:46.555 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:46.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:46.615 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:46 smithi170 ceph-mon[33394]: 3.11 scrub starts 2024-02-10T07:36:46.615 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:46 smithi170 ceph-mon[33394]: 3.11 scrub ok 2024-02-10T07:36:46.640 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:36:46.656 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:46.724 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:46.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:46.825 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:46.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:46.860 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:46.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:46.960 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:47.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:47.003 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:47.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:47.104 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:47.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:47.138 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:47.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:47.160 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:36:47.160 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:36:47.239 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:47.256 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:47 smithi170 ceph-mon[33394]: pgmap v211: 158 pgs: 4 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 149 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s; 22246/203223 objects misplaced (10.947%); 0 B/s, 192 objects/s recovering 2024-02-10T07:36:47.257 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:47 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:47.257 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:47 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:47.277 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:47.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:47.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:47.380 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:47.412 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:47.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:47.513 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:47.556 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:47.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:47.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:47.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:47 smithi008 ceph-mon[29816]: pgmap v211: 158 pgs: 4 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 149 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s; 22246/203223 objects misplaced (10.947%); 0 B/s, 192 objects/s recovering 2024-02-10T07:36:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:47 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:47 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:47.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:47 smithi008 ceph-mon[40743]: pgmap v211: 158 pgs: 4 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 149 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s; 22246/203223 objects misplaced (10.947%); 0 B/s, 192 objects/s recovering 2024-02-10T07:36:47.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:47 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:36:47.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:47 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:36:47.659 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:47.696 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:47.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:47.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:47.798 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:47.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:47.838 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:47.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:47.940 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:47.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:47.974 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:47.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:48.006 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-10T07:36:48.018 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:36:48.076 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:48.128 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:48.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:48.230 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:48.284 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:48.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:48.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:48 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:36:48.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:48 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:36:48.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:48 smithi008 ceph-mon[29816]: 6.8 scrub starts 2024-02-10T07:36:48.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:48 smithi008 ceph-mon[29816]: 6.8 scrub ok 2024-02-10T07:36:48.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:48 smithi008 ceph-mon[29816]: osdmap e164: 8 total, 8 up, 8 in 2024-02-10T07:36:48.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:48 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:36:48.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:48 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:36:48.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:48 smithi008 ceph-mon[40743]: 6.8 scrub starts 2024-02-10T07:36:48.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:48 smithi008 ceph-mon[40743]: 6.8 scrub ok 2024-02-10T07:36:48.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:48 smithi008 ceph-mon[40743]: osdmap e164: 8 total, 8 up, 8 in 2024-02-10T07:36:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:48 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:36:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:48 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:36:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:48 smithi170 ceph-mon[33394]: 6.8 scrub starts 2024-02-10T07:36:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:48 smithi170 ceph-mon[33394]: 6.8 scrub ok 2024-02-10T07:36:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:48 smithi170 ceph-mon[33394]: osdmap e164: 8 total, 8 up, 8 in 2024-02-10T07:36:48.386 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:48.399 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:36:48.427 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:48.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:48.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:48.529 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:48.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:48.563 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:48.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:48.607 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:36:48.664 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:48.697 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:48.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:48.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:48.801 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:48.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:48.825 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:48.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:48.927 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:48.970 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:48.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:49.071 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:49.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:36:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:36:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:36:49.097 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:49.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:49.199 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:49.230 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:49.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:49.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:49.332 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:49.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:49.372 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:49.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:49.473 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:49.518 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:49.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:49.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:49 smithi008 ceph-mon[29816]: pgmap v213: 158 pgs: 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.08k op/s; 22115/198183 objects misplaced (11.159%); 0 B/s, 239 objects/s recovering 2024-02-10T07:36:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:49 smithi008 ceph-mon[29816]: 6.1c scrub starts 2024-02-10T07:36:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:49 smithi008 ceph-mon[29816]: 6.1c scrub ok 2024-02-10T07:36:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:49 smithi008 ceph-mon[40743]: pgmap v213: 158 pgs: 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.08k op/s; 22115/198183 objects misplaced (11.159%); 0 B/s, 239 objects/s recovering 2024-02-10T07:36:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:49 smithi008 ceph-mon[40743]: 6.1c scrub starts 2024-02-10T07:36:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:49 smithi008 ceph-mon[40743]: 6.1c scrub ok 2024-02-10T07:36:49.621 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:49 smithi170 ceph-mon[33394]: pgmap v213: 158 pgs: 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.08k op/s; 22115/198183 objects misplaced (11.159%); 0 B/s, 239 objects/s recovering 2024-02-10T07:36:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:49 smithi170 ceph-mon[33394]: 6.1c scrub starts 2024-02-10T07:36:49.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:49 smithi170 ceph-mon[33394]: 6.1c scrub ok 2024-02-10T07:36:49.666 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:49.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:49.768 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:49.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:49.807 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:49.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:49.908 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:49.938 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:49.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:49.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:50.040 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:50.066 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:50.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:50.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:50.169 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:50.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:50.222 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:50.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:50.323 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:50 smithi008 ceph-mon[29816]: 6.2 deep-scrub starts 2024-02-10T07:36:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:50 smithi008 ceph-mon[29816]: 6.2 deep-scrub ok 2024-02-10T07:36:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:50 smithi008 ceph-mon[40743]: 6.2 deep-scrub starts 2024-02-10T07:36:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:50 smithi008 ceph-mon[40743]: 6.2 deep-scrub ok 2024-02-10T07:36:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:50.361 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:50.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:50.462 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:50.497 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:50.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:50.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:50.599 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:50.627 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:50.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:50.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:50 smithi170 ceph-mon[33394]: 6.2 deep-scrub starts 2024-02-10T07:36:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:50 smithi170 ceph-mon[33394]: 6.2 deep-scrub ok 2024-02-10T07:36:50.664 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:36:50.729 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:50.767 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:36:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:50.802 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:50.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:50.902 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:50.943 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:50.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:50.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:51.045 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:51.090 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:51.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:51.192 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:51.223 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:51.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:51.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:51.325 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:51.370 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:51.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:51.372 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:51 smithi170 ceph-mon[33394]: pgmap v214: 158 pgs: 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 150 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 728 B/s rd, 0 B/s wr, 1.08k op/s; 20844/189828 objects misplaced (10.980%); 0 B/s, 446 objects/s recovering 2024-02-10T07:36:51.472 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:51.517 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:51.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:51.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:51.538 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:51 smithi008 ceph-mon[29816]: pgmap v214: 158 pgs: 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 150 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 728 B/s rd, 0 B/s wr, 1.08k op/s; 20844/189828 objects misplaced (10.980%); 0 B/s, 446 objects/s recovering 2024-02-10T07:36:51.538 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:51 smithi008 ceph-mon[40743]: pgmap v214: 158 pgs: 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 150 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 728 B/s rd, 0 B/s wr, 1.08k op/s; 20844/189828 objects misplaced (10.980%); 0 B/s, 446 objects/s recovering 2024-02-10T07:36:51.551 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:36:51.552 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:36:51.619 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:51.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:51.667 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:51.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:51.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:51.804 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:51.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:51.905 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:51.933 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:51.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:51.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:52.035 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:52.062 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:52.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:52.163 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:52.188 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:52.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:52.241 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-10T07:36:52.289 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:52.316 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:52.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:52.418 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:52.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:52.444 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:52.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:52.506 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:36:52.546 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:52.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:52.570 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:52.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:52.671 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:52.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:52.698 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:52.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:52.745 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.0 to 1 (10000) 2024-02-10T07:36:52.799 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:52.836 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:52.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:52.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:52 smithi170 ceph-mon[33394]: osdmap e165: 8 total, 8 up, 8 in 2024-02-10T07:36:52.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:52 smithi170 ceph-mon[33394]: pgmap v216: 158 pgs: 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 150 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 578 op/s; 20844/189354 objects misplaced (11.008%); 0 B/s, 281 objects/s recovering 2024-02-10T07:36:52.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:52 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2608115666' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:36:52.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:52 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:36:52.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:52 smithi170 ceph-mon[33394]: 4.16 scrub starts 2024-02-10T07:36:52.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:52 smithi170 ceph-mon[33394]: 4.16 scrub ok 2024-02-10T07:36:52.938 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:52.972 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:52.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:53.074 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:53.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:52 smithi008 ceph-mon[40743]: osdmap e165: 8 total, 8 up, 8 in 2024-02-10T07:36:53.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:52 smithi008 ceph-mon[40743]: pgmap v216: 158 pgs: 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 150 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 578 op/s; 20844/189354 objects misplaced (11.008%); 0 B/s, 281 objects/s recovering 2024-02-10T07:36:53.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:52 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2608115666' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:36:53.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:52 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:36:53.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:52 smithi008 ceph-mon[40743]: 4.16 scrub starts 2024-02-10T07:36:53.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:52 smithi008 ceph-mon[40743]: 4.16 scrub ok 2024-02-10T07:36:53.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:52 smithi008 ceph-mon[29816]: osdmap e165: 8 total, 8 up, 8 in 2024-02-10T07:36:53.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:52 smithi008 ceph-mon[29816]: pgmap v216: 158 pgs: 3 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 150 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 578 op/s; 20844/189354 objects misplaced (11.008%); 0 B/s, 281 objects/s recovering 2024-02-10T07:36:53.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:52 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2608115666' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:36:53.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:52 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:36:53.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:52 smithi008 ceph-mon[29816]: 4.16 scrub starts 2024-02-10T07:36:53.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:52 smithi008 ceph-mon[29816]: 4.16 scrub ok 2024-02-10T07:36:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:53.098 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:53.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:53.199 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:53.224 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:53.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:53.325 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:53.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:53.349 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:53.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:53.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:36:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:36:53.450 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:53.474 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:53.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:53.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:53.576 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:53.601 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:53.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:53.703 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:53.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:53.729 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:53.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:53.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:53 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:36:53.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:53 smithi008 ceph-mon[29816]: osdmap e166: 8 total, 8 up, 8 in 2024-02-10T07:36:53.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:53 smithi008 ceph-mon[29816]: 3.13 scrub starts 2024-02-10T07:36:53.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:53 smithi008 ceph-mon[29816]: 3.13 scrub ok 2024-02-10T07:36:53.805 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:53 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:53.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:53 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:36:53.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:53 smithi008 ceph-mon[40743]: osdmap e166: 8 total, 8 up, 8 in 2024-02-10T07:36:53.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:53 smithi008 ceph-mon[40743]: 3.13 scrub starts 2024-02-10T07:36:53.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:53 smithi008 ceph-mon[40743]: 3.13 scrub ok 2024-02-10T07:36:53.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:53 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:53.830 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:53.868 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:53.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:53.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:53.970 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:54.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:54.001 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:54.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:54.028 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:36:54.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:53 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:36:54.046 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:53 smithi170 ceph-mon[33394]: osdmap e166: 8 total, 8 up, 8 in 2024-02-10T07:36:54.046 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:53 smithi170 ceph-mon[33394]: 3.13 scrub starts 2024-02-10T07:36:54.046 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:53 smithi170 ceph-mon[33394]: 3.13 scrub ok 2024-02-10T07:36:54.046 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:53 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:36:54.068 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:36:54.101 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:36:54.102 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:54.125 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:36:54.130 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:54.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:54.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:54.155 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:36:54.176 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:36:54.199 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:36:54.223 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:36:54.225 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:36:54.225 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:36:54.225 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:36:54.225 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:36:54.225 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:36:54.225 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:36:54.225 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:36:54.225 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:36:54.232 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:54.257 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:54.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:54.317 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:36:54.317 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2024-02-10T07:36:54.317 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2024-02-10T07:36:54.317 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2024-02-10T07:36:54.317 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:36:54.360 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:54.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:54.385 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:54.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:54.486 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:54.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:54.511 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:54.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:54.612 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:54.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:54.638 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:54.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:54.739 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:54.768 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:54.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:54.870 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:54.895 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:54.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:54.996 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:55.034 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:55.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:55.066 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:54 smithi008 ceph-mon[29816]: 7.b scrub starts 2024-02-10T07:36:55.066 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:54 smithi008 ceph-mon[29816]: osdmap e167: 8 total, 8 up, 8 in 2024-02-10T07:36:55.067 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:54 smithi008 ceph-mon[29816]: pgmap v219: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 151 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s; 19567/177756 objects misplaced (11.008%); 0 B/s, 608 objects/s recovering 2024-02-10T07:36:55.067 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:54 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3386572995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:36:55.067 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:54 smithi008 ceph-mon[29816]: 7.b scrub ok 2024-02-10T07:36:55.067 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:54 smithi008 ceph-mon[40743]: 7.b scrub starts 2024-02-10T07:36:55.067 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:54 smithi008 ceph-mon[40743]: osdmap e167: 8 total, 8 up, 8 in 2024-02-10T07:36:55.067 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:54 smithi008 ceph-mon[40743]: pgmap v219: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 151 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s; 19567/177756 objects misplaced (11.008%); 0 B/s, 608 objects/s recovering 2024-02-10T07:36:55.067 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:54 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3386572995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:36:55.067 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:54 smithi008 ceph-mon[40743]: 7.b scrub ok 2024-02-10T07:36:55.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:54 smithi170 ceph-mon[33394]: 7.b scrub starts 2024-02-10T07:36:55.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:54 smithi170 ceph-mon[33394]: osdmap e167: 8 total, 8 up, 8 in 2024-02-10T07:36:55.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:54 smithi170 ceph-mon[33394]: pgmap v219: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 151 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s; 19567/177756 objects misplaced (11.008%); 0 B/s, 608 objects/s recovering 2024-02-10T07:36:55.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:54 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3386572995' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:36:55.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:54 smithi170 ceph-mon[33394]: 7.b scrub ok 2024-02-10T07:36:55.129 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-10T07:36:55.136 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:55.209 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:55.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:55.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:55.312 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:55.339 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:55.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:55.440 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:55.474 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:36:55.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:55.489 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:55.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:55.590 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:55.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:55.614 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:55.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:55.715 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:55.740 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:55.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:55.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:55.842 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:55.870 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:55.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:55 smithi170 ceph-mon[33394]: osdmap e168: 8 total, 8 up, 8 in 2024-02-10T07:36:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:55 smithi170 ceph-mon[33394]: 6.15 scrub starts 2024-02-10T07:36:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:55 smithi170 ceph-mon[33394]: 6.15 scrub ok 2024-02-10T07:36:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:55 smithi170 ceph-mon[33394]: 3.b scrub starts 2024-02-10T07:36:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:55 smithi170 ceph-mon[33394]: 3.b scrub ok 2024-02-10T07:36:55.972 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:56.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:56.003 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:56.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:56.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:55 smithi008 ceph-mon[29816]: osdmap e168: 8 total, 8 up, 8 in 2024-02-10T07:36:56.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:55 smithi008 ceph-mon[29816]: 6.15 scrub starts 2024-02-10T07:36:56.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:55 smithi008 ceph-mon[29816]: 6.15 scrub ok 2024-02-10T07:36:56.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:55 smithi008 ceph-mon[29816]: 3.b scrub starts 2024-02-10T07:36:56.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:55 smithi008 ceph-mon[29816]: 3.b scrub ok 2024-02-10T07:36:56.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:55 smithi008 ceph-mon[40743]: osdmap e168: 8 total, 8 up, 8 in 2024-02-10T07:36:56.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:55 smithi008 ceph-mon[40743]: 6.15 scrub starts 2024-02-10T07:36:56.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:55 smithi008 ceph-mon[40743]: 6.15 scrub ok 2024-02-10T07:36:56.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:55 smithi008 ceph-mon[40743]: 3.b scrub starts 2024-02-10T07:36:56.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:55 smithi008 ceph-mon[40743]: 3.b scrub ok 2024-02-10T07:36:56.104 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:56.143 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:56.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:56.244 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:56.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:56.287 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:56.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:56.388 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:56.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:56.420 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:56.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:56.521 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:56.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:56.567 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:56.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:56.669 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:56.717 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:56.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:56.818 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:56.851 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:56.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:56.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:56.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:56 smithi170 ceph-mon[33394]: pgmap v221: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 151 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 995 op/s; 19567/177756 objects misplaced (11.008%); 0 B/s, 481 objects/s recovering 2024-02-10T07:36:56.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:56 smithi170 ceph-mon[33394]: 4.13 scrub starts 2024-02-10T07:36:56.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:56 smithi170 ceph-mon[33394]: 4.13 scrub ok 2024-02-10T07:36:56.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:56 smithi170 ceph-mon[33394]: osdmap e169: 8 total, 8 up, 8 in 2024-02-10T07:36:56.953 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:56.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:56.994 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:56.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:57.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:56 smithi008 ceph-mon[29816]: pgmap v221: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 151 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 995 op/s; 19567/177756 objects misplaced (11.008%); 0 B/s, 481 objects/s recovering 2024-02-10T07:36:57.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:56 smithi008 ceph-mon[29816]: 4.13 scrub starts 2024-02-10T07:36:57.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:56 smithi008 ceph-mon[29816]: 4.13 scrub ok 2024-02-10T07:36:57.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:56 smithi008 ceph-mon[29816]: osdmap e169: 8 total, 8 up, 8 in 2024-02-10T07:36:57.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:56 smithi008 ceph-mon[40743]: pgmap v221: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 151 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 995 op/s; 19567/177756 objects misplaced (11.008%); 0 B/s, 481 objects/s recovering 2024-02-10T07:36:57.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:56 smithi008 ceph-mon[40743]: 4.13 scrub starts 2024-02-10T07:36:57.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:56 smithi008 ceph-mon[40743]: 4.13 scrub ok 2024-02-10T07:36:57.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:56 smithi008 ceph-mon[40743]: osdmap e169: 8 total, 8 up, 8 in 2024-02-10T07:36:57.095 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:57.133 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:57.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:57.235 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:36:57.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:57.281 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:57.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:36:57.363 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 1 1 2024-02-10T07:36:57.382 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:57.412 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:57.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:57.514 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:57.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:57.549 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:57.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:57.650 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:57.713 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:57.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:57.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:57.815 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:57.853 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:57.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:57.877 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:57 smithi008 ceph-mon[29816]: 3.1c scrub starts 2024-02-10T07:36:57.877 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:57 smithi008 ceph-mon[29816]: 3.1c scrub ok 2024-02-10T07:36:57.877 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:57 smithi008 ceph-mon[29816]: 3.2 scrub starts 2024-02-10T07:36:57.877 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:57 smithi008 ceph-mon[29816]: 3.2 scrub ok 2024-02-10T07:36:57.878 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:57 smithi008 ceph-mon[40743]: 3.1c scrub starts 2024-02-10T07:36:57.878 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:57 smithi008 ceph-mon[40743]: 3.1c scrub ok 2024-02-10T07:36:57.878 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:57 smithi008 ceph-mon[40743]: 3.2 scrub starts 2024-02-10T07:36:57.878 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:57 smithi008 ceph-mon[40743]: 3.2 scrub ok 2024-02-10T07:36:57.953 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:57.982 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:57.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:57.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:57.987 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:57 smithi170 ceph-mon[33394]: 3.1c scrub starts 2024-02-10T07:36:57.987 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:57 smithi170 ceph-mon[33394]: 3.1c scrub ok 2024-02-10T07:36:57.987 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:57 smithi170 ceph-mon[33394]: 3.2 scrub starts 2024-02-10T07:36:57.987 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:57 smithi170 ceph-mon[33394]: 3.2 scrub ok 2024-02-10T07:36:58.037 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:36:58.084 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:58.116 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:58.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:58.163 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:36:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2024-02-10T07:36:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2024-02-10T07:36:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:36:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:45487 7.7 7:e2c2b0c4:::benchmark_data_smithi008_82046_object45486:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-02-10T07:36:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:34:36.379384+0000", 2024-02-10T07:36:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 141.779242799, 2024-02-10T07:36:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.14030762499999999, 2024-02-10T07:36:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.164 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 45487 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.379384+0000", 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.379384+0000", 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.165 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.379386+0000", 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5600000000000001e-06 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.379429+0000", 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3568e-05 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.379445+0000", 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6245000000000001e-05 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.166 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.379454+0000", 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.8619999999999992e-06 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.379516+0000", 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1692000000000005e-05 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.379673+0000", 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015672400000000001 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,3", 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.379831+0000", 2024-02-10T07:36:58.167 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000158073 2024-02-10T07:36:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.381494+0000", 2024-02-10T07:36:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001662813 2024-02-10T07:36:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.509290+0000", 2024-02-10T07:36:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12779623100000001 2024-02-10T07:36:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.168 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.519617+0000", 2024-02-10T07:36:58.169 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010327443 2024-02-10T07:36:58.169 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.169 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.169 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.169 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.519655+0000", 2024-02-10T07:36:58.169 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7867000000000002e-05 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.519692+0000", 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6547000000000001e-05 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:45497 7.d 7:b45e73d4:::benchmark_data_smithi008_82046_object45496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:34:36.384508+0000", 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 141.77411879499999, 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.1248157, 2024-02-10T07:36:58.170 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 45497 2024-02-10T07:36:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.384508+0000", 2024-02-10T07:36:58.171 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.384508+0000", 2024-02-10T07:36:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.384510+0000", 2024-02-10T07:36:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6869999999999999e-06 2024-02-10T07:36:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.172 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.384563+0000", 2024-02-10T07:36:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3322999999999998e-05 2024-02-10T07:36:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.384578+0000", 2024-02-10T07:36:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5228000000000001e-05 2024-02-10T07:36:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.384588+0000", 2024-02-10T07:36:58.173 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.7899999999999994e-06 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.384654+0000", 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5468000000000004e-05 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.384843+0000", 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00018956800000000001 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.385038+0000", 2024-02-10T07:36:58.174 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000194372 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.386717+0000", 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0016793310000000001 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.386756+0000", 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9227999999999998e-05 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.509256+0000", 2024-02-10T07:36:58.175 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.122500149 2024-02-10T07:36:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.509299+0000", 2024-02-10T07:36:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2920999999999998e-05 2024-02-10T07:36:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.509324+0000", 2024-02-10T07:36:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4635e-05 2024-02-10T07:36:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.176 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:45499 7.c 7:33d477c7:::benchmark_data_smithi008_82046_object45498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-02-10T07:36:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:34:36.386514+0000", 2024-02-10T07:36:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 141.77211269, 2024-02-10T07:36:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.13799698799999999, 2024-02-10T07:36:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 45499 2024-02-10T07:36:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.177 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.386514+0000", 2024-02-10T07:36:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.386514+0000", 2024-02-10T07:36:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.178 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.179 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.179 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.386515+0000", 2024-02-10T07:36:58.179 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3900000000000004e-07 2024-02-10T07:36:58.180 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.180 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.180 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.180 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.386570+0000", 2024-02-10T07:36:58.180 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5421999999999999e-05 2024-02-10T07:36:58.180 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.180 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.180 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.386582+0000", 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1484999999999999e-05 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.386588+0000", 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.302e-06 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.386630+0000", 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2039999999999997e-05 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.181 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.386721+0000", 2024-02-10T07:36:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0635000000000002e-05 2024-02-10T07:36:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:36:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.386820+0000", 2024-02-10T07:36:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9020999999999999e-05 2024-02-10T07:36:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.388203+0000", 2024-02-10T07:36:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0013830069999999999 2024-02-10T07:36:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.182 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.183 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.183 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.388266+0000", 2024-02-10T07:36:58.183 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3439e-05 2024-02-10T07:36:58.183 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.183 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.183 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.183 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.524408+0000", 2024-02-10T07:36:58.183 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.13614164300000001 2024-02-10T07:36:58.184 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.184 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.184 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.184 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.524473+0000", 2024-02-10T07:36:58.184 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4708000000000005e-05 2024-02-10T07:36:58.185 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.185 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.185 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.185 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:34:36.524511+0000", 2024-02-10T07:36:58.185 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8447e-05 2024-02-10T07:36:58.185 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:84223 7.7 7:e4ba1dda:::benchmark_data_smithi008_82046_object84222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-10T07:36:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:35:08.988946+0000", 2024-02-10T07:36:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 109.16968100299999, 2024-02-10T07:36:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.13621433299999999, 2024-02-10T07:36:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.186 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.187 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.187 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.187 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 84223 2024-02-10T07:36:58.187 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.187 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.187 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.187 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.187 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.988946+0000", 2024-02-10T07:36:58.187 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.188 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.188 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.188 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.988946+0000", 2024-02-10T07:36:58.188 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.188 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.188 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.988947+0000", 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.237e-06 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.988995+0000", 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8168999999999999e-05 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.989007+0000", 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1362000000000001e-05 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.189 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.989012+0000", 2024-02-10T07:36:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4709999999999998e-06 2024-02-10T07:36:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.989063+0000", 2024-02-10T07:36:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0717999999999998e-05 2024-02-10T07:36:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.190 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.989178+0000", 2024-02-10T07:36:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000114879 2024-02-10T07:36:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,3", 2024-02-10T07:36:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.989264+0000", 2024-02-10T07:36:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5925000000000001e-05 2024-02-10T07:36:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.991754+0000", 2024-02-10T07:36:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0024905930000000001 2024-02-10T07:36:58.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.192 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.192 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.192 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.001634+0000", 2024-02-10T07:36:58.192 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0098796130000000006 2024-02-10T07:36:58.192 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.192 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.192 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.192 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.125092+0000", 2024-02-10T07:36:58.192 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.123458072 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.125133+0000", 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1001000000000003e-05 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.125160+0000", 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7293000000000001e-05 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.193 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:84230 7.7 7:e1144ae5:::benchmark_data_smithi008_82046_object84229:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:35:08.994378+0000", 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 109.164249286, 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.133198963, 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 84230 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.994378+0000", 2024-02-10T07:36:58.194 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.994378+0000", 2024-02-10T07:36:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.994379+0000", 2024-02-10T07:36:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.314e-06 2024-02-10T07:36:58.195 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.196 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.196 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.196 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.994422+0000", 2024-02-10T07:36:58.197 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2928e-05 2024-02-10T07:36:58.197 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.197 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.197 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.197 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.994437+0000", 2024-02-10T07:36:58.197 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4775e-05 2024-02-10T07:36:58.197 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.197 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.197 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.197 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.994446+0000", 2024-02-10T07:36:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.9190000000000005e-06 2024-02-10T07:36:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.994499+0000", 2024-02-10T07:36:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3171000000000003e-05 2024-02-10T07:36:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.994615+0000", 2024-02-10T07:36:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000115942 2024-02-10T07:36:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.198 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,3", 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.994749+0000", 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013384200000000001 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.997697+0000", 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0029485729999999999 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.003995+0000", 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0062980529999999996 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.199 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.127490+0000", 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.123495032 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.127553+0000", 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2403000000000005e-05 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.127577+0000", 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4011000000000001e-05 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.200 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:84235 7.7 7:ed3b8013:::benchmark_data_smithi008_82046_object84234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-10T07:36:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:35:08.998002+0000", 2024-02-10T07:36:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 109.160624912, 2024-02-10T07:36:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.13261448000000001, 2024-02-10T07:36:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.201 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.202 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.202 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 84235 2024-02-10T07:36:58.202 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.202 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.202 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.202 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.202 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.998002+0000", 2024-02-10T07:36:58.202 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.202 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.203 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.203 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.203 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.998002+0000", 2024-02-10T07:36:58.203 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.203 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.203 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.203 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.204 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.998003+0000", 2024-02-10T07:36:58.204 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.446e-06 2024-02-10T07:36:58.204 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.204 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.204 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.204 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.998056+0000", 2024-02-10T07:36:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2726999999999998e-05 2024-02-10T07:36:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.998066+0000", 2024-02-10T07:36:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0261999999999999e-05 2024-02-10T07:36:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.998072+0000", 2024-02-10T07:36:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0299999999999999e-06 2024-02-10T07:36:58.205 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.998141+0000", 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8176999999999995e-05 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.998252+0000", 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011110199999999999 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,3", 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:08.998398+0000", 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000145833 2024-02-10T07:36:58.206 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.001701+0000", 2024-02-10T07:36:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033031269999999999 2024-02-10T07:36:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.004251+0000", 2024-02-10T07:36:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0025498830000000002 2024-02-10T07:36:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.130517+0000", 2024-02-10T07:36:58.207 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.126266092 2024-02-10T07:36:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.130578+0000", 2024-02-10T07:36:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0973000000000001e-05 2024-02-10T07:36:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.130616+0000", 2024-02-10T07:36:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8828000000000002e-05 2024-02-10T07:36:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.208 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.209 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.209 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.209 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.209 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:84241 7.c 7:3c763e1a:::benchmark_data_smithi008_82046_object84240:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-10T07:36:58.209 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:35:09.003506+0000", 2024-02-10T07:36:58.209 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 109.15512098799999, 2024-02-10T07:36:58.209 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11052640399999999, 2024-02-10T07:36:58.210 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.210 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.210 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.210 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.210 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.210 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 84241 2024-02-10T07:36:58.210 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.211 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.211 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.211 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.211 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.003506+0000", 2024-02-10T07:36:58.211 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.211 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.211 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.003506+0000", 2024-02-10T07:36:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.212 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.213 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.003507+0000", 2024-02-10T07:36:58.213 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2729999999999999e-06 2024-02-10T07:36:58.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.213 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.213 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.213 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.003579+0000", 2024-02-10T07:36:58.213 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.1673000000000003e-05 2024-02-10T07:36:58.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.213 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.003592+0000", 2024-02-10T07:36:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2709e-05 2024-02-10T07:36:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.003598+0000", 2024-02-10T07:36:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9020000000000002e-06 2024-02-10T07:36:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.003649+0000", 2024-02-10T07:36:58.214 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0129000000000001e-05 2024-02-10T07:36:58.215 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.215 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.215 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.215 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.003750+0000", 2024-02-10T07:36:58.215 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000101229 2024-02-10T07:36:58.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.216 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.216 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:36:58.216 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.003840+0000", 2024-02-10T07:36:58.216 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0514999999999997e-05 2024-02-10T07:36:58.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.216 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.216 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.006485+0000", 2024-02-10T07:36:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002644211 2024-02-10T07:36:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.006591+0000", 2024-02-10T07:36:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010682400000000001 2024-02-10T07:36:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.113944+0000", 2024-02-10T07:36:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.10735246699999999 2024-02-10T07:36:58.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.114003+0000", 2024-02-10T07:36:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8674000000000001e-05 2024-02-10T07:36:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.114032+0000", 2024-02-10T07:36:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9798000000000002e-05 2024-02-10T07:36:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.218 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:84242 7.7 7:ee010f5e:::benchmark_data_smithi008_82046_object84241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:35:09.004481+0000", 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 109.15414602600001, 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12900229099999999, 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 84242 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.219 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.004481+0000", 2024-02-10T07:36:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.004481+0000", 2024-02-10T07:36:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.004482+0000", 2024-02-10T07:36:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5000000000000002e-07 2024-02-10T07:36:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.220 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.004540+0000", 2024-02-10T07:36:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7921999999999999e-05 2024-02-10T07:36:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.004550+0000", 2024-02-10T07:36:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0097e-05 2024-02-10T07:36:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.004555+0000", 2024-02-10T07:36:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5929999999999996e-06 2024-02-10T07:36:58.221 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.004602+0000", 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6254999999999997e-05 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.004726+0000", 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000124604 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,3", 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.004865+0000", 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013867799999999999 2024-02-10T07:36:58.222 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.007206+0000", 2024-02-10T07:36:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0023414970000000001 2024-02-10T07:36:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.113958+0000", 2024-02-10T07:36:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.106751336 2024-02-10T07:36:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.133408+0000", 2024-02-10T07:36:58.223 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019450469000000001 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.133445+0000", 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6689000000000001e-05 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:09.133483+0000", 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8401e-05 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:127379 7.d 7:b624a2e6:::benchmark_data_smithi008_82046_object12241:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:35:57.301980+0000", 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 60.856646505000001, 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.110337113, 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 127379 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.225 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:57.301980+0000", 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:57.301980+0000", 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:57.301982+0000", 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.252e-06 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.226 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:57.301987+0000", 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2129999999999999e-06 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:57.302003+0000", 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5756e-05 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:57.302011+0000", 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8709999999999997e-06 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.227 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:57.302084+0000", 2024-02-10T07:36:58.228 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3813999999999995e-05 2024-02-10T07:36:58.228 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.228 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.228 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.228 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:57.305166+0000", 2024-02-10T07:36:58.228 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0030817309999999999 2024-02-10T07:36:58.228 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.228 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.228 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:36:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:57.305312+0000", 2024-02-10T07:36:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014605400000000001 2024-02-10T07:36:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:57.308814+0000", 2024-02-10T07:36:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0035023960000000001 2024-02-10T07:36:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.229 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:57.309304+0000", 2024-02-10T07:36:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00048980300000000003 2024-02-10T07:36:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:57.412194+0000", 2024-02-10T07:36:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.102889522 2024-02-10T07:36:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:57.412277+0000", 2024-02-10T07:36:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2706000000000004e-05 2024-02-10T07:36:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.230 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:35:57.412318+0000", 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0995000000000003e-05 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:133136 7.7 7:e02d8470:::benchmark_data_smithi008_82046_object17998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:08.110007+0000", 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 50.048620256, 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12975399500000001, 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.231 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 133136 2024-02-10T07:36:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:08.110007+0000", 2024-02-10T07:36:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.233 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.235 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.235 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:08.110007+0000", 2024-02-10T07:36:58.235 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:08.110008+0000", 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.562e-06 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:08.110012+0000", 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.07e-06 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:08.110023+0000", 2024-02-10T07:36:58.236 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0638e-05 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:08.110028+0000", 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4970000000000001e-06 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:08.110076+0000", 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7583000000000002e-05 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:08.110189+0000", 2024-02-10T07:36:58.237 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011281900000000001 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,3", 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:08.110478+0000", 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00028872099999999998 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:08.111737+0000", 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0012591130000000001 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:08.198752+0000", 2024-02-10T07:36:58.238 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.087015044 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:08.239652+0000", 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.040900451999999997 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:08.239710+0000", 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7513000000000003e-05 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:08.239761+0000", 2024-02-10T07:36:58.239 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0983000000000003e-05 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:133971 7.d 7:b90d9b42:::benchmark_data_smithi008_82046_object18833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:09.014490+0000", 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 49.144136502000002, 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.113155511, 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.240 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 133971 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:09.014490+0000", 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:09.014490+0000", 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:09.014492+0000", 2024-02-10T07:36:58.241 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2279999999999999e-06 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:09.014496+0000", 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.758e-06 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:09.014508+0000", 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1270999999999999e-05 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:09.014515+0000", 2024-02-10T07:36:58.242 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.401e-06 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:09.014578+0000", 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3107000000000004e-05 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:09.014695+0000", 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011658399999999999 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:09.014776+0000", 2024-02-10T07:36:58.243 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1703e-05 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:09.017260+0000", 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002483477 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:09.023322+0000", 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0060617529999999996 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:09.127536+0000", 2024-02-10T07:36:58.244 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.104214293 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:09.127601+0000", 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4516000000000004e-05 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:09.127646+0000", 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5420000000000002e-05 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.245 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:149716 7.d 7:bc9d36fa:::benchmark_data_smithi008_82046_object34578:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:27.887801+0000", 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 30.270825697999999, 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.113076144, 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 149716 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.887801+0000", 2024-02-10T07:36:58.246 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.887801+0000", 2024-02-10T07:36:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.887802+0000", 2024-02-10T07:36:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2529999999999999e-06 2024-02-10T07:36:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.247 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.887806+0000", 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6279999999999998e-06 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.887816+0000", 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0111e-05 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.887824+0000", 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5320000000000001e-06 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.248 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.887872+0000", 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8736000000000002e-05 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.887958+0000", 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5561000000000003e-05 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.888033+0000", 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4788000000000003e-05 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.249 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.890227+0000", 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002194437 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.893023+0000", 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0027953079999999998 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:28.000817+0000", 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.107794265 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.250 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:28.000851+0000", 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4359000000000001e-05 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:28.000877+0000", 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6166e-05 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:149720 7.d 7:b9e1504f:::benchmark_data_smithi008_82046_object34582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:27.890384+0000", 2024-02-10T07:36:58.251 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 30.268242961999999, 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11061354399999999, 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 149720 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.890384+0000", 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.252 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.890384+0000", 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.890385+0000", 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4199999999999995e-07 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.890388+0000", 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.523e-06 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.253 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.890399+0000", 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0756e-05 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.890405+0000", 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9009999999999999e-06 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.890442+0000", 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6869000000000002e-05 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.254 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.890552+0000", 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000110128 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.890678+0000", 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012625099999999999 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.893039+0000", 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0023610430000000002 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:27.893491+0000", 2024-02-10T07:36:58.255 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000451953 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:28.000916+0000", 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.10742500100000001 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:28.000965+0000", 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8562e-05 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:28.000997+0000", 2024-02-10T07:36:58.256 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2914999999999999e-05 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:154796 7.7 7:e78e2e7b:::benchmark_data_smithi008_82046_object39658:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:32.732902+0000", 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 25.425725193000002, 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.122669376, 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 154796 2024-02-10T07:36:58.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.258 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:58.259 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.259 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.259 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.259 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.732902+0000", 2024-02-10T07:36:58.259 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.259 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.732902+0000", 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.732903+0000", 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.212e-06 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.732907+0000", 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9890000000000003e-06 2024-02-10T07:36:58.260 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.732918+0000", 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.061e-05 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.732923+0000", 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.857e-06 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.732967+0000", 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3545999999999999e-05 2024-02-10T07:36:58.261 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.733062+0000", 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4581999999999997e-05 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,3", 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.733351+0000", 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000289914 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.737923+0000", 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0045715169999999998 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.262 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.740647+0000", 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0027237310000000001 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.855481+0000", 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.114834638 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.855532+0000", 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1084000000000001e-05 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.263 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.855571+0000", 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8695999999999997e-05 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:154799 7.d 7:b874f55b:::benchmark_data_smithi008_82046_object39661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:32.737101+0000", 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 25.421525888000001, 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.116901802, 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.264 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 154799 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.737101+0000", 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.737101+0000", 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.265 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.737103+0000", 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6330000000000001e-06 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.737109+0000", 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3990000000000001e-06 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.737128+0000", 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8538000000000001e-05 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.266 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.737138+0000", 2024-02-10T07:36:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0186999999999999e-05 2024-02-10T07:36:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.737219+0000", 2024-02-10T07:36:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1366999999999995e-05 2024-02-10T07:36:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.737360+0000", 2024-02-10T07:36:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014122000000000001 2024-02-10T07:36:58.267 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.737488+0000", 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012805600000000001 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.740694+0000", 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0032051480000000001 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.853768+0000", 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.113074493 2024-02-10T07:36:58.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.853911+0000", 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014273000000000001 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.853962+0000", 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1437000000000002e-05 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.854003+0000", 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0593999999999998e-05 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.269 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:154802 7.c 7:342edd0c:::benchmark_data_smithi008_82046_object39664:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:32.742020+0000", 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 25.416607349, 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11984452800000001, 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 154802 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.270 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742020+0000", 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742020+0000", 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742021+0000", 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.53e-06 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.271 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742025+0000", 2024-02-10T07:36:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0419999999999996e-06 2024-02-10T07:36:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742039+0000", 2024-02-10T07:36:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4219000000000001e-05 2024-02-10T07:36:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742049+0000", 2024-02-10T07:36:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.2070000000000001e-06 2024-02-10T07:36:58.272 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742119+0000", 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0715000000000003e-05 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742278+0000", 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015845200000000001 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742404+0000", 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012643999999999999 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.273 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.850521+0000", 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.10811643899999999 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.851737+0000", 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0012166259999999999 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.861702+0000", 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0099648170000000008 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.861823+0000", 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000121395 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.861864+0000", 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0645999999999997e-05 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.24740.0:154803 7.b e155/140 7:dde2e294:::benchmark_data_smithi008_82046_object39665:head v 155'9739, mlcod=155'9733)", 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:32.742809+0000", 2024-02-10T07:36:58.275 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 25.415817711999999, 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.107852243, 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742809+0000", 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742809+0000", 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742810+0000", 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.1999999999999999e-07 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742817+0000", 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9979999999999997e-06 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742835+0000", 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7748e-05 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742842+0000", 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0299999999999996e-06 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742900+0000", 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8295999999999999e-05 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.742950+0000", 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0095000000000001e-05 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.278 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.850606+0000", 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.107655796 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.850661+0000", 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5560000000000003e-05 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:171766 7.c 7:3b5c76f9:::benchmark_data_smithi008_82046_object56628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:54.822144+0000", 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 3.3364828489999998, 2024-02-10T07:36:58.279 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.116187627, 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 171766 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.822144+0000", 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.280 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.822144+0000", 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.822145+0000", 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5300000000000003e-07 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.822148+0000", 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4620000000000001e-06 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.281 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.822157+0000", 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6449999999999994e-06 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.822162+0000", 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4500000000000003e-06 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.822209+0000", 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6594999999999997e-05 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.822282+0000", 2024-02-10T07:36:58.282 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3305000000000004e-05 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.822346+0000", 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4097999999999995e-05 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.825944+0000", 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0035970820000000001 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.828824+0000", 2024-02-10T07:36:58.283 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002880615 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.938210+0000", 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.109385522 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.938279+0000", 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9024999999999997e-05 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.938332+0000", 2024-02-10T07:36:58.284 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2975e-05 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:171770 7.c 7:34eada2b:::benchmark_data_smithi008_82046_object56632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:54.825978+0000", 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 3.332648823, 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.112462543, 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 171770 2024-02-10T07:36:58.285 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.825978+0000", 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.825978+0000", 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.825979+0000", 2024-02-10T07:36:58.286 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0750000000000001e-06 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.825983+0000", 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8759999999999998e-06 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.825992+0000", 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6819999999999992e-06 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.825999+0000", 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.1389999999999998e-06 2024-02-10T07:36:58.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.826094+0000", 2024-02-10T07:36:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5056999999999995e-05 2024-02-10T07:36:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.826205+0000", 2024-02-10T07:36:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011130500000000001 2024-02-10T07:36:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:36:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.826322+0000", 2024-02-10T07:36:58.288 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011631 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.828927+0000", 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0026053109999999999 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.829284+0000", 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00035741000000000001 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.938368+0000", 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.109084203 2024-02-10T07:36:58.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.938403+0000", 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4547e-05 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.938441+0000", 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7628e-05 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.290 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:171776 7.c 7:36a9ef93:::benchmark_data_smithi008_82046_object56638:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:54.835417+0000", 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 3.32320985, 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.117467587, 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 171776 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.835417+0000", 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.835417+0000", 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.835418+0000", 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.004e-06 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.835423+0000", 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.493e-06 2024-02-10T07:36:58.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.835435+0000", 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1997999999999999e-05 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.835442+0000", 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9310000000000008e-06 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.835492+0000", 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9051999999999997e-05 2024-02-10T07:36:58.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.835579+0000", 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7103999999999998e-05 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.835655+0000", 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.682e-05 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.838401+0000", 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002745623 2024-02-10T07:36:58.294 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.839182+0000", 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00078137699999999998 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.952816+0000", 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11363324 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.952856+0000", 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9925999999999997e-05 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:36:58.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:36:58.296 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:36:58.296 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:54.952885+0000", 2024-02-10T07:36:58.296 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9019000000000001e-05 2024-02-10T07:36:58.296 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.296 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.296 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.296 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:36:58.296 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:36:58.296 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:36:58.307 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:58.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:58.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:58.409 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:58.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:58.455 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:58.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:58.556 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:36:58.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:58.591 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:58.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:36:58.692 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:58.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:58.725 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:58.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:58.800 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:36:58.826 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:36:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:58.866 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:58.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:36:58.915 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:36:58.967 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:58.996 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:58.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:59.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:36:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:36:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:36:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:58 smithi008 ceph-mon[29816]: pgmap v223: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 151 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1015 B/s rd, 0 B/s wr, 751 op/s; 19444/173739 objects misplaced (11.191%); 0 B/s, 386 objects/s recovering 2024-02-10T07:36:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:58 smithi008 ceph-mon[29816]: 2.0 scrub starts 2024-02-10T07:36:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:58 smithi008 ceph-mon[29816]: 2.0 scrub ok 2024-02-10T07:36:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:58 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3714265236' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:36:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:58 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:36:59.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:58 smithi008 ceph-mon[40743]: pgmap v223: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 151 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1015 B/s rd, 0 B/s wr, 751 op/s; 19444/173739 objects misplaced (11.191%); 0 B/s, 386 objects/s recovering 2024-02-10T07:36:59.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:58 smithi008 ceph-mon[40743]: 2.0 scrub starts 2024-02-10T07:36:59.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:58 smithi008 ceph-mon[40743]: 2.0 scrub ok 2024-02-10T07:36:59.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:58 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3714265236' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:36:59.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:58 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:36:59.098 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:59.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:59.126 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:59.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:59.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:58 smithi170 ceph-mon[33394]: pgmap v223: 158 pgs: 1 peering, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_toofull, 151 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1015 B/s rd, 0 B/s wr, 751 op/s; 19444/173739 objects misplaced (11.191%); 0 B/s, 386 objects/s recovering 2024-02-10T07:36:59.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:58 smithi170 ceph-mon[33394]: 2.0 scrub starts 2024-02-10T07:36:59.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:58 smithi170 ceph-mon[33394]: 2.0 scrub ok 2024-02-10T07:36:59.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:58 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3714265236' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:36:59.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:58 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:36:59.227 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:36:59.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:59.257 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:59.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:36:59.335 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-10T07:36:59.358 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:36:59.396 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:59.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:36:59.498 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:36:59.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:59.542 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:36:59.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:36:59.634 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:36:59.643 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:59.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:59.668 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:59.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:36:59.769 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:36:59.799 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:59.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:36:59.901 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:36:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:36:59.925 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:36:59.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:00.026 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:00.068 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:00.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:00.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:59 smithi008 ceph-mon[40743]: 4.f scrub starts 2024-02-10T07:37:00.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:59 smithi008 ceph-mon[40743]: 4.f scrub ok 2024-02-10T07:37:00.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:59 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:37:00.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:59 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:37:00.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:36:59 smithi008 ceph-mon[40743]: osdmap e170: 8 total, 8 up, 8 in 2024-02-10T07:37:00.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:59 smithi008 ceph-mon[29816]: 4.f scrub starts 2024-02-10T07:37:00.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:59 smithi008 ceph-mon[29816]: 4.f scrub ok 2024-02-10T07:37:00.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:59 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:37:00.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:59 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:37:00.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:36:59 smithi008 ceph-mon[29816]: osdmap e170: 8 total, 8 up, 8 in 2024-02-10T07:37:00.111 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:59 smithi170 ceph-mon[33394]: 4.f scrub starts 2024-02-10T07:37:00.111 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:59 smithi170 ceph-mon[33394]: 4.f scrub ok 2024-02-10T07:37:00.111 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:59 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:37:00.111 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:59 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:37:00.112 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:36:59 smithi170 ceph-mon[33394]: osdmap e170: 8 total, 8 up, 8 in 2024-02-10T07:37:00.169 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:00.214 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:00.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:00.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:00.316 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:00.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:00.374 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:00.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:00.475 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:00.518 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:00.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:00.620 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:00.661 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:00.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:00.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:00.763 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:00.806 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:00.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:00.866 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:00 smithi008 ceph-mon[40743]: pgmap v225: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 626 op/s; 17741/166479 objects misplaced (10.657%); 2.9 MiB/s, 91 objects/s recovering 2024-02-10T07:37:00.867 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:00 smithi008 ceph-mon[29816]: pgmap v225: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 626 op/s; 17741/166479 objects misplaced (10.657%); 2.9 MiB/s, 91 objects/s recovering 2024-02-10T07:37:00.907 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:00.940 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:00.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:00.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:01.042 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:01.095 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:01.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:01.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:00 smithi170 ceph-mon[33394]: pgmap v225: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 626 op/s; 17741/166479 objects misplaced (10.657%); 2.9 MiB/s, 91 objects/s recovering 2024-02-10T07:37:01.196 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:01.228 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:01.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:01.330 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:01.369 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:01.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:01.470 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:01.513 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:01.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:01.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:01.615 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:01.639 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:01.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:01.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:01.741 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:01.781 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:01.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:01.882 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:01.924 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:01.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:01.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:02.026 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:02.054 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:02.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:02.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:02.071 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:02.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:01 smithi008 ceph-mon[40743]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:37:02.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:01 smithi008 ceph-mon[40743]: osdmap e171: 8 total, 8 up, 8 in 2024-02-10T07:37:02.084 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:01 smithi008 ceph-mon[29816]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:37:02.084 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:01 smithi008 ceph-mon[29816]: osdmap e171: 8 total, 8 up, 8 in 2024-02-10T07:37:02.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:01 smithi170 ceph-mon[33394]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:37:02.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:01 smithi170 ceph-mon[33394]: osdmap e171: 8 total, 8 up, 8 in 2024-02-10T07:37:02.157 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:02.184 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:02.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:02.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:02.286 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:02.339 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:02.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:02.441 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:02.471 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:02.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:02.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:02.573 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:02.599 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:02.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:02.642 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:37:02.643 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2024-02-10T07:37:02.643 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2024-02-10T07:37:02.643 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:37:02.702 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:02.743 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:02.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:02.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:02.845 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:02.890 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:02.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:02.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:02.913 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:37:02.991 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:03.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:03.061 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:03.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:03.162 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:03.190 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:03.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:03.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:03.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:37:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:37:03.217 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:03 smithi170 ceph-mon[33394]: pgmap v227: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 650 op/s; 17741/166056 objects misplaced (10.684%); 2.9 MiB/s, 91 objects/s recovering 2024-02-10T07:37:03.217 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:03.217 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:03.292 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:03.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:03.321 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:03.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:03.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:03 smithi008 ceph-mon[29816]: pgmap v227: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 650 op/s; 17741/166056 objects misplaced (10.684%); 2.9 MiB/s, 91 objects/s recovering 2024-02-10T07:37:03.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:03.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:03.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:03 smithi008 ceph-mon[40743]: pgmap v227: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 650 op/s; 17741/166056 objects misplaced (10.684%); 2.9 MiB/s, 91 objects/s recovering 2024-02-10T07:37:03.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:03.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:03.422 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:03.455 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:03.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:03.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:03.477 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-10T07:37:03.557 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:03.601 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:03.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:03.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:03.703 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:03.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:03.742 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:03.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:03.843 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:03.844 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:37:03.909 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:03.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:04.010 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:04.043 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:04.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:04.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:04.144 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:04.180 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:04.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:04.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:04.282 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:04.292 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:04 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:04.318 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:04.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:04.399 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:04 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:04.399 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:04 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:04.420 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:04.471 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:04.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:04.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:04.573 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:04.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:04.607 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:04.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:04.708 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:04.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:04.740 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:04.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:04.841 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:04.878 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:04.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:04.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:04.956 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:04.981 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:05.012 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:05.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:05.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:05.114 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:05.157 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:05.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:05.180 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.1 to 1 (10000) 2024-02-10T07:37:05.259 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:05.303 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:05.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:05.404 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:05.437 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:05.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:05.451 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:05 smithi170 ceph-mon[33394]: pgmap v228: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.19k op/s; 15442/152523 objects misplaced (10.124%); 6.1 MiB/s, 136 objects/s recovering 2024-02-10T07:37:05.451 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:05 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1275468579' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:37:05.451 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:05 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:37:05.539 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:05.581 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:05.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:05 smithi008 ceph-mon[29816]: pgmap v228: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.19k op/s; 15442/152523 objects misplaced (10.124%); 6.1 MiB/s, 136 objects/s recovering 2024-02-10T07:37:05.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:05 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1275468579' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:37:05.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:05 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:37:05.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:05 smithi008 ceph-mon[40743]: pgmap v228: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.19k op/s; 15442/152523 objects misplaced (10.124%); 6.1 MiB/s, 136 objects/s recovering 2024-02-10T07:37:05.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:05 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1275468579' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:37:05.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:05 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:37:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:05.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:05.684 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:05.721 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:05.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:05.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:05.823 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:05.853 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:05.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:05.955 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:05.989 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:05.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:06.032 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:37:06.032 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2024-02-10T07:37:06.032 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2024-02-10T07:37:06.032 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2024-02-10T07:37:06.032 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:37:06.091 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:06.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:06.115 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:06.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:06.216 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:06.257 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:06.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:06.359 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:06.393 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:06.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:06.455 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:06 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:37:06.455 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:06 smithi170 ceph-mon[33394]: osdmap e172: 8 total, 8 up, 8 in 2024-02-10T07:37:06.455 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:06 smithi170 ceph-mon[33394]: osdmap e173: 8 total, 8 up, 8 in 2024-02-10T07:37:06.495 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:06.539 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:06.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:06 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:37:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:06 smithi008 ceph-mon[29816]: osdmap e172: 8 total, 8 up, 8 in 2024-02-10T07:37:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:06 smithi008 ceph-mon[29816]: osdmap e173: 8 total, 8 up, 8 in 2024-02-10T07:37:06.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:06 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:37:06.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:06 smithi008 ceph-mon[40743]: osdmap e172: 8 total, 8 up, 8 in 2024-02-10T07:37:06.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:06 smithi008 ceph-mon[40743]: osdmap e173: 8 total, 8 up, 8 in 2024-02-10T07:37:06.641 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:06.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:06.677 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:06.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:06.778 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:06.813 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:06.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:06.915 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:06.954 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:06.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:07.055 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:07.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:07.094 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:07.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:07.195 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:07.226 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:07.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:07.328 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:07.362 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:07.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:07.463 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:07.495 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:07.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:07.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:07.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:07 smithi008 ceph-mon[29816]: pgmap v230: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 731 B/s rd, 0 B/s wr, 664 op/s; 15442/152523 objects misplaced (10.124%); 3.6 MiB/s, 58 objects/s recovering 2024-02-10T07:37:07.529 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:07 smithi008 ceph-mon[40743]: pgmap v230: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 731 B/s rd, 0 B/s wr, 664 op/s; 15442/152523 objects misplaced (10.124%); 3.6 MiB/s, 58 objects/s recovering 2024-02-10T07:37:07.577 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 2 1 2024-02-10T07:37:07.597 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:07.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:07.627 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:07.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:07.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:07 smithi170 ceph-mon[33394]: pgmap v230: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 731 B/s rd, 0 B/s wr, 664 op/s; 15442/152523 objects misplaced (10.124%); 3.6 MiB/s, 58 objects/s recovering 2024-02-10T07:37:07.728 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:07.748 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-10T07:37:07.796 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:07.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:07.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:07.898 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:07.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:07.953 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:07.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:08.054 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:08.088 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:08.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:08.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:08.120 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:37:08.190 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:08.236 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:08.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:08.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:08.339 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:08.373 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:08.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:08.420 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:08 smithi008 ceph-mon[29816]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:37:08.421 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:08 smithi008 ceph-mon[29816]: osdmap e174: 8 total, 8 up, 8 in 2024-02-10T07:37:08.421 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:08 smithi008 ceph-mon[40743]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:37:08.421 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:08 smithi008 ceph-mon[40743]: osdmap e174: 8 total, 8 up, 8 in 2024-02-10T07:37:08.474 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:08.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:08.510 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:08.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:08.611 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:08.615 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:08 smithi170 ceph-mon[33394]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:37:08.615 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:08 smithi170 ceph-mon[33394]: osdmap e174: 8 total, 8 up, 8 in 2024-02-10T07:37:08.642 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:08.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:08.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:08.744 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:08.773 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:08.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:08.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:08.875 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:08.903 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:08.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:08.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:09.005 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:09.009 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:37:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:37:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:37:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:09.035 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:09.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:09.136 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:09.161 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:09.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:09.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:09.263 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:09.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:09.330 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:09.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:09.352 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:37:09.397 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:37:09.431 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:09.440 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:37:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:09.473 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:09.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:09.481 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:37:09.518 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:37:09.518 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:09 smithi008 ceph-mon[29816]: pgmap v232: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.08k op/s; 13853/146979 objects misplaced (9.425%); 20 MiB/s, 320 objects/s recovering 2024-02-10T07:37:09.518 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:09 smithi008 ceph-mon[29816]: osdmap e175: 8 total, 8 up, 8 in 2024-02-10T07:37:09.518 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:09 smithi008 ceph-mon[40743]: pgmap v232: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.08k op/s; 13853/146979 objects misplaced (9.425%); 20 MiB/s, 320 objects/s recovering 2024-02-10T07:37:09.519 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:09 smithi008 ceph-mon[40743]: osdmap e175: 8 total, 8 up, 8 in 2024-02-10T07:37:09.553 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:37:09.574 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:09.586 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:37:09.601 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:09.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:09.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:09.616 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:37:09.618 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:37:09.619 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:37:09.619 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:37:09.619 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:37:09.619 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:37:09.619 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:37:09.619 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:37:09.619 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:37:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:09 smithi170 ceph-mon[33394]: pgmap v232: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 153 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.08k op/s; 13853/146979 objects misplaced (9.425%); 20 MiB/s, 320 objects/s recovering 2024-02-10T07:37:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:09 smithi170 ceph-mon[33394]: osdmap e175: 8 total, 8 up, 8 in 2024-02-10T07:37:09.703 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:09.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:09.728 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:09.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:09.829 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:09.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:09.854 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:09.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:09.955 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:09.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:09.980 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:09.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:09.981 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:10.075 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:37:10.075 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:37:10.081 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:10.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:10.109 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:10.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:10.210 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:10.228 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:37:10.236 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:10.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:10.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:10.338 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:10.380 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:10.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:10.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:10.482 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:10.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:10.524 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:10.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:10.572 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:10 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1767308398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:37:10.572 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:10 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1039138949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:37:10.572 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:10 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:37:10.572 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:10 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:37:10.573 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:10 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:37:10.573 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:10 smithi008 ceph-mon[29816]: osdmap e176: 8 total, 8 up, 8 in 2024-02-10T07:37:10.573 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:10 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1767308398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:37:10.573 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:10 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1039138949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:37:10.573 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:10 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:37:10.573 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:10 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:37:10.573 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:10 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:37:10.573 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:10 smithi008 ceph-mon[40743]: osdmap e176: 8 total, 8 up, 8 in 2024-02-10T07:37:10.627 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:10 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1767308398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:37:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:10 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1039138949' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:37:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:10 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:37:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:10 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:37:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:10 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:37:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:10 smithi170 ceph-mon[33394]: osdmap e176: 8 total, 8 up, 8 in 2024-02-10T07:37:10.669 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:10.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:10.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:10.771 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:10.814 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:10.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:10.916 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:10.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:10.955 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:10.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:11.056 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:11.088 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:11.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:11.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:11.190 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:11.220 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:11.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:11.322 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:11.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:11.349 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:11.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:11.439 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T07:37:11.450 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:11.459 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:11 smithi008 ceph-mon[29816]: pgmap v235: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 153 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s; 11616/137310 objects misplaced (8.460%); 27 MiB/s, 431 objects/s recovering 2024-02-10T07:37:11.460 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:11 smithi008 ceph-mon[40743]: pgmap v235: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 153 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s; 11616/137310 objects misplaced (8.460%); 27 MiB/s, 431 objects/s recovering 2024-02-10T07:37:11.493 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:11.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:11.595 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:11.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:11 smithi170 ceph-mon[33394]: pgmap v235: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 153 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.10k op/s; 11616/137310 objects misplaced (8.460%); 27 MiB/s, 431 objects/s recovering 2024-02-10T07:37:11.635 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:11.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:11.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:11.737 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:11.770 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:11.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:11.776 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:37:11.872 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:11.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:11.904 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:11.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:12.005 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:12.040 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:12.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:12.142 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:12.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:12.183 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:12.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:12.284 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:12.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:12.327 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:12.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:12.428 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:12.459 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:12.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:12.562 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:12.603 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:12.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:12.705 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:12.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:12.750 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:12.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:12.851 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:12.882 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:12.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:12.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:12.931 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:12 smithi170 ceph-mon[33394]: osdmap e177: 8 total, 8 up, 8 in 2024-02-10T07:37:12.931 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:12 smithi170 ceph-mon[33394]: pgmap v238: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 153 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 11616/136881 objects misplaced (8.486%); 7.1 MiB/s, 115 objects/s recovering 2024-02-10T07:37:12.931 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:12 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:37:12.931 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:12 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:37:12.980 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:12 smithi008 ceph-mon[29816]: osdmap e177: 8 total, 8 up, 8 in 2024-02-10T07:37:12.980 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:12 smithi008 ceph-mon[29816]: pgmap v238: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 153 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 11616/136881 objects misplaced (8.486%); 7.1 MiB/s, 115 objects/s recovering 2024-02-10T07:37:12.981 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:12 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:37:12.981 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:12 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:37:12.981 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:12 smithi008 ceph-mon[40743]: osdmap e177: 8 total, 8 up, 8 in 2024-02-10T07:37:12.981 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:12 smithi008 ceph-mon[40743]: pgmap v238: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 153 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 11616/136881 objects misplaced (8.486%); 7.1 MiB/s, 115 objects/s recovering 2024-02-10T07:37:12.981 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:12 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:37:12.981 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:12 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:37:12.984 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:13.020 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:13.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:13.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:13.122 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:13.157 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:13.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:13.258 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:13.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:13.299 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:13.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:13.377 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:37:13.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:37:12 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:37:13.400 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:13.469 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:13.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:13.570 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:13.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:13.628 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:13.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:13.729 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:13.773 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:13.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:13.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:13.775 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:37:13.868 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:13.876 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:13.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:13 smithi170 ceph-mon[33394]: osdmap e178: 8 total, 8 up, 8 in 2024-02-10T07:37:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:13 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:13.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:13.926 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:13.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:14.027 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:14.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:14.082 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:14.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:14.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:13 smithi008 ceph-mon[29816]: osdmap e178: 8 total, 8 up, 8 in 2024-02-10T07:37:14.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:13 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:14.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:13 smithi008 ceph-mon[40743]: osdmap e178: 8 total, 8 up, 8 in 2024-02-10T07:37:14.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:13 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:14.183 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:14.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:14.225 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:14.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:14.326 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:14.356 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:14.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:14.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:14.458 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:14.488 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:14.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:14.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:14.590 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:14.621 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:14.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:14.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:14.723 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:14.753 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:14.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:14.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:14.827 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:37:14.828 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:37:14.856 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:14.875 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.2 to 1 (10000) 2024-02-10T07:37:14.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:14.913 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:14.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:15.014 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:15.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:15.057 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:15.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:14 smithi170 ceph-mon[33394]: pgmap v240: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 905 op/s; 9758/124932 objects misplaced (7.811%); 5.6 MiB/s, 89 objects/s recovering 2024-02-10T07:37:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:14 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/479360067' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:37:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:14 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:37:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:14 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:37:15.158 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:15.192 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:15.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:15.247 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:14 smithi008 ceph-mon[29816]: pgmap v240: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 905 op/s; 9758/124932 objects misplaced (7.811%); 5.6 MiB/s, 89 objects/s recovering 2024-02-10T07:37:15.247 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:14 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/479360067' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:37:15.247 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:14 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:37:15.247 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:14 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:37:15.247 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:14 smithi008 ceph-mon[40743]: pgmap v240: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 905 op/s; 9758/124932 objects misplaced (7.811%); 5.6 MiB/s, 89 objects/s recovering 2024-02-10T07:37:15.247 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:14 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/479360067' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:37:15.247 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:14 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:37:15.248 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:14 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:37:15.293 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:15.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:15.322 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:15.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:15.423 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:15.451 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:15.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:15.520 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-10T07:37:15.553 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:15.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:15.603 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:15.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:15.704 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:15.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:15.743 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:15.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:15.844 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:15.873 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:37:15.877 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:15.894 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:15.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:15.995 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:16.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:16.046 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:16.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:16.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:15 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/479360067' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:37:16.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:15 smithi170 ceph-mon[33394]: osdmap e179: 8 total, 8 up, 8 in 2024-02-10T07:37:16.147 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:16.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:16.181 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:16.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:16.271 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:15 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/479360067' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:37:16.271 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:15 smithi008 ceph-mon[29816]: osdmap e179: 8 total, 8 up, 8 in 2024-02-10T07:37:16.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:15 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/479360067' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:37:16.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:15 smithi008 ceph-mon[40743]: osdmap e179: 8 total, 8 up, 8 in 2024-02-10T07:37:16.282 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:16.318 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 3 1 2024-02-10T07:37:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:16.338 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:16.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:16.439 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:16.495 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:16.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:16.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:16.597 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:16.626 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:16.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:16.728 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:16.757 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:16.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:16.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:16.862 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:16.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:16.914 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:16.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:17.015 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:17.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:17.056 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:17.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:17.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:16 smithi170 ceph-mon[33394]: pgmap v242: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 920 B/s rd, 0 B/s wr, 742 op/s; 9758/124932 objects misplaced (7.811%); 4.6 MiB/s, 73 objects/s recovering 2024-02-10T07:37:17.157 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:17.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:17.199 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:17.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:17.297 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:16 smithi008 ceph-mon[29816]: pgmap v242: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 920 B/s rd, 0 B/s wr, 742 op/s; 9758/124932 objects misplaced (7.811%); 4.6 MiB/s, 73 objects/s recovering 2024-02-10T07:37:17.298 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:16 smithi008 ceph-mon[40743]: pgmap v242: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 920 B/s rd, 0 B/s wr, 742 op/s; 9758/124932 objects misplaced (7.811%); 4.6 MiB/s, 73 objects/s recovering 2024-02-10T07:37:17.301 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:17.339 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:17.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:17.441 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:17.485 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:17.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:17.586 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:17.624 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:17.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:17.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:17.726 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:17.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:17.768 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:17.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:17.869 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:17.911 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:37:17.919 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:17.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:17.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:18.021 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:18.039 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:17 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:18.039 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:17 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/95329423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:37:18.062 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:18.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:18.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:18.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:17 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:18.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:17 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/95329423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:37:18.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:17 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:18.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:17 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/95329423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:37:18.164 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:18.197 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:18.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:18.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:18.230 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:37:18.230 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:37:18.300 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:18.330 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:18.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:18.431 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:18.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:18.470 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:18.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:18.572 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:18.611 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:18.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:18.713 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:18.749 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:18.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:18.851 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:18.896 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:18.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:18.998 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:19.042 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:19.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:19.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:18 smithi170 ceph-mon[33394]: pgmap v243: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.4 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.00k op/s; 9758/119085 objects misplaced (8.194%); 4.2 MiB/s, 67 objects/s recovering 2024-02-10T07:37:19.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:18 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:37:19.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:18 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/95329423' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:37:19.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:18 smithi170 ceph-mon[33394]: osdmap e180: 8 total, 8 up, 8 in 2024-02-10T07:37:19.046 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:18 smithi170 ceph-mon[33394]: 5.16 scrub starts 2024-02-10T07:37:19.046 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:18 smithi170 ceph-mon[33394]: 5.16 scrub ok 2024-02-10T07:37:19.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:37:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:37:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:37:19.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:18 smithi008 ceph-mon[40743]: pgmap v243: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.4 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.00k op/s; 9758/119085 objects misplaced (8.194%); 4.2 MiB/s, 67 objects/s recovering 2024-02-10T07:37:19.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:18 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:37:19.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:18 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/95329423' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:37:19.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:18 smithi008 ceph-mon[40743]: osdmap e180: 8 total, 8 up, 8 in 2024-02-10T07:37:19.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:18 smithi008 ceph-mon[40743]: 5.16 scrub starts 2024-02-10T07:37:19.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:18 smithi008 ceph-mon[40743]: 5.16 scrub ok 2024-02-10T07:37:19.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:18 smithi008 ceph-mon[29816]: pgmap v243: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.4 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.00k op/s; 9758/119085 objects misplaced (8.194%); 4.2 MiB/s, 67 objects/s recovering 2024-02-10T07:37:19.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:18 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:37:19.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:18 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/95329423' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:37:19.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:18 smithi008 ceph-mon[29816]: osdmap e180: 8 total, 8 up, 8 in 2024-02-10T07:37:19.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:18 smithi008 ceph-mon[29816]: 5.16 scrub starts 2024-02-10T07:37:19.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:18 smithi008 ceph-mon[29816]: 5.16 scrub ok 2024-02-10T07:37:19.084 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-10T07:37:19.145 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:19.186 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:19.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:19.361 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:19.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:19.399 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:19.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:19.418 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:37:19.500 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:19.529 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:19.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:19.631 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:19.671 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:19.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:19.774 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:19.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:19.800 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:19.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:19.809 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:19.901 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:19.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:19.946 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:19.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:20.038 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:19 smithi008 ceph-mon[40743]: 5.b scrub starts 2024-02-10T07:37:20.038 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:19 smithi008 ceph-mon[40743]: 5.b scrub ok 2024-02-10T07:37:20.038 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:19 smithi008 ceph-mon[40743]: 6.a scrub starts 2024-02-10T07:37:20.038 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:19 smithi008 ceph-mon[40743]: 6.a scrub ok 2024-02-10T07:37:20.038 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:19 smithi008 ceph-mon[40743]: 6.b scrub starts 2024-02-10T07:37:20.038 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:19 smithi008 ceph-mon[40743]: 6.b scrub ok 2024-02-10T07:37:20.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:19 smithi008 ceph-mon[29816]: 5.b scrub starts 2024-02-10T07:37:20.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:19 smithi008 ceph-mon[29816]: 5.b scrub ok 2024-02-10T07:37:20.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:19 smithi008 ceph-mon[29816]: 6.a scrub starts 2024-02-10T07:37:20.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:19 smithi008 ceph-mon[29816]: 6.a scrub ok 2024-02-10T07:37:20.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:19 smithi008 ceph-mon[29816]: 6.b scrub starts 2024-02-10T07:37:20.039 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:19 smithi008 ceph-mon[29816]: 6.b scrub ok 2024-02-10T07:37:20.047 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:20.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:20.077 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:20.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:19 smithi170 ceph-mon[33394]: 5.b scrub starts 2024-02-10T07:37:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:19 smithi170 ceph-mon[33394]: 5.b scrub ok 2024-02-10T07:37:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:19 smithi170 ceph-mon[33394]: 6.a scrub starts 2024-02-10T07:37:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:19 smithi170 ceph-mon[33394]: 6.a scrub ok 2024-02-10T07:37:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:19 smithi170 ceph-mon[33394]: 6.b scrub starts 2024-02-10T07:37:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:19 smithi170 ceph-mon[33394]: 6.b scrub ok 2024-02-10T07:37:20.178 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:20.216 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:20.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:20.318 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:20.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:20.348 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:20.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:20.449 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:20.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:20.493 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:20.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:20.595 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:20.625 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:20.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:20.673 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:37:20.724 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:37:20.726 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:20.770 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:37:20.772 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:20.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:20.817 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:37:20.864 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:37:20.877 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:20.880 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:37:20.906 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:37:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:20.912 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:20.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:20.948 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:37:20.992 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:37:21.004 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:37:21.004 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:37:21.004 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:37:21.004 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:37:21.004 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:37:21.004 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:37:21.005 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:37:21.005 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:37:21.013 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:21.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:21.069 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:21.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:21.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:20 smithi170 ceph-mon[33394]: pgmap v245: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 736 B/s rd, 0 B/s wr, 1.14k op/s; 8447/108942 objects misplaced (7.754%); 3.7 MiB/s, 58 objects/s recovering 2024-02-10T07:37:21.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:20 smithi170 ceph-mon[33394]: 5.1 scrub starts 2024-02-10T07:37:21.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:20 smithi170 ceph-mon[33394]: 5.1 scrub ok 2024-02-10T07:37:21.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:20 smithi170 ceph-mon[33394]: 6.11 scrub starts 2024-02-10T07:37:21.169 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:20 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:37:21.169 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:20 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:37:21.169 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:20 smithi170 ceph-mon[33394]: 6.11 scrub ok 2024-02-10T07:37:21.169 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:20 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4168448536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:37:21.170 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:21.205 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:21.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:21.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:21.307 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:21.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[40743]: pgmap v245: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 736 B/s rd, 0 B/s wr, 1.14k op/s; 8447/108942 objects misplaced (7.754%); 3.7 MiB/s, 58 objects/s recovering 2024-02-10T07:37:21.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[40743]: 5.1 scrub starts 2024-02-10T07:37:21.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[40743]: 5.1 scrub ok 2024-02-10T07:37:21.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[40743]: 6.11 scrub starts 2024-02-10T07:37:21.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:37:21.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:37:21.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[40743]: 6.11 scrub ok 2024-02-10T07:37:21.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4168448536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:37:21.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[29816]: pgmap v245: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 736 B/s rd, 0 B/s wr, 1.14k op/s; 8447/108942 objects misplaced (7.754%); 3.7 MiB/s, 58 objects/s recovering 2024-02-10T07:37:21.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[29816]: 5.1 scrub starts 2024-02-10T07:37:21.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[29816]: 5.1 scrub ok 2024-02-10T07:37:21.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[29816]: 6.11 scrub starts 2024-02-10T07:37:21.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:37:21.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:37:21.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[29816]: 6.11 scrub ok 2024-02-10T07:37:21.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:20 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4168448536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:37:21.347 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:37:21.347 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2024-02-10T07:37:21.347 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2024-02-10T07:37:21.347 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:37:21.348 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.348 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:154826 7.0 7.577e9d00 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:37:21.348 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:32.885749+0000", 2024-02-10T07:37:21.348 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 48.459093983000002, 2024-02-10T07:37:21.348 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.86189586600000001, 2024-02-10T07:37:21.348 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.348 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:37:21.348 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.348 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.348 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.348 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 154826 2024-02-10T07:37:21.348 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885749+0000", 2024-02-10T07:37:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885749+0000", 2024-02-10T07:37:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.349 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885751+0000", 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2109999999999999e-06 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885755+0000", 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7960000000000003e-06 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885765+0000", 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0101999999999999e-05 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.350 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885775+0000", 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9359999999999994e-06 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885836+0000", 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0615999999999997e-05 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885856+0000", 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0418999999999999e-05 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.351 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:33.747631+0000", 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.86177504800000004 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:33.747645+0000", 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3738e-05 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.24740.0:155750 7.e e158/157 7:7b6197ca:::benchmark_data_smithi008_82046_object40612:head v 158'9709, mlcod=141'8002)", 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:35.634302+0000", 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 45.710541104999997, 2024-02-10T07:37:21.352 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.036202830999999998, 2024-02-10T07:37:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.634302+0000", 2024-02-10T07:37:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.634302+0000", 2024-02-10T07:37:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.353 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.634303+0000", 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.2099999999999995e-07 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.634311+0000", 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0450000000000001e-06 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.634324+0000", 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2469e-05 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.354 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.634335+0000", 2024-02-10T07:37:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1731e-05 2024-02-10T07:37:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.636050+0000", 2024-02-10T07:37:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001714791 2024-02-10T07:37:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.636074+0000", 2024-02-10T07:37:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3608000000000002e-05 2024-02-10T07:37:21.355 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.356 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.356 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.356 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.670432+0000", 2024-02-10T07:37:21.356 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.034358445000000001 2024-02-10T07:37:21.356 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.357 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.357 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.357 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.670505+0000", 2024-02-10T07:37:21.357 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2821e-05 2024-02-10T07:37:21.357 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.357 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.357 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.357 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.357 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.357 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.24740.0:155751 7.e e158/157 7:7d63a5ee:::benchmark_data_smithi008_82046_object40613:head v 158'9710, mlcod=141'8002)", 2024-02-10T07:37:21.357 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:35.638083+0000", 2024-02-10T07:37:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 45.706760094000003, 2024-02-10T07:37:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.032478677999999997, 2024-02-10T07:37:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.638083+0000", 2024-02-10T07:37:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.358 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.638083+0000", 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.638084+0000", 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.75e-07 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.638090+0000", 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5699999999999998e-06 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.359 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.638101+0000", 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0263e-05 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.638107+0000", 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6610000000000003e-06 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.640094+0000", 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0019868619999999998 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.640117+0000", 2024-02-10T07:37:21.360 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2637999999999999e-05 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.670508+0000", 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030391264000000001 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.670562+0000", 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3745000000000001e-05 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.361 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.24740.0:155755 7.e e158/157 7:74de3371:::benchmark_data_smithi008_82046_object40617:head v 158'9711, mlcod=141'8002)", 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:35.641981+0000", 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 45.702862420000002, 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.028630035000000002, 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.641981+0000", 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.362 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.641981+0000", 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.641982+0000", 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0999999999999998e-07 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.641989+0000", 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6389999999999992e-06 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.641999+0000", 2024-02-10T07:37:21.363 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0002e-05 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.642006+0000", 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4819999999999999e-06 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.648692+0000", 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0066859689999999999 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.648716+0000", 2024-02-10T07:37:21.364 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4212000000000001e-05 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.670564+0000", 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.021848303999999999 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:35.670611+0000", 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6717e-05 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.365 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.366 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24356.0:1470 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 93827140445184 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-02-10T07:37:21.366 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:36.420360+0000", 2024-02-10T07:37:21.366 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 44.924483725000002, 2024-02-10T07:37:21.366 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029466708000000001, 2024-02-10T07:37:21.366 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.366 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:37:21.366 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.366 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24356", 2024-02-10T07:37:21.366 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1933562162", 2024-02-10T07:37:21.367 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 1470 2024-02-10T07:37:21.367 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.367 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.367 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.367 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.367 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:36.420360+0000", 2024-02-10T07:37:21.367 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.368 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.368 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.368 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.368 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:36.420360+0000", 2024-02-10T07:37:21.368 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.368 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.368 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.368 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.368 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:36.420361+0000", 2024-02-10T07:37:21.368 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.359e-06 2024-02-10T07:37:21.368 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.368 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.368 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.368 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:36.420364+0000", 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.225e-06 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:36.420371+0000", 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6980000000000001e-06 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:36.420375+0000", 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5979999999999999e-06 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.369 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:36.420420+0000", 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4597e-05 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:36.420552+0000", 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013173300000000001 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:36.449826+0000", 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029274497999999999 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.370 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:172217 7.0 7:011f3673:::benchmark_data_smithi008_82046_object57079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:55.310295+0000", 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 26.034548494999999, 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.057396206999999998, 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 172217 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.371 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.372 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:55.310295+0000", 2024-02-10T07:37:21.372 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.372 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.372 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.372 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.372 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:55.310295+0000", 2024-02-10T07:37:21.372 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.372 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.372 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.372 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.372 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:55.310296+0000", 2024-02-10T07:37:21.372 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.167e-06 2024-02-10T07:37:21.372 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.372 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:55.310304+0000", 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5129999999999999e-06 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:55.310313+0000", 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4390000000000006e-06 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:55.310321+0000", 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7700000000000001e-06 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.373 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:55.310366+0000", 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5649000000000003e-05 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:55.310479+0000", 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000112929 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:55.310748+0000", 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00026855200000000003 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.374 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:37:21.375 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:55.312611+0000", 2024-02-10T07:37:21.375 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0018629639999999999 2024-02-10T07:37:21.375 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.375 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.375 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.375 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:55.317705+0000", 2024-02-10T07:37:21.375 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0050938379999999998 2024-02-10T07:37:21.375 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.375 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.375 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.375 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:55.367583+0000", 2024-02-10T07:37:21.375 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.04987859 2024-02-10T07:37:21.375 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.375 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:21.378 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:21.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:21.379 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.379 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:55.367646+0000", 2024-02-10T07:37:21.379 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3089999999999994e-05 2024-02-10T07:37:21.379 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.379 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.379 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.379 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:55.367691+0000", 2024-02-10T07:37:21.379 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4706000000000002e-05 2024-02-10T07:37:21.379 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.379 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.379 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.379 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.380 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.380 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:176105 7.0 7:06736203:::benchmark_data_smithi008_82046_object60967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-02-10T07:37:21.380 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:59.114412+0000", 2024-02-10T07:37:21.380 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.230431054, 2024-02-10T07:37:21.380 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.05299835, 2024-02-10T07:37:21.380 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.380 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.380 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.380 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.380 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.380 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 176105 2024-02-10T07:37:21.380 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.380 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.380 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114412+0000", 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114412+0000", 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114414+0000", 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.21e-06 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.381 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.382 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.382 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114420+0000", 2024-02-10T07:37:21.382 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7139999999999998e-06 2024-02-10T07:37:21.382 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.382 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.382 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.382 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114441+0000", 2024-02-10T07:37:21.382 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1174000000000001e-05 2024-02-10T07:37:21.382 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.382 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.382 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.382 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114450+0000", 2024-02-10T07:37:21.382 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1929999999999993e-06 2024-02-10T07:37:21.382 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114506+0000", 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6677e-05 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114603+0000", 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6729000000000002e-05 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114889+0000", 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000285545 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.383 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.384 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:37:21.384 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.117755+0000", 2024-02-10T07:37:21.384 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0028664160000000001 2024-02-10T07:37:21.384 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.384 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.384 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.384 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.119154+0000", 2024-02-10T07:37:21.384 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0013991190000000001 2024-02-10T07:37:21.384 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.384 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.384 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.384 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.167318+0000", 2024-02-10T07:37:21.384 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.048163922999999997 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.167373+0000", 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4568999999999999e-05 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.167411+0000", 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8080999999999997e-05 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.385 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:176106 7.0 7:07e42078:::benchmark_data_smithi008_82046_object60968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:59.114522+0000", 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 22.230321215, 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.053538765000000002, 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 176106 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.386 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114522+0000", 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114522+0000", 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114523+0000", 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.7699999999999992e-07 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.387 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114527+0000", 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3059999999999999e-06 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114538+0000", 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0556e-05 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.114546+0000", 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6539999999999999e-06 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.116341+0000", 2024-02-10T07:37:21.388 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0017949260000000001 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.116465+0000", 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000124039 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.116639+0000", 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000174506 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.117852+0000", 2024-02-10T07:37:21.389 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0012130750000000001 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.119183+0000", 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0013312549999999999 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.167947+0000", 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.048763125999999997 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.168008+0000", 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1636000000000004e-05 2024-02-10T07:37:21.390 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.168061+0000", 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2709e-05 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:176688 7.0 7:04f873bb:::benchmark_data_smithi008_82046_object61550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:59.801084+0000", 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.543759830999999, 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.063910385, 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.391 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.392 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.392 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.392 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 176688 2024-02-10T07:37:21.392 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.392 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.392 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.392 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.392 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.801084+0000", 2024-02-10T07:37:21.392 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.392 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.392 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.392 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.392 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.801084+0000", 2024-02-10T07:37:21.392 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.393 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.393 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.393 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.393 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.801084+0000", 2024-02-10T07:37:21.393 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9699999999999995e-07 2024-02-10T07:37:21.393 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.393 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.393 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.393 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.801088+0000", 2024-02-10T07:37:21.393 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3790000000000002e-06 2024-02-10T07:37:21.393 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.396 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.396 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.396 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.801096+0000", 2024-02-10T07:37:21.396 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1030000000000002e-06 2024-02-10T07:37:21.396 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.396 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.396 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.396 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.801103+0000", 2024-02-10T07:37:21.396 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.7290000000000003e-06 2024-02-10T07:37:21.396 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.396 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.396 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.396 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.801134+0000", 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.1579999999999999e-05 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.801224+0000", 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.9874e-05 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.801481+0000", 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00025724999999999999 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.803186+0000", 2024-02-10T07:37:21.397 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0017049719999999999 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.804271+0000", 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001084523 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.864870+0000", 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.060599277 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.864956+0000", 2024-02-10T07:37:21.398 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5790000000000004e-05 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:59.864994+0000", 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8111000000000002e-05 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:177879 7.0 7:05f7f0bb:::benchmark_data_smithi008_82046_object62741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:00.961785+0000", 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 20.383058789, 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.042795162999999997, 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.399 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 177879 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.961785+0000", 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.961785+0000", 2024-02-10T07:37:21.400 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.961786+0000", 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1820000000000001e-06 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.961790+0000", 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5889999999999996e-06 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.961803+0000", 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2672e-05 2024-02-10T07:37:21.401 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.961811+0000", 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.4109999999999996e-06 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.961869+0000", 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7187e-05 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.961996+0000", 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012696800000000001 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.402 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.998144+0000", 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.036148221000000001 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.004383+0000", 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0062395940000000002 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.004396+0000", 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2669e-05 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.403 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.004435+0000", 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9051999999999998e-05 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.004534+0000", 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.8436000000000004e-05 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.004580+0000", 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6181999999999999e-05 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.404 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.24740.0:177882 7.b e170/165 7:de340868:::benchmark_data_smithi008_82046_object62744:head v 170'11155, mlcod=170'11153)", 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:00.963343+0000", 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 20.381499921, 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.036952656, 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.963343+0000", 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.405 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.963343+0000", 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.963345+0000", 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.15e-06 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.963350+0000", 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9179999999999999e-06 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.406 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.997804+0000", 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.034453504000000003 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.997818+0000", 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4358999999999999e-05 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.997871+0000", 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2741999999999997e-05 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.407 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.997891+0000", 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9456000000000001e-05 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.000204+0000", 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0023138910000000002 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.000296+0000", 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1636000000000002e-05 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.408 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:177885 7.0 7:000ce93d:::benchmark_data_smithi008_82046_object62747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:00.971033+0000", 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 20.373810262999999, 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.034406243000000003, 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 177885 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.409 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.971033+0000", 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.971033+0000", 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.971034+0000", 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2219999999999999e-06 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.410 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.971040+0000", 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2310000000000004e-06 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.997803+0000", 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026763042000000001 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:00.997819+0000", 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6648000000000001e-05 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.000053+0000", 2024-02-10T07:37:21.411 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0022339980000000001 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.000185+0000", 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000131588 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.000485+0000", 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00030044300000000001 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.004400+0000", 2024-02-10T07:37:21.412 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0039150390000000004 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.004622+0000", 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00022213199999999999 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.005294+0000", 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00067120799999999996 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.005401+0000", 2024-02-10T07:37:21.413 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000106946 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:01.005439+0000", 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8745999999999998e-05 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:179346 7.0 7:026b3150:::benchmark_data_smithi008_82046_object64208:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:02.516967+0000", 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 18.827875879, 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.082144761999999996, 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.414 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 179346 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:02.516967+0000", 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:02.516967+0000", 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.415 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:02.516969+0000", 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4020000000000001e-06 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:02.516975+0000", 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7169999999999996e-06 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:02.516991+0000", 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6589999999999999e-05 2024-02-10T07:37:21.416 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:02.517001+0000", 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0097e-05 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:02.517060+0000", 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8992e-05 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:02.517183+0000", 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012264 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.417 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:02.517512+0000", 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000328661 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:02.520334+0000", 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0028220039999999999 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:02.520627+0000", 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00029320300000000002 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.418 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:02.598977+0000", 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.078350660000000003 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:02.599067+0000", 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.9918000000000002e-05 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:02.599112+0000", 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4878e-05 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.419 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:179712 7.0 7:02bacbf3:::benchmark_data_smithi008_82046_object64574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:03.028391+0000", 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 18.316452283, 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.028833213999999999, 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 179712 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.420 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:03.028391+0000", 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:03.028391+0000", 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:03.028392+0000", 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0109999999999999e-06 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.421 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:03.028396+0000", 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2540000000000003e-06 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:03.028409+0000", 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3096e-05 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:03.028417+0000", 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3259999999999998e-06 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.422 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:03.028465+0000", 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.791e-05 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:03.028547+0000", 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2711000000000001e-05 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:03.028687+0000", 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000140121 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.423 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:03.029861+0000", 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001173278 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:03.031495+0000", 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00163378 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:03.057122+0000", 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.025627827999999998 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.424 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:03.057183+0000", 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0628000000000003e-05 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:03.057224+0000", 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1270999999999998e-05 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:180804 7.0 7:07417419:::benchmark_data_smithi008_82046_object65666:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:04.281706+0000", 2024-02-10T07:37:21.425 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 17.063136891999999, 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029880902000000001, 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 180804 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.281706+0000", 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.426 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.281706+0000", 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.281708+0000", 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4580000000000001e-06 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.281713+0000", 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1080000000000001e-06 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.427 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.281727+0000", 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3645000000000001e-05 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.281745+0000", 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8136999999999999e-05 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.281801+0000", 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6362e-05 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.428 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.281932+0000", 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013112099999999999 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.282289+0000", 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000357124 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.286009+0000", 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0037198719999999999 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.302697+0000", 2024-02-10T07:37:21.429 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016688194 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.311488+0000", 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.008790272 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.311546+0000", 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8544000000000001e-05 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.311587+0000", 2024-02-10T07:37:21.430 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1065000000000001e-05 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:180806 7.0 7:00e28560:::benchmark_data_smithi008_82046_object65668:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:04.282244+0000", 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 17.062599586000001, 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.029494811999999999, 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.431 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 180806 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.282244+0000", 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.282244+0000", 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.432 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.282245+0000", 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8400000000000003e-07 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.282249+0000", 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3200000000000001e-06 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.282259+0000", 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0246e-05 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.433 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.282267+0000", 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4580000000000004e-06 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.283945+0000", 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001678655 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.284071+0000", 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000125606 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.284311+0000", 2024-02-10T07:37:21.434 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000240504 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.290342+0000", 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0060308219999999999 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.302749+0000", 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.01240661 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.311636+0000", 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0088873230000000008 2024-02-10T07:37:21.435 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.311688+0000", 2024-02-10T07:37:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1879000000000003e-05 2024-02-10T07:37:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.311739+0000", 2024-02-10T07:37:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0605000000000001e-05 2024-02-10T07:37:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.436 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:180807 7.0 7:06ec96f6:::benchmark_data_smithi008_82046_object65669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:04.283198+0000", 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 17.061644854000001, 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.028634698, 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 180807 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.283198+0000", 2024-02-10T07:37:21.437 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.283198+0000", 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.283199+0000", 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5199999999999995e-07 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.283204+0000", 2024-02-10T07:37:21.438 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6029999999999998e-06 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.283217+0000", 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2835e-05 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.283224+0000", 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6650000000000003e-06 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.286042+0000", 2024-02-10T07:37:21.439 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0028179519999999999 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.286164+0000", 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000121635 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.286406+0000", 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024236200000000001 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.290349+0000", 2024-02-10T07:37:21.440 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0039428120000000004 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.302799+0000", 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012449531999999999 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.311778+0000", 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0089792710000000005 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.311814+0000", 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6452999999999998e-05 2024-02-10T07:37:21.441 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.311833+0000", 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8726e-05 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:180812 7.0 7:05abd8c8:::benchmark_data_smithi008_82046_object65674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:04.286048+0000", 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 17.05879526, 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.031491910999999997, 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.442 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 180812 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.286048+0000", 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.286048+0000", 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.443 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.286049+0000", 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.373e-06 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.286053+0000", 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7349999999999998e-06 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.286064+0000", 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1274000000000001e-05 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.444 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.286072+0000", 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0890000000000002e-06 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.288078+0000", 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020066369999999999 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.288243+0000", 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016467599999999999 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.445 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.288529+0000", 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00028603899999999998 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.290470+0000", 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0019415299999999999 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.302834+0000", 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012363424 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.446 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.317415+0000", 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.014581297 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.317498+0000", 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2744999999999995e-05 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:04.317540+0000", 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2092000000000003e-05 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.447 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:182765 7.0 7:02e686aa:::benchmark_data_smithi008_82046_object67627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:06.052259+0000", 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 15.292584692, 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.111276601, 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 182765 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.448 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052259+0000", 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052259+0000", 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052260+0000", 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8080000000000001e-06 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.449 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052266+0000", 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4620000000000004e-06 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052283+0000", 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7297000000000001e-05 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052292+0000", 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6379999999999993e-06 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.450 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052351+0000", 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8943999999999997e-05 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052465+0000", 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011377100000000001 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052868+0000", 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00040303099999999998 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.451 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055300+0000", 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0024328169999999999 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.058232+0000", 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002931978 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.163436+0000", 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.10520370699999999 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.452 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.163494+0000", 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7556999999999998e-05 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.163535+0000", 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1591000000000002e-05 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:182767 7.0 7:063d4601:::benchmark_data_smithi008_82046_object67629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:06.055797+0000", 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 15.289046344000001, 2024-02-10T07:37:21.453 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.10783342999999999, 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 182767 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055797+0000", 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.454 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055797+0000", 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055798+0000", 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3340000000000001e-06 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055804+0000", 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4360000000000001e-06 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:37:21.455 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055817+0000", 2024-02-10T07:37:21.456 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3298999999999999e-05 2024-02-10T07:37:21.456 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.456 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.456 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:37:21.456 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055825+0000", 2024-02-10T07:37:21.456 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9950000000000005e-06 2024-02-10T07:37:21.456 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.456 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.456 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:37:21.456 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055877+0000", 2024-02-10T07:37:21.456 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1536999999999998e-05 2024-02-10T07:37:21.456 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.456 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.456 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.056012+0000", 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013553799999999999 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.056365+0000", 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000352686 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.058275+0000", 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001910192 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.457 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.058520+0000", 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024463000000000002 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.163565+0000", 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.105045084 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.163594+0000", 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8890999999999998e-05 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.163630+0000", 2024-02-10T07:37:21.458 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6807999999999997e-05 2024-02-10T07:37:21.459 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.459 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.459 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.459 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:37:21.459 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:37:21.459 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:37:21.479 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:21.519 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:21.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:21.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:21.620 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:21.657 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:21.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:21.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:21.759 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:21.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:21.801 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:21.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:21.901 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:21.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:21.947 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:21.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:22.048 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:22.081 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:22.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:22.183 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:22.227 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:22.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:22.329 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:22.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:21 smithi008 ceph-mon[29816]: 7.5 scrub starts 2024-02-10T07:37:22.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:21 smithi008 ceph-mon[29816]: 7.5 scrub ok 2024-02-10T07:37:22.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:21 smithi008 ceph-mon[29816]: osdmap e181: 8 total, 8 up, 8 in 2024-02-10T07:37:22.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:21 smithi008 ceph-mon[40743]: 7.5 scrub starts 2024-02-10T07:37:22.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:21 smithi008 ceph-mon[40743]: 7.5 scrub ok 2024-02-10T07:37:22.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:21 smithi008 ceph-mon[40743]: osdmap e181: 8 total, 8 up, 8 in 2024-02-10T07:37:22.352 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:22.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:21 smithi170 ceph-mon[33394]: 7.5 scrub starts 2024-02-10T07:37:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:21 smithi170 ceph-mon[33394]: 7.5 scrub ok 2024-02-10T07:37:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:21 smithi170 ceph-mon[33394]: osdmap e181: 8 total, 8 up, 8 in 2024-02-10T07:37:22.454 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:22.487 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:22.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:22.589 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:22.634 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:22.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:22.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:22.700 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-10T07:37:22.736 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:22.762 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:22.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:22.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:22.865 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:22.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:22.908 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:22.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:23.009 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:23.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:23.041 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:23.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:23.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[29816]: 7.18 scrub starts 2024-02-10T07:37:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[29816]: 6.f scrub starts 2024-02-10T07:37:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[29816]: 6.f scrub ok 2024-02-10T07:37:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[29816]: pgmap v247: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.2 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 796 op/s; 8447/108426 objects misplaced (7.791%) 2024-02-10T07:37:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[29816]: 7.18 scrub ok 2024-02-10T07:37:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[29816]: 6.19 scrub starts 2024-02-10T07:37:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[29816]: 6.19 scrub ok 2024-02-10T07:37:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[29816]: 4.10 scrub starts 2024-02-10T07:37:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[29816]: 4.10 scrub ok 2024-02-10T07:37:23.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[40743]: 7.18 scrub starts 2024-02-10T07:37:23.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[40743]: 6.f scrub starts 2024-02-10T07:37:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[40743]: 6.f scrub ok 2024-02-10T07:37:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[40743]: pgmap v247: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.2 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 796 op/s; 8447/108426 objects misplaced (7.791%) 2024-02-10T07:37:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[40743]: 7.18 scrub ok 2024-02-10T07:37:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[40743]: 6.19 scrub starts 2024-02-10T07:37:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[40743]: 6.19 scrub ok 2024-02-10T07:37:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[40743]: 4.10 scrub starts 2024-02-10T07:37:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:22 smithi008 ceph-mon[40743]: 4.10 scrub ok 2024-02-10T07:37:23.143 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:23.183 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:23.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:23.285 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:23.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:23.315 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:23.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:23.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:37:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:37:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:22 smithi170 ceph-mon[33394]: 7.18 scrub starts 2024-02-10T07:37:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:22 smithi170 ceph-mon[33394]: 6.f scrub starts 2024-02-10T07:37:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:22 smithi170 ceph-mon[33394]: 6.f scrub ok 2024-02-10T07:37:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:22 smithi170 ceph-mon[33394]: pgmap v247: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 2.2 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 796 op/s; 8447/108426 objects misplaced (7.791%) 2024-02-10T07:37:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:22 smithi170 ceph-mon[33394]: 7.18 scrub ok 2024-02-10T07:37:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:22 smithi170 ceph-mon[33394]: 6.19 scrub starts 2024-02-10T07:37:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:22 smithi170 ceph-mon[33394]: 6.19 scrub ok 2024-02-10T07:37:23.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:22 smithi170 ceph-mon[33394]: 4.10 scrub starts 2024-02-10T07:37:23.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:22 smithi170 ceph-mon[33394]: 4.10 scrub ok 2024-02-10T07:37:23.416 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:23.444 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:23.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:23.545 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:23.575 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:23.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:23.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:23.677 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:23.709 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:23.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:23.810 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:23.838 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:23.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:23.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:23.940 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:23.964 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:23.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:24.065 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:24.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:24.098 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:24.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:24.199 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:24.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:24.240 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:24.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:24.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[29816]: 6.4 scrub starts 2024-02-10T07:37:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[29816]: 6.4 scrub ok 2024-02-10T07:37:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[29816]: 3.d scrub starts 2024-02-10T07:37:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[29816]: 3.d scrub ok 2024-02-10T07:37:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[29816]: 5.2 scrub starts 2024-02-10T07:37:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[29816]: 5.2 scrub ok 2024-02-10T07:37:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[29816]: 5.1b scrub starts 2024-02-10T07:37:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[29816]: 5.1b scrub ok 2024-02-10T07:37:24.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[40743]: 6.4 scrub starts 2024-02-10T07:37:24.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[40743]: 6.4 scrub ok 2024-02-10T07:37:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[40743]: 3.d scrub starts 2024-02-10T07:37:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[40743]: 3.d scrub ok 2024-02-10T07:37:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[40743]: 5.2 scrub starts 2024-02-10T07:37:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[40743]: 5.2 scrub ok 2024-02-10T07:37:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[40743]: 5.1b scrub starts 2024-02-10T07:37:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:23 smithi008 ceph-mon[40743]: 5.1b scrub ok 2024-02-10T07:37:24.341 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:24.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:24.379 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:24.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:23 smithi170 ceph-mon[33394]: 6.4 scrub starts 2024-02-10T07:37:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:23 smithi170 ceph-mon[33394]: 6.4 scrub ok 2024-02-10T07:37:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:23 smithi170 ceph-mon[33394]: 3.d scrub starts 2024-02-10T07:37:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:23 smithi170 ceph-mon[33394]: 3.d scrub ok 2024-02-10T07:37:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:23 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:23 smithi170 ceph-mon[33394]: 5.2 scrub starts 2024-02-10T07:37:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:23 smithi170 ceph-mon[33394]: 5.2 scrub ok 2024-02-10T07:37:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:23 smithi170 ceph-mon[33394]: 5.1b scrub starts 2024-02-10T07:37:24.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:23 smithi170 ceph-mon[33394]: 5.1b scrub ok 2024-02-10T07:37:24.481 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:24.522 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:24.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:24.624 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:24.656 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:24.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:24.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:24.758 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:24.788 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:24.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:24.890 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:24.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:24.929 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:24.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:25.030 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:25.068 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:25.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:25.171 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:25.215 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:25.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:25.318 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[29816]: 3.6 scrub starts 2024-02-10T07:37:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[29816]: 3.6 scrub ok 2024-02-10T07:37:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[29816]: pgmap v248: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 1.9 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 554 op/s; 7250/93066 objects misplaced (7.790%) 2024-02-10T07:37:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[29816]: 7.a scrub starts 2024-02-10T07:37:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[29816]: 4.1f scrub starts 2024-02-10T07:37:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[29816]: 4.1f scrub ok 2024-02-10T07:37:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[29816]: 3.5 scrub starts 2024-02-10T07:37:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[29816]: 3.5 scrub ok 2024-02-10T07:37:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[29816]: 3.17 scrub starts 2024-02-10T07:37:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[29816]: 3.17 scrub ok 2024-02-10T07:37:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[40743]: 3.6 scrub starts 2024-02-10T07:37:25.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[40743]: 3.6 scrub ok 2024-02-10T07:37:25.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[40743]: pgmap v248: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 1.9 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 554 op/s; 7250/93066 objects misplaced (7.790%) 2024-02-10T07:37:25.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[40743]: 7.a scrub starts 2024-02-10T07:37:25.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[40743]: 4.1f scrub starts 2024-02-10T07:37:25.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[40743]: 4.1f scrub ok 2024-02-10T07:37:25.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[40743]: 3.5 scrub starts 2024-02-10T07:37:25.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[40743]: 3.5 scrub ok 2024-02-10T07:37:25.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[40743]: 3.17 scrub starts 2024-02-10T07:37:25.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:24 smithi008 ceph-mon[40743]: 3.17 scrub ok 2024-02-10T07:37:25.346 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:25.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:24 smithi170 ceph-mon[33394]: 3.6 scrub starts 2024-02-10T07:37:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:24 smithi170 ceph-mon[33394]: 3.6 scrub ok 2024-02-10T07:37:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:24 smithi170 ceph-mon[33394]: pgmap v248: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 1.9 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 554 op/s; 7250/93066 objects misplaced (7.790%) 2024-02-10T07:37:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:24 smithi170 ceph-mon[33394]: 7.a scrub starts 2024-02-10T07:37:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:24 smithi170 ceph-mon[33394]: 4.1f scrub starts 2024-02-10T07:37:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:24 smithi170 ceph-mon[33394]: 4.1f scrub ok 2024-02-10T07:37:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:24 smithi170 ceph-mon[33394]: 3.5 scrub starts 2024-02-10T07:37:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:24 smithi170 ceph-mon[33394]: 3.5 scrub ok 2024-02-10T07:37:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:24 smithi170 ceph-mon[33394]: 3.17 scrub starts 2024-02-10T07:37:25.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:24 smithi170 ceph-mon[33394]: 3.17 scrub ok 2024-02-10T07:37:25.448 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:25.472 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:25.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:25.574 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:25.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:25.616 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:25.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:25.717 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:25.755 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:25.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:25.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:25.857 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:25.906 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:25.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:26.007 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:26.037 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:26.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:25 smithi008 ceph-mon[29816]: 3.15 scrub starts 2024-02-10T07:37:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:25 smithi008 ceph-mon[29816]: 3.15 scrub ok 2024-02-10T07:37:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:25 smithi008 ceph-mon[29816]: 7.a scrub ok 2024-02-10T07:37:26.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:25 smithi008 ceph-mon[40743]: 3.15 scrub starts 2024-02-10T07:37:26.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:25 smithi008 ceph-mon[40743]: 3.15 scrub ok 2024-02-10T07:37:26.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:25 smithi008 ceph-mon[40743]: 7.a scrub ok 2024-02-10T07:37:26.139 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:26.167 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:26.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:26.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:26.269 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:26.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:26.303 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:26.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:25 smithi170 ceph-mon[33394]: 3.15 scrub starts 2024-02-10T07:37:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:25 smithi170 ceph-mon[33394]: 3.15 scrub ok 2024-02-10T07:37:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:25 smithi170 ceph-mon[33394]: 7.a scrub ok 2024-02-10T07:37:26.404 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:26.473 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:26.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:26.489 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:26.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:26.502 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:26.590 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:26.619 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:26.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:26.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:26.721 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:26.769 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:26.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:26.870 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:26.920 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:26.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:27.021 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:27.051 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:27.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:27.152 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:26 smithi008 ceph-mon[29816]: pgmap v249: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 1.9 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 310 op/s; 7250/93066 objects misplaced (7.790%) 2024-02-10T07:37:27.152 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:26 smithi008 ceph-mon[29816]: 3.8 scrub starts 2024-02-10T07:37:27.152 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:26 smithi008 ceph-mon[29816]: 3.8 scrub ok 2024-02-10T07:37:27.152 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:26 smithi008 ceph-mon[40743]: pgmap v249: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 1.9 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 310 op/s; 7250/93066 objects misplaced (7.790%) 2024-02-10T07:37:27.152 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:26 smithi008 ceph-mon[40743]: 3.8 scrub starts 2024-02-10T07:37:27.152 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:26 smithi008 ceph-mon[40743]: 3.8 scrub ok 2024-02-10T07:37:27.153 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:27.197 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:27.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:27.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:27.299 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:27.335 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:27.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:26 smithi170 ceph-mon[33394]: pgmap v249: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 1.9 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 310 op/s; 7250/93066 objects misplaced (7.790%) 2024-02-10T07:37:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:26 smithi170 ceph-mon[33394]: 3.8 scrub starts 2024-02-10T07:37:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:26 smithi170 ceph-mon[33394]: 3.8 scrub ok 2024-02-10T07:37:27.437 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:27.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:27.491 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:27.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:27.592 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:27.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:27.621 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:27.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:27.722 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:27.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:27.761 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:27.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:27.862 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:27.892 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:27.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:27.971 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.3 to 1 (10000) 2024-02-10T07:37:27.993 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:28.021 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:28.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:28.122 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:28.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:28.161 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:28.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:28.230 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:37:28.262 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:28.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:27 smithi008 ceph-mon[29816]: 4.14 scrub starts 2024-02-10T07:37:28.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:27 smithi008 ceph-mon[29816]: 4.14 scrub ok 2024-02-10T07:37:28.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:27 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/282799845' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:37:28.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:27 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:37:28.273 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:27 smithi008 ceph-mon[40743]: 4.14 scrub starts 2024-02-10T07:37:28.273 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:27 smithi008 ceph-mon[40743]: 4.14 scrub ok 2024-02-10T07:37:28.273 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:27 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/282799845' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:37:28.274 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:27 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:37:28.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:27 smithi170 ceph-mon[33394]: 4.14 scrub starts 2024-02-10T07:37:28.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:27 smithi170 ceph-mon[33394]: 4.14 scrub ok 2024-02-10T07:37:28.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:27 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/282799845' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:37:28.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:27 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:37:28.318 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:28.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:28.420 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:28.451 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:28.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:28.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:28.553 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:28.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:28.621 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:28.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:28.722 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:28.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:28.759 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:28.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:28.860 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:28.890 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:28.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:28.992 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:29.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:29.036 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:29.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:29.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:37:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:37:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:37:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:28 smithi008 ceph-mon[29816]: pgmap v250: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 434 op/s; 7250/87624 objects misplaced (8.274%) 2024-02-10T07:37:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:28 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:37:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:28 smithi008 ceph-mon[29816]: osdmap e182: 8 total, 8 up, 8 in 2024-02-10T07:37:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:28 smithi008 ceph-mon[40743]: pgmap v250: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 434 op/s; 7250/87624 objects misplaced (8.274%) 2024-02-10T07:37:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:28 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:37:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:28 smithi008 ceph-mon[40743]: osdmap e182: 8 total, 8 up, 8 in 2024-02-10T07:37:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:28 smithi170 ceph-mon[33394]: pgmap v250: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 434 op/s; 7250/87624 objects misplaced (8.274%) 2024-02-10T07:37:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:28 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:37:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:28 smithi170 ceph-mon[33394]: osdmap e182: 8 total, 8 up, 8 in 2024-02-10T07:37:29.136 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:29.180 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:29.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:29.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:29.282 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:29.316 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:29.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:29.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:29.418 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:29.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:29.442 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:29.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:29.543 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:29.581 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:29.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:29.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:29.683 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:29.716 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:29.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:29.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:29.818 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:29.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:29.852 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:29.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:29.914 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 4 1 2024-02-10T07:37:29.952 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:30.005 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:30.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:30.107 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:30.129 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:29 smithi008 ceph-mon[29816]: 6.1e scrub starts 2024-02-10T07:37:30.129 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:29 smithi008 ceph-mon[29816]: 6.1e scrub ok 2024-02-10T07:37:30.129 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:29 smithi008 ceph-mon[40743]: 6.1e scrub starts 2024-02-10T07:37:30.129 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:29 smithi008 ceph-mon[40743]: 6.1e scrub ok 2024-02-10T07:37:30.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:30.174 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:30.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:30.275 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:30.307 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:30.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:29 smithi170 ceph-mon[33394]: 6.1e scrub starts 2024-02-10T07:37:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:29 smithi170 ceph-mon[33394]: 6.1e scrub ok 2024-02-10T07:37:30.408 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:30.437 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:30.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:30.539 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:30.581 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:30.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:30.682 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:30.740 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:30.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:30.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:30.845 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:30.885 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:30.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:30.933 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:37:30.934 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:37:30.934 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.934 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDPGPushReply(7.19 182/138 [PushReplyOp(7:9c6a01c8:::benchmark_data_smithi008_82046_object92375:head)])", 2024-02-10T07:37:30.934 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:30.930981+0000", 2024-02-10T07:37:30.934 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0012673350000000001, 2024-02-10T07:37:30.934 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0012943, 2024-02-10T07:37:30.934 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:30.934 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:37:30.934 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:30.934 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.935 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:30.935 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930981+0000", 2024-02-10T07:37:30.935 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.935 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.935 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.935 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:30.935 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930981+0000", 2024-02-10T07:37:30.935 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.935 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.935 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.935 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:30.935 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930981+0000", 2024-02-10T07:37:30.936 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1799999999999995e-07 2024-02-10T07:37:30.936 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.936 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.936 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:30.936 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930989+0000", 2024-02-10T07:37:30.936 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.221e-06 2024-02-10T07:37:30.936 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.936 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.936 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:30.936 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930998+0000", 2024-02-10T07:37:30.936 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8800000000000003e-06 2024-02-10T07:37:30.936 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.937 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.937 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:30.937 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931007+0000", 2024-02-10T07:37:30.937 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3799999999999994e-06 2024-02-10T07:37:30.937 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.937 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:30.937 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.937 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.937 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.937 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:207109 7.14 7.df287e94 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-02-10T07:37:30.937 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:30.931037+0000", 2024-02-10T07:37:30.937 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0012113060000000001, 2024-02-10T07:37:30.937 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001295346, 2024-02-10T07:37:30.938 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:30.938 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:37:30.938 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:30.938 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:30.938 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:30.938 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 207109 2024-02-10T07:37:30.938 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.938 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:30.938 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.938 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:30.938 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931037+0000", 2024-02-10T07:37:30.938 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.939 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.939 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.939 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:30.939 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931037+0000", 2024-02-10T07:37:30.939 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.939 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.939 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.939 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:30.939 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931038+0000", 2024-02-10T07:37:30.939 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5009999999999999e-06 2024-02-10T07:37:30.939 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.939 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.939 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:30.940 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931042+0000", 2024-02-10T07:37:30.940 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7220000000000001e-06 2024-02-10T07:37:30.940 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.940 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.940 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:30.940 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931052+0000", 2024-02-10T07:37:30.940 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0128e-05 2024-02-10T07:37:30.940 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.940 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.940 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:30.940 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931067+0000", 2024-02-10T07:37:30.941 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5277e-05 2024-02-10T07:37:30.941 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.941 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:30.941 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.941 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.941 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.941 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:207110 7.19 7:9fe816b0:::benchmark_data_smithi008_82046_object91972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-02-10T07:37:30.941 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:30.931089+0000", 2024-02-10T07:37:30.941 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0011593160000000001, 2024-02-10T07:37:30.941 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0013072769999999999, 2024-02-10T07:37:30.941 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:30.941 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:37:30.942 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:30.942 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:30.942 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:30.942 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 207110 2024-02-10T07:37:30.942 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.942 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:30.942 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.942 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:30.942 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931089+0000", 2024-02-10T07:37:30.942 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.942 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.942 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.943 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:30.943 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931089+0000", 2024-02-10T07:37:30.943 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.943 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.943 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.943 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:30.943 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931089+0000", 2024-02-10T07:37:30.943 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5000000000000003e-07 2024-02-10T07:37:30.943 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.943 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.943 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:30.943 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931092+0000", 2024-02-10T07:37:30.943 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.117e-06 2024-02-10T07:37:30.944 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.944 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.944 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:30.944 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931100+0000", 2024-02-10T07:37:30.944 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0199999999999994e-06 2024-02-10T07:37:30.944 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.944 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.944 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:30.944 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931107+0000", 2024-02-10T07:37:30.944 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7009999999999999e-06 2024-02-10T07:37:30.944 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.944 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.944 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:30.945 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931262+0000", 2024-02-10T07:37:30.945 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000154322 2024-02-10T07:37:30.945 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.945 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.945 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:30.945 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931401+0000", 2024-02-10T07:37:30.945 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000139546 2024-02-10T07:37:30.945 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.945 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.945 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-02-10T07:37:30.945 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931887+0000", 2024-02-10T07:37:30.945 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000485725 2024-02-10T07:37:30.945 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.946 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:30.946 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.946 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.946 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.946 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:207090 7.12 e182/138 7:4cd57185:::benchmark_data_smithi008_82046_object91952:head v 182'10609, mlcod=182'10605)", 2024-02-10T07:37:30.946 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:30.919257+0000", 2024-02-10T07:37:30.946 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.012991160999999999, 2024-02-10T07:37:30.946 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013218870000000001, 2024-02-10T07:37:30.946 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:30.946 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:37:30.946 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:30.946 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.946 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:30.946 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919257+0000", 2024-02-10T07:37:30.947 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.947 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.947 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.947 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:30.947 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919257+0000", 2024-02-10T07:37:30.947 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.947 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.947 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.947 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:30.947 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919257+0000", 2024-02-10T07:37:30.947 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7899999999999999e-07 2024-02-10T07:37:30.948 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.948 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.948 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:30.948 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919263+0000", 2024-02-10T07:37:30.948 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2140000000000004e-06 2024-02-10T07:37:30.948 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.948 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.948 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:30.948 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919270+0000", 2024-02-10T07:37:30.948 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5170000000000002e-06 2024-02-10T07:37:30.948 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.948 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.948 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:30.949 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919275+0000", 2024-02-10T07:37:30.949 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2379999999999997e-06 2024-02-10T07:37:30.949 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.949 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.949 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:30.949 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919413+0000", 2024-02-10T07:37:30.949 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013742999999999999 2024-02-10T07:37:30.949 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.949 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.949 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:30.951 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919432+0000", 2024-02-10T07:37:30.951 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9428000000000001e-05 2024-02-10T07:37:30.951 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.951 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:30.951 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.951 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.951 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.951 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:207099 7.4 7:2313bc08:::benchmark_data_smithi008_82046_object91961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-02-10T07:37:30.951 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:30.919848+0000", 2024-02-10T07:37:30.951 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.012399893, 2024-02-10T07:37:30.951 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012693795000000001, 2024-02-10T07:37:30.951 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:30.951 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:37:30.951 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:30.952 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:30.952 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:30.952 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 207099 2024-02-10T07:37:30.952 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.952 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:30.952 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.952 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:30.952 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919848+0000", 2024-02-10T07:37:30.952 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.952 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.952 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.952 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:30.952 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919848+0000", 2024-02-10T07:37:30.952 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.953 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.953 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.953 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:30.953 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919849+0000", 2024-02-10T07:37:30.953 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.22e-06 2024-02-10T07:37:30.953 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.953 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.953 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:30.953 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919853+0000", 2024-02-10T07:37:30.953 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7330000000000001e-06 2024-02-10T07:37:30.953 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.953 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.953 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:30.953 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919863+0000", 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0273e-05 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919871+0000", 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.9759999999999995e-06 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.919926+0000", 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.486e-05 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.920037+0000", 2024-02-10T07:37:30.954 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011048 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.920479+0000", 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00044221700000000001 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929514+0000", 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0090349590000000004 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930226+0000", 2024-02-10T07:37:30.955 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00071213400000000001 2024-02-10T07:37:30.956 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.956 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:30.956 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.956 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.956 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.956 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:207092 7.e e182/166 7:7e6f1266:::benchmark_data_smithi008_82046_object91954:head v 182'12931, mlcod=182'12928)", 2024-02-10T07:37:30.956 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:30.920840+0000", 2024-02-10T07:37:30.956 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.011407916000000001, 2024-02-10T07:37:30.956 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011789391, 2024-02-10T07:37:30.956 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:30.956 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:37:30.956 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:30.956 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.920840+0000", 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.920840+0000", 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.920841+0000", 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1699999999999998e-07 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.957 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.920846+0000", 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2580000000000003e-06 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.920855+0000", 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0340000000000002e-06 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.920862+0000", 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.691e-06 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.958 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.922122+0000", 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0012606010000000001 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.922146+0000", 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3459999999999999e-05 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDPGPush(7.12 182/138 [PushOp(7:4d34343a:::benchmark_data_smithi008_82046_object106121:head, version: 128'6751, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:4d34343a:::benchmark_data_smithi008_82046_object106121:head@128'6751, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:30.928193+0000", 2024-02-10T07:37:30.959 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.004055108, 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0045252859999999999, 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.928193+0000", 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.928193+0000", 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.960 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.928194+0000", 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8599999999999998e-07 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.928242+0000", 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7960000000000002e-05 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.928265+0000", 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3740999999999999e-05 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:30.961 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.928284+0000", 2024-02-10T07:37:30.962 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8553e-05 2024-02-10T07:37:30.962 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.962 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:30.962 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.962 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.962 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.962 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:207098 7.12 e182/138 7:4e31a417:::benchmark_data_smithi008_82046_object91960:head v 182'10610, mlcod=182'10605)", 2024-02-10T07:37:30.962 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:30.928308+0000", 2024-02-10T07:37:30.962 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0039399229999999997, 2024-02-10T07:37:30.962 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0044618890000000001, 2024-02-10T07:37:30.962 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:30.962 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:37:30.962 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:30.963 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.963 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:30.963 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.928308+0000", 2024-02-10T07:37:30.963 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.963 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.963 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.963 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:30.963 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.928308+0000", 2024-02-10T07:37:30.963 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.963 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.963 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.963 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:30.963 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.928309+0000", 2024-02-10T07:37:30.964 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1600000000000001e-07 2024-02-10T07:37:30.964 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.964 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.964 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:30.964 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.928313+0000", 2024-02-10T07:37:30.964 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4939999999999997e-06 2024-02-10T07:37:30.964 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.964 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.964 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:30.964 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.928321+0000", 2024-02-10T07:37:30.964 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.14e-06 2024-02-10T07:37:30.964 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.964 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.965 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:30.965 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.928327+0000", 2024-02-10T07:37:30.965 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.784e-06 2024-02-10T07:37:30.965 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.965 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.965 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:30.965 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.928398+0000", 2024-02-10T07:37:30.965 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0873999999999999e-05 2024-02-10T07:37:30.965 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.965 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.965 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:30.965 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.928421+0000", 2024-02-10T07:37:30.966 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2810999999999999e-05 2024-02-10T07:37:30.966 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.966 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:30.966 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.966 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.966 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.966 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:207093 7.e e182/166 7:795116d1:::benchmark_data_smithi008_82046_object91955:head v 182'12932, mlcod=182'12928)", 2024-02-10T07:37:30.966 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:30.929650+0000", 2024-02-10T07:37:30.966 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0025983299999999998, 2024-02-10T07:37:30.966 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003185149, 2024-02-10T07:37:30.966 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:30.966 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:37:30.967 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:30.967 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.967 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:30.967 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929650+0000", 2024-02-10T07:37:30.967 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.967 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.967 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.967 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:30.967 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929650+0000", 2024-02-10T07:37:30.967 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.967 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.967 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.968 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:30.968 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929651+0000", 2024-02-10T07:37:30.968 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2999999999999999e-07 2024-02-10T07:37:30.968 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.968 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.968 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:30.968 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929658+0000", 2024-02-10T07:37:30.968 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2470000000000002e-06 2024-02-10T07:37:30.968 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.968 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.968 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:30.969 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929668+0000", 2024-02-10T07:37:30.969 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0513000000000001e-05 2024-02-10T07:37:30.969 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.969 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.969 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:30.969 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929675+0000", 2024-02-10T07:37:30.969 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6340000000000003e-06 2024-02-10T07:37:30.969 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.969 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.969 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:30.969 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929786+0000", 2024-02-10T07:37:30.969 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000110849 2024-02-10T07:37:30.969 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.970 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.970 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:30.970 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929806+0000", 2024-02-10T07:37:30.970 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9743000000000001e-05 2024-02-10T07:37:30.970 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.970 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:30.970 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.970 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.970 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.970 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:207100 7.12 e182/138 7:4ceba7f7:::benchmark_data_smithi008_82046_object91962:head v 182'10611, mlcod=182'10607)", 2024-02-10T07:37:30.970 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:30.929854+0000", 2024-02-10T07:37:30.970 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.002394087, 2024-02-10T07:37:30.971 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0030420859999999998, 2024-02-10T07:37:30.971 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:30.971 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:37:30.971 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:30.971 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.971 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:30.971 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929854+0000", 2024-02-10T07:37:30.971 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.971 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.971 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.971 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:30.971 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929854+0000", 2024-02-10T07:37:30.972 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.972 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.972 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.972 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:30.972 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929854+0000", 2024-02-10T07:37:30.972 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9900000000000001e-07 2024-02-10T07:37:30.972 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.972 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.972 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:30.972 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929859+0000", 2024-02-10T07:37:30.972 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9309999999999999e-06 2024-02-10T07:37:30.972 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.972 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.972 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929868+0000", 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3699999999999995e-06 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.929874+0000", 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5520000000000001e-06 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931087+0000", 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0012131279999999999 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.973 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:30.974 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931109+0000", 2024-02-10T07:37:30.974 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1613000000000001e-05 2024-02-10T07:37:30.974 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.974 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:30.974 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.974 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.974 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.974 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:207104 7.d e182/98 7:bf7df337:::benchmark_data_smithi008_82046_object91966:head v 182'12757, mlcod=182'12755)", 2024-02-10T07:37:30.974 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:30.930330+0000", 2024-02-10T07:37:30.974 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0019176410000000001, 2024-02-10T07:37:30.974 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002628876, 2024-02-10T07:37:30.974 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:30.974 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:37:30.975 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:30.975 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.975 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:30.975 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930330+0000", 2024-02-10T07:37:30.975 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.975 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.975 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.975 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:30.975 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930330+0000", 2024-02-10T07:37:30.975 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.975 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.975 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.975 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:30.976 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930331+0000", 2024-02-10T07:37:30.976 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7299999999999996e-07 2024-02-10T07:37:30.976 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.976 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.976 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:30.976 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930336+0000", 2024-02-10T07:37:30.976 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9860000000000002e-06 2024-02-10T07:37:30.976 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.976 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.976 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:30.976 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930344+0000", 2024-02-10T07:37:30.976 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7409999999999992e-06 2024-02-10T07:37:30.977 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.977 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.977 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:30.977 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930350+0000", 2024-02-10T07:37:30.977 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8390000000000003e-06 2024-02-10T07:37:30.977 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.977 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.977 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:30.977 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931364+0000", 2024-02-10T07:37:30.977 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0010143800000000001 2024-02-10T07:37:30.977 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.977 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.978 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:30.978 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.931384+0000", 2024-02-10T07:37:30.978 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9633999999999999e-05 2024-02-10T07:37:30.978 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.978 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:30.978 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.978 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.978 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.978 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDPGPushReply(7.19 182/138 [PushReplyOp(7:9c6a01c8:::benchmark_data_smithi008_82046_object92375:head)])", 2024-02-10T07:37:30.978 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:30.930573+0000", 2024-02-10T07:37:30.978 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0016748620000000001, 2024-02-10T07:37:30.978 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002448104, 2024-02-10T07:37:30.978 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:30.978 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:37:30.979 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:30.979 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.979 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:30.979 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930573+0000", 2024-02-10T07:37:30.979 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.979 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.979 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.979 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:30.979 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930573+0000", 2024-02-10T07:37:30.979 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:30.979 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.979 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.980 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:30.980 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930574+0000", 2024-02-10T07:37:30.980 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8700000000000001e-07 2024-02-10T07:37:30.980 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.980 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.980 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:30.980 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930580+0000", 2024-02-10T07:37:30.980 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.336e-06 2024-02-10T07:37:30.980 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.980 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.980 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:30.981 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930588+0000", 2024-02-10T07:37:30.981 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4679999999999992e-06 2024-02-10T07:37:30.981 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:30.981 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:30.981 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:30.981 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:30.930596+0000", 2024-02-10T07:37:30.981 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6960000000000005e-06 2024-02-10T07:37:30.981 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.981 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:30.981 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.981 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:30.981 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:37:30.981 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 12 2024-02-10T07:37:30.981 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:37:30.986 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:31.015 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:31.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:31.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:31.117 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:31.145 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:31.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:31.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:31.247 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:31.259 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:30 smithi008 ceph-mon[40743]: 5.18 scrub starts 2024-02-10T07:37:31.259 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:30 smithi008 ceph-mon[40743]: 5.18 scrub ok 2024-02-10T07:37:31.259 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:30 smithi008 ceph-mon[40743]: pgmap v252: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 1.6 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 440 op/s; 4917/78672 objects misplaced (6.250%) 2024-02-10T07:37:31.259 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:30 smithi008 ceph-mon[40743]: 4.8 scrub starts 2024-02-10T07:37:31.259 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:30 smithi008 ceph-mon[40743]: 4.8 scrub ok 2024-02-10T07:37:31.260 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:30 smithi008 ceph-mon[40743]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:37:31.260 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:30 smithi008 ceph-mon[29816]: 5.18 scrub starts 2024-02-10T07:37:31.260 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:30 smithi008 ceph-mon[29816]: 5.18 scrub ok 2024-02-10T07:37:31.260 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:30 smithi008 ceph-mon[29816]: pgmap v252: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 1.6 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 440 op/s; 4917/78672 objects misplaced (6.250%) 2024-02-10T07:37:31.260 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:30 smithi008 ceph-mon[29816]: 4.8 scrub starts 2024-02-10T07:37:31.260 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:30 smithi008 ceph-mon[29816]: 4.8 scrub ok 2024-02-10T07:37:31.260 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:30 smithi008 ceph-mon[29816]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:37:31.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:31.279 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:31.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:31.380 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:30 smithi170 ceph-mon[33394]: 5.18 scrub starts 2024-02-10T07:37:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:30 smithi170 ceph-mon[33394]: 5.18 scrub ok 2024-02-10T07:37:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:30 smithi170 ceph-mon[33394]: pgmap v252: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 1.6 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 440 op/s; 4917/78672 objects misplaced (6.250%) 2024-02-10T07:37:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:30 smithi170 ceph-mon[33394]: 4.8 scrub starts 2024-02-10T07:37:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:30 smithi170 ceph-mon[33394]: 4.8 scrub ok 2024-02-10T07:37:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:30 smithi170 ceph-mon[33394]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-10T07:37:31.421 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:31.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:31.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:31.523 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:31.555 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:31.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:31.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:31.658 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:31.689 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:31.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:31.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:31.791 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:31.828 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:31.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:31.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:31.973 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:31.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:31.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:32.051 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-10T07:37:32.076 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:32.176 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:32.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:32.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:31 smithi008 ceph-mon[29816]: 4.17 scrub starts 2024-02-10T07:37:32.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:31 smithi008 ceph-mon[29816]: 4.17 scrub ok 2024-02-10T07:37:32.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:31 smithi008 ceph-mon[29816]: osdmap e183: 8 total, 8 up, 8 in 2024-02-10T07:37:32.273 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:31 smithi008 ceph-mon[40743]: 4.17 scrub starts 2024-02-10T07:37:32.274 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:31 smithi008 ceph-mon[40743]: 4.17 scrub ok 2024-02-10T07:37:32.274 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:31 smithi008 ceph-mon[40743]: osdmap e183: 8 total, 8 up, 8 in 2024-02-10T07:37:32.277 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:32.305 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:32.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:32.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:32.382 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:32.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:31 smithi170 ceph-mon[33394]: 4.17 scrub starts 2024-02-10T07:37:32.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:31 smithi170 ceph-mon[33394]: 4.17 scrub ok 2024-02-10T07:37:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:31 smithi170 ceph-mon[33394]: osdmap e183: 8 total, 8 up, 8 in 2024-02-10T07:37:32.407 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:32.434 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:32.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:32.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:32.472 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:32.536 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:32.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:32.592 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:32.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:32.693 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:32.723 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:32.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:32.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:32.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:32.852 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:32.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:32.954 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:32.985 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:32.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:32.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:33.087 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:33.115 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:33.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:33.189 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:32 smithi008 ceph-mon[29816]: pgmap v254: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 1.6 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 617 op/s; 4917/78252 objects misplaced (6.284%); 4.1 MiB/s, 65 objects/s recovering 2024-02-10T07:37:33.189 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:32 smithi008 ceph-mon[29816]: 4.e scrub starts 2024-02-10T07:37:33.189 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:32 smithi008 ceph-mon[29816]: 4.e scrub ok 2024-02-10T07:37:33.189 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:33.189 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:32 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2478684872' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:37:33.189 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:32 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:37:33.190 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:32 smithi008 ceph-mon[40743]: pgmap v254: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 1.6 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 617 op/s; 4917/78252 objects misplaced (6.284%); 4.1 MiB/s, 65 objects/s recovering 2024-02-10T07:37:33.190 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:32 smithi008 ceph-mon[40743]: 4.e scrub starts 2024-02-10T07:37:33.190 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:32 smithi008 ceph-mon[40743]: 4.e scrub ok 2024-02-10T07:37:33.190 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:33.190 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:32 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2478684872' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:37:33.190 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:32 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:37:33.218 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:33.261 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:33.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:33.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:33.363 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:33.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:37:32 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:37:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:32 smithi170 ceph-mon[33394]: pgmap v254: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 1.6 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 617 op/s; 4917/78252 objects misplaced (6.284%); 4.1 MiB/s, 65 objects/s recovering 2024-02-10T07:37:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:32 smithi170 ceph-mon[33394]: 4.e scrub starts 2024-02-10T07:37:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:32 smithi170 ceph-mon[33394]: 4.e scrub ok 2024-02-10T07:37:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:32 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2478684872' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:37:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:32 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:37:33.392 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:33.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:33.494 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:33.537 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:33.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:33.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:33.639 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:33.667 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:33.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:33.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:33.769 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:33.771 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:37:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:33.812 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:33.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:33.913 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:33.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:33.939 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:33.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:34.040 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:34.079 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:34.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:34.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:34.181 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:34.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:34.222 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:34.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:34.323 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:34.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:37:34.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:37:34.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[29816]: 4.19 scrub starts 2024-02-10T07:37:34.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[29816]: 5.f scrub starts 2024-02-10T07:37:34.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[29816]: 5.f scrub ok 2024-02-10T07:37:34.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[29816]: 4.19 scrub ok 2024-02-10T07:37:34.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:34.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:37:34.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:37:34.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:37:34.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:37:34.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[29816]: osdmap e184: 8 total, 8 up, 8 in 2024-02-10T07:37:34.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:37:34.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:37:34.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[40743]: 4.19 scrub starts 2024-02-10T07:37:34.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[40743]: 5.f scrub starts 2024-02-10T07:37:34.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[40743]: 5.f scrub ok 2024-02-10T07:37:34.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[40743]: 4.19 scrub ok 2024-02-10T07:37:34.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:34.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:37:34.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:37:34.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:37:34.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:37:34.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:33 smithi008 ceph-mon[40743]: osdmap e184: 8 total, 8 up, 8 in 2024-02-10T07:37:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:34.361 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:34.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:33 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:37:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:33 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:37:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:33 smithi170 ceph-mon[33394]: 4.19 scrub starts 2024-02-10T07:37:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:33 smithi170 ceph-mon[33394]: 5.f scrub starts 2024-02-10T07:37:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:33 smithi170 ceph-mon[33394]: 5.f scrub ok 2024-02-10T07:37:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:33 smithi170 ceph-mon[33394]: 4.19 scrub ok 2024-02-10T07:37:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:33 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:34.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:33 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:37:34.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:33 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:37:34.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:33 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:37:34.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:33 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:37:34.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:33 smithi170 ceph-mon[33394]: osdmap e184: 8 total, 8 up, 8 in 2024-02-10T07:37:34.462 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:34.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:34.494 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:34.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:34.595 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:34.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:34.631 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:34.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:34.732 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:34.773 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:34.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:34.873 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:34.902 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:34.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:34.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:35.004 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:35.044 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:35.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:35.146 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:35.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:35.179 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:35.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:35.280 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:35.312 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:35.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:35.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:35.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:34 smithi008 ceph-mon[29816]: 5.1c scrub starts 2024-02-10T07:37:35.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:35 smithi008 ceph-mon[29816]: 5.1c scrub ok 2024-02-10T07:37:35.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:35 smithi008 ceph-mon[29816]: pgmap v256: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 1.3 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.29k op/s; 2253/64395 objects misplaced (3.499%); 14 MiB/s, 222 objects/s recovering 2024-02-10T07:37:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:34 smithi008 ceph-mon[40743]: 5.1c scrub starts 2024-02-10T07:37:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:35 smithi008 ceph-mon[40743]: 5.1c scrub ok 2024-02-10T07:37:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:35 smithi008 ceph-mon[40743]: pgmap v256: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 1.3 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.29k op/s; 2253/64395 objects misplaced (3.499%); 14 MiB/s, 222 objects/s recovering 2024-02-10T07:37:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:34 smithi170 ceph-mon[33394]: 5.1c scrub starts 2024-02-10T07:37:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:34 smithi170 ceph-mon[33394]: 5.1c scrub ok 2024-02-10T07:37:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:34 smithi170 ceph-mon[33394]: pgmap v256: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 1.3 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.29k op/s; 2253/64395 objects misplaced (3.499%); 14 MiB/s, 222 objects/s recovering 2024-02-10T07:37:35.414 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:35.446 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:35.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:35.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:35.548 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:35.580 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:35.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:35.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:35.681 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:35.705 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:35.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:35.807 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:35.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:35.832 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:35.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:35.933 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:35.964 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:35.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:35.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:36.065 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:36.110 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:36.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:36.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:36.212 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:36.248 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:36.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:36.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:36.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:35 smithi008 ceph-mon[29816]: 7.12 scrub starts 2024-02-10T07:37:36.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:36 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:37:36.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:36 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:37:36.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:36 smithi008 ceph-mon[29816]: osdmap e185: 8 total, 8 up, 8 in 2024-02-10T07:37:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:35 smithi008 ceph-mon[40743]: 7.12 scrub starts 2024-02-10T07:37:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:36 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:37:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:36 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:37:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:36 smithi008 ceph-mon[40743]: osdmap e185: 8 total, 8 up, 8 in 2024-02-10T07:37:36.350 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:36.377 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:36.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:36.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:35 smithi170 ceph-mon[33394]: 7.12 scrub starts 2024-02-10T07:37:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:35 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:37:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:35 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:37:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:35 smithi170 ceph-mon[33394]: osdmap e185: 8 total, 8 up, 8 in 2024-02-10T07:37:36.480 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:36.509 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:36.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:36.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:36.611 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:36.642 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:36.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:36.744 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:36.773 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:36.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:36.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:36.875 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:36.909 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:36.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:36.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:37.010 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:37.042 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:37.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:37.144 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:37.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:37.184 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:37.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:37.285 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:37.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:37.330 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:37.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:37.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:36 smithi008 ceph-mon[29816]: pgmap v258: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 1.3 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 793 op/s; 2253/64395 objects misplaced (3.499%); 8.5 MiB/s, 135 objects/s recovering 2024-02-10T07:37:37.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:37 smithi008 ceph-mon[29816]: 5.11 scrub starts 2024-02-10T07:37:37.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:37 smithi008 ceph-mon[29816]: 5.11 scrub ok 2024-02-10T07:37:37.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:37 smithi008 ceph-mon[29816]: osdmap e186: 8 total, 8 up, 8 in 2024-02-10T07:37:37.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:37 smithi008 ceph-mon[29816]: osdmap e187: 8 total, 8 up, 8 in 2024-02-10T07:37:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:36 smithi008 ceph-mon[40743]: pgmap v258: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 1.3 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 793 op/s; 2253/64395 objects misplaced (3.499%); 8.5 MiB/s, 135 objects/s recovering 2024-02-10T07:37:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:37 smithi008 ceph-mon[40743]: 5.11 scrub starts 2024-02-10T07:37:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:37 smithi008 ceph-mon[40743]: 5.11 scrub ok 2024-02-10T07:37:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:37 smithi008 ceph-mon[40743]: osdmap e186: 8 total, 8 up, 8 in 2024-02-10T07:37:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:37 smithi008 ceph-mon[40743]: osdmap e187: 8 total, 8 up, 8 in 2024-02-10T07:37:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:36 smithi170 ceph-mon[33394]: pgmap v258: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 1.3 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 793 op/s; 2253/64395 objects misplaced (3.499%); 8.5 MiB/s, 135 objects/s recovering 2024-02-10T07:37:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:36 smithi170 ceph-mon[33394]: 5.11 scrub starts 2024-02-10T07:37:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:36 smithi170 ceph-mon[33394]: 5.11 scrub ok 2024-02-10T07:37:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:36 smithi170 ceph-mon[33394]: osdmap e186: 8 total, 8 up, 8 in 2024-02-10T07:37:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:36 smithi170 ceph-mon[33394]: osdmap e187: 8 total, 8 up, 8 in 2024-02-10T07:37:37.431 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:37.460 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:37.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:37.561 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:37.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:37.595 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:37.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:37.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:37.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:37.721 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:37.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:37.822 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:37.854 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:37.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:37.956 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:37.987 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:37.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:37.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:38.089 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:38.118 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:38.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:38.220 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:38.253 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:38.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:38.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:37 smithi008 ceph-mon[29816]: 3.9 deep-scrub starts 2024-02-10T07:37:38.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[29816]: 3.9 deep-scrub ok 2024-02-10T07:37:38.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[29816]: 7.12 scrub starts 2024-02-10T07:37:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:37 smithi008 ceph-mon[40743]: 3.9 deep-scrub starts 2024-02-10T07:37:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[40743]: 3.9 deep-scrub ok 2024-02-10T07:37:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[40743]: 7.12 scrub starts 2024-02-10T07:37:38.354 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:38.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:37 smithi170 ceph-mon[33394]: 3.9 deep-scrub starts 2024-02-10T07:37:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:37 smithi170 ceph-mon[33394]: 3.9 deep-scrub ok 2024-02-10T07:37:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:37 smithi170 ceph-mon[33394]: 7.12 scrub starts 2024-02-10T07:37:38.402 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:38.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:38.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:38.504 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:38.550 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:38.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:38.651 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:38.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:38.688 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:38.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:38.789 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:38.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:38.825 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:38.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:38.926 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:38.965 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:38.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:38.969 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:37:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:37:39.067 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:39.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:39.119 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:39.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:39.222 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:39.252 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:39.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[29816]: 7.12 scrub ok 2024-02-10T07:37:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[29816]: 7.19 scrub starts 2024-02-10T07:37:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[29816]: pgmap v261: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1695/59877 objects misplaced (2.831%) 2024-02-10T07:37:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:37:39.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:37:39.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[29816]: 7.19 scrub ok 2024-02-10T07:37:39.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[29816]: osdmap e188: 8 total, 8 up, 8 in 2024-02-10T07:37:39.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:37:39.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[29816]: 7.14 scrub starts 2024-02-10T07:37:39.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[40743]: 7.12 scrub ok 2024-02-10T07:37:39.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[40743]: 7.19 scrub starts 2024-02-10T07:37:39.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[40743]: pgmap v261: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1695/59877 objects misplaced (2.831%) 2024-02-10T07:37:39.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:37:39.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:37:39.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[40743]: 7.19 scrub ok 2024-02-10T07:37:39.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[40743]: osdmap e188: 8 total, 8 up, 8 in 2024-02-10T07:37:39.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:37:39.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:38 smithi008 ceph-mon[40743]: 7.14 scrub starts 2024-02-10T07:37:39.354 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:38 smithi170 ceph-mon[33394]: 7.12 scrub ok 2024-02-10T07:37:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:38 smithi170 ceph-mon[33394]: 7.19 scrub starts 2024-02-10T07:37:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:38 smithi170 ceph-mon[33394]: pgmap v261: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 1695/59877 objects misplaced (2.831%) 2024-02-10T07:37:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:38 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:37:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:38 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:37:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:38 smithi170 ceph-mon[33394]: 7.19 scrub ok 2024-02-10T07:37:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:38 smithi170 ceph-mon[33394]: osdmap e188: 8 total, 8 up, 8 in 2024-02-10T07:37:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:38 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:37:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:38 smithi170 ceph-mon[33394]: 7.14 scrub starts 2024-02-10T07:37:39.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:39.395 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:39.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:39.496 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:39.523 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:39.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:39.625 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:39.648 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:39.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:39.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:39.751 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:39.776 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:39.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:39.877 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:39.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:39.902 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:39.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:39.961 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:37:39.995 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:37:40.003 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:40.019 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:37:40.033 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:40.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:40.043 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:37:40.083 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:37:40.125 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:37:40.135 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:40.172 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:37:40.173 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:40.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:40.209 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:37:40.219 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:37:40.219 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:37:40.219 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:37:40.220 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:37:40.220 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:37:40.220 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:37:40.220 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:37:40.220 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:37:40.220 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[29816]: 3.0 scrub starts 2024-02-10T07:37:40.220 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[29816]: 7.14 scrub ok 2024-02-10T07:37:40.221 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[29816]: 3.0 scrub ok 2024-02-10T07:37:40.221 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[29816]: osdmap e189: 8 total, 8 up, 8 in 2024-02-10T07:37:40.221 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[29816]: 6.6 deep-scrub starts 2024-02-10T07:37:40.221 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[29816]: 6.6 deep-scrub ok 2024-02-10T07:37:40.221 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:40.221 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:40.221 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/460135910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:37:40.221 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[40743]: 3.0 scrub starts 2024-02-10T07:37:40.221 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[40743]: 7.14 scrub ok 2024-02-10T07:37:40.221 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[40743]: 3.0 scrub ok 2024-02-10T07:37:40.221 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[40743]: osdmap e189: 8 total, 8 up, 8 in 2024-02-10T07:37:40.221 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[40743]: 6.6 deep-scrub starts 2024-02-10T07:37:40.221 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[40743]: 6.6 deep-scrub ok 2024-02-10T07:37:40.222 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:40.222 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:40.222 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:40 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/460135910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:37:40.275 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:40.303 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:40.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:40.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:40 smithi170 ceph-mon[33394]: 3.0 scrub starts 2024-02-10T07:37:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:40 smithi170 ceph-mon[33394]: 7.14 scrub ok 2024-02-10T07:37:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:40 smithi170 ceph-mon[33394]: 3.0 scrub ok 2024-02-10T07:37:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:40 smithi170 ceph-mon[33394]: osdmap e189: 8 total, 8 up, 8 in 2024-02-10T07:37:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:40 smithi170 ceph-mon[33394]: 6.6 deep-scrub starts 2024-02-10T07:37:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:40 smithi170 ceph-mon[33394]: 6.6 deep-scrub ok 2024-02-10T07:37:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:40 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:40 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:40 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/460135910' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:37:40.405 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:40.435 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:40.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:40.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:40.537 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:40.577 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:40.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:40.679 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:40.716 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:40.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:40.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:40.818 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:40.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:40.849 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:40.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:40.950 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:40.979 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:40.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:40.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:41.081 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:41.111 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:41.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:41.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:41.195 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:41 smithi008 ceph-mon[29816]: 1.0 scrub starts 2024-02-10T07:37:41.195 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:41 smithi008 ceph-mon[29816]: 1.0 scrub ok 2024-02-10T07:37:41.195 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:41 smithi008 ceph-mon[29816]: pgmap v264: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.1 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 156/53868 objects misplaced (0.290%) 2024-02-10T07:37:41.195 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:41 smithi008 ceph-mon[29816]: osdmap e190: 8 total, 8 up, 8 in 2024-02-10T07:37:41.195 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:41 smithi008 ceph-mon[29816]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-10T07:37:41.195 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:41 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:37:41.195 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:41 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:41.196 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:41 smithi008 ceph-mon[40743]: 1.0 scrub starts 2024-02-10T07:37:41.196 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:41 smithi008 ceph-mon[40743]: 1.0 scrub ok 2024-02-10T07:37:41.196 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:41 smithi008 ceph-mon[40743]: pgmap v264: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.1 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 156/53868 objects misplaced (0.290%) 2024-02-10T07:37:41.196 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:41 smithi008 ceph-mon[40743]: osdmap e190: 8 total, 8 up, 8 in 2024-02-10T07:37:41.196 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:41 smithi008 ceph-mon[40743]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-10T07:37:41.196 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:41 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:37:41.196 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:41 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:41.213 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:41.245 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:41.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:41.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:41.347 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:41 smithi170 ceph-mon[33394]: 1.0 scrub starts 2024-02-10T07:37:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:41 smithi170 ceph-mon[33394]: 1.0 scrub ok 2024-02-10T07:37:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:41 smithi170 ceph-mon[33394]: pgmap v264: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.1 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 156/53868 objects misplaced (0.290%) 2024-02-10T07:37:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:41 smithi170 ceph-mon[33394]: osdmap e190: 8 total, 8 up, 8 in 2024-02-10T07:37:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:41 smithi170 ceph-mon[33394]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-10T07:37:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:41 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:37:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:41 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:37:41.390 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:41.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:41.427 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:37:41.492 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:41.519 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:41.556 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:41.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:41.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:41.657 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:41.718 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:41.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:41.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:41.821 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:41.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:41.880 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:41.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:41.981 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:42.030 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:42.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:42.131 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:42.174 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:42.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:42.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:42.276 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:42.322 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:42.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:42.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[29816]: 6.5 scrub starts 2024-02-10T07:37:42.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[29816]: 6.5 scrub ok 2024-02-10T07:37:42.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[29816]: 3.14 scrub starts 2024-02-10T07:37:42.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[29816]: 3.14 scrub ok 2024-02-10T07:37:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[29816]: 3.16 scrub starts 2024-02-10T07:37:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[29816]: 3.16 scrub ok 2024-02-10T07:37:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:37:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[29816]: osdmap e191: 8 total, 8 up, 8 in 2024-02-10T07:37:42.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[40743]: 6.5 scrub starts 2024-02-10T07:37:42.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[40743]: 6.5 scrub ok 2024-02-10T07:37:42.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[40743]: 3.14 scrub starts 2024-02-10T07:37:42.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[40743]: 3.14 scrub ok 2024-02-10T07:37:42.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[40743]: 3.16 scrub starts 2024-02-10T07:37:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[40743]: 3.16 scrub ok 2024-02-10T07:37:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:37:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:42 smithi008 ceph-mon[40743]: osdmap e191: 8 total, 8 up, 8 in 2024-02-10T07:37:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:42 smithi170 ceph-mon[33394]: 6.5 scrub starts 2024-02-10T07:37:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:42 smithi170 ceph-mon[33394]: 6.5 scrub ok 2024-02-10T07:37:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:42 smithi170 ceph-mon[33394]: 3.14 scrub starts 2024-02-10T07:37:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:42 smithi170 ceph-mon[33394]: 3.14 scrub ok 2024-02-10T07:37:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:42 smithi170 ceph-mon[33394]: 3.16 scrub starts 2024-02-10T07:37:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:42 smithi170 ceph-mon[33394]: 3.16 scrub ok 2024-02-10T07:37:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:42 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:37:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:42 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:42 smithi170 ceph-mon[33394]: osdmap e191: 8 total, 8 up, 8 in 2024-02-10T07:37:42.424 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:42.455 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:42.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:42.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:42.557 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:42.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:42.601 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:42.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:42.702 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:42.754 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:42.760 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:42.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:42.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:42.861 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:42.905 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:42.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:43.006 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:43.049 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:43.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[29816]: 4.2 scrub starts 2024-02-10T07:37:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[29816]: 4.2 scrub ok 2024-02-10T07:37:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[29816]: 4.18 scrub starts 2024-02-10T07:37:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[29816]: 4.18 scrub ok 2024-02-10T07:37:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[29816]: pgmap v267: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.1 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 535 op/s; 156/53445 objects misplaced (0.292%); 7.3 MiB/s, 116 objects/s recovering 2024-02-10T07:37:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[29816]: 4.1e scrub starts 2024-02-10T07:37:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[29816]: 4.1e scrub ok 2024-02-10T07:37:43.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2177675335' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:37:43.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[40743]: 4.2 scrub starts 2024-02-10T07:37:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[40743]: 4.2 scrub ok 2024-02-10T07:37:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[40743]: 4.18 scrub starts 2024-02-10T07:37:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[40743]: 4.18 scrub ok 2024-02-10T07:37:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[40743]: pgmap v267: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.1 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 535 op/s; 156/53445 objects misplaced (0.292%); 7.3 MiB/s, 116 objects/s recovering 2024-02-10T07:37:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[40743]: 4.1e scrub starts 2024-02-10T07:37:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[40743]: 4.1e scrub ok 2024-02-10T07:37:43.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:43 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2177675335' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:37:43.149 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:43.188 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:43.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:43.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:43.291 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:43.335 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:43.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:43.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:37:42 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:37:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:43 smithi170 ceph-mon[33394]: 4.2 scrub starts 2024-02-10T07:37:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:43 smithi170 ceph-mon[33394]: 4.2 scrub ok 2024-02-10T07:37:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:43 smithi170 ceph-mon[33394]: 4.18 scrub starts 2024-02-10T07:37:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:43 smithi170 ceph-mon[33394]: 4.18 scrub ok 2024-02-10T07:37:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:43 smithi170 ceph-mon[33394]: pgmap v267: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.1 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 535 op/s; 156/53445 objects misplaced (0.292%); 7.3 MiB/s, 116 objects/s recovering 2024-02-10T07:37:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:43 smithi170 ceph-mon[33394]: 4.1e scrub starts 2024-02-10T07:37:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:43 smithi170 ceph-mon[33394]: 4.1e scrub ok 2024-02-10T07:37:43.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:43 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2177675335' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:37:43.436 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:43.470 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:43.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:43.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:43.538 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:37:43.573 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:43.599 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:43.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:43.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:43.701 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:43.752 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:43.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:43.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:43.767 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.4 to 1 (10000) 2024-02-10T07:37:43.853 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:43.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:43.907 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:43.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:44.008 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:44.031 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:44.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:44.134 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:44.161 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:44.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:44.262 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:44.278 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:44 smithi008 ceph-mon[40743]: 3.10 scrub starts 2024-02-10T07:37:44.278 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:44 smithi008 ceph-mon[40743]: 3.10 scrub ok 2024-02-10T07:37:44.278 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:44 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:44.278 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:44 smithi008 ceph-mon[40743]: 3.12 scrub starts 2024-02-10T07:37:44.278 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:44 smithi008 ceph-mon[40743]: 3.12 scrub ok 2024-02-10T07:37:44.279 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:44 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2177675335' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:37:44.279 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:44 smithi008 ceph-mon[40743]: osdmap e192: 8 total, 8 up, 8 in 2024-02-10T07:37:44.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:44 smithi008 ceph-mon[29816]: 3.10 scrub starts 2024-02-10T07:37:44.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:44 smithi008 ceph-mon[29816]: 3.10 scrub ok 2024-02-10T07:37:44.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:44 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:44.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:44 smithi008 ceph-mon[29816]: 3.12 scrub starts 2024-02-10T07:37:44.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:44 smithi008 ceph-mon[29816]: 3.12 scrub ok 2024-02-10T07:37:44.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:44 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2177675335' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:37:44.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:44 smithi008 ceph-mon[29816]: osdmap e192: 8 total, 8 up, 8 in 2024-02-10T07:37:44.293 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:44.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:44.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:44 smithi170 ceph-mon[33394]: 3.10 scrub starts 2024-02-10T07:37:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:44 smithi170 ceph-mon[33394]: 3.10 scrub ok 2024-02-10T07:37:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:44 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:44 smithi170 ceph-mon[33394]: 3.12 scrub starts 2024-02-10T07:37:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:44 smithi170 ceph-mon[33394]: 3.12 scrub ok 2024-02-10T07:37:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:44 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2177675335' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:37:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:44 smithi170 ceph-mon[33394]: osdmap e192: 8 total, 8 up, 8 in 2024-02-10T07:37:44.396 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:44.424 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:44.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:44.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:44.526 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:44.563 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:44.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:44.664 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:44.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:44.694 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:44.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:44.795 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:44.838 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:44.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:44.940 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:44.968 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:44.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:45.070 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:45.099 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:45.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:45.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:45.230 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:45.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:45.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:45 smithi008 ceph-mon[29816]: 4.d scrub starts 2024-02-10T07:37:45.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:45 smithi008 ceph-mon[29816]: 4.d scrub ok 2024-02-10T07:37:45.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:45 smithi008 ceph-mon[29816]: 2.2 scrub starts 2024-02-10T07:37:45.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:45 smithi008 ceph-mon[29816]: 2.2 scrub ok 2024-02-10T07:37:45.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:45 smithi008 ceph-mon[29816]: pgmap v269: 158 pgs: 158 active+clean; 798 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:37:45.270 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:45 smithi008 ceph-mon[40743]: 4.d scrub starts 2024-02-10T07:37:45.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:45 smithi008 ceph-mon[40743]: 4.d scrub ok 2024-02-10T07:37:45.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:45 smithi008 ceph-mon[40743]: 2.2 scrub starts 2024-02-10T07:37:45.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:45 smithi008 ceph-mon[40743]: 2.2 scrub ok 2024-02-10T07:37:45.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:45 smithi008 ceph-mon[40743]: pgmap v269: 158 pgs: 158 active+clean; 798 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:37:45.332 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:45.362 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:45.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:45.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:45.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:45 smithi170 ceph-mon[33394]: 4.d scrub starts 2024-02-10T07:37:45.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:45 smithi170 ceph-mon[33394]: 4.d scrub ok 2024-02-10T07:37:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:45 smithi170 ceph-mon[33394]: 2.2 scrub starts 2024-02-10T07:37:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:45 smithi170 ceph-mon[33394]: 2.2 scrub ok 2024-02-10T07:37:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:45 smithi170 ceph-mon[33394]: pgmap v269: 158 pgs: 158 active+clean; 798 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:37:45.464 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:45.480 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:37:45.481 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:37:45.508 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:45.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:45.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:45.610 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:45.648 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:45.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:45.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:45.750 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:45.781 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:45.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:45.883 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:45.919 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:45.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:46.020 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:46.049 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:46.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:46.152 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:46.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:46.189 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:46.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:46.290 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:46.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:46 smithi008 ceph-mon[40743]: 4.9 scrub starts 2024-02-10T07:37:46.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:46 smithi008 ceph-mon[40743]: 4.9 scrub ok 2024-02-10T07:37:46.333 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:46.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:46 smithi008 ceph-mon[29816]: 4.9 scrub starts 2024-02-10T07:37:46.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:46 smithi008 ceph-mon[29816]: 4.9 scrub ok 2024-02-10T07:37:46.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:46 smithi170 ceph-mon[33394]: 4.9 scrub starts 2024-02-10T07:37:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:46 smithi170 ceph-mon[33394]: 4.9 scrub ok 2024-02-10T07:37:46.434 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:46.469 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:46.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:46.571 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:46.600 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:46.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:46.673 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 5 1 2024-02-10T07:37:46.701 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:46.730 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:46.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:46.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:46.832 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:46.891 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:46.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:46.992 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:47.043 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:47.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:47.144 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:47.196 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-10T07:37:47.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:47.203 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:47.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:47.304 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:47.321 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:47 smithi008 ceph-mon[40743]: 5.d scrub starts 2024-02-10T07:37:47.321 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:47 smithi008 ceph-mon[40743]: 5.d scrub ok 2024-02-10T07:37:47.321 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:47 smithi008 ceph-mon[40743]: pgmap v270: 158 pgs: 158 active+clean; 798 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:47.321 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:47 smithi008 ceph-mon[40743]: 7.c deep-scrub starts 2024-02-10T07:37:47.321 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:47 smithi008 ceph-mon[40743]: osdmap e193: 8 total, 8 up, 8 in 2024-02-10T07:37:47.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:47 smithi008 ceph-mon[29816]: 5.d scrub starts 2024-02-10T07:37:47.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:47 smithi008 ceph-mon[29816]: 5.d scrub ok 2024-02-10T07:37:47.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:47 smithi008 ceph-mon[29816]: pgmap v270: 158 pgs: 158 active+clean; 798 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:47.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:47 smithi008 ceph-mon[29816]: 7.c deep-scrub starts 2024-02-10T07:37:47.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:47 smithi008 ceph-mon[29816]: osdmap e193: 8 total, 8 up, 8 in 2024-02-10T07:37:47.335 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:47.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:47.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:47.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:47 smithi170 ceph-mon[33394]: 5.d scrub starts 2024-02-10T07:37:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:47 smithi170 ceph-mon[33394]: 5.d scrub ok 2024-02-10T07:37:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:47 smithi170 ceph-mon[33394]: pgmap v270: 158 pgs: 158 active+clean; 798 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:37:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:47 smithi170 ceph-mon[33394]: 7.c deep-scrub starts 2024-02-10T07:37:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:47 smithi170 ceph-mon[33394]: osdmap e193: 8 total, 8 up, 8 in 2024-02-10T07:37:47.437 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:47.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:47.485 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:47.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:47.586 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:47.617 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:47.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:47.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:47.719 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:47.748 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:47.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:47.772 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:47.849 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:47.857 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:47.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:47.906 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:47.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:48.007 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:48.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:48.064 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:48.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:48.103 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:48.165 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:48.196 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[29816]: 2.1 scrub starts 2024-02-10T07:37:48.196 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[29816]: 2.1 scrub ok 2024-02-10T07:37:48.196 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[29816]: 7.d scrub starts 2024-02-10T07:37:48.196 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[29816]: 7.d scrub ok 2024-02-10T07:37:48.196 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[29816]: 4.11 scrub starts 2024-02-10T07:37:48.196 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[29816]: 4.11 scrub ok 2024-02-10T07:37:48.196 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:48.196 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:48.197 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[40743]: 2.1 scrub starts 2024-02-10T07:37:48.197 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[40743]: 2.1 scrub ok 2024-02-10T07:37:48.197 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[40743]: 7.d scrub starts 2024-02-10T07:37:48.197 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[40743]: 7.d scrub ok 2024-02-10T07:37:48.197 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[40743]: 4.11 scrub starts 2024-02-10T07:37:48.197 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[40743]: 4.11 scrub ok 2024-02-10T07:37:48.197 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:48.197 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:48.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:48.224 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:48.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:48.325 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:48.358 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:48.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:48 smithi170 ceph-mon[33394]: 2.1 scrub starts 2024-02-10T07:37:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:48 smithi170 ceph-mon[33394]: 2.1 scrub ok 2024-02-10T07:37:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:48 smithi170 ceph-mon[33394]: 7.d scrub starts 2024-02-10T07:37:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:48 smithi170 ceph-mon[33394]: 7.d scrub ok 2024-02-10T07:37:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:48 smithi170 ceph-mon[33394]: 4.11 scrub starts 2024-02-10T07:37:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:48 smithi170 ceph-mon[33394]: 4.11 scrub ok 2024-02-10T07:37:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:48 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:37:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:48 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:37:48.460 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:48.494 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:48.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:48.596 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:48.643 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:48.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:48.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:48.744 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:48.783 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:48.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:48.885 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:48.913 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:48.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:49.015 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:49.022 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:37:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:37:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:37:49.042 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:49.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:49.144 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:49.179 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:49.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:49.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:49.281 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:49.318 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:49.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:49.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:49.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:49 smithi008 ceph-mon[29816]: pgmap v272: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 644 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:37:49.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:49 smithi008 ceph-mon[29816]: 5.1f scrub starts 2024-02-10T07:37:49.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:49 smithi008 ceph-mon[29816]: 5.1f scrub ok 2024-02-10T07:37:49.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:49 smithi008 ceph-mon[29816]: 7.c deep-scrub ok 2024-02-10T07:37:49.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:49 smithi008 ceph-mon[40743]: pgmap v272: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 644 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:37:49.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:49 smithi008 ceph-mon[40743]: 5.1f scrub starts 2024-02-10T07:37:49.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:49 smithi008 ceph-mon[40743]: 5.1f scrub ok 2024-02-10T07:37:49.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:49 smithi008 ceph-mon[40743]: 7.c deep-scrub ok 2024-02-10T07:37:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:49 smithi170 ceph-mon[33394]: pgmap v272: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 644 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:37:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:49 smithi170 ceph-mon[33394]: 5.1f scrub starts 2024-02-10T07:37:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:49 smithi170 ceph-mon[33394]: 5.1f scrub ok 2024-02-10T07:37:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:49 smithi170 ceph-mon[33394]: 7.c deep-scrub ok 2024-02-10T07:37:49.419 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:49.448 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:49.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:49.550 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:49.579 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:49.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:49.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:49.681 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:49.710 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:49.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:49.812 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:49.855 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:49.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:49.957 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:50.006 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:50.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:50.079 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:37:50.107 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:50.149 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:50.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:50.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:50.251 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:50.279 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:50.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:50.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[29816]: 4.c scrub starts 2024-02-10T07:37:50.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[29816]: 6.9 scrub starts 2024-02-10T07:37:50.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[29816]: 6.9 scrub ok 2024-02-10T07:37:50.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[29816]: 4.c scrub ok 2024-02-10T07:37:50.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[29816]: 5.6 scrub starts 2024-02-10T07:37:50.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[29816]: 5.6 scrub ok 2024-02-10T07:37:50.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/313556496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:37:50.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[29816]: 5.0 scrub starts 2024-02-10T07:37:50.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[29816]: 5.0 scrub ok 2024-02-10T07:37:50.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[29816]: 5.15 scrub starts 2024-02-10T07:37:50.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[29816]: 5.15 scrub ok 2024-02-10T07:37:50.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[40743]: 4.c scrub starts 2024-02-10T07:37:50.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[40743]: 6.9 scrub starts 2024-02-10T07:37:50.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[40743]: 6.9 scrub ok 2024-02-10T07:37:50.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[40743]: 4.c scrub ok 2024-02-10T07:37:50.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[40743]: 5.6 scrub starts 2024-02-10T07:37:50.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[40743]: 5.6 scrub ok 2024-02-10T07:37:50.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/313556496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:37:50.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[40743]: 5.0 scrub starts 2024-02-10T07:37:50.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[40743]: 5.0 scrub ok 2024-02-10T07:37:50.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[40743]: 5.15 scrub starts 2024-02-10T07:37:50.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:50 smithi008 ceph-mon[40743]: 5.15 scrub ok 2024-02-10T07:37:50.381 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:50 smithi170 ceph-mon[33394]: 4.c scrub starts 2024-02-10T07:37:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:50 smithi170 ceph-mon[33394]: 6.9 scrub starts 2024-02-10T07:37:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:50 smithi170 ceph-mon[33394]: 6.9 scrub ok 2024-02-10T07:37:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:50 smithi170 ceph-mon[33394]: 4.c scrub ok 2024-02-10T07:37:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:50 smithi170 ceph-mon[33394]: 5.6 scrub starts 2024-02-10T07:37:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:50 smithi170 ceph-mon[33394]: 5.6 scrub ok 2024-02-10T07:37:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:50 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/313556496' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:37:50.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:50 smithi170 ceph-mon[33394]: 5.0 scrub starts 2024-02-10T07:37:50.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:50 smithi170 ceph-mon[33394]: 5.0 scrub ok 2024-02-10T07:37:50.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:50 smithi170 ceph-mon[33394]: 5.15 scrub starts 2024-02-10T07:37:50.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:50 smithi170 ceph-mon[33394]: 5.15 scrub ok 2024-02-10T07:37:50.412 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:50.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:50.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:50.514 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:50.543 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:50.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:50.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:50.647 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:50.693 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:50.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:50.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:50.795 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:50.836 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:50.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:50.938 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:50.970 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:50.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:51.071 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:51.113 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:51.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:51.215 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:51.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:51.259 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:51.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:51.297 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[29816]: 4.1a scrub starts 2024-02-10T07:37:51.297 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[29816]: 5.5 deep-scrub starts 2024-02-10T07:37:51.297 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[29816]: 4.1a scrub ok 2024-02-10T07:37:51.297 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[29816]: 5.5 deep-scrub ok 2024-02-10T07:37:51.298 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[29816]: 3.f scrub starts 2024-02-10T07:37:51.298 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[29816]: 3.f scrub ok 2024-02-10T07:37:51.298 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[29816]: pgmap v273: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 456 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:37:51.298 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:37:51.298 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/313556496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:37:51.298 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[29816]: osdmap e194: 8 total, 8 up, 8 in 2024-02-10T07:37:51.298 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[40743]: 4.1a scrub starts 2024-02-10T07:37:51.298 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[40743]: 5.5 deep-scrub starts 2024-02-10T07:37:51.298 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[40743]: 4.1a scrub ok 2024-02-10T07:37:51.298 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[40743]: 5.5 deep-scrub ok 2024-02-10T07:37:51.298 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[40743]: 3.f scrub starts 2024-02-10T07:37:51.298 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[40743]: 3.f scrub ok 2024-02-10T07:37:51.299 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[40743]: pgmap v273: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 456 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:37:51.299 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:37:51.299 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/313556496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:37:51.299 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:51 smithi008 ceph-mon[40743]: osdmap e194: 8 total, 8 up, 8 in 2024-02-10T07:37:51.360 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:51 smithi170 ceph-mon[33394]: 4.1a scrub starts 2024-02-10T07:37:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:51 smithi170 ceph-mon[33394]: 5.5 deep-scrub starts 2024-02-10T07:37:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:51 smithi170 ceph-mon[33394]: 4.1a scrub ok 2024-02-10T07:37:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:51 smithi170 ceph-mon[33394]: 5.5 deep-scrub ok 2024-02-10T07:37:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:51 smithi170 ceph-mon[33394]: 3.f scrub starts 2024-02-10T07:37:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:51 smithi170 ceph-mon[33394]: 3.f scrub ok 2024-02-10T07:37:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:51 smithi170 ceph-mon[33394]: pgmap v273: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 456 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:37:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:51 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:37:51.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:51 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/313556496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:37:51.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:51 smithi170 ceph-mon[33394]: osdmap e194: 8 total, 8 up, 8 in 2024-02-10T07:37:51.395 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:51.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:51.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:51.497 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:51.538 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:51.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:51.639 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:51.678 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:51.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:51.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:51.779 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:51.809 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:51.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:51.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:51.911 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:51.954 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:51.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:51.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:52.056 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:52.093 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:52.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:52.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:52.194 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:52.223 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:52.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:52.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:52.325 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:52.369 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:52.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:52.471 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:52.500 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:52.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:52.602 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:52.638 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:52.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:52.740 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:52.773 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:52.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:52.874 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:52.933 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:52.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:52.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:53.018 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:52 smithi008 ceph-mon[29816]: osdmap e195: 8 total, 8 up, 8 in 2024-02-10T07:37:53.018 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:52 smithi008 ceph-mon[29816]: pgmap v276: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 447 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 937 op/s 2024-02-10T07:37:53.018 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:52 smithi008 ceph-mon[40743]: osdmap e195: 8 total, 8 up, 8 in 2024-02-10T07:37:53.018 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:52 smithi008 ceph-mon[40743]: pgmap v276: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 447 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 937 op/s 2024-02-10T07:37:53.035 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:53.044 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:37:53.061 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:53.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:53.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:52 smithi170 ceph-mon[33394]: osdmap e195: 8 total, 8 up, 8 in 2024-02-10T07:37:53.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:52 smithi170 ceph-mon[33394]: pgmap v276: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 447 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 937 op/s 2024-02-10T07:37:53.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:37:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:37:53.162 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:53.163 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:37:53.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:53.207 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:53.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:53.217 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:37:53.261 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:37:53.306 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:37:53.308 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:53.351 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:37:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:53.360 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:53.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:53.399 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:37:53.443 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:37:53.461 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:53.485 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:37:53.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:53.492 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:53.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:53.498 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:37:53.498 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:37:53.498 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:37:53.499 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:37:53.499 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:37:53.499 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:37:53.499 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:37:53.499 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:37:53.593 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:53.625 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:53.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:53.727 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:53.759 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:53.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:53.861 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:53.896 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:53.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:53.902 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:53 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:53.902 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:53 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/997831154' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:37:53.902 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:53 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:53.902 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:53 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/997831154' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:37:53.998 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:54.028 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:54.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:54.129 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:54.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:53 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:37:54.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:53 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/997831154' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:37:54.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:54.171 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:54.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:54.236 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Removed 115135 objects 2024-02-10T07:37:54.236 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Clean up completed and total clean up time :134.004 2024-02-10T07:37:54.272 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:54.296 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:54.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:54.397 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:54.422 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:54.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:54.523 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:54.547 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:54.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:54.648 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:54.673 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:54.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127184 7.9 7:930e04c1:::benchmark_data_smithi008_82046_object12046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:55.989856+0000", 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 118.70729361399999, 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.25485934199999999, 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:54.699 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127184 2024-02-10T07:37:54.700 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.700 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.700 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.700 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.700 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989856+0000", 2024-02-10T07:37:54.700 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.700 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.700 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.700 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.700 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989856+0000", 2024-02-10T07:37:54.700 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.700 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.700 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.700 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.701 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989857+0000", 2024-02-10T07:37:54.701 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3799999999999996e-07 2024-02-10T07:37:54.701 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.701 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.701 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.701 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989861+0000", 2024-02-10T07:37:54.701 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.703e-06 2024-02-10T07:37:54.701 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.701 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.701 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.701 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989870+0000", 2024-02-10T07:37:54.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0869999999999996e-06 2024-02-10T07:37:54.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.702 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.702 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.702 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989877+0000", 2024-02-10T07:37:54.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.153e-06 2024-02-10T07:37:54.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.702 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.702 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.702 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.015176+0000", 2024-02-10T07:37:54.702 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.025298960999999998 2024-02-10T07:37:54.702 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.703 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.703 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.703 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.015291+0000", 2024-02-10T07:37:54.703 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000115123 2024-02-10T07:37:54.703 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.703 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.703 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:37:54.703 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.015667+0000", 2024-02-10T07:37:54.703 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00037586599999999998 2024-02-10T07:37:54.703 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.703 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.703 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.020630+0000", 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0049631440000000001 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.020900+0000", 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026987000000000002 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244581+0000", 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.22368058900000001 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244640+0000", 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9453999999999997e-05 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244716+0000", 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5654000000000004e-05 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127211 7.9 7:97623005:::benchmark_data_smithi008_82046_object12073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.027194+0000", 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 118.669955519, 2024-02-10T07:37:54.705 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.217660456, 2024-02-10T07:37:54.706 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.706 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.706 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:54.706 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:54.706 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:54.706 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127211 2024-02-10T07:37:54.706 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.706 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.706 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.706 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.706 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027194+0000", 2024-02-10T07:37:54.706 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027194+0000", 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027195+0000", 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7499999999999998e-07 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027200+0000", 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6829999999999999e-06 2024-02-10T07:37:54.707 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027212+0000", 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1522000000000001e-05 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027219+0000", 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6979999999999998e-06 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.031348+0000", 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0041287870000000001 2024-02-10T07:37:54.708 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.709 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.709 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.709 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.031490+0000", 2024-02-10T07:37:54.709 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014215199999999999 2024-02-10T07:37:54.709 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.709 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.709 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:37:54.709 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.031908+0000", 2024-02-10T07:37:54.709 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00041737399999999998 2024-02-10T07:37:54.709 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.709 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.709 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.709 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.038097+0000", 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0061895500000000003 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.038176+0000", 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8963999999999998e-05 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244761+0000", 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.206584868 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244806+0000", 2024-02-10T07:37:54.710 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5368000000000003e-05 2024-02-10T07:37:54.711 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.711 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.711 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.711 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244855+0000", 2024-02-10T07:37:54.711 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8514999999999999e-05 2024-02-10T07:37:54.711 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.711 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.711 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.711 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.711 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.711 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127243 7.14 7:2d50c885:::benchmark_data_smithi008_82046_object12105:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-10T07:37:54.711 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.076957+0000", 2024-02-10T07:37:54.711 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 118.620192409, 2024-02-10T07:37:54.711 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.074139102, 2024-02-10T07:37:54.712 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.712 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-02-10T07:37:54.712 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:54.712 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:54.712 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:54.712 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127243 2024-02-10T07:37:54.712 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.712 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.712 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.712 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.713 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076957+0000", 2024-02-10T07:37:54.713 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.714 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.714 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.714 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.714 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076957+0000", 2024-02-10T07:37:54.714 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.714 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.714 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.714 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.714 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076958+0000", 2024-02-10T07:37:54.714 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9300000000000006e-07 2024-02-10T07:37:54.714 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.714 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.714 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.714 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076963+0000", 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2520000000000001e-06 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076974+0000", 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0808999999999999e-05 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076981+0000", 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.391e-06 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.715 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.077036+0000", 2024-02-10T07:37:54.716 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5529999999999999e-05 2024-02-10T07:37:54.716 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.716 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.716 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.716 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.077172+0000", 2024-02-10T07:37:54.716 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013559100000000001 2024-02-10T07:37:54.716 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.716 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.716 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,3,4,5", 2024-02-10T07:37:54.716 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.077478+0000", 2024-02-10T07:37:54.716 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00030572100000000001 2024-02-10T07:37:54.716 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.716 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.716 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.717 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.080684+0000", 2024-02-10T07:37:54.717 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032062390000000001 2024-02-10T07:37:54.717 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.717 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.717 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:37:54.717 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.081086+0000", 2024-02-10T07:37:54.717 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00040183900000000003 2024-02-10T07:37:54.717 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.717 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.081152+0000", 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6185000000000004e-05 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.083905+0000", 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027525420000000002 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.090838+0000", 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0069333470000000003 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.718 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.233892+0000", 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.143053929 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.233903+0000", 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1412e-05 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.151012+0000", 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.91710869900000003 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.719 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.720 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.720 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.151097+0000", 2024-02-10T07:37:54.720 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4623000000000006e-05 2024-02-10T07:37:54.720 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.720 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.720 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.720 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.720 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.720 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127271 7.17 7:ee10418c:::benchmark_data_smithi008_82046_object12133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:37:54.720 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.283167+0000", 2024-02-10T07:37:54.720 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 118.41398287200001, 2024-02-10T07:37:54.720 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.87381412300000005, 2024-02-10T07:37:54.720 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127271 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283167+0000", 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283167+0000", 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.721 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.722 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.722 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.722 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283168+0000", 2024-02-10T07:37:54.722 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6199999999999996e-07 2024-02-10T07:37:54.722 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.722 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.722 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.722 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283173+0000", 2024-02-10T07:37:54.722 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6589999999999998e-06 2024-02-10T07:37:54.722 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.722 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283184+0000", 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1175999999999999e-05 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283192+0000", 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1049999999999995e-06 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283249+0000", 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7491000000000002e-05 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.723 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.724 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:37:54.724 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283259+0000", 2024-02-10T07:37:54.724 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2790000000000004e-06 2024-02-10T07:37:54.724 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.724 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.724 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.724 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.145677+0000", 2024-02-10T07:37:54.724 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.86241815099999997 2024-02-10T07:37:54.724 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.724 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.724 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.724 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.145793+0000", 2024-02-10T07:37:54.724 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011595 2024-02-10T07:37:54.724 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.725 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.725 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,3,6,7", 2024-02-10T07:37:54.725 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.146056+0000", 2024-02-10T07:37:54.725 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026355900000000002 2024-02-10T07:37:54.725 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.725 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.725 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.725 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.147787+0000", 2024-02-10T07:37:54.725 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0017307329999999999 2024-02-10T07:37:54.725 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.725 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.725 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.725 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.155806+0000", 2024-02-10T07:37:54.725 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0080188640000000005 2024-02-10T07:37:54.726 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.726 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.726 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.726 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.155945+0000", 2024-02-10T07:37:54.726 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013933700000000001 2024-02-10T07:37:54.726 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.726 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.726 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:37:54.726 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156537+0000", 2024-02-10T07:37:54.726 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00059228199999999996 2024-02-10T07:37:54.726 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.726 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.726 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.726 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156801+0000", 2024-02-10T07:37:54.727 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026387999999999997 2024-02-10T07:37:54.727 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.727 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.727 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.727 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156860+0000", 2024-02-10T07:37:54.727 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8900999999999997e-05 2024-02-10T07:37:54.727 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.727 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.727 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.727 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156981+0000", 2024-02-10T07:37:54.727 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000120894 2024-02-10T07:37:54.728 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.728 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.728 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.728 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.728 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.728 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127281 7.17 7:ec58e9f8:::benchmark_data_smithi008_82046_object12143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:37:54.728 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.333207+0000", 2024-02-10T07:37:54.728 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 118.363942568, 2024-02-10T07:37:54.728 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.82508648100000004, 2024-02-10T07:37:54.728 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.728 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.728 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:54.728 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127281 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333207+0000", 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333207+0000", 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.729 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333209+0000", 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.415e-06 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333213+0000", 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2230000000000001e-06 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333224+0000", 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1483e-05 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.730 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333232+0000", 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.4970000000000004e-06 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333278+0000", 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6362000000000001e-05 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333287+0000", 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6230000000000003e-06 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.731 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.732 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.147857+0000", 2024-02-10T07:37:54.732 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.81456973200000005 2024-02-10T07:37:54.732 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.732 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.732 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.732 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.147984+0000", 2024-02-10T07:37:54.732 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012703200000000001 2024-02-10T07:37:54.732 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.732 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.732 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,3,6,7", 2024-02-10T07:37:54.732 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.148252+0000", 2024-02-10T07:37:54.732 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000268278 2024-02-10T07:37:54.733 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.733 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.733 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.733 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.155857+0000", 2024-02-10T07:37:54.733 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0076047550000000004 2024-02-10T07:37:54.733 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.733 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.733 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.733 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.155901+0000", 2024-02-10T07:37:54.733 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4283000000000003e-05 2024-02-10T07:37:54.733 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.733 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.733 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156520+0000", 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00061886199999999997 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156552+0000", 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1786999999999999e-05 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.158034+0000", 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001482141 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.734 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.158140+0000", 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010587 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.158294+0000", 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015413800000000001 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24356.0:1373 5.3 5:c609908c:::notify.5:head [watch ping cookie 93827140438272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.411579+0000", 2024-02-10T07:37:54.735 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 118.285570777, 2024-02-10T07:37:54.736 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.71068318600000002, 2024-02-10T07:37:54.736 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.736 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:37:54.736 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:54.736 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24356", 2024-02-10T07:37:54.736 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1933562162", 2024-02-10T07:37:54.736 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 1373 2024-02-10T07:37:54.736 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.736 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.736 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.736 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.736 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411579+0000", 2024-02-10T07:37:54.736 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411579+0000", 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411581+0000", 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4839999999999999e-06 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411585+0000", 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8799999999999999e-06 2024-02-10T07:37:54.737 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.738 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.738 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.738 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411598+0000", 2024-02-10T07:37:54.738 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2388999999999999e-05 2024-02-10T07:37:54.738 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.738 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.738 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.738 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411606+0000", 2024-02-10T07:37:54.738 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4659999999999999e-06 2024-02-10T07:37:54.738 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411704+0000", 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8095000000000002e-05 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411714+0000", 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0044000000000001e-05 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.115532+0000", 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.70381718199999999 2024-02-10T07:37:54.739 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.740 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.740 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.740 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.115689+0000", 2024-02-10T07:37:54.740 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015699900000000001 2024-02-10T07:37:54.740 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.740 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.740 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.740 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.122262+0000", 2024-02-10T07:37:54.740 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0065736470000000002 2024-02-10T07:37:54.740 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.740 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.740 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.740 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24356.0:1383 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 93827140438272 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.412062+0000", 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 118.28508774399999, 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.725469484, 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24356", 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1933562162", 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 1383 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.741 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.742 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412062+0000", 2024-02-10T07:37:54.742 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.742 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.742 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.742 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.742 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412062+0000", 2024-02-10T07:37:54.742 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.742 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.742 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.742 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.742 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412063+0000", 2024-02-10T07:37:54.742 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7700000000000004e-07 2024-02-10T07:37:54.742 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.742 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.743 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.743 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412067+0000", 2024-02-10T07:37:54.743 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9330000000000003e-06 2024-02-10T07:37:54.743 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.743 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.743 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.743 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412081+0000", 2024-02-10T07:37:54.743 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3991e-05 2024-02-10T07:37:54.743 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.744 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.744 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.744 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412088+0000", 2024-02-10T07:37:54.744 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6459999999999993e-06 2024-02-10T07:37:54.744 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.744 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.744 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.744 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.498465+0000", 2024-02-10T07:37:54.744 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.086376877000000005 2024-02-10T07:37:54.744 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.744 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.744 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:37:54.744 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.498476+0000", 2024-02-10T07:37:54.744 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0975000000000001e-05 2024-02-10T07:37:54.745 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.745 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.745 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.745 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.122295+0000", 2024-02-10T07:37:54.745 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.62381883100000002 2024-02-10T07:37:54.745 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.745 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.745 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.745 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.122389+0000", 2024-02-10T07:37:54.745 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3941000000000001e-05 2024-02-10T07:37:54.745 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.745 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.745 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.745 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.137532+0000", 2024-02-10T07:37:54.746 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015142513 2024-02-10T07:37:54.746 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.746 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.746 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.746 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.746 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.746 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:131746 7.9 7:97040099:::benchmark_data_smithi008_82046_object16608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:37:54.746 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:05.798493+0000", 2024-02-10T07:37:54.747 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 108.898656471, 2024-02-10T07:37:54.747 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.26161331199999999, 2024-02-10T07:37:54.747 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.747 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.747 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:54.747 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:54.747 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:54.747 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 131746 2024-02-10T07:37:54.747 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.747 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.748 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.748 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.748 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798493+0000", 2024-02-10T07:37:54.748 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.748 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.748 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.748 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.748 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798493+0000", 2024-02-10T07:37:54.749 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.749 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.749 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.749 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.749 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798495+0000", 2024-02-10T07:37:54.749 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2240000000000001e-06 2024-02-10T07:37:54.749 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.749 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.749 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.749 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798499+0000", 2024-02-10T07:37:54.749 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2359999999999998e-06 2024-02-10T07:37:54.749 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.749 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.749 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.750 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798518+0000", 2024-02-10T07:37:54.750 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9519000000000001e-05 2024-02-10T07:37:54.750 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.750 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.750 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.750 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798530+0000", 2024-02-10T07:37:54.750 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.184e-05 2024-02-10T07:37:54.750 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.750 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.750 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.750 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.830695+0000", 2024-02-10T07:37:54.750 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.032164618999999998 2024-02-10T07:37:54.750 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.750 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.830907+0000", 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00021249600000000001 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.831383+0000", 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000476016 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.834488+0000", 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0031048759999999999 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.751 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.752 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.752 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.834673+0000", 2024-02-10T07:37:54.752 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00018433600000000001 2024-02-10T07:37:54.752 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.752 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.752 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.752 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.059995+0000", 2024-02-10T07:37:54.752 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.22532233199999999 2024-02-10T07:37:54.752 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.752 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.752 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.752 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060046+0000", 2024-02-10T07:37:54.752 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1076999999999999e-05 2024-02-10T07:37:54.752 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.753 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.753 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.753 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060107+0000", 2024-02-10T07:37:54.753 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0741e-05 2024-02-10T07:37:54.753 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.753 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.753 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.753 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.753 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.753 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:131775 7.19 7:9fe66b4c:::benchmark_data_smithi008_82046_object16637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:37:54.753 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:05.844733+0000", 2024-02-10T07:37:54.753 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 108.852416744, 2024-02-10T07:37:54.753 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.216159669, 2024-02-10T07:37:54.754 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.754 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.754 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:54.754 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:54.754 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:54.754 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 131775 2024-02-10T07:37:54.754 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.754 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.754 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.754 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.754 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844733+0000", 2024-02-10T07:37:54.755 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.755 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.755 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.755 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.755 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844733+0000", 2024-02-10T07:37:54.755 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.755 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.755 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.755 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.755 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844734+0000", 2024-02-10T07:37:54.755 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3340000000000001e-06 2024-02-10T07:37:54.755 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.756 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.756 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.756 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844739+0000", 2024-02-10T07:37:54.756 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.065e-06 2024-02-10T07:37:54.756 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.756 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.756 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.756 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844748+0000", 2024-02-10T07:37:54.756 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0960000000000007e-06 2024-02-10T07:37:54.756 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.756 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.756 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.756 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844754+0000", 2024-02-10T07:37:54.756 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6900000000000003e-06 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844936+0000", 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00018217400000000001 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.845059+0000", 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000122815 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.845348+0000", 2024-02-10T07:37:54.757 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00028826999999999998 2024-02-10T07:37:54.758 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.758 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.758 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.758 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.856837+0000", 2024-02-10T07:37:54.758 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011489256999999999 2024-02-10T07:37:54.758 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.758 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.758 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.758 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.856879+0000", 2024-02-10T07:37:54.758 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2268000000000003e-05 2024-02-10T07:37:54.758 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.758 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.758 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.758 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.856919+0000", 2024-02-10T07:37:54.759 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9681000000000002e-05 2024-02-10T07:37:54.759 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.759 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.759 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.759 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.856958+0000", 2024-02-10T07:37:54.759 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9623999999999999e-05 2024-02-10T07:37:54.759 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.759 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.760 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:37:54.760 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.860314+0000", 2024-02-10T07:37:54.760 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033553839999999999 2024-02-10T07:37:54.760 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.760 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.760 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.760 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060424+0000", 2024-02-10T07:37:54.760 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.20010988099999999 2024-02-10T07:37:54.760 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.760 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.760 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.760 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060484+0000", 2024-02-10T07:37:54.760 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0136000000000002e-05 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060893+0000", 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00040899399999999998 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:131874 7.9 7:90ce5fa2:::benchmark_data_smithi008_82046_object16736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:06.060485+0000", 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 108.63666502, 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.22169967700000001, 2024-02-10T07:37:54.761 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 131874 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060485+0000", 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060485+0000", 2024-02-10T07:37:54.762 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060486+0000", 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7999999999999993e-07 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060491+0000", 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.814e-06 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060503+0000", 2024-02-10T07:37:54.763 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1915e-05 2024-02-10T07:37:54.764 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.764 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.764 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.764 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060511+0000", 2024-02-10T07:37:54.764 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6479999999999992e-06 2024-02-10T07:37:54.764 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.764 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.764 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.764 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.082347+0000", 2024-02-10T07:37:54.764 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.021835391999999999 2024-02-10T07:37:54.765 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.765 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.765 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.765 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.082588+0000", 2024-02-10T07:37:54.765 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00024096600000000001 2024-02-10T07:37:54.765 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.765 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.765 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:37:54.765 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.083045+0000", 2024-02-10T07:37:54.765 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00045718000000000001 2024-02-10T07:37:54.765 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.765 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.765 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.765 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.091368+0000", 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0083230149999999996 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.091406+0000", 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8444e-05 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.282075+0000", 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.19066877500000001 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.282128+0000", 2024-02-10T07:37:54.766 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3090000000000002e-05 2024-02-10T07:37:54.767 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.767 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.767 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.767 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.282185+0000", 2024-02-10T07:37:54.767 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6458000000000001e-05 2024-02-10T07:37:54.767 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.767 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.767 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.767 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.767 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.768 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:132742 7.19 7:991f4e26:::benchmark_data_smithi008_82046_object17604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:37:54.768 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:07.446202+0000", 2024-02-10T07:37:54.768 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 107.25094815600001, 2024-02-10T07:37:54.768 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.25343312400000001, 2024-02-10T07:37:54.768 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.768 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.768 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:54.768 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:54.769 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:54.769 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 132742 2024-02-10T07:37:54.769 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.769 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.769 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.769 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.769 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446202+0000", 2024-02-10T07:37:54.770 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.770 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.770 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.770 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.770 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446202+0000", 2024-02-10T07:37:54.770 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.770 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.771 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.771 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.771 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446203+0000", 2024-02-10T07:37:54.771 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.159e-06 2024-02-10T07:37:54.771 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.771 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.772 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.772 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446207+0000", 2024-02-10T07:37:54.772 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.738e-06 2024-02-10T07:37:54.772 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.772 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.772 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.772 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446217+0000", 2024-02-10T07:37:54.773 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0518e-05 2024-02-10T07:37:54.773 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.773 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.773 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.773 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446225+0000", 2024-02-10T07:37:54.773 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3760000000000002e-06 2024-02-10T07:37:54.773 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.775 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.775 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.775 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446280+0000", 2024-02-10T07:37:54.775 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5883e-05 2024-02-10T07:37:54.775 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.776 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.776 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.776 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446423+0000", 2024-02-10T07:37:54.776 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014267300000000001 2024-02-10T07:37:54.776 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.776 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.776 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-02-10T07:37:54.777 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446714+0000", 2024-02-10T07:37:54.777 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00029046200000000001 2024-02-10T07:37:54.777 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.777 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.777 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.777 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450612+0000", 2024-02-10T07:37:54.778 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0038986860000000002 2024-02-10T07:37:54.778 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.778 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.778 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:37:54.778 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450626+0000", 2024-02-10T07:37:54.778 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4151999999999999e-05 2024-02-10T07:37:54.778 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.778 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.779 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.779 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450672+0000", 2024-02-10T07:37:54.779 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6119999999999999e-05 2024-02-10T07:37:54.779 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.779 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.779 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.779 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450720+0000", 2024-02-10T07:37:54.779 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7803999999999998e-05 2024-02-10T07:37:54.779 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.779 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.779 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.779 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450765+0000", 2024-02-10T07:37:54.779 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4594e-05 2024-02-10T07:37:54.780 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.780 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.780 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.780 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699383+0000", 2024-02-10T07:37:54.780 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24861809200000001 2024-02-10T07:37:54.780 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.780 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.780 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.780 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699507+0000", 2024-02-10T07:37:54.780 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000124156 2024-02-10T07:37:54.780 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.780 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.780 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.780 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699635+0000", 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000127711 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:132743 7.19 7:9cfb864f:::benchmark_data_smithi008_82046_object17605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:07.446450+0000", 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 107.250699844, 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.25339128999999999, 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:54.781 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 132743 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446450+0000", 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446450+0000", 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.782 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446451+0000", 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8599999999999997e-07 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446455+0000", 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.5839999999999999e-06 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446466+0000", 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1242e-05 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.783 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.784 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446474+0000", 2024-02-10T07:37:54.784 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2309999999999996e-06 2024-02-10T07:37:54.784 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.784 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.784 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.784 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.448986+0000", 2024-02-10T07:37:54.784 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0025125059999999999 2024-02-10T07:37:54.784 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.784 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.784 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.784 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.449131+0000", 2024-02-10T07:37:54.784 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014447200000000001 2024-02-10T07:37:54.784 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.784 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.449392+0000", 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026119499999999998 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450809+0000", 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0014168469999999999 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450905+0000", 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6237999999999996e-05 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.785 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450997+0000", 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1774999999999995e-05 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.452168+0000", 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00117117 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.452212+0000", 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3548999999999999e-05 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.786 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699673+0000", 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24746167399999999 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699723+0000", 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9421e-05 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699841+0000", 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011849999999999999 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.787 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.788 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.788 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154815 7.14 7:28f3695e:::benchmark_data_smithi008_82046_object39677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:37:54.788 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.856040+0000", 2024-02-10T07:37:54.788 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 81.841110228999995, 2024-02-10T07:37:54.788 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96816402099999999, 2024-02-10T07:37:54.788 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.788 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.788 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:54.788 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:54.788 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:54.788 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154815 2024-02-10T07:37:54.788 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.788 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.788 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856040+0000", 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856040+0000", 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856042+0000", 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8700000000000001e-06 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.789 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.790 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856046+0000", 2024-02-10T07:37:54.790 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9649999999999999e-06 2024-02-10T07:37:54.790 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.790 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.790 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.790 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856063+0000", 2024-02-10T07:37:54.790 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6838999999999999e-05 2024-02-10T07:37:54.790 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.790 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.790 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.790 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856083+0000", 2024-02-10T07:37:54.791 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9191e-05 2024-02-10T07:37:54.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.791 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.791 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.791 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856152+0000", 2024-02-10T07:37:54.791 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9622999999999994e-05 2024-02-10T07:37:54.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.791 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.791 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:37:54.791 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856165+0000", 2024-02-10T07:37:54.791 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2604e-05 2024-02-10T07:37:54.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.791 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.791 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.792 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.820015+0000", 2024-02-10T07:37:54.792 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.963850031 2024-02-10T07:37:54.792 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.792 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.792 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.820144+0000", 2024-02-10T07:37:54.792 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012971799999999999 2024-02-10T07:37:54.792 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.792 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,3,4,5", 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.820489+0000", 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00034403500000000002 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.822408+0000", 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001919624 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.822852+0000", 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00044381899999999997 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.793 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.794 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.822915+0000", 2024-02-10T07:37:54.794 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.279e-05 2024-02-10T07:37:54.794 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.794 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.794 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.794 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.822955+0000", 2024-02-10T07:37:54.794 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0460999999999998e-05 2024-02-10T07:37:54.794 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.794 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.794 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.794 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.823010+0000", 2024-02-10T07:37:54.794 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4537999999999999e-05 2024-02-10T07:37:54.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.823994+0000", 2024-02-10T07:37:54.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00098379999999999995 2024-02-10T07:37:54.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.824060+0000", 2024-02-10T07:37:54.796 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6548e-05 2024-02-10T07:37:54.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.824204+0000", 2024-02-10T07:37:54.796 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014356500000000001 2024-02-10T07:37:54.797 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.797 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.797 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.797 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:212345 7.f 7:f80fa090:::benchmark_data_smithi008_82046_object97207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-10T07:37:54.797 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.578926+0000", 2024-02-10T07:37:54.797 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.118223516, 2024-02-10T07:37:54.797 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24282195200000001, 2024-02-10T07:37:54.797 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.797 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.797 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:37:54.797 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:37:54.798 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:37:54.798 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 212345 2024-02-10T07:37:54.798 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.798 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.798 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.798 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.798 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578926+0000", 2024-02-10T07:37:54.798 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.798 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.798 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.798 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.798 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578926+0000", 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578928+0000", 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.486e-06 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578932+0000", 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2699999999999998e-06 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578945+0000", 2024-02-10T07:37:54.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2703e-05 2024-02-10T07:37:54.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.800 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.800 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578954+0000", 2024-02-10T07:37:54.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8710000000000003e-06 2024-02-10T07:37:54.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.800 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.800 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.579336+0000", 2024-02-10T07:37:54.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00038255500000000003 2024-02-10T07:37:54.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.579461+0000", 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000124621 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.579564+0000", 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000103317 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.582702+0000", 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0031373400000000002 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583326+0000", 2024-02-10T07:37:54.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00062478000000000002 2024-02-10T07:37:54.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:37:54.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821642+0000", 2024-02-10T07:37:54.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.238315844 2024-02-10T07:37:54.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821699+0000", 2024-02-10T07:37:54.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6635000000000002e-05 2024-02-10T07:37:54.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.803 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.803 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821748+0000", 2024-02-10T07:37:54.803 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9530000000000002e-05 2024-02-10T07:37:54.803 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.803 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.803 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.803 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.803 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212347 7.12 e187/185 7:4b6dc533:::benchmark_data_smithi008_82046_object97209:head v 187'10763, mlcod=187'10761)", 2024-02-10T07:37:54.803 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.583166+0000", 2024-02-10T07:37:54.803 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.113983812000001, 2024-02-10T07:37:54.804 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24110515900000001, 2024-02-10T07:37:54.804 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.804 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.804 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.804 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.804 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.804 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583166+0000", 2024-02-10T07:37:54.804 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.804 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.804 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.804 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.804 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583166+0000", 2024-02-10T07:37:54.805 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.805 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583167+0000", 2024-02-10T07:37:54.805 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1299999999999999e-07 2024-02-10T07:37:54.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.805 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583172+0000", 2024-02-10T07:37:54.805 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6740000000000002e-06 2024-02-10T07:37:54.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.806 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583182+0000", 2024-02-10T07:37:54.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8130000000000007e-06 2024-02-10T07:37:54.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.806 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.806 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583189+0000", 2024-02-10T07:37:54.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9140000000000002e-06 2024-02-10T07:37:54.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.806 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.806 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583624+0000", 2024-02-10T07:37:54.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00043515399999999999 2024-02-10T07:37:54.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.807 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583646+0000", 2024-02-10T07:37:54.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1474000000000001e-05 2024-02-10T07:37:54.807 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.807 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.824236+0000", 2024-02-10T07:37:54.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24059001699999999 2024-02-10T07:37:54.807 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.807 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.824271+0000", 2024-02-10T07:37:54.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.54e-05 2024-02-10T07:37:54.808 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.808 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.808 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.808 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.808 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.808 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212349 7.e e187/166 7:7df8d3ee:::benchmark_data_smithi008_82046_object97211:head v 187'13215, mlcod=187'13213)", 2024-02-10T07:37:54.808 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.583663+0000", 2024-02-10T07:37:54.808 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.113486541, 2024-02-10T07:37:54.808 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23797284099999999, 2024-02-10T07:37:54.808 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.808 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.808 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.808 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583663+0000", 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583663+0000", 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583664+0000", 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3900000000000004e-07 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583670+0000", 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7790000000000001e-06 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583680+0000", 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7419999999999992e-06 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583686+0000", 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5740000000000001e-06 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.584048+0000", 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00036165999999999998 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.584076+0000", 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8121000000000001e-05 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821584+0000", 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.237508318 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.811 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821636+0000", 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1907999999999998e-05 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop_reply(client.24740.0:212345 7.f e187/98 ondisk, result = 0)", 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.585629+0000", 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.111521204999999, 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23612559999999999, 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.812 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.813 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.813 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585629+0000", 2024-02-10T07:37:54.813 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.813 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.813 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585629+0000", 2024-02-10T07:37:54.813 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.813 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.813 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585630+0000", 2024-02-10T07:37:54.813 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0999999999999997e-07 2024-02-10T07:37:54.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585635+0000", 2024-02-10T07:37:54.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1050000000000003e-06 2024-02-10T07:37:54.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585645+0000", 2024-02-10T07:37:54.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0484e-05 2024-02-10T07:37:54.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585652+0000", 2024-02-10T07:37:54.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2570000000000001e-06 2024-02-10T07:37:54.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821625+0000", 2024-02-10T07:37:54.815 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23597299399999999 2024-02-10T07:37:54.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821635+0000", 2024-02-10T07:37:54.815 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9240000000000002e-06 2024-02-10T07:37:54.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821754+0000", 2024-02-10T07:37:54.816 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011892600000000001 2024-02-10T07:37:54.816 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.816 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.816 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.816 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.816 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.816 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDPGPushReply(7.14 187/156 [PushReplyOp(7:2f8c8aed:::benchmark_data_smithi008_82046_object99852:head)])", 2024-02-10T07:37:54.816 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:37:54.816 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.108959763000001, 2024-02-10T07:37:54.816 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23338334399999999, 2024-02-10T07:37:54.816 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.816 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-02-10T07:37:54.816 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2499999999999999e-07 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588193+0000", 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.881e-06 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588200+0000", 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.703e-06 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588206+0000", 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4869999999999999e-06 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588865+0000", 2024-02-10T07:37:54.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00065820199999999998 2024-02-10T07:37:54.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821573+0000", 2024-02-10T07:37:54.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.232708846 2024-02-10T07:37:54.819 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.819 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.819 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.819 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212353 7.12 e187/185 7:4af281cd:::benchmark_data_smithi008_82046_object97215:head v 187'10764, mlcod=187'10761)", 2024-02-10T07:37:54.820 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.589498+0000", 2024-02-10T07:37:54.820 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.107652027, 2024-02-10T07:37:54.820 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23719131800000001, 2024-02-10T07:37:54.820 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.820 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.820 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.820 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.820 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.820 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589498+0000", 2024-02-10T07:37:54.820 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.820 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.820 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.820 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.820 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589498+0000", 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589499+0000", 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8500000000000001e-07 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589505+0000", 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4060000000000002e-06 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821581+0000", 2024-02-10T07:37:54.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23207629699999999 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821593+0000", 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1387e-05 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821991+0000", 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00039868400000000001 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.822013+0000", 2024-02-10T07:37:54.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1191000000000001e-05 2024-02-10T07:37:54.823 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.823 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.823 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.823 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.826650+0000", 2024-02-10T07:37:54.823 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046371650000000004 2024-02-10T07:37:54.823 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.823 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.823 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.823 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.826689+0000", 2024-02-10T07:37:54.823 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9502999999999998e-05 2024-02-10T07:37:54.823 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.823 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.824 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.824 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.824 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.824 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212363 7.5 e187/135 7:a6287109:::benchmark_data_smithi008_82046_object97225:head v 187'10252, mlcod=187'10250)", 2024-02-10T07:37:54.824 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.594086+0000", 2024-02-10T07:37:54.824 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 17.103064255, 2024-02-10T07:37:54.824 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23397765000000001, 2024-02-10T07:37:54.824 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:37:54.824 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:37:54.824 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:37:54.824 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.824 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:37:54.824 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594086+0000", 2024-02-10T07:37:54.824 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.825 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.825 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.825 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:37:54.825 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594086+0000", 2024-02-10T07:37:54.825 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:37:54.825 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.825 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.825 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:37:54.825 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594087+0000", 2024-02-10T07:37:54.825 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.595e-06 2024-02-10T07:37:54.825 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.825 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.825 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:37:54.825 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594097+0000", 2024-02-10T07:37:54.826 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4560000000000006e-06 2024-02-10T07:37:54.826 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.826 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.826 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:37:54.826 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821570+0000", 2024-02-10T07:37:54.826 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.22747376999999999 2024-02-10T07:37:54.826 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.826 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.827 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:37:54.827 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821582+0000", 2024-02-10T07:37:54.827 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1093e-05 2024-02-10T07:37:54.827 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.827 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.827 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:37:54.827 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821684+0000", 2024-02-10T07:37:54.827 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010225699999999999 2024-02-10T07:37:54.827 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.827 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.828 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:37:54.828 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821708+0000", 2024-02-10T07:37:54.828 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4641e-05 2024-02-10T07:37:54.828 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.828 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.828 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:37:54.828 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.828021+0000", 2024-02-10T07:37:54.828 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0063127080000000002 2024-02-10T07:37:54.828 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:37:54.829 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:37:54.829 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:37:54.829 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.828063+0000", 2024-02-10T07:37:54.829 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2129999999999998e-05 2024-02-10T07:37:54.829 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.829 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.829 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.830 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:37:54.830 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:37:54.830 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:37:54.830 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:54.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:54 smithi008 ceph-mon[29816]: pgmap v277: 158 pgs: 158 active+clean; 90 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-02-10T07:37:54.831 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:54 smithi008 ceph-mon[40743]: pgmap v277: 158 pgs: 158 active+clean; 90 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-02-10T07:37:54.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:54.855 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:54.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:54.956 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:54.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:54.981 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:54.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:55.083 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:55.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:55.109 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:55.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:55.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:54 smithi170 ceph-mon[33394]: pgmap v277: 158 pgs: 158 active+clean; 90 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-02-10T07:37:55.210 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:55.239 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:55.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:55.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:55.341 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:55.372 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:55.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:55.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:55.474 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:55.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:55.500 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:55.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:55.602 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:55.631 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:55.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:55.732 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:55.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:55.756 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:55.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:55.857 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:55.885 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:55.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:55.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:55.900 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-02-10T07:37:55.900 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-02-10T07:37:55.925 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-10T07:37:55.987 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:56.015 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:56.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:56.117 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:56.144 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:56.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:56.246 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:56.263 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:37:56.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:56.276 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:56.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:56.277 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:56.277 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:56.282 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:37:56.377 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:56.405 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:56.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:56.507 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:56.532 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:56.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:56.633 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:56.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:56.658 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:56.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:56.759 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:56.786 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:56.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:56.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:56.863 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.5 to 1 (10000) 2024-02-10T07:37:56.889 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:56.915 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:56.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:56.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:57.017 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:57.043 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:57.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:57.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:57.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:56 smithi170 ceph-mon[33394]: pgmap v278: 158 pgs: 158 active+clean; 90 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-10T07:37:57.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:56 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3500741987' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:37:57.145 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:57.171 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:57.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:57.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:57.273 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:57.298 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:57.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:57.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:57.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:56 smithi008 ceph-mon[29816]: pgmap v278: 158 pgs: 158 active+clean; 90 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-10T07:37:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:56 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3500741987' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:37:57.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:56 smithi008 ceph-mon[40743]: pgmap v278: 158 pgs: 158 active+clean; 90 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-10T07:37:57.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:56 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3500741987' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:37:57.400 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:37:57.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:57.428 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:57.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:37:57.529 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:57.557 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:57.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:57.659 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:57.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:57.687 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:57.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:57.788 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:57.817 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:57.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:57.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:37:57.915 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 6 1 2024-02-10T07:37:57.918 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:57.945 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:57.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:58.047 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:37:58.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:58.073 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:58.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:37:58.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:57 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3500741987' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:37:58.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:57 smithi008 ceph-mon[29816]: osdmap e196: 8 total, 8 up, 8 in 2024-02-10T07:37:58.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:57 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3500741987' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:37:58.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:57 smithi008 ceph-mon[40743]: osdmap e196: 8 total, 8 up, 8 in 2024-02-10T07:37:58.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:57 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3500741987' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:37:58.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:57 smithi170 ceph-mon[33394]: osdmap e196: 8 total, 8 up, 8 in 2024-02-10T07:37:58.174 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:58.199 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:58.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:58.300 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:37:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:58.325 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:58.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:37:58.426 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:58.450 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:58.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:58.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:58.552 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:58.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:58.577 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:58.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:58.678 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:58.704 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:58.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:58.806 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:58.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:58.830 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:58.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:58.899 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:37:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:37:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:37:58.899 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:37:58 smithi008 ceph-mon[40743]: pgmap v280: 158 pgs: 158 active+clean; 41 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 860 op/s 2024-02-10T07:37:58.899 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:37:58 smithi008 ceph-mon[29816]: pgmap v280: 158 pgs: 158 active+clean; 41 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 860 op/s 2024-02-10T07:37:58.931 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:58.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:58.955 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:58.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:59.056 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:59.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:59.080 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:59.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:59.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:37:58 smithi170 ceph-mon[33394]: pgmap v280: 158 pgs: 158 active+clean; 41 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 860 op/s 2024-02-10T07:37:59.181 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:59.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:59.205 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:59.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:59.306 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:37:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:59.331 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:59.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:37:59.431 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:59.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:59.461 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:59.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:59.562 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:37:59.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:59.587 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:59.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:37:59.688 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:37:59.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:59.716 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:59.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:37:59.817 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:37:59.845 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:37:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:59.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:37:59.947 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:37:59.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:37:59.973 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:37:59.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:00.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:00.101 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:00.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:00.203 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:00.226 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:00.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:00.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:00.328 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:00.355 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:00.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:00.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:00.470 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:00.485 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:00.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:00.586 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:00.612 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:00.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:00.713 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:00.738 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:00.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:00.839 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:00.865 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:00.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:00.966 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:00.982 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:00 smithi008 ceph-mon[29816]: pgmap v281: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 906 op/s 2024-02-10T07:38:00.983 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:00 smithi008 ceph-mon[40743]: pgmap v281: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 906 op/s 2024-02-10T07:38:00.990 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:00.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:00.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:01.092 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:01.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:01.117 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:01.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:01.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:00 smithi170 ceph-mon[33394]: pgmap v281: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 906 op/s 2024-02-10T07:38:01.218 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:01.245 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:01.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:01.347 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:01.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:01.371 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:01.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:01.472 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:01.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:01.501 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:01.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:01.602 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:01.629 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:01.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:01.732 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:01.758 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:01.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:01.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:01.859 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:01.893 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:01.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:01.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:01.895 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'unique_pool_0' removed 2024-02-10T07:38:01.994 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:02.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:02.022 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:02.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:02.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:01 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3221170962' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:38:02.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:01 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:38:02.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:01 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3221170962' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:38:02.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:01 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:38:02.123 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:02.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:01 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3221170962' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:38:02.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:01 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:38:02.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:02.169 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:02.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:02.270 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:02.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:02.295 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:02.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:02.396 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:02.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:02.420 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:02.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:02.522 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:02.548 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:02.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:02.649 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:02.673 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:02.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:02.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:02.775 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:02.798 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:02.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:02.900 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:02.923 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:02.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:02.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:02.953 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:02 smithi008 ceph-mon[40743]: pgmap v282: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 714 op/s 2024-02-10T07:38:02.953 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:02 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:38:02.953 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:02 smithi008 ceph-mon[40743]: osdmap e197: 8 total, 8 up, 8 in 2024-02-10T07:38:02.953 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:02 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:02.953 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:02 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:02.953 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:02 smithi008 ceph-mon[29816]: pgmap v282: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 714 op/s 2024-02-10T07:38:02.953 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:02 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:38:02.953 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:02 smithi008 ceph-mon[29816]: osdmap e197: 8 total, 8 up, 8 in 2024-02-10T07:38:02.953 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:02 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:02.953 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:02 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:03.025 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:03.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:03.055 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:03.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:03.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:38:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:38:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:02 smithi170 ceph-mon[33394]: pgmap v282: 158 pgs: 158 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 714 op/s 2024-02-10T07:38:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:02 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:38:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:02 smithi170 ceph-mon[33394]: osdmap e197: 8 total, 8 up, 8 in 2024-02-10T07:38:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:02 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:02 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:03.156 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:03.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:03.185 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:03.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:03.286 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:03.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:03.311 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:03.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:03.412 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:03.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:03.441 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:03.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:03.542 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:03.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:03.570 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:03.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:03.671 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:03.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:03.696 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:03.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:03.797 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:03.826 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:03.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:03.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:03.887 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:38:03.928 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:03.956 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:03.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:03.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:04.058 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:04.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:03 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:04.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:03 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2709291180' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:38:04.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:03 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:04.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:03 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2709291180' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:38:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:04.083 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:04.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:04.184 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:04.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:04.209 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:04.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:04.310 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:04.335 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:04.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:04.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:03 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:04.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:03 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2709291180' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:38:04.435 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:04.464 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:04.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:04.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:04.565 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:04.593 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:04.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:04.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:04.694 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:04.721 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:04.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:04.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:04.823 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:04.850 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:04.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:04.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:04.925 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-10T07:38:04.926 INFO:tasks.radosbench:Beginning radosbench... 2024-02-10T07:38:04.926 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-02-10T07:38:04.926 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-02-10T07:38:04.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:04.979 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:04.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:05.081 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:05.110 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:05.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:05.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:05.211 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:05.235 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:05.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:04 smithi008 ceph-mon[29816]: pgmap v284: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:38:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:04 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:38:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:04 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2709291180' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:38:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:04 smithi008 ceph-mon[29816]: osdmap e198: 8 total, 8 up, 8 in 2024-02-10T07:38:05.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:04 smithi008 ceph-mon[40743]: pgmap v284: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:38:05.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:04 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:38:05.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:04 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2709291180' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:38:05.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:04 smithi008 ceph-mon[40743]: osdmap e198: 8 total, 8 up, 8 in 2024-02-10T07:38:05.338 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:05.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:05.369 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:05.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:04 smithi170 ceph-mon[33394]: pgmap v284: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:38:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:04 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:38:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:04 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2709291180' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:38:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:04 smithi170 ceph-mon[33394]: osdmap e198: 8 total, 8 up, 8 in 2024-02-10T07:38:05.471 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:05.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:05.498 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:05.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:05.599 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:05.628 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:05.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:05.730 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:05.758 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:05.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:05.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:05.860 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:05.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:05.889 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:05.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:05.990 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:06.018 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:06.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:06.120 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:06.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:06.149 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:06.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:06.250 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:06.277 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:06.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:06.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:06.379 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:06.387 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:38:06.387 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:38:06.387 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-02-10T07:38:06.387 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:38:06.403 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:06.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:06.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:06.505 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:06.529 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:06.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:06.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:06.631 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:06.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:06.655 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:06.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:06.756 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:06.781 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:06.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:06.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:06.883 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:06.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:06.907 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:06.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:07.008 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:07.037 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:07.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:07.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:07.139 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:07.163 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:07.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:07.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:07.265 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:07.293 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:07.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:07.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:07.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:06 smithi008 ceph-mon[29816]: pgmap v286: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:38:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:06 smithi008 ceph-mon[40743]: pgmap v286: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:38:07.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:06 smithi170 ceph-mon[33394]: pgmap v286: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-10T07:38:07.395 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:07.420 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:07.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:07.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:07.522 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:07.550 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:07.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:07.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:07.652 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:07.681 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:07.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:07.782 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:07.809 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:07.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:07.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:07.911 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:07.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:07.938 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:07.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:08.039 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:08.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:08.068 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:08.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:08.169 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:08.196 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:08.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:08.298 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:08.325 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:08.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:08.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:08.356 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:08.388 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-10T07:38:08.427 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:08.455 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:08.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:08.556 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:08.580 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:08.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:08.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:08.682 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:08.712 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:08.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:08.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:08.814 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:08.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:08.839 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:08.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:08.892 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:38:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:38:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:38:08.941 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:08.966 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:08.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:08.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:09.069 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:09.094 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:09.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:09.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:08 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:38:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:08 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:38:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:08 smithi170 ceph-mon[33394]: pgmap v287: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:38:09.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:08 smithi170 ceph-mon[33394]: osdmap e199: 8 total, 8 up, 8 in 2024-02-10T07:38:09.161 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:08 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:38:09.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:08 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:38:09.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:08 smithi008 ceph-mon[29816]: pgmap v287: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:38:09.162 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:08 smithi008 ceph-mon[29816]: osdmap e199: 8 total, 8 up, 8 in 2024-02-10T07:38:09.162 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:08 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:38:09.162 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:08 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:38:09.162 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:08 smithi008 ceph-mon[40743]: pgmap v287: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:38:09.162 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:08 smithi008 ceph-mon[40743]: osdmap e199: 8 total, 8 up, 8 in 2024-02-10T07:38:09.197 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:09.212 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:38:09.222 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:09.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:09.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:09.232 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:38:09.251 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:38:09.270 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:38:09.291 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:38:09.311 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:38:09.324 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:09.332 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:38:09.347 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:09.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:09.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:09.352 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:38:09.354 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:38:09.354 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:38:09.354 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:38:09.354 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:38:09.354 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:38:09.354 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:38:09.354 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:38:09.354 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:38:09.449 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:09.473 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:09.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:09.574 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:09.576 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:38:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:09.599 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:09.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:09.700 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:09.724 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:09.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:09.825 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:09.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:09.849 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:09.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:09.950 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:09.977 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:09.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:10.079 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:10.103 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:10.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:10.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:10.205 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:10.211 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:09 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/276797581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:38:10.211 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:09 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/276797581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:38:10.233 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:10.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:10.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:10.306 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:10.306 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:10.307 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:10.335 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:10.364 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:10.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:10.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:10.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:09 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/276797581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:38:10.466 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:10.490 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:10.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:10.591 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:10.624 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:10.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:10.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:10.728 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:10.757 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:10.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:10.859 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:10.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:10.885 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:10.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:10.986 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:11.011 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:11.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:11.113 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:11.141 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:11.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:11.243 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:11.268 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:11.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:11.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:10 smithi008 ceph-mon[29816]: pgmap v289: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:38:11.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:10 smithi008 ceph-mon[40743]: pgmap v289: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:38:11.369 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:11.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:10 smithi170 ceph-mon[33394]: pgmap v289: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:38:11.393 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:11.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:11.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:11.495 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:11.524 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:11.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:11.625 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:11.649 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:11.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:11.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:11.751 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:11.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:11.780 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:11.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:11.881 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:11.909 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:11.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:12.011 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:12.039 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:12.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:12.141 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:12.169 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:12.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:12.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:12.237 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:38:12.271 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:12.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:12.305 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:12.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:12.406 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:12.431 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:12.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:12.532 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:12.558 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:12.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:12.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:12.660 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:12.684 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:12.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:12.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:12.786 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:12.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:12 smithi008 ceph-mon[40743]: pgmap v290: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:38:12.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:12 smithi008 ceph-mon[29816]: pgmap v290: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:38:12.814 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:12.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:12.832 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:12.916 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:12.939 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:12.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:13.041 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:13.067 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:13.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:13.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:12 smithi170 ceph-mon[33394]: pgmap v290: 132 pgs: 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:38:13.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:38:12 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:38:13.169 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:13.194 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:13.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:13.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:13.296 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:13.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:13.325 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:13.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:13.426 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:13.450 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:13.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:13.552 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:13.580 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:13.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:13.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:13.682 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:13.706 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:13.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:13.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:13.775 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'unique_pool_1' created 2024-02-10T07:38:13.808 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:13.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:13.838 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:13.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:13.940 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:13.969 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:13.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:14.014 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:13 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3036847956' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-10T07:38:14.014 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:13 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-10T07:38:14.014 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:13 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:14.014 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:13 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3036847956' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-10T07:38:14.014 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:13 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-10T07:38:14.014 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:13 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:14.070 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:14.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:14.096 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:14.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:14.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:13 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3036847956' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-10T07:38:14.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:13 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-10T07:38:14.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:13 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:14.197 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:14.224 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:14.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:14.325 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:14.353 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:14.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:14.455 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:14.481 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:14.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:14.582 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:14.611 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:14.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:14.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:14.713 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:14.740 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:14.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:14.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:14.843 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:14.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:14.871 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:14.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:14.884 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-02-10T07:38:14.972 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:14.976 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:14 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-10T07:38:14.977 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:14 smithi008 ceph-mon[40743]: osdmap e200: 8 total, 8 up, 8 in 2024-02-10T07:38:14.977 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:14 smithi008 ceph-mon[40743]: pgmap v292: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:14.977 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:14 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-10T07:38:14.977 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:14 smithi008 ceph-mon[29816]: osdmap e200: 8 total, 8 up, 8 in 2024-02-10T07:38:14.977 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:14 smithi008 ceph-mon[29816]: pgmap v292: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:15.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:15.001 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:15.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:15.102 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:15.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:15.130 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:15.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:15.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:14 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-10T07:38:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:14 smithi170 ceph-mon[33394]: osdmap e200: 8 total, 8 up, 8 in 2024-02-10T07:38:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:14 smithi170 ceph-mon[33394]: pgmap v292: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:15.231 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:15.258 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:15.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:15.360 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:15.386 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:15.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:15.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:15.489 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:15.515 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:15.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:15.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:15.617 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:15.641 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:15.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:15.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:15.697 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:38:15.717 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:38:15.736 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:38:15.743 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:15.756 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:38:15.766 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:15.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:15.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:15.777 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:38:15.801 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:38:15.822 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:38:15.823 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:15 smithi008 ceph-mon[29816]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:38:15.823 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:15 smithi008 ceph-mon[29816]: osdmap e201: 8 total, 8 up, 8 in 2024-02-10T07:38:15.823 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:15 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/769357943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:38:15.823 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:15 smithi008 ceph-mon[40743]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:38:15.823 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:15 smithi008 ceph-mon[40743]: osdmap e201: 8 total, 8 up, 8 in 2024-02-10T07:38:15.823 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:15 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/769357943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:38:15.843 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:38:15.845 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:38:15.845 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:38:15.845 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:38:15.845 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:38:15.845 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:38:15.845 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:38:15.845 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:38:15.845 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:38:15.869 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:15.897 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:15.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:15.999 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:16.023 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:16.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:16.125 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:16.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:15 smithi170 ceph-mon[33394]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:38:16.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:15 smithi170 ceph-mon[33394]: osdmap e201: 8 total, 8 up, 8 in 2024-02-10T07:38:16.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:15 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/769357943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:38:16.153 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:16.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:16.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:16.255 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:16.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:16.282 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:16.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:16.383 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:16.409 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:16.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:16.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:16.512 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:16.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:16.537 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:16.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:16.639 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:16.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:16.663 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:16.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:16.764 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:16.791 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:16.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:16.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:16.893 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:16.921 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:16.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:17.022 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:17.046 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:17.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:17.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:16 smithi008 ceph-mon[29816]: osdmap e202: 8 total, 8 up, 8 in 2024-02-10T07:38:17.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:16 smithi008 ceph-mon[29816]: pgmap v295: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:17.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:16 smithi008 ceph-mon[29816]: osdmap e203: 8 total, 8 up, 8 in 2024-02-10T07:38:17.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:16 smithi008 ceph-mon[40743]: osdmap e202: 8 total, 8 up, 8 in 2024-02-10T07:38:17.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:16 smithi008 ceph-mon[40743]: pgmap v295: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:17.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:16 smithi008 ceph-mon[40743]: osdmap e203: 8 total, 8 up, 8 in 2024-02-10T07:38:17.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:16 smithi170 ceph-mon[33394]: osdmap e202: 8 total, 8 up, 8 in 2024-02-10T07:38:17.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:16 smithi170 ceph-mon[33394]: pgmap v295: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:17.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:16 smithi170 ceph-mon[33394]: osdmap e203: 8 total, 8 up, 8 in 2024-02-10T07:38:17.147 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:17.175 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:17.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:17.277 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:17.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:17.301 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:17.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:17.403 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:17.431 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:17.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:17.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:17.533 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:17.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:17.557 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:17.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:17.658 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:17.687 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:17.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:17.788 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:17.799 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.6 to 1 (10000) 2024-02-10T07:38:17.815 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:17.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:17.917 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:17.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:17.942 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:17.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:18.030 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:17 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2163917693' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:38:18.031 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:17 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:18.031 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:17 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:18.031 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:17 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2163917693' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:38:18.031 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:17 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:18.031 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:17 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:18.043 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:18.066 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:18.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:18.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:17 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2163917693' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:38:18.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:17 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:18.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:17 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:18.169 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:18.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:18.198 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:18.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:18.299 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:18.326 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:18.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:18.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:18.428 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:18.434 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:18.455 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:18.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:18.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:18.457 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:38:18.457 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:38:18.457 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-02-10T07:38:18.457 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-02-10T07:38:18.457 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:38:18.557 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:18.584 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:18.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:18.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:18.686 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:18.710 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:18.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:18.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:18.812 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:18.837 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:18.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:18.938 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:18.965 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:18.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:18.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:19.029 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:18 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2163917693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:38:19.029 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:18 smithi008 ceph-mon[29816]: osdmap e204: 8 total, 8 up, 8 in 2024-02-10T07:38:19.029 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:18 smithi008 ceph-mon[29816]: pgmap v298: 148 pgs: 10 unknown, 138 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:19.029 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:38:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:38:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:38:19.029 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:18 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2163917693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:38:19.030 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:18 smithi008 ceph-mon[40743]: osdmap e204: 8 total, 8 up, 8 in 2024-02-10T07:38:19.030 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:18 smithi008 ceph-mon[40743]: pgmap v298: 148 pgs: 10 unknown, 138 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:19.054 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 7 1 2024-02-10T07:38:19.067 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:19.120 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:19.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:19.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:18 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2163917693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:38:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:18 smithi170 ceph-mon[33394]: osdmap e204: 8 total, 8 up, 8 in 2024-02-10T07:38:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:18 smithi170 ceph-mon[33394]: pgmap v298: 148 pgs: 10 unknown, 138 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:19.221 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:19.245 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:19.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:19.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:19.347 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:19.375 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:19.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:19.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:19.477 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:19.504 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:19.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:19.606 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:19.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:19.633 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:19.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:19.734 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:19.761 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:19.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:19.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:19.810 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:38:19.864 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:19.892 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:19.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:19.902 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:19 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1763899370' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:38:19.902 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:19 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1763899370' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:38:19.993 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:20.018 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:20.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:20.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:20.058 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:38:20.119 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:19 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1763899370' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:38:20.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:20.144 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:20.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:20.245 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:20.274 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:20.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:20.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:20.375 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:20.403 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:20.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:20.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:20.505 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:20.533 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:20.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:20.576 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-10T07:38:20.634 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:20.662 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:20.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:20.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:20.763 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:20.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:20.790 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:20.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:20.891 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:20.947 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:20.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:21.049 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:21.073 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:21.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:21.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:21.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:20 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:38:21.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:20 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1763899370' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:38:21.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:20 smithi008 ceph-mon[29816]: osdmap e205: 8 total, 8 up, 8 in 2024-02-10T07:38:21.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:20 smithi008 ceph-mon[29816]: pgmap v300: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:21.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:20 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:38:21.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:20 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1763899370' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:38:21.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:20 smithi008 ceph-mon[40743]: osdmap e205: 8 total, 8 up, 8 in 2024-02-10T07:38:21.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:20 smithi008 ceph-mon[40743]: pgmap v300: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:20 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:38:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:20 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1763899370' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:38:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:20 smithi170 ceph-mon[33394]: osdmap e205: 8 total, 8 up, 8 in 2024-02-10T07:38:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:20 smithi170 ceph-mon[33394]: pgmap v300: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:38:21.174 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:21.199 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:21.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:21.300 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:21.329 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:21.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:21.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:21.430 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:21.458 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:21.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:21.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:21.560 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:21.587 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:21.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:21.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:21.688 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:21.715 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:21.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:21.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:21.817 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:21.845 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:21.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:21.946 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:21.972 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:21.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:22.022 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:22.074 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:22.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:21 smithi008 ceph-mon[29816]: 4.12 scrub starts 2024-02-10T07:38:22.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:21 smithi008 ceph-mon[29816]: 4.12 scrub ok 2024-02-10T07:38:22.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:21 smithi008 ceph-mon[40743]: 4.12 scrub starts 2024-02-10T07:38:22.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:21 smithi008 ceph-mon[40743]: 4.12 scrub ok 2024-02-10T07:38:22.102 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:22.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:21 smithi170 ceph-mon[33394]: 4.12 scrub starts 2024-02-10T07:38:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:21 smithi170 ceph-mon[33394]: 4.12 scrub ok 2024-02-10T07:38:22.204 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:22.228 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:22.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:22.329 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:22.344 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:22.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:22.356 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:22.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:22.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:22.483 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:22.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:22.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:22.585 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:22.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:22.612 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:22.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:22.713 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:22.738 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:22.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:22.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:22.839 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:22.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:22.864 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:22.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:22.965 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:22.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:22.989 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:22.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:23.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:22 smithi008 ceph-mon[29816]: 2.0 scrub starts 2024-02-10T07:38:23.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:22 smithi008 ceph-mon[29816]: 2.0 scrub ok 2024-02-10T07:38:23.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:22 smithi008 ceph-mon[29816]: 3.a scrub starts 2024-02-10T07:38:23.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:22 smithi008 ceph-mon[29816]: 3.a scrub ok 2024-02-10T07:38:23.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:22 smithi008 ceph-mon[29816]: pgmap v301: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:23.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:22 smithi008 ceph-mon[29816]: 6.1f scrub starts 2024-02-10T07:38:23.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:22 smithi008 ceph-mon[29816]: 6.1f scrub ok 2024-02-10T07:38:23.060 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:22 smithi008 ceph-mon[40743]: 2.0 scrub starts 2024-02-10T07:38:23.060 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:22 smithi008 ceph-mon[40743]: 2.0 scrub ok 2024-02-10T07:38:23.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:22 smithi008 ceph-mon[40743]: 3.a scrub starts 2024-02-10T07:38:23.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:22 smithi008 ceph-mon[40743]: 3.a scrub ok 2024-02-10T07:38:23.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:22 smithi008 ceph-mon[40743]: pgmap v301: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:23.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:22 smithi008 ceph-mon[40743]: 6.1f scrub starts 2024-02-10T07:38:23.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:22 smithi008 ceph-mon[40743]: 6.1f scrub ok 2024-02-10T07:38:23.090 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:23.113 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:23.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:23.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:38:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:38:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:22 smithi170 ceph-mon[33394]: 2.0 scrub starts 2024-02-10T07:38:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:22 smithi170 ceph-mon[33394]: 2.0 scrub ok 2024-02-10T07:38:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:22 smithi170 ceph-mon[33394]: 3.a scrub starts 2024-02-10T07:38:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:22 smithi170 ceph-mon[33394]: 3.a scrub ok 2024-02-10T07:38:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:22 smithi170 ceph-mon[33394]: pgmap v301: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:38:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:22 smithi170 ceph-mon[33394]: 6.1f scrub starts 2024-02-10T07:38:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:22 smithi170 ceph-mon[33394]: 6.1f scrub ok 2024-02-10T07:38:23.215 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:23.243 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:23.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:23.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:23.345 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:23.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:23.370 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:23.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:23.471 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:23.500 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:23.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:23.512 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:38:23.602 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:23.629 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:23.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:23.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:23.731 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:23.759 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:23.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:23.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:23.824 INFO:teuthology.orchestra.run.smithi008.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-02-10T07:38:23.861 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:23.885 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:23.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:23.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:23.958 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:23 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:23.959 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:23 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2474895357' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:38:23.959 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:23 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:38:23.959 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:23 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:23.959 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:23 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2474895357' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:38:23.959 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:23 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:38:23.987 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:24.011 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:24.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:24.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:24.113 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:24.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:23 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:24.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:23 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2474895357' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:38:24.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:23 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:38:24.137 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:24.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:24.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:24.238 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:24.266 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:24.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:24.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:24.368 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:24.395 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:24.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:24.497 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:24.520 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:24.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:24.622 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:24.650 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:24.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:24.752 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:24.779 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:24.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:24.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:24.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[29816]: 3.19 scrub starts 2024-02-10T07:38:24.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[29816]: 3.19 scrub ok 2024-02-10T07:38:24.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[29816]: pgmap v302: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:38:24.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:38:24.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[29816]: osdmap e206: 8 total, 8 up, 8 in 2024-02-10T07:38:24.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[29816]: 6.1b scrub starts 2024-02-10T07:38:24.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[29816]: 6.1b scrub ok 2024-02-10T07:38:24.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[29816]: 6.16 scrub starts 2024-02-10T07:38:24.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[40743]: 3.19 scrub starts 2024-02-10T07:38:24.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[40743]: 3.19 scrub ok 2024-02-10T07:38:24.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[40743]: pgmap v302: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:38:24.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:38:24.834 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[40743]: osdmap e206: 8 total, 8 up, 8 in 2024-02-10T07:38:24.834 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[40743]: 6.1b scrub starts 2024-02-10T07:38:24.834 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[40743]: 6.1b scrub ok 2024-02-10T07:38:24.834 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:24 smithi008 ceph-mon[40743]: 6.16 scrub starts 2024-02-10T07:38:24.881 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:24.909 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:24.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:24.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:25.011 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:25.037 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:25.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:25.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:24 smithi170 ceph-mon[33394]: 3.19 scrub starts 2024-02-10T07:38:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:24 smithi170 ceph-mon[33394]: 3.19 scrub ok 2024-02-10T07:38:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:24 smithi170 ceph-mon[33394]: pgmap v302: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:38:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:24 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:38:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:24 smithi170 ceph-mon[33394]: osdmap e206: 8 total, 8 up, 8 in 2024-02-10T07:38:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:24 smithi170 ceph-mon[33394]: 6.1b scrub starts 2024-02-10T07:38:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:24 smithi170 ceph-mon[33394]: 6.1b scrub ok 2024-02-10T07:38:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:24 smithi170 ceph-mon[33394]: 6.16 scrub starts 2024-02-10T07:38:25.139 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:25.164 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:25.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:25.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:25.267 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:25.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:25.296 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:25.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:25.397 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:25.422 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:25.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:25.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:25.524 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:25.550 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:25.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:25.597 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:25.597 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:25.653 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:25.682 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:25.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:25.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:25.784 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:25.811 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:25.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:25.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:25.830 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.7 to 1 (10000) 2024-02-10T07:38:25.913 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:25.940 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:25.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:26.041 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:26.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:26.066 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:26.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[29816]: 3.1d scrub starts 2024-02-10T07:38:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[29816]: 3.1d scrub ok 2024-02-10T07:38:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[29816]: 6.16 scrub ok 2024-02-10T07:38:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[29816]: 5.12 scrub starts 2024-02-10T07:38:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[29816]: 5.12 scrub ok 2024-02-10T07:38:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[29816]: 4.1c scrub starts 2024-02-10T07:38:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[29816]: 4.1c scrub ok 2024-02-10T07:38:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1133288202' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:38:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:38:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[29816]: 5.13 scrub starts 2024-02-10T07:38:26.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[29816]: 5.13 scrub ok 2024-02-10T07:38:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[40743]: 3.1d scrub starts 2024-02-10T07:38:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[40743]: 3.1d scrub ok 2024-02-10T07:38:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[40743]: 6.16 scrub ok 2024-02-10T07:38:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[40743]: 5.12 scrub starts 2024-02-10T07:38:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[40743]: 5.12 scrub ok 2024-02-10T07:38:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[40743]: 4.1c scrub starts 2024-02-10T07:38:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[40743]: 4.1c scrub ok 2024-02-10T07:38:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1133288202' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:38:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:38:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[40743]: 5.13 scrub starts 2024-02-10T07:38:26.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:25 smithi008 ceph-mon[40743]: 5.13 scrub ok 2024-02-10T07:38:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:25 smithi170 ceph-mon[33394]: 3.1d scrub starts 2024-02-10T07:38:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:25 smithi170 ceph-mon[33394]: 3.1d scrub ok 2024-02-10T07:38:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:25 smithi170 ceph-mon[33394]: 6.16 scrub ok 2024-02-10T07:38:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:25 smithi170 ceph-mon[33394]: 5.12 scrub starts 2024-02-10T07:38:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:25 smithi170 ceph-mon[33394]: 5.12 scrub ok 2024-02-10T07:38:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:25 smithi170 ceph-mon[33394]: 4.1c scrub starts 2024-02-10T07:38:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:25 smithi170 ceph-mon[33394]: 4.1c scrub ok 2024-02-10T07:38:26.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:25 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1133288202' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:38:26.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:25 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:38:26.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:25 smithi170 ceph-mon[33394]: 5.13 scrub starts 2024-02-10T07:38:26.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:25 smithi170 ceph-mon[33394]: 5.13 scrub ok 2024-02-10T07:38:26.167 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:26.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:26.192 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:26.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:26.293 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:26.322 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:26.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:26.423 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:26.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:26.448 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:26.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:26.549 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:26.577 DEBUG:teuthology.orchestra.run.smithi008:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-02-10T07:38:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:26.578 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:26.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:26.580 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-10T07:38:26.580 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-10T07:38:26.627 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:hints = 1 2024-02-10T07:38:26.627 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-10T07:38:26.627 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Object prefix: benchmark_data_smithi008_109566 2024-02-10T07:38:26.628 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:38:26.628 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 0 0 0 0 0 0 - 0 2024-02-10T07:38:26.678 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:26.733 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:26.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:26.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:26.742 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-10T07:38:26.742 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:38:26.835 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:26.869 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:26.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:26.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:26.945 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[29816]: 4.15 scrub starts 2024-02-10T07:38:26.945 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[29816]: 4.15 scrub ok 2024-02-10T07:38:26.945 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[29816]: 6.10 scrub starts 2024-02-10T07:38:26.946 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[29816]: 4.7 scrub starts 2024-02-10T07:38:26.946 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[29816]: 4.7 scrub ok 2024-02-10T07:38:26.946 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[29816]: 6.10 scrub ok 2024-02-10T07:38:26.946 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[29816]: pgmap v304: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:38:26.946 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[29816]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:38:26.946 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:38:26.946 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[29816]: osdmap e207: 8 total, 8 up, 8 in 2024-02-10T07:38:26.946 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[29816]: 4.a scrub starts 2024-02-10T07:38:26.946 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[29816]: 4.a scrub ok 2024-02-10T07:38:26.946 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[40743]: 4.15 scrub starts 2024-02-10T07:38:26.946 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[40743]: 4.15 scrub ok 2024-02-10T07:38:26.946 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[40743]: 6.10 scrub starts 2024-02-10T07:38:26.946 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[40743]: 4.7 scrub starts 2024-02-10T07:38:26.947 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[40743]: 4.7 scrub ok 2024-02-10T07:38:26.947 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[40743]: 6.10 scrub ok 2024-02-10T07:38:26.947 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[40743]: pgmap v304: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:38:26.947 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[40743]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:38:26.947 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:38:26.947 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[40743]: osdmap e207: 8 total, 8 up, 8 in 2024-02-10T07:38:26.947 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[40743]: 4.a scrub starts 2024-02-10T07:38:26.947 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:26 smithi008 ceph-mon[40743]: 4.a scrub ok 2024-02-10T07:38:26.970 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:26.998 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:26.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:27.100 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:26 smithi170 ceph-mon[33394]: 4.15 scrub starts 2024-02-10T07:38:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:26 smithi170 ceph-mon[33394]: 4.15 scrub ok 2024-02-10T07:38:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:26 smithi170 ceph-mon[33394]: 6.10 scrub starts 2024-02-10T07:38:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:26 smithi170 ceph-mon[33394]: 4.7 scrub starts 2024-02-10T07:38:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:26 smithi170 ceph-mon[33394]: 4.7 scrub ok 2024-02-10T07:38:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:26 smithi170 ceph-mon[33394]: 6.10 scrub ok 2024-02-10T07:38:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:26 smithi170 ceph-mon[33394]: pgmap v304: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:38:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:26 smithi170 ceph-mon[33394]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:38:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:26 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:38:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:26 smithi170 ceph-mon[33394]: osdmap e207: 8 total, 8 up, 8 in 2024-02-10T07:38:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:26 smithi170 ceph-mon[33394]: 4.a scrub starts 2024-02-10T07:38:27.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:26 smithi170 ceph-mon[33394]: 4.a scrub ok 2024-02-10T07:38:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:27.134 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:27.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:27.235 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:27.269 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:27.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:27.371 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:27.414 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:27.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:27.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:27.515 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:27.560 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:27.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:27.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:27.629 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 1 15 3011 2996 186.966 187.25 0.00371076 0.00532357 2024-02-10T07:38:27.662 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:27.704 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:27.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:27.805 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:27.831 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:38:27.831 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-02-10T07:38:27.831 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-02-10T07:38:27.831 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:38:27.831 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.831 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-02-10T07:38:27.831 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:29:34.373659+0000", 2024-02-10T07:38:27.831 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 533.12148354600004, 2024-02-10T07:38:27.831 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.69914913400000001, 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.14158", 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3905205183", 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 1 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:34.373659+0000", 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.832 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:34.373659+0000", 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:34.373661+0000", 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3319999999999999e-06 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:34.373670+0000", 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0690000000000008e-06 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.833 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:34.373687+0000", 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6413e-05 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:34.373700+0000", 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2775000000000001e-05 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:34.373784+0000", 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4270000000000005e-05 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:34.373794+0000", 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9189999999999994e-06 2024-02-10T07:38:27.834 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:35.071846+0000", 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.69805253499999997 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:35.071987+0000", 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014106000000000001 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:35.072124+0000", 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013668400000000001 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.835 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.836 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:38:27.836 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:35.072519+0000", 2024-02-10T07:38:27.836 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00039502199999999997 2024-02-10T07:38:27.836 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.836 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.836 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.836 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:35.072685+0000", 2024-02-10T07:38:27.836 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016614599999999999 2024-02-10T07:38:27.836 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.836 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.836 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.836 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:35.072746+0000", 2024-02-10T07:38:27.836 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0253000000000001e-05 2024-02-10T07:38:27.836 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:35.072782+0000", 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6177999999999998e-05 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:29:35.072808+0000", 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6478e-05 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(osd.0.0:33270 7.9 e139/135 7:937d747e:::benchmark_data_smithi008_82046_object28887:head v 139'9316, mlcod=139'9302)", 2024-02-10T07:38:27.837 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.078637+0000", 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 151.41650537199999, 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.158943063, 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.078637+0000", 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.078637+0000", 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.838 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.078637+0000", 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8799999999999999e-07 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.078641+0000", 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.14e-06 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.078646+0000", 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9599999999999999e-06 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.839 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.078650+0000", 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9870000000000001e-06 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.088740+0000", 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010090346 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.088759+0000", 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8853000000000001e-05 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.840 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.237545+0000", 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.148786272 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.237580+0000", 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.5317000000000001e-05 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127245 7.b 7:db9643ad:::benchmark_data_smithi008_82046_object12107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.078906+0000", 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 151.416237016, 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0410415989999999, 2024-02-10T07:38:27.841 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127245 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.078906+0000", 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.078906+0000", 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.842 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.078907+0000", 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2890000000000001e-06 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.078911+0000", 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2710000000000003e-06 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.078923+0000", 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1369000000000001e-05 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.843 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.078930+0000", 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5079999999999999e-06 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.079107+0000", 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017721800000000001 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.079227+0000", 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011987199999999999 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:38:27.844 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.079348+0000", 2024-02-10T07:38:27.845 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012117600000000001 2024-02-10T07:38:27.845 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.845 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.845 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.845 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.094680+0000", 2024-02-10T07:38:27.845 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015331404 2024-02-10T07:38:27.845 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.845 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.845 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.845 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.231490+0000", 2024-02-10T07:38:27.845 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.13681054500000001 2024-02-10T07:38:27.845 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.845 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.231502+0000", 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1447000000000001e-05 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.119883+0000", 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.88838167400000001 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.119947+0000", 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3825999999999995e-05 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.846 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127247 7.0 7:04e8d55c:::benchmark_data_smithi008_82046_object12109:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:38:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.233047+0000", 2024-02-10T07:38:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 151.26209589499999, 2024-02-10T07:38:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.91480609700000004, 2024-02-10T07:38:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-02-10T07:38:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127247 2024-02-10T07:38:27.847 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.848 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.848 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.848 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.848 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.233047+0000", 2024-02-10T07:38:27.848 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.848 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.848 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.848 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.848 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.233047+0000", 2024-02-10T07:38:27.848 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.848 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.848 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.848 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.849 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.233048+0000", 2024-02-10T07:38:27.849 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4529999999999999e-06 2024-02-10T07:38:27.849 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.849 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.849 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.849 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.233053+0000", 2024-02-10T07:38:27.849 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6190000000000002e-06 2024-02-10T07:38:27.849 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.849 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.849 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.849 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.233063+0000", 2024-02-10T07:38:27.849 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.008e-05 2024-02-10T07:38:27.850 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.850 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.850 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.850 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.233075+0000", 2024-02-10T07:38:27.850 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2559e-05 2024-02-10T07:38:27.850 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.850 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.850 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.850 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.233230+0000", 2024-02-10T07:38:27.850 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015488899999999999 2024-02-10T07:38:27.850 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.850 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.850 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.851 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.233239+0000", 2024-02-10T07:38:27.851 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8089999999999999e-06 2024-02-10T07:38:27.851 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.851 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.851 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.851 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.133049+0000", 2024-02-10T07:38:27.851 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.89980981599999998 2024-02-10T07:38:27.851 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.851 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.851 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for missing object", 2024-02-10T07:38:27.851 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.133599+0000", 2024-02-10T07:38:27.851 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00055033899999999995 2024-02-10T07:38:27.851 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.147794+0000", 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014194277999999999 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.147853+0000", 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9255000000000001e-05 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.852 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127268 7.b 7:d3a62516:::benchmark_data_smithi008_82046_object12130:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.282586+0000", 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 151.21255652599999, 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.85278400499999996, 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127268 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.282586+0000", 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.853 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.854 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.854 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.854 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.282586+0000", 2024-02-10T07:38:27.854 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.854 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.854 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.854 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.854 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.282588+0000", 2024-02-10T07:38:27.854 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0710000000000002e-06 2024-02-10T07:38:27.854 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.854 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.854 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.854 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.282593+0000", 2024-02-10T07:38:27.854 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2549999999999997e-06 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.282606+0000", 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2812e-05 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.282615+0000", 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3000000000000002e-06 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.282674+0000", 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9333000000000003e-05 2024-02-10T07:38:27.855 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.856 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.856 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.856 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.282683+0000", 2024-02-10T07:38:27.856 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8699999999999998e-06 2024-02-10T07:38:27.856 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.856 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.856 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.856 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.120213+0000", 2024-02-10T07:38:27.856 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.83753064899999996 2024-02-10T07:38:27.856 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.856 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.856 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.856 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.120325+0000", 2024-02-10T07:38:27.856 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0001121 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 6", 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.120456+0000", 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013097900000000001 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.135097+0000", 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014640966 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.135172+0000", 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.4674999999999999e-05 2024-02-10T07:38:27.857 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.135276+0000", 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000103618 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.135370+0000", 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4376999999999994e-05 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.858 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127272 7.b 7:d26e648b:::benchmark_data_smithi008_82046_object12134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.289272+0000", 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 151.20587105000001, 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.84639729600000002, 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127272 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.289272+0000", 2024-02-10T07:38:27.859 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.289272+0000", 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.289274+0000", 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9769999999999999e-06 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.289280+0000", 2024-02-10T07:38:27.860 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2020000000000001e-06 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.289303+0000", 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3552000000000001e-05 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.289313+0000", 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6600000000000007e-06 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.289376+0000", 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2607999999999994e-05 2024-02-10T07:38:27.861 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.289386+0000", 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0396e-05 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.123327+0000", 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.83394067100000002 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.123442+0000", 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011552000000000001 2024-02-10T07:38:27.862 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 6", 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.123574+0000", 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013231399999999999 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.135142+0000", 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011567707999999999 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.135378+0000", 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000236054 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.863 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.135578+0000", 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000200249 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.135669+0000", 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0384999999999996e-05 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127276 7.b 7:d9e863b6:::benchmark_data_smithi008_82046_object12138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:38:27.864 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.293247+0000", 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 151.201895369, 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.84780041299999997, 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127276 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.293247+0000", 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.865 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.293247+0000", 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.293249+0000", 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4190000000000001e-06 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.293254+0000", 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6679999999999998e-06 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.866 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.293271+0000", 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6609999999999999e-05 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.293280+0000", 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4029999999999996e-06 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.293343+0000", 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2908000000000001e-05 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.293354+0000", 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0644e-05 2024-02-10T07:38:27.867 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.868 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.868 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.868 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.129185+0000", 2024-02-10T07:38:27.868 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.83583098300000003 2024-02-10T07:38:27.868 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.868 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.868 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.868 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.129298+0000", 2024-02-10T07:38:27.868 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000112982 2024-02-10T07:38:27.868 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.868 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.868 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 6", 2024-02-10T07:38:27.868 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.129431+0000", 2024-02-10T07:38:27.868 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000133428 2024-02-10T07:38:27.869 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.869 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.869 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:38:27.869 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.135678+0000", 2024-02-10T07:38:27.869 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0062464829999999997 2024-02-10T07:38:27.869 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.869 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.869 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.869 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.140866+0000", 2024-02-10T07:38:27.869 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0051877809999999998 2024-02-10T07:38:27.869 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.869 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.869 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.869 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.140960+0000", 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4333000000000006e-05 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.141048+0000", 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7770999999999997e-05 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127289 7.b 7:de47480c:::benchmark_data_smithi008_82046_object12151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.364939+0000", 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 151.13020337899999, 2024-02-10T07:38:27.870 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.77630776499999998, 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127289 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.364939+0000", 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.871 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.364939+0000", 2024-02-10T07:38:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.364941+0000", 2024-02-10T07:38:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.917e-06 2024-02-10T07:38:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.364945+0000", 2024-02-10T07:38:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2579999999999997e-06 2024-02-10T07:38:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.872 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.873 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.873 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.364961+0000", 2024-02-10T07:38:27.873 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5265999999999999e-05 2024-02-10T07:38:27.873 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.873 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.873 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.873 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.364970+0000", 2024-02-10T07:38:27.873 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7589999999999992e-06 2024-02-10T07:38:27.873 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.873 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.873 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.873 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.365033+0000", 2024-02-10T07:38:27.873 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2959000000000005e-05 2024-02-10T07:38:27.873 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.365043+0000", 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.5449999999999992e-06 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.132085+0000", 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.76704220199999995 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.132199+0000", 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000113384 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.874 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 6", 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.132322+0000", 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012325899999999999 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.136083+0000", 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0037612349999999999 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.141094+0000", 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0050110959999999996 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.875 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.141172+0000", 2024-02-10T07:38:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7854000000000004e-05 2024-02-10T07:38:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.141247+0000", 2024-02-10T07:38:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5030999999999994e-05 2024-02-10T07:38:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.876 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24356.0:1386 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 93827140445184 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.412019+0000", 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 151.08312353599999, 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.73117822399999999, 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24356", 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1933562162", 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 1386 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412019+0000", 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.877 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412019+0000", 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412020+0000", 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1039999999999999e-06 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412027+0000", 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7290000000000003e-06 2024-02-10T07:38:27.878 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412039+0000", 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.166e-05 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412046+0000", 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5780000000000001e-06 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412108+0000", 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1996999999999996e-05 2024-02-10T07:38:27.879 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412117+0000", 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9500000000000007e-06 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.117798+0000", 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.70568117600000002 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.117949+0000", 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015026199999999999 2024-02-10T07:38:27.880 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.143197+0000", 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.025248768000000001 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:132729 7.12 7:4b76b4a5:::benchmark_data_smithi008_82046_object17591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:07.433849+0000", 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 140.061293181, 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.17748874100000001, 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.881 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 132729 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.433849+0000", 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.433849+0000", 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.882 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.433852+0000", 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.429e-06 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.433858+0000", 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3439999999999997e-06 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.516533+0000", 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.082674592000000005 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.883 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.884 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.884 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.516543+0000", 2024-02-10T07:38:27.884 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0312999999999999e-05 2024-02-10T07:38:27.884 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.884 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.884 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.884 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.516770+0000", 2024-02-10T07:38:27.884 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00022679199999999999 2024-02-10T07:38:27.884 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.884 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.884 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.884 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.516920+0000", 2024-02-10T07:38:27.885 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015015300000000001 2024-02-10T07:38:27.885 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.885 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.885 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,2,3,6", 2024-02-10T07:38:27.885 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.517227+0000", 2024-02-10T07:38:27.885 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00030733900000000001 2024-02-10T07:38:27.885 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.885 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.885 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:38:27.885 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.520236+0000", 2024-02-10T07:38:27.885 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003008574 2024-02-10T07:38:27.885 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.885 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.520277+0000", 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0537000000000002e-05 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.610300+0000", 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.090023304999999998 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.610347+0000", 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7290999999999998e-05 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.886 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.611159+0000", 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00081185700000000003 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.611218+0000", 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9323000000000001e-05 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.611338+0000", 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000119892 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.887 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.888 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.888 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154803 7.b 7:dde2e294:::benchmark_data_smithi008_82046_object39665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-02-10T07:38:27.888 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.741827+0000", 2024-02-10T07:38:27.888 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 114.753315641, 2024-02-10T07:38:27.888 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1131070689999998, 2024-02-10T07:38:27.888 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.888 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-02-10T07:38:27.888 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.888 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.888 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.888 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154803 2024-02-10T07:38:27.888 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.888 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.888 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.741827+0000", 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.741827+0000", 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.741830+0000", 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5459999999999998e-06 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.889 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.741835+0000", 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6949999999999997e-06 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.741849+0000", 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3368e-05 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.741858+0000", 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3910000000000004e-06 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.890 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.741968+0000", 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000109792 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.742094+0000", 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000126608 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 6", 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.742549+0000", 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00045454099999999998 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.891 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.766191+0000", 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.023641716 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.766202+0000", 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1637999999999999e-05 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739123+0000", 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97292105200000001 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.892 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739138+0000", 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.436e-05 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.842319+0000", 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1031809809999999 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for missing object", 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.842642+0000", 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00032379400000000002 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.893 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.894 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.854849+0000", 2024-02-10T07:38:27.894 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012207015 2024-02-10T07:38:27.894 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.894 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.894 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.894 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.854934+0000", 2024-02-10T07:38:27.894 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4572000000000003e-05 2024-02-10T07:38:27.894 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.894 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.894 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.894 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.894 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.894 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154809 7.b 7:d67084d0:::benchmark_data_smithi008_82046_object39671:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.750199+0000", 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 114.744943345, 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1160224900000002, 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154809 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.750199+0000", 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.895 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.896 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.896 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.896 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.750199+0000", 2024-02-10T07:38:27.896 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.896 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.896 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.896 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.896 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.750200+0000", 2024-02-10T07:38:27.896 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.13e-07 2024-02-10T07:38:27.896 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.896 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.896 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.896 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.750203+0000", 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9380000000000001e-06 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.750211+0000", 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6519999999999995e-06 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.750218+0000", 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7190000000000004e-06 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.897 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.750506+0000", 2024-02-10T07:38:27.898 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00028781800000000001 2024-02-10T07:38:27.898 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.898 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.898 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.898 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.750640+0000", 2024-02-10T07:38:27.898 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0001338 2024-02-10T07:38:27.898 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.898 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.898 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 6", 2024-02-10T07:38:27.898 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.750762+0000", 2024-02-10T07:38:27.898 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000122438 2024-02-10T07:38:27.898 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.898 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.898 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.766234+0000", 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015471983 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.766243+0000", 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.969e-06 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739175+0000", 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97293163199999999 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.899 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.900 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739185+0000", 2024-02-10T07:38:27.900 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9140000000000003e-06 2024-02-10T07:38:27.900 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.900 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.900 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.900 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.843027+0000", 2024-02-10T07:38:27.900 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1038422050000001 2024-02-10T07:38:27.900 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.900 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.900 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for missing object", 2024-02-10T07:38:27.900 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.843165+0000", 2024-02-10T07:38:27.900 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013781899999999999 2024-02-10T07:38:27.900 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.866144+0000", 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.022979481999999999 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.866222+0000", 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7608000000000006e-05 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.901 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154812 7.b 7:d57cbec3:::benchmark_data_smithi008_82046_object39674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.759780+0000", 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 114.735363053, 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1050681550000001, 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154812 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.759780+0000", 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.902 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.759780+0000", 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.759781+0000", 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7409999999999999e-06 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.759787+0000", 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5300000000000004e-06 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.903 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.904 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.760305+0000", 2024-02-10T07:38:27.904 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00051850500000000005 2024-02-10T07:38:27.904 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.904 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.904 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.904 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.760319+0000", 2024-02-10T07:38:27.904 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.323e-05 2024-02-10T07:38:27.904 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.904 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.904 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.904 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.766272+0000", 2024-02-10T07:38:27.904 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0059538890000000004 2024-02-10T07:38:27.904 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.766281+0000", 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5210000000000002e-06 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739214+0000", 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97293327299999999 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739223+0000", 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6400000000000003e-06 2024-02-10T07:38:27.905 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.843191+0000", 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.103968311 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.843307+0000", 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000115585 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.843438+0000", 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000131464 2024-02-10T07:38:27.906 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:38:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.854394+0000", 2024-02-10T07:38:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010955321000000001 2024-02-10T07:38:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.855124+0000", 2024-02-10T07:38:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00073064899999999999 2024-02-10T07:38:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.864409+0000", 2024-02-10T07:38:27.907 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0092850550000000004 2024-02-10T07:38:27.908 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.908 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.908 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.908 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.864683+0000", 2024-02-10T07:38:27.908 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00027325 2024-02-10T07:38:27.908 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.908 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.908 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.908 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.864848+0000", 2024-02-10T07:38:27.908 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000165191 2024-02-10T07:38:27.908 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.908 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.908 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.908 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154821 7.b 7:d0f1635f:::benchmark_data_smithi008_82046_object39683:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.881516+0000", 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 114.613626311, 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.983838153, 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154821 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.909 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.881516+0000", 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.881516+0000", 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.881518+0000", 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8929999999999999e-06 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.910 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.881523+0000", 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1290000000000004e-06 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.881536+0000", 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2279999999999999e-05 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.881544+0000", 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5040000000000002e-06 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.911 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.881615+0000", 2024-02-10T07:38:27.912 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0834999999999995e-05 2024-02-10T07:38:27.912 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.912 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.912 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.912 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.881626+0000", 2024-02-10T07:38:27.912 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0855e-05 2024-02-10T07:38:27.912 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.912 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.912 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.912 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739250+0000", 2024-02-10T07:38:27.912 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.85762451100000003 2024-02-10T07:38:27.912 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.912 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.912 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.913 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739259+0000", 2024-02-10T07:38:27.913 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.354e-06 2024-02-10T07:38:27.913 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.913 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.913 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.913 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.845975+0000", 2024-02-10T07:38:27.913 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1067161699999999 2024-02-10T07:38:27.913 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.913 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.913 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.913 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.846100+0000", 2024-02-10T07:38:27.913 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000125532 2024-02-10T07:38:27.913 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.913 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.846246+0000", 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014550199999999999 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.854817+0000", 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0085713520000000008 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.855228+0000", 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00041086300000000003 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.914 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.864896+0000", 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0096678809999999997 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.865214+0000", 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00031762700000000001 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.865354+0000", 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000140865 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.915 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.916 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.916 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.916 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154825 7.b 7:db0d2cd8:::benchmark_data_smithi008_82046_object39687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:38:27.916 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.883106+0000", 2024-02-10T07:38:27.916 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 114.61203666, 2024-02-10T07:38:27.916 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.982686902, 2024-02-10T07:38:27.916 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.916 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.916 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.916 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.916 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.916 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154825 2024-02-10T07:38:27.916 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.916 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883106+0000", 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883106+0000", 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883107+0000", 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.128e-06 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.917 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883111+0000", 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3379999999999998e-06 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883122+0000", 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0577000000000001e-05 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883129+0000", 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0740000000000004e-06 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.918 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883184+0000", 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5173000000000002e-05 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883194+0000", 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9760000000000007e-06 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739287+0000", 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.85609274400000002 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.919 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.920 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739296+0000", 2024-02-10T07:38:27.920 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6470000000000004e-06 2024-02-10T07:38:27.920 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.920 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.920 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.920 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.849875+0000", 2024-02-10T07:38:27.920 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.110579089 2024-02-10T07:38:27.920 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.920 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.920 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.920 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.850003+0000", 2024-02-10T07:38:27.920 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012874499999999999 2024-02-10T07:38:27.920 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.920 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.850155+0000", 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015133799999999999 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.855138+0000", 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0049831440000000001 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.855272+0000", 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013383000000000001 2024-02-10T07:38:27.921 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.865409+0000", 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010137591 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.865636+0000", 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00022707900000000001 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.922 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.865793+0000", 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000156429 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154845 7.b 7:dd327772:::benchmark_data_smithi008_82046_object39707:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.911915+0000", 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 114.58322721, 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9545649219999999, 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.923 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.924 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.924 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154845 2024-02-10T07:38:27.924 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.924 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.924 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.924 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.924 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.911915+0000", 2024-02-10T07:38:27.924 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.924 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.924 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.924 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.924 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.911915+0000", 2024-02-10T07:38:27.924 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.924 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.911917+0000", 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.248e-06 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.911921+0000", 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1960000000000001e-06 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.911932+0000", 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1428e-05 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.925 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.911941+0000", 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3100000000000001e-06 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.911998+0000", 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7021000000000002e-05 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.912006+0000", 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0190000000000006e-06 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.926 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739332+0000", 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.82732605599999998 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739341+0000", 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8810000000000002e-06 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.855302+0000", 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1159616809999999 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.855421+0000", 2024-02-10T07:38:27.927 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000119129 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.855564+0000", 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014231000000000001 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.864365+0000", 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.008801372 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.866258+0000", 2024-02-10T07:38:27.928 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001892878 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.866298+0000", 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0460000000000002e-05 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.866350+0000", 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1313000000000002e-05 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.866480+0000", 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013061999999999999 2024-02-10T07:38:27.929 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154826 7.0 7:00b97eea:::benchmark_data_smithi008_82046_object39688:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:33.741727+0000", 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 113.753415515, 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0666621080000001, 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154826 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.930 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.741727+0000", 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.741727+0000", 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.741728+0000", 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.37e-06 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.931 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.741732+0000", 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7500000000000001e-06 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.741744+0000", 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1847e-05 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.741754+0000", 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0193e-05 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.932 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.742432+0000", 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00067759099999999996 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.742443+0000", 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1442e-05 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.801402+0000", 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0589584999999999 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.801544+0000", 2024-02-10T07:38:27.933 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014232399999999999 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 3,5", 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.801885+0000", 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00034105000000000002 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.803572+0000", 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0016866369999999999 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.805238+0000", 2024-02-10T07:38:27.934 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0016662879999999999 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.808145+0000", 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0029071560000000001 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.808256+0000", 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011033500000000001 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.808389+0000", 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013362500000000001 2024-02-10T07:38:27.935 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154852 7.b 7:dc19ea1b:::benchmark_data_smithi008_82046_object39714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:33.826022+0000", 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 113.6691208, 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0411249499999999, 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154852 2024-02-10T07:38:27.936 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.826022+0000", 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.826022+0000", 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.826023+0000", 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.548e-06 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.937 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.826028+0000", 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3599999999999998e-06 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.826039+0000", 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1758e-05 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.826045+0000", 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9889999999999997e-06 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.938 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.826123+0000", 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7269999999999997e-05 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.826136+0000", 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3587999999999999e-05 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.860697+0000", 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.034560506 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.939 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.860831+0000", 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000134258 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.860977+0000", 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014543100000000001 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.865366+0000", 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0043895330000000001 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.866524+0000", 2024-02-10T07:38:27.940 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0011583139999999999 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.866568+0000", 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3479999999999997e-05 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.867017+0000", 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00044903400000000001 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.867147+0000", 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012988100000000001 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.941 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154866 7.0 7:01d38255:::benchmark_data_smithi008_82046_object39728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:33.849561+0000", 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 113.64558116000001, 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96027641900000005, 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154866 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.942 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.849561+0000", 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.849561+0000", 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.849563+0000", 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6929999999999999e-06 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.943 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.849570+0000", 2024-02-10T07:38:27.944 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6279999999999999e-06 2024-02-10T07:38:27.944 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.944 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.944 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.944 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.849590+0000", 2024-02-10T07:38:27.944 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.986e-05 2024-02-10T07:38:27.944 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.944 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.944 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.944 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.849601+0000", 2024-02-10T07:38:27.944 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1022000000000001e-05 2024-02-10T07:38:27.944 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.944 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.849673+0000", 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2828000000000001e-05 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.849688+0000", 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4157e-05 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.803646+0000", 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.95395848500000002 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.945 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.946 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.803787+0000", 2024-02-10T07:38:27.946 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000140514 2024-02-10T07:38:27.946 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.946 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.946 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 3,5", 2024-02-10T07:38:27.946 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.804018+0000", 2024-02-10T07:38:27.946 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00023100200000000001 2024-02-10T07:38:27.946 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.946 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.946 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:38:27.946 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.805274+0000", 2024-02-10T07:38:27.946 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0012563629999999999 2024-02-10T07:38:27.946 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.946 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.805764+0000", 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000490015 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.809377+0000", 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0036127630000000002 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.809434+0000", 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6968000000000001e-05 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.947 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.948 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.809838+0000", 2024-02-10T07:38:27.948 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00040412100000000002 2024-02-10T07:38:27.948 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.948 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.948 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.948 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.948 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.948 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154869 7.b 7:dc5f020b:::benchmark_data_smithi008_82046_object39731:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-02-10T07:38:27.948 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:33.856867+0000", 2024-02-10T07:38:27.948 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 113.638275467, 2024-02-10T07:38:27.948 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0271456400000001, 2024-02-10T07:38:27.948 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:38:27.948 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154869 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.856867+0000", 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.856867+0000", 2024-02-10T07:38:27.949 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.856868+0000", 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2619999999999999e-06 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.856873+0000", 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1470000000000003e-06 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.856884+0000", 2024-02-10T07:38:27.950 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1107000000000001e-05 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.856890+0000", 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2049999999999999e-06 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.856947+0000", 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7268000000000001e-05 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.856955+0000", 2024-02-10T07:38:27.951 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5560000000000002e-06 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.867195+0000", 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.010240421 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.867308+0000", 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000113319 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.867436+0000", 2024-02-10T07:38:27.952 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012728899999999999 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.879892+0000", 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.01245608 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.879936+0000", 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4320000000000003e-05 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.883733+0000", 2024-02-10T07:38:27.953 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0037966660000000002 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.883784+0000", 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1273999999999997e-05 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.884013+0000", 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00022872600000000001 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:38:27.954 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:38:27.955 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:38:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:27.956 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:27.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:28.057 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:28.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[29816]: 3.c scrub starts 2024-02-10T07:38:28.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[29816]: 3.c scrub ok 2024-02-10T07:38:28.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[29816]: 3.3 scrub starts 2024-02-10T07:38:28.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[29816]: 3.3 scrub ok 2024-02-10T07:38:28.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[29816]: 6.1d scrub starts 2024-02-10T07:38:28.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[29816]: 6.1d scrub ok 2024-02-10T07:38:28.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[29816]: 5.9 scrub starts 2024-02-10T07:38:28.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[29816]: 5.9 scrub ok 2024-02-10T07:38:28.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[40743]: 3.c scrub starts 2024-02-10T07:38:28.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[40743]: 3.c scrub ok 2024-02-10T07:38:28.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[40743]: 3.3 scrub starts 2024-02-10T07:38:28.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[40743]: 3.3 scrub ok 2024-02-10T07:38:28.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[40743]: 6.1d scrub starts 2024-02-10T07:38:28.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[40743]: 6.1d scrub ok 2024-02-10T07:38:28.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[40743]: 5.9 scrub starts 2024-02-10T07:38:28.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:27 smithi008 ceph-mon[40743]: 5.9 scrub ok 2024-02-10T07:38:28.096 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:28.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:28.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:27 smithi170 ceph-mon[33394]: 3.c scrub starts 2024-02-10T07:38:28.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:27 smithi170 ceph-mon[33394]: 3.c scrub ok 2024-02-10T07:38:28.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:27 smithi170 ceph-mon[33394]: 3.3 scrub starts 2024-02-10T07:38:28.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:27 smithi170 ceph-mon[33394]: 3.3 scrub ok 2024-02-10T07:38:28.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:27 smithi170 ceph-mon[33394]: 6.1d scrub starts 2024-02-10T07:38:28.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:27 smithi170 ceph-mon[33394]: 6.1d scrub ok 2024-02-10T07:38:28.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:27 smithi170 ceph-mon[33394]: 5.9 scrub starts 2024-02-10T07:38:28.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:27 smithi170 ceph-mon[33394]: 5.9 scrub ok 2024-02-10T07:38:28.198 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:28.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:28.245 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:28.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:28.345 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:28.385 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:28.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:28.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:28.487 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:28.522 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:28.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:28.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:28.624 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:28.629 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 2 16 6291 6275 195.936 204.938 0.0097792 0.00508979 2024-02-10T07:38:28.663 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:28.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:28.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:28.765 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:28.808 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:28.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:28.910 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:28.957 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:28.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:29.058 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:29.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:38:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:38:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:38:29.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:28 smithi008 ceph-mon[29816]: 6.d scrub starts 2024-02-10T07:38:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:28 smithi008 ceph-mon[29816]: 6.d scrub ok 2024-02-10T07:38:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:28 smithi008 ceph-mon[29816]: pgmap v306: 148 pgs: 148 active+clean; 46 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.7 MiB/s wr, 92 op/s 2024-02-10T07:38:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:28 smithi008 ceph-mon[29816]: 6.0 scrub starts 2024-02-10T07:38:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:28 smithi008 ceph-mon[29816]: 6.0 scrub ok 2024-02-10T07:38:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:28 smithi008 ceph-mon[40743]: 6.d scrub starts 2024-02-10T07:38:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:28 smithi008 ceph-mon[40743]: 6.d scrub ok 2024-02-10T07:38:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:28 smithi008 ceph-mon[40743]: pgmap v306: 148 pgs: 148 active+clean; 46 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.7 MiB/s wr, 92 op/s 2024-02-10T07:38:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:28 smithi008 ceph-mon[40743]: 6.0 scrub starts 2024-02-10T07:38:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:28 smithi008 ceph-mon[40743]: 6.0 scrub ok 2024-02-10T07:38:29.092 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:29.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:29.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:28 smithi170 ceph-mon[33394]: 6.d scrub starts 2024-02-10T07:38:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:28 smithi170 ceph-mon[33394]: 6.d scrub ok 2024-02-10T07:38:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:28 smithi170 ceph-mon[33394]: pgmap v306: 148 pgs: 148 active+clean; 46 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.7 MiB/s wr, 92 op/s 2024-02-10T07:38:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:28 smithi170 ceph-mon[33394]: 6.0 scrub starts 2024-02-10T07:38:29.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:28 smithi170 ceph-mon[33394]: 6.0 scrub ok 2024-02-10T07:38:29.195 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:29.229 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:29.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:29.330 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:29.365 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:29.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:29.466 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:29.511 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:29.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:29.613 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:29.630 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 3 16 9395 9379 195.275 194 0.00388049 0.00511109 2024-02-10T07:38:29.653 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:29.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:29.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:29.755 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:29.788 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:29.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:29.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:29.890 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:29.923 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:29.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:30.025 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:30.061 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:30.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:30.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:29 smithi170 ceph-mon[33394]: 5.1a scrub starts 2024-02-10T07:38:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:29 smithi170 ceph-mon[33394]: 5.1a scrub ok 2024-02-10T07:38:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:29 smithi170 ceph-mon[33394]: 3.1f scrub starts 2024-02-10T07:38:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:29 smithi170 ceph-mon[33394]: 3.1f scrub ok 2024-02-10T07:38:30.163 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:30.197 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:30.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:30.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:30.299 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:29 smithi008 ceph-mon[29816]: 5.1a scrub starts 2024-02-10T07:38:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:29 smithi008 ceph-mon[29816]: 5.1a scrub ok 2024-02-10T07:38:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:29 smithi008 ceph-mon[29816]: 3.1f scrub starts 2024-02-10T07:38:30.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:29 smithi008 ceph-mon[29816]: 3.1f scrub ok 2024-02-10T07:38:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:29 smithi008 ceph-mon[40743]: 5.1a scrub starts 2024-02-10T07:38:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:29 smithi008 ceph-mon[40743]: 5.1a scrub ok 2024-02-10T07:38:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:29 smithi008 ceph-mon[40743]: 3.1f scrub starts 2024-02-10T07:38:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:29 smithi008 ceph-mon[40743]: 3.1f scrub ok 2024-02-10T07:38:30.344 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:30.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:30.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:30.446 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:30.477 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-10T07:38:30.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:30.478 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:30.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:30.579 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:30.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:30.626 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:30.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:30.630 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 4 16 12304 12288 191.904 181.812 0.00443363 0.00520232 2024-02-10T07:38:30.726 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:30.774 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:30.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:30.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:30.876 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:30.908 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:30.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:31.009 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:31.058 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:31.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:31.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:31.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[29816]: 6.e scrub starts 2024-02-10T07:38:31.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[29816]: 6.e scrub ok 2024-02-10T07:38:31.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[29816]: 5.a scrub starts 2024-02-10T07:38:31.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[29816]: 5.a scrub ok 2024-02-10T07:38:31.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[29816]: 5.1d scrub starts 2024-02-10T07:38:31.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[29816]: 5.1d scrub ok 2024-02-10T07:38:31.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[29816]: pgmap v307: 148 pgs: 148 active+clean; 239 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 30 MiB/s wr, 477 op/s 2024-02-10T07:38:31.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:38:31.116 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:38:31.117 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[29816]: 3.18 scrub starts 2024-02-10T07:38:31.117 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[29816]: 3.18 scrub ok 2024-02-10T07:38:31.117 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[40743]: 6.e scrub starts 2024-02-10T07:38:31.117 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[40743]: 6.e scrub ok 2024-02-10T07:38:31.117 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[40743]: 5.a scrub starts 2024-02-10T07:38:31.117 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[40743]: 5.a scrub ok 2024-02-10T07:38:31.117 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[40743]: 5.1d scrub starts 2024-02-10T07:38:31.117 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[40743]: 5.1d scrub ok 2024-02-10T07:38:31.117 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[40743]: pgmap v307: 148 pgs: 148 active+clean; 239 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 30 MiB/s wr, 477 op/s 2024-02-10T07:38:31.117 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:38:31.117 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:38:31.117 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[40743]: 3.18 scrub starts 2024-02-10T07:38:31.118 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:30 smithi008 ceph-mon[40743]: 3.18 scrub ok 2024-02-10T07:38:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:30 smithi170 ceph-mon[33394]: 6.e scrub starts 2024-02-10T07:38:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:30 smithi170 ceph-mon[33394]: 6.e scrub ok 2024-02-10T07:38:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:30 smithi170 ceph-mon[33394]: 5.a scrub starts 2024-02-10T07:38:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:30 smithi170 ceph-mon[33394]: 5.a scrub ok 2024-02-10T07:38:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:30 smithi170 ceph-mon[33394]: 5.1d scrub starts 2024-02-10T07:38:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:30 smithi170 ceph-mon[33394]: 5.1d scrub ok 2024-02-10T07:38:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:30 smithi170 ceph-mon[33394]: pgmap v307: 148 pgs: 148 active+clean; 239 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 30 MiB/s wr, 477 op/s 2024-02-10T07:38:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:30 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:38:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:30 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:38:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:30 smithi170 ceph-mon[33394]: 3.18 scrub starts 2024-02-10T07:38:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:30 smithi170 ceph-mon[33394]: 3.18 scrub ok 2024-02-10T07:38:31.160 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:31.193 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:31.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:31.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:31.246 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:38:31.295 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:31.312 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:38:31.327 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:31.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:31.352 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:38:31.394 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:38:31.429 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:31.441 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:38:31.463 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:31.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:31.475 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:38:31.495 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:38:31.515 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:38:31.518 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:38:31.519 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:38:31.519 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:38:31.519 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:38:31.519 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:38:31.519 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:38:31.519 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:38:31.519 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:38:31.565 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:31.588 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:31.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:31.630 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 5 16 14539 14523 181.462 139.688 0.00381745 0.00550226 2024-02-10T07:38:31.691 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:31.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:31.729 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:31.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:31.830 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:31.863 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:31.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:31.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:31.965 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:32.004 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:32.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:32.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:32.106 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:31 smithi170 ceph-mon[33394]: 4.b scrub starts 2024-02-10T07:38:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:31 smithi170 ceph-mon[33394]: 4.b scrub ok 2024-02-10T07:38:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:31 smithi170 ceph-mon[33394]: osdmap e208: 8 total, 8 up, 8 in 2024-02-10T07:38:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:31 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3025550650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:38:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:31 smithi170 ceph-mon[33394]: 5.10 scrub starts 2024-02-10T07:38:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:31 smithi170 ceph-mon[33394]: 5.10 scrub ok 2024-02-10T07:38:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:31 smithi170 ceph-mon[33394]: 5.3 scrub starts 2024-02-10T07:38:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:31 smithi170 ceph-mon[33394]: 5.3 scrub ok 2024-02-10T07:38:32.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:31 smithi170 ceph-mon[33394]: osdmap e209: 8 total, 8 up, 8 in 2024-02-10T07:38:32.141 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:32.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:32.242 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:32.277 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:32.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:32.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:32.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[29816]: 4.b scrub starts 2024-02-10T07:38:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[29816]: 4.b scrub ok 2024-02-10T07:38:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[29816]: osdmap e208: 8 total, 8 up, 8 in 2024-02-10T07:38:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3025550650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:38:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[29816]: 5.10 scrub starts 2024-02-10T07:38:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[29816]: 5.10 scrub ok 2024-02-10T07:38:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[29816]: 5.3 scrub starts 2024-02-10T07:38:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[29816]: 5.3 scrub ok 2024-02-10T07:38:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[29816]: osdmap e209: 8 total, 8 up, 8 in 2024-02-10T07:38:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[40743]: 4.b scrub starts 2024-02-10T07:38:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[40743]: 4.b scrub ok 2024-02-10T07:38:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[40743]: osdmap e208: 8 total, 8 up, 8 in 2024-02-10T07:38:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3025550650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:38:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[40743]: 5.10 scrub starts 2024-02-10T07:38:32.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[40743]: 5.10 scrub ok 2024-02-10T07:38:32.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[40743]: 5.3 scrub starts 2024-02-10T07:38:32.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[40743]: 5.3 scrub ok 2024-02-10T07:38:32.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:31 smithi008 ceph-mon[40743]: osdmap e209: 8 total, 8 up, 8 in 2024-02-10T07:38:32.379 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:32.410 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:32.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:32.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:32.512 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:32.553 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:32.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:32.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:32.630 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 6 16 17096 17080 177.852 159.812 0.0041285 0.00561589 2024-02-10T07:38:32.655 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:32.698 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:32.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:32.799 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:32.843 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:32.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:32.920 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:32.945 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:33.007 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:33.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:33.062 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[29816]: 5.7 scrub starts 2024-02-10T07:38:33.062 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[29816]: 5.7 scrub ok 2024-02-10T07:38:33.062 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[29816]: 6.14 scrub starts 2024-02-10T07:38:33.062 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[29816]: 6.14 scrub ok 2024-02-10T07:38:33.062 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[29816]: pgmap v310: 148 pgs: 148 active+clean; 408 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-10T07:38:33.062 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:33.062 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:33.062 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[29816]: 6.18 scrub starts 2024-02-10T07:38:33.062 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[29816]: 6.18 scrub ok 2024-02-10T07:38:33.062 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T07:38:33.063 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[40743]: 5.7 scrub starts 2024-02-10T07:38:33.063 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[40743]: 5.7 scrub ok 2024-02-10T07:38:33.063 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[40743]: 6.14 scrub starts 2024-02-10T07:38:33.063 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[40743]: 6.14 scrub ok 2024-02-10T07:38:33.063 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[40743]: pgmap v310: 148 pgs: 148 active+clean; 408 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-10T07:38:33.063 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:33.063 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:33.063 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[40743]: 6.18 scrub starts 2024-02-10T07:38:33.063 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[40743]: 6.18 scrub ok 2024-02-10T07:38:33.063 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T07:38:33.108 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:33.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:38:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:38:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:32 smithi170 ceph-mon[33394]: 5.7 scrub starts 2024-02-10T07:38:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:32 smithi170 ceph-mon[33394]: 5.7 scrub ok 2024-02-10T07:38:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:32 smithi170 ceph-mon[33394]: 6.14 scrub starts 2024-02-10T07:38:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:32 smithi170 ceph-mon[33394]: 6.14 scrub ok 2024-02-10T07:38:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:32 smithi170 ceph-mon[33394]: pgmap v310: 148 pgs: 148 active+clean; 408 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-10T07:38:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:32 smithi170 ceph-mon[33394]: 6.18 scrub starts 2024-02-10T07:38:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:32 smithi170 ceph-mon[33394]: 6.18 scrub ok 2024-02-10T07:38:33.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-02-10T07:38:33.142 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:33.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:33.245 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:33.288 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:33.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:33.390 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:33.431 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:33.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:33.532 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:33.572 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:33.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:33.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:33.630 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 7 16 19383 19367 172.863 142.938 0.00486424 0.0057787 2024-02-10T07:38:33.674 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:33.714 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:33.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:33.790 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:38:33.816 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:33.860 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:33.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:33.962 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:33.996 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:33.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[29816]: 4.5 scrub starts 2024-02-10T07:38:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[29816]: 4.5 scrub ok 2024-02-10T07:38:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1049086624' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:38:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:38:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:38:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:38:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-02-10T07:38:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:38:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[29816]: osdmap e210: 8 total, 8 up, 8 in 2024-02-10T07:38:34.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[40743]: 4.5 scrub starts 2024-02-10T07:38:34.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[40743]: 4.5 scrub ok 2024-02-10T07:38:34.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:34.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1049086624' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:38:34.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:38:34.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:38:34.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:38:34.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-02-10T07:38:34.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:38:34.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:33 smithi008 ceph-mon[40743]: osdmap e210: 8 total, 8 up, 8 in 2024-02-10T07:38:34.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:34.131 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:34.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:34.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:34.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:33 smithi170 ceph-mon[33394]: 4.5 scrub starts 2024-02-10T07:38:34.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:33 smithi170 ceph-mon[33394]: 4.5 scrub ok 2024-02-10T07:38:34.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:33 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:34.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:33 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1049086624' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:38:34.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:33 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:38:34.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:33 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:38:34.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:33 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:38:34.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-02-10T07:38:34.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:33 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:38:34.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:33 smithi170 ceph-mon[33394]: osdmap e210: 8 total, 8 up, 8 in 2024-02-10T07:38:34.233 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:34.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:34.258 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:34.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:34.358 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:34.391 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:34.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:34.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:34.495 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:34.529 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:34.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:34.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:34.630 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 8 16 21716 21700 169.481 145.812 0.00837582 0.0058939 2024-02-10T07:38:34.631 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:34.669 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:34.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:34.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:34.757 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:34.771 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:34.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:34.828 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:34.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:34.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:34.970 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:34.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:34.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:35.073 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:35.108 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:35.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:35.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:35.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:34 smithi170 ceph-mon[33394]: 6.7 scrub starts 2024-02-10T07:38:35.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:34 smithi170 ceph-mon[33394]: 6.7 scrub ok 2024-02-10T07:38:35.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:34 smithi170 ceph-mon[33394]: 5.c scrub starts 2024-02-10T07:38:35.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:34 smithi170 ceph-mon[33394]: 5.c scrub ok 2024-02-10T07:38:35.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:34 smithi170 ceph-mon[33394]: pgmap v312: 148 pgs: 148 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 175 MiB/s wr, 2.80k op/s 2024-02-10T07:38:35.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:34 smithi170 ceph-mon[33394]: osdmap e211: 8 total, 8 up, 8 in 2024-02-10T07:38:35.211 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:35.229 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:34 smithi008 ceph-mon[29816]: 6.7 scrub starts 2024-02-10T07:38:35.230 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:34 smithi008 ceph-mon[29816]: 6.7 scrub ok 2024-02-10T07:38:35.230 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:34 smithi008 ceph-mon[29816]: 5.c scrub starts 2024-02-10T07:38:35.230 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:34 smithi008 ceph-mon[29816]: 5.c scrub ok 2024-02-10T07:38:35.230 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:34 smithi008 ceph-mon[29816]: pgmap v312: 148 pgs: 148 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 175 MiB/s wr, 2.80k op/s 2024-02-10T07:38:35.230 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:34 smithi008 ceph-mon[29816]: osdmap e211: 8 total, 8 up, 8 in 2024-02-10T07:38:35.231 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:34 smithi008 ceph-mon[40743]: 6.7 scrub starts 2024-02-10T07:38:35.231 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:34 smithi008 ceph-mon[40743]: 6.7 scrub ok 2024-02-10T07:38:35.231 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:34 smithi008 ceph-mon[40743]: 5.c scrub starts 2024-02-10T07:38:35.231 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:34 smithi008 ceph-mon[40743]: 5.c scrub ok 2024-02-10T07:38:35.231 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:34 smithi008 ceph-mon[40743]: pgmap v312: 148 pgs: 148 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 175 MiB/s wr, 2.80k op/s 2024-02-10T07:38:35.231 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:34 smithi008 ceph-mon[40743]: osdmap e211: 8 total, 8 up, 8 in 2024-02-10T07:38:35.243 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:35.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:35.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:35.345 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:35.396 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:35.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:35.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:35.498 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:35.539 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:35.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:35.630 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 9 16 24089 24073 167.128 148.312 0.0059181 0.00597847 2024-02-10T07:38:35.641 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:35.679 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:35.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:35.781 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:35.813 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:35.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:35.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:35.912 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:38:35.915 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:35.946 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:35.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:36.048 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:36.078 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:36.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:36.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:36.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:35 smithi170 ceph-mon[33394]: 5.19 scrub starts 2024-02-10T07:38:36.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:35 smithi170 ceph-mon[33394]: 5.19 scrub ok 2024-02-10T07:38:36.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:35 smithi170 ceph-mon[33394]: 4.6 scrub starts 2024-02-10T07:38:36.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:35 smithi170 ceph-mon[33394]: 4.6 scrub ok 2024-02-10T07:38:36.181 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:36.189 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:35 smithi008 ceph-mon[29816]: 5.19 scrub starts 2024-02-10T07:38:36.189 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:35 smithi008 ceph-mon[29816]: 5.19 scrub ok 2024-02-10T07:38:36.189 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:35 smithi008 ceph-mon[29816]: 4.6 scrub starts 2024-02-10T07:38:36.190 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:35 smithi008 ceph-mon[29816]: 4.6 scrub ok 2024-02-10T07:38:36.190 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:35 smithi008 ceph-mon[40743]: 5.19 scrub starts 2024-02-10T07:38:36.190 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:35 smithi008 ceph-mon[40743]: 5.19 scrub ok 2024-02-10T07:38:36.190 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:35 smithi008 ceph-mon[40743]: 4.6 scrub starts 2024-02-10T07:38:36.190 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:35 smithi008 ceph-mon[40743]: 4.6 scrub ok 2024-02-10T07:38:36.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:36.217 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:36.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:36.318 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:36.366 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:36.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:36.467 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:36.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:36.512 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:36.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:36.613 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:36.631 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 10 16 26356 26340 164.577 141.688 0.00340544 0.00606886 2024-02-10T07:38:36.654 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:36.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:36.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:36.756 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:36.787 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:36.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:36.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:36.889 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:36.924 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:36.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:37.027 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:37.065 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:37.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:37.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:36 smithi008 ceph-mon[29816]: 3.1c scrub starts 2024-02-10T07:38:37.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:36 smithi008 ceph-mon[29816]: 3.1c scrub ok 2024-02-10T07:38:37.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:36 smithi008 ceph-mon[29816]: 3.e scrub starts 2024-02-10T07:38:37.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:36 smithi008 ceph-mon[29816]: 3.e scrub ok 2024-02-10T07:38:37.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:36 smithi008 ceph-mon[29816]: pgmap v314: 148 pgs: 148 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 173 MiB/s wr, 2.77k op/s 2024-02-10T07:38:37.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:36 smithi008 ceph-mon[29816]: 4.3 scrub starts 2024-02-10T07:38:37.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:36 smithi008 ceph-mon[29816]: 4.3 scrub ok 2024-02-10T07:38:37.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:36 smithi008 ceph-mon[40743]: 3.1c scrub starts 2024-02-10T07:38:37.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:36 smithi008 ceph-mon[40743]: 3.1c scrub ok 2024-02-10T07:38:37.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:36 smithi008 ceph-mon[40743]: 3.e scrub starts 2024-02-10T07:38:37.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:36 smithi008 ceph-mon[40743]: 3.e scrub ok 2024-02-10T07:38:37.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:36 smithi008 ceph-mon[40743]: pgmap v314: 148 pgs: 148 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 173 MiB/s wr, 2.77k op/s 2024-02-10T07:38:37.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:36 smithi008 ceph-mon[40743]: 4.3 scrub starts 2024-02-10T07:38:37.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:36 smithi008 ceph-mon[40743]: 4.3 scrub ok 2024-02-10T07:38:37.166 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:37.205 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:37.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:37.307 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:37.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:37.343 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:37.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:37.372 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:38:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:36 smithi170 ceph-mon[33394]: 3.1c scrub starts 2024-02-10T07:38:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:36 smithi170 ceph-mon[33394]: 3.1c scrub ok 2024-02-10T07:38:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:36 smithi170 ceph-mon[33394]: 3.e scrub starts 2024-02-10T07:38:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:36 smithi170 ceph-mon[33394]: 3.e scrub ok 2024-02-10T07:38:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:36 smithi170 ceph-mon[33394]: pgmap v314: 148 pgs: 148 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 173 MiB/s wr, 2.77k op/s 2024-02-10T07:38:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:36 smithi170 ceph-mon[33394]: 4.3 scrub starts 2024-02-10T07:38:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:36 smithi170 ceph-mon[33394]: 4.3 scrub ok 2024-02-10T07:38:37.444 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:37.467 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:37.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:37.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:37.569 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:37.603 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:37.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:37.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:37.632 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 11 16 28152 28136 159.797 112.25 0.00789512 0.00625193 2024-02-10T07:38:37.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:37.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:37.741 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:37.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:37.842 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:37.867 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:37.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:37.968 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:38.002 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:38.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:38.104 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:38.138 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:38.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:38.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:38.240 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:38.273 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:38.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:38.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:38.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:37 smithi008 ceph-mon[29816]: 5.8 scrub starts 2024-02-10T07:38:38.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:37 smithi008 ceph-mon[29816]: 5.8 scrub ok 2024-02-10T07:38:38.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:37 smithi008 ceph-mon[29816]: 3.4 deep-scrub starts 2024-02-10T07:38:38.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:37 smithi008 ceph-mon[29816]: 3.4 deep-scrub ok 2024-02-10T07:38:38.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:37 smithi008 ceph-mon[29816]: 3.1 deep-scrub starts 2024-02-10T07:38:38.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:37 smithi008 ceph-mon[29816]: 3.1 deep-scrub ok 2024-02-10T07:38:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:37 smithi008 ceph-mon[40743]: 5.8 scrub starts 2024-02-10T07:38:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:37 smithi008 ceph-mon[40743]: 5.8 scrub ok 2024-02-10T07:38:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:37 smithi008 ceph-mon[40743]: 3.4 deep-scrub starts 2024-02-10T07:38:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:37 smithi008 ceph-mon[40743]: 3.4 deep-scrub ok 2024-02-10T07:38:38.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:37 smithi008 ceph-mon[40743]: 3.1 deep-scrub starts 2024-02-10T07:38:38.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:37 smithi008 ceph-mon[40743]: 3.1 deep-scrub ok 2024-02-10T07:38:38.375 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:37 smithi170 ceph-mon[33394]: 5.8 scrub starts 2024-02-10T07:38:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:37 smithi170 ceph-mon[33394]: 5.8 scrub ok 2024-02-10T07:38:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:37 smithi170 ceph-mon[33394]: 3.4 deep-scrub starts 2024-02-10T07:38:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:37 smithi170 ceph-mon[33394]: 3.4 deep-scrub ok 2024-02-10T07:38:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:37 smithi170 ceph-mon[33394]: 3.1 deep-scrub starts 2024-02-10T07:38:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:37 smithi170 ceph-mon[33394]: 3.1 deep-scrub ok 2024-02-10T07:38:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:38.409 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:38.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:38.510 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:38.534 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:38.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:38.633 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 12 16 29607 29591 154.06 90.9375 0.00947365 0.00648534 2024-02-10T07:38:38.636 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:38.672 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:38.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:38.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:38.774 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:38.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:38.809 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:38.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:38.910 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:38.948 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:38.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:39.050 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:39.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:38:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:38:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[29816]: 4.1b scrub starts 2024-02-10T07:38:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[29816]: 4.1b scrub ok 2024-02-10T07:38:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[29816]: 5.b scrub starts 2024-02-10T07:38:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[29816]: 5.b scrub ok 2024-02-10T07:38:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[29816]: 4.4 scrub starts 2024-02-10T07:38:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[29816]: pgmap v315: 148 pgs: 148 active+clean; 1.3 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 187 MiB/s wr, 2.99k op/s; 18 B/s, 0 objects/s recovering 2024-02-10T07:38:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[29816]: 4.4 scrub ok 2024-02-10T07:38:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[29816]: 3.1b scrub starts 2024-02-10T07:38:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[29816]: 3.1b scrub ok 2024-02-10T07:38:39.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[40743]: 4.1b scrub starts 2024-02-10T07:38:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[40743]: 4.1b scrub ok 2024-02-10T07:38:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[40743]: 5.b scrub starts 2024-02-10T07:38:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[40743]: 5.b scrub ok 2024-02-10T07:38:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[40743]: 4.4 scrub starts 2024-02-10T07:38:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[40743]: pgmap v315: 148 pgs: 148 active+clean; 1.3 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 187 MiB/s wr, 2.99k op/s; 18 B/s, 0 objects/s recovering 2024-02-10T07:38:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[40743]: 4.4 scrub ok 2024-02-10T07:38:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[40743]: 3.1b scrub starts 2024-02-10T07:38:39.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:38 smithi008 ceph-mon[40743]: 3.1b scrub ok 2024-02-10T07:38:39.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:39.084 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:39.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:39.185 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:39.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:39.209 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:39.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:39.310 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:39.349 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:39.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:39.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:38 smithi170 ceph-mon[33394]: 4.1b scrub starts 2024-02-10T07:38:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:38 smithi170 ceph-mon[33394]: 4.1b scrub ok 2024-02-10T07:38:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:38 smithi170 ceph-mon[33394]: 5.b scrub starts 2024-02-10T07:38:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:38 smithi170 ceph-mon[33394]: 5.b scrub ok 2024-02-10T07:38:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:38 smithi170 ceph-mon[33394]: 4.4 scrub starts 2024-02-10T07:38:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:38 smithi170 ceph-mon[33394]: pgmap v315: 148 pgs: 148 active+clean; 1.3 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 187 MiB/s wr, 2.99k op/s; 18 B/s, 0 objects/s recovering 2024-02-10T07:38:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:38 smithi170 ceph-mon[33394]: 4.4 scrub ok 2024-02-10T07:38:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:38 smithi170 ceph-mon[33394]: 3.1b scrub starts 2024-02-10T07:38:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:38 smithi170 ceph-mon[33394]: 3.1b scrub ok 2024-02-10T07:38:39.451 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:39.474 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:39.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:39.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:39.576 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:39.609 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:39.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:39.633 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 13 15 31330 31315 150.498 107.75 0.00389248 0.00663885 2024-02-10T07:38:39.710 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:39.751 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:39.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:39.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:39.853 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:39.876 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:39.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:39.978 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:40.019 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:40.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:40.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:40.121 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:40.160 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:40.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:40.262 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:40.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:40.287 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:40.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:40.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:39 smithi008 ceph-mon[29816]: 6.12 deep-scrub starts 2024-02-10T07:38:40.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:39 smithi008 ceph-mon[29816]: 6.12 deep-scrub ok 2024-02-10T07:38:40.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:39 smithi008 ceph-mon[29816]: 6.c deep-scrub starts 2024-02-10T07:38:40.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:39 smithi008 ceph-mon[29816]: 6.c deep-scrub ok 2024-02-10T07:38:40.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:39 smithi008 ceph-mon[29816]: 5.1e scrub starts 2024-02-10T07:38:40.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:39 smithi008 ceph-mon[29816]: 5.1e scrub ok 2024-02-10T07:38:40.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:39 smithi008 ceph-mon[40743]: 6.12 deep-scrub starts 2024-02-10T07:38:40.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:39 smithi008 ceph-mon[40743]: 6.12 deep-scrub ok 2024-02-10T07:38:40.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:39 smithi008 ceph-mon[40743]: 6.c deep-scrub starts 2024-02-10T07:38:40.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:39 smithi008 ceph-mon[40743]: 6.c deep-scrub ok 2024-02-10T07:38:40.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:39 smithi008 ceph-mon[40743]: 5.1e scrub starts 2024-02-10T07:38:40.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:39 smithi008 ceph-mon[40743]: 5.1e scrub ok 2024-02-10T07:38:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:39 smithi170 ceph-mon[33394]: 6.12 deep-scrub starts 2024-02-10T07:38:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:39 smithi170 ceph-mon[33394]: 6.12 deep-scrub ok 2024-02-10T07:38:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:39 smithi170 ceph-mon[33394]: 6.c deep-scrub starts 2024-02-10T07:38:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:39 smithi170 ceph-mon[33394]: 6.c deep-scrub ok 2024-02-10T07:38:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:39 smithi170 ceph-mon[33394]: 5.1e scrub starts 2024-02-10T07:38:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:39 smithi170 ceph-mon[33394]: 5.1e scrub ok 2024-02-10T07:38:40.388 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:40.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:40.428 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:40.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:40.529 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:40.568 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:40.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:40.633 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 14 16 32800 32784 146.306 91.8125 0.00361935 0.00681491 2024-02-10T07:38:40.670 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:40.694 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:40.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:40.796 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:40.819 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:40.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:40.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:40.921 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:40.944 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:40.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:41.046 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:41.069 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:41.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:41.171 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:41.203 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:41.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:41.232 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:40 smithi008 ceph-mon[29816]: 3.5 scrub starts 2024-02-10T07:38:41.233 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:40 smithi008 ceph-mon[29816]: 3.5 scrub ok 2024-02-10T07:38:41.233 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:40 smithi008 ceph-mon[29816]: 6.3 scrub starts 2024-02-10T07:38:41.233 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:40 smithi008 ceph-mon[29816]: 6.3 scrub ok 2024-02-10T07:38:41.233 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:40 smithi008 ceph-mon[29816]: pgmap v316: 148 pgs: 148 active+clean; 1.7 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 172 MiB/s wr, 2.75k op/s; 13 B/s, 0 objects/s recovering 2024-02-10T07:38:41.233 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:40 smithi008 ceph-mon[29816]: 5.11 scrub starts 2024-02-10T07:38:41.233 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:40 smithi008 ceph-mon[29816]: 5.11 scrub ok 2024-02-10T07:38:41.233 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:40 smithi008 ceph-mon[40743]: 3.5 scrub starts 2024-02-10T07:38:41.233 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:40 smithi008 ceph-mon[40743]: 3.5 scrub ok 2024-02-10T07:38:41.233 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:40 smithi008 ceph-mon[40743]: 6.3 scrub starts 2024-02-10T07:38:41.234 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:40 smithi008 ceph-mon[40743]: 6.3 scrub ok 2024-02-10T07:38:41.234 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:40 smithi008 ceph-mon[40743]: pgmap v316: 148 pgs: 148 active+clean; 1.7 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 172 MiB/s wr, 2.75k op/s; 13 B/s, 0 objects/s recovering 2024-02-10T07:38:41.234 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:40 smithi008 ceph-mon[40743]: 5.11 scrub starts 2024-02-10T07:38:41.234 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:40 smithi008 ceph-mon[40743]: 5.11 scrub ok 2024-02-10T07:38:41.304 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:41.336 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:41.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:41.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:40 smithi170 ceph-mon[33394]: 3.5 scrub starts 2024-02-10T07:38:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:40 smithi170 ceph-mon[33394]: 3.5 scrub ok 2024-02-10T07:38:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:40 smithi170 ceph-mon[33394]: 6.3 scrub starts 2024-02-10T07:38:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:40 smithi170 ceph-mon[33394]: 6.3 scrub ok 2024-02-10T07:38:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:40 smithi170 ceph-mon[33394]: pgmap v316: 148 pgs: 148 active+clean; 1.7 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 172 MiB/s wr, 2.75k op/s; 13 B/s, 0 objects/s recovering 2024-02-10T07:38:41.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:40 smithi170 ceph-mon[33394]: 5.11 scrub starts 2024-02-10T07:38:41.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:40 smithi170 ceph-mon[33394]: 5.11 scrub ok 2024-02-10T07:38:41.439 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:41.465 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:41.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:41.567 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:41.613 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:41.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:41.633 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 15 16 33755 33739 140.533 59.6875 0.00927177 0.00711105 2024-02-10T07:38:41.715 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:41.756 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:41.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:41.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:41.858 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:41.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:41.909 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:41.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:42.010 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:42.059 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:42.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:42.161 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:42.194 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:42.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:42.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:42.274 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:41 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:38:42.274 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:41 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:38:42.296 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:42.332 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:42.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:41 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:38:42.434 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:42.473 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:42.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:42.574 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:42.611 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:42.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:42.633 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 16 16 35044 35028 136.785 80.5625 0.0031817 0.007295 2024-02-10T07:38:42.712 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:42.751 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:42.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:42.852 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:42.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:42.915 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:42.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:43.016 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:43.045 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:43.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:43.147 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:43.180 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:43.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:43.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:43.282 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:43.305 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:43.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:43.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:43.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:42 smithi008 ceph-mon[29816]: 5.4 scrub starts 2024-02-10T07:38:43.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:43 smithi008 ceph-mon[29816]: 5.4 scrub ok 2024-02-10T07:38:43.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:43 smithi008 ceph-mon[29816]: pgmap v317: 148 pgs: 148 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 636 B/s rd, 95 MiB/s wr, 1.53k op/s; 13 B/s, 0 objects/s recovering 2024-02-10T07:38:43.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:43 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:43.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:43 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:43.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:43 smithi008 ceph-mon[29816]: from='client.25028 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:38:43.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:42 smithi008 ceph-mon[40743]: 5.4 scrub starts 2024-02-10T07:38:43.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:43 smithi008 ceph-mon[40743]: 5.4 scrub ok 2024-02-10T07:38:43.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:43 smithi008 ceph-mon[40743]: pgmap v317: 148 pgs: 148 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 636 B/s rd, 95 MiB/s wr, 1.53k op/s; 13 B/s, 0 objects/s recovering 2024-02-10T07:38:43.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:43 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:43.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:43 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:43.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:43 smithi008 ceph-mon[40743]: from='client.25028 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:38:43.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:38:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:38:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:42 smithi170 ceph-mon[33394]: 5.4 scrub starts 2024-02-10T07:38:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:42 smithi170 ceph-mon[33394]: 5.4 scrub ok 2024-02-10T07:38:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:42 smithi170 ceph-mon[33394]: pgmap v317: 148 pgs: 148 active+clean; 1.8 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 636 B/s rd, 95 MiB/s wr, 1.53k op/s; 13 B/s, 0 objects/s recovering 2024-02-10T07:38:43.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:42 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:43.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:42 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:43.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:42 smithi170 ceph-mon[33394]: from='client.25028 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:38:43.407 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:43.446 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:43.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:43.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:43.548 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:43.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:43.574 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:43.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:43.633 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 17 15 36381 36366 133.658 83.625 0.00665326 0.00747687 2024-02-10T07:38:43.675 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:43.708 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:43.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:43.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:43.717 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:38:43.724 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:38:43.811 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:43.843 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:43.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:43.945 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:43.987 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:43.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:44.089 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:44.123 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:44.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:44.225 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:44.268 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:44.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[29816]: 3.1a scrub starts 2024-02-10T07:38:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[29816]: 3.1a scrub ok 2024-02-10T07:38:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:38:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:38:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[29816]: 5.2 scrub starts 2024-02-10T07:38:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[29816]: 5.2 scrub ok 2024-02-10T07:38:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[29816]: osdmap e212: 8 total, 8 up, 8 in 2024-02-10T07:38:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:38:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:38:44.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[40743]: 3.1a scrub starts 2024-02-10T07:38:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[40743]: 3.1a scrub ok 2024-02-10T07:38:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:38:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:38:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[40743]: 5.2 scrub starts 2024-02-10T07:38:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[40743]: 5.2 scrub ok 2024-02-10T07:38:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[40743]: osdmap e212: 8 total, 8 up, 8 in 2024-02-10T07:38:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:38:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:38:44.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:44 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:44.370 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:44 smithi170 ceph-mon[33394]: 3.1a scrub starts 2024-02-10T07:38:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:44 smithi170 ceph-mon[33394]: 3.1a scrub ok 2024-02-10T07:38:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:44 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:38:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:44 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:38:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:44 smithi170 ceph-mon[33394]: 5.2 scrub starts 2024-02-10T07:38:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:44 smithi170 ceph-mon[33394]: 5.2 scrub ok 2024-02-10T07:38:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:44 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:44 smithi170 ceph-mon[33394]: osdmap e212: 8 total, 8 up, 8 in 2024-02-10T07:38:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:44 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:38:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:44 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:38:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:44 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:44.401 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:44.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:44.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:44.503 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:44.575 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:44.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:44.596 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:38:44.596 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:38:44.633 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 18 16 38245 38229 132.701 116.438 0.00878412 0.00753021 2024-02-10T07:38:44.676 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:44.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:44.720 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:44.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:44.821 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:44.855 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:44.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:44.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:44.957 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:44.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:44.993 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:44.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:45.094 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:45.119 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-02-10T07:38:41.821949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1956256092,"num_objects":30039,"num_object_clones":0,"num_object_copies":90117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30039,"num_whiteouts":0,"num_read":1330,"num_read_kb":1176,"num_write":30456,"num_write_kb":1912285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":1772204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30495,"ondisk_log_size":30495,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7103744,"kb_used_data":5571132,"kb_used_omap":15,"kb_used_meta":1532528,"kb_avail":742857472,"statfs":{"total":767960285184,"available":760686051328,"internally_reserved":0,"allocated":5704839168,"data_stored":5701799492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15602,"internal_metadata":1569309454},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":805437440,"num_objects":12290,"num_object_clones":0,"num_object_copies":36870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12290,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12290,"num_write_kb":786560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.044363"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":314,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.781379+0000","last_change":"2024-02-10T07:36:35.613197+0000","last_active":"2024-02-10T07:38:34.781379+0000","last_peered":"2024-02-10T07:38:34.781379+0000","last_clean":"2024-02-10T07:38:34.781379+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:38:34.781379+0000","last_undegraded":"2024-02-10T07:38:34.781379+0000","last_fullsized":"2024-02-10T07:38:34.781379+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:35.613074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:36:35.613074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":324,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786216+0000","last_change":"2024-02-10T07:38:34.709543+0000","last_active":"2024-02-10T07:38:34.786216+0000","last_peered":"2024-02-10T07:38:34.786216+0000","last_clean":"2024-02-10T07:38:34.786216+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:38:34.786216+0000","last_undegraded":"2024-02-10T07:38:34.786216+0000","last_fullsized":"2024-02-10T07:38:34.786216+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:34.709422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:34.709422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:41.456368+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":340,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.791423+0000","last_change":"2024-02-10T07:38:29.585901+0000","last_active":"2024-02-10T07:38:34.791423+0000","last_peered":"2024-02-10T07:38:34.791423+0000","last_clean":"2024-02-10T07:38:34.791423+0000","last_became_active":"2024-02-10T07:36:33.760016+0000","last_became_peered":"2024-02-10T07:36:33.760016+0000","last_unstale":"2024-02-10T07:38:34.791423+0000","last_undegraded":"2024-02-10T07:38:34.791423+0000","last_fullsized":"2024-02-10T07:38:34.791423+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:38:29.585764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:29.585764+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:50.749980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":353,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.797625+0000","last_change":"2024-02-10T07:38:34.797625+0000","last_active":"2024-02-10T07:38:34.797625+0000","last_peered":"2024-02-10T07:38:34.797625+0000","last_clean":"2024-02-10T07:38:34.797625+0000","last_became_active":"2024-02-10T07:38:34.790379+0000","last_became_peered":"2024-02-10T07:38:34.790379+0000","last_unstale":"2024-02-10T07:38:34.797625+0000","last_undegraded":"2024-02-10T07:38:34.797625+0000","last_fullsized":"2024-02-10T07:38:34.797625+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:37:41.734697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:41.734697+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:51.173162+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":313,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.792124+0000","last_change":"2024-02-10T07:38:24.207588+0000","last_active":"2024-02-10T07:38:34.792124+0000","last_peered":"2024-02-10T07:38:34.792124+0000","last_clean":"2024-02-10T07:38:34.792124+0000","last_became_active":"2024-02-10T07:31:57.300056+0000","last_became_peered":"2024-02-10T07:31:57.300056+0000","last_unstale":"2024-02-10T07:38:34.792124+0000","last_undegraded":"2024-02-10T07:38:34.792124+0000","last_fullsized":"2024-02-10T07:38:34.792124+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:24.207501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:24.207501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:40.217247+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":317,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.780303+0000","last_change":"2024-02-10T07:37:29.763145+0000","last_active":"2024-02-10T07:38:34.780303+0000","last_peered":"2024-02-10T07:38:34.780303+0000","last_clean":"2024-02-10T07:38:34.780303+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:38:34.780303+0000","last_undegraded":"2024-02-10T07:38:34.780303+0000","last_fullsized":"2024-02-10T07:38:34.780303+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:29.763023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:29.763023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:37.708169+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":318,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783450+0000","last_change":"2024-02-10T07:35:43.965606+0000","last_active":"2024-02-10T07:38:34.783450+0000","last_peered":"2024-02-10T07:38:34.783450+0000","last_clean":"2024-02-10T07:38:34.783450+0000","last_became_active":"2024-02-10T07:31:51.323331+0000","last_became_peered":"2024-02-10T07:31:51.323331+0000","last_unstale":"2024-02-10T07:38:34.783450+0000","last_undegraded":"2024-02-10T07:38:34.783450+0000","last_fullsized":"2024-02-10T07:38:34.783450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:35:43.965338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:35:43.965338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":346,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783983+0000","last_change":"2024-02-10T07:37:32.922146+0000","last_active":"2024-02-10T07:38:34.783983+0000","last_peered":"2024-02-10T07:38:34.783983+0000","last_clean":"2024-02-10T07:38:34.783983+0000","last_became_active":"2024-02-10T07:31:53.286957+0000","last_became_peered":"2024-02-10T07:31:53.286957+0000","last_unstale":"2024-02-10T07:38:34.783983+0000","last_undegraded":"2024-02-10T07:38:34.783983+0000","last_fullsized":"2024-02-10T07:38:34.783983+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:37:32.921838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:32.921838+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:02.913956+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":312,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783974+0000","last_change":"2024-02-10T07:38:32.551419+0000","last_active":"2024-02-10T07:38:34.783974+0000","last_peered":"2024-02-10T07:38:34.783974+0000","last_clean":"2024-02-10T07:38:34.783974+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:38:34.783974+0000","last_undegraded":"2024-02-10T07:38:34.783974+0000","last_fullsized":"2024-02-10T07:38:34.783974+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:32.551293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:32.551293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:51.987665+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":277,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.784763+0000","last_change":"2024-02-10T07:37:23.021878+0000","last_active":"2024-02-10T07:38:34.784763+0000","last_peered":"2024-02-10T07:38:34.784763+0000","last_clean":"2024-02-10T07:38:34.784763+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:38:34.784763+0000","last_undegraded":"2024-02-10T07:38:34.784763+0000","last_fullsized":"2024-02-10T07:38:34.784763+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:23.021803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:23.021803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":266,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785016+0000","last_change":"2024-02-10T07:38:24.115694+0000","last_active":"2024-02-10T07:38:34.785016+0000","last_peered":"2024-02-10T07:38:34.785016+0000","last_clean":"2024-02-10T07:38:34.785016+0000","last_became_active":"2024-02-10T07:36:33.771340+0000","last_became_peered":"2024-02-10T07:36:33.771340+0000","last_unstale":"2024-02-10T07:38:34.785016+0000","last_undegraded":"2024-02-10T07:38:34.785016+0000","last_fullsized":"2024-02-10T07:38:34.785016+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:38:24.115550+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:34:49.349360+0000","last_clean_scrub_stamp":"2024-02-10T07:38:24.115550+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:50.858815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":337,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.780976+0000","last_change":"2024-02-10T07:37:49.609721+0000","last_active":"2024-02-10T07:38:34.780976+0000","last_peered":"2024-02-10T07:38:34.780976+0000","last_clean":"2024-02-10T07:38:34.780976+0000","last_became_active":"2024-02-10T07:31:53.287858+0000","last_became_peered":"2024-02-10T07:31:53.287858+0000","last_unstale":"2024-02-10T07:38:34.780976+0000","last_undegraded":"2024-02-10T07:38:34.780976+0000","last_fullsized":"2024-02-10T07:38:34.780976+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:37:49.609631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:49.609631+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:01.288487+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":277,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.784633+0000","last_change":"2024-02-10T07:37:21.968926+0000","last_active":"2024-02-10T07:38:34.784633+0000","last_peered":"2024-02-10T07:38:34.784633+0000","last_clean":"2024-02-10T07:38:34.784633+0000","last_became_active":"2024-02-10T07:36:33.757073+0000","last_became_peered":"2024-02-10T07:36:33.757073+0000","last_unstale":"2024-02-10T07:38:34.784633+0000","last_undegraded":"2024-02-10T07:38:34.784633+0000","last_fullsized":"2024-02-10T07:38:34.784633+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:21.968825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:21.968825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":317,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786566+0000","last_change":"2024-02-10T07:38:28.455565+0000","last_active":"2024-02-10T07:38:34.786566+0000","last_peered":"2024-02-10T07:38:34.786566+0000","last_clean":"2024-02-10T07:38:34.786566+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:38:34.786566+0000","last_undegraded":"2024-02-10T07:38:34.786566+0000","last_fullsized":"2024-02-10T07:38:34.786566+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:28.455461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:28.455461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:37.658639+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":289,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:35.189800+0000","last_change":"2024-02-10T07:38:35.189838+0000","last_active":"2024-02-10T07:38:35.189800+0000","last_peered":"2024-02-10T07:38:35.189800+0000","last_clean":"2024-02-10T07:38:35.189800+0000","last_became_active":"2024-02-10T07:36:33.767970+0000","last_became_peered":"2024-02-10T07:36:33.767970+0000","last_unstale":"2024-02-10T07:38:35.189800+0000","last_undegraded":"2024-02-10T07:38:35.189800+0000","last_fullsized":"2024-02-10T07:38:35.189800+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:35.189720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:35.189720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:01.666908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":331,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:37.597841+0000","last_change":"2024-02-10T07:38:37.597887+0000","last_active":"2024-02-10T07:38:37.597841+0000","last_peered":"2024-02-10T07:38:37.597841+0000","last_clean":"2024-02-10T07:38:37.597841+0000","last_became_active":"2024-02-10T07:31:53.292417+0000","last_became_peered":"2024-02-10T07:31:53.292417+0000","last_unstale":"2024-02-10T07:38:37.597841+0000","last_undegraded":"2024-02-10T07:38:37.597841+0000","last_fullsized":"2024-02-10T07:38:37.597841+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-10T07:38:37.597762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:37.597762+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:59.682048+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":323,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.781419+0000","last_change":"2024-02-10T07:37:28.737774+0000","last_active":"2024-02-10T07:38:34.781419+0000","last_peered":"2024-02-10T07:38:34.781419+0000","last_clean":"2024-02-10T07:38:34.781419+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:38:34.781419+0000","last_undegraded":"2024-02-10T07:38:34.781419+0000","last_fullsized":"2024-02-10T07:38:34.781419+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:28.737622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:28.737622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:44.951148+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":315,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.784567+0000","last_change":"2024-02-10T07:38:29.698707+0000","last_active":"2024-02-10T07:38:34.784567+0000","last_peered":"2024-02-10T07:38:34.784567+0000","last_clean":"2024-02-10T07:38:34.784567+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:38:34.784567+0000","last_undegraded":"2024-02-10T07:38:34.784567+0000","last_fullsized":"2024-02-10T07:38:34.784567+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:29.698581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:29.698581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:47.178090+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":316,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.784647+0000","last_change":"2024-02-10T07:35:46.640362+0000","last_active":"2024-02-10T07:38:34.784647+0000","last_peered":"2024-02-10T07:38:34.784647+0000","last_clean":"2024-02-10T07:38:34.784647+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:38:34.784647+0000","last_undegraded":"2024-02-10T07:38:34.784647+0000","last_fullsized":"2024-02-10T07:38:34.784647+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:35:46.640232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:35:46.640232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"89'22","reported_seq":352,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.789530+0000","last_change":"2024-02-10T07:38:24.669466+0000","last_active":"2024-02-10T07:38:34.789530+0000","last_peered":"2024-02-10T07:38:34.789530+0000","last_clean":"2024-02-10T07:38:34.789530+0000","last_became_active":"2024-02-10T07:31:53.295527+0000","last_became_peered":"2024-02-10T07:31:53.295527+0000","last_unstale":"2024-02-10T07:38:34.789530+0000","last_undegraded":"2024-02-10T07:38:34.789530+0000","last_fullsized":"2024-02-10T07:38:34.789530+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2024-02-10T07:38:24.669327+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:38:24.669327+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:54.083623+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":315,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785283+0000","last_change":"2024-02-10T07:38:22.233892+0000","last_active":"2024-02-10T07:38:34.785283+0000","last_peered":"2024-02-10T07:38:34.785283+0000","last_clean":"2024-02-10T07:38:34.785283+0000","last_became_active":"2024-02-10T07:36:33.754341+0000","last_became_peered":"2024-02-10T07:36:33.754341+0000","last_unstale":"2024-02-10T07:38:34.785283+0000","last_undegraded":"2024-02-10T07:38:34.785283+0000","last_fullsized":"2024-02-10T07:38:34.785283+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:22.233765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:22.233765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:34.922632+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":317,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.781305+0000","last_change":"2024-02-10T07:37:33.771341+0000","last_active":"2024-02-10T07:38:34.781305+0000","last_peered":"2024-02-10T07:38:34.781305+0000","last_clean":"2024-02-10T07:38:34.781305+0000","last_became_active":"2024-02-10T07:31:55.298474+0000","last_became_peered":"2024-02-10T07:31:55.298474+0000","last_unstale":"2024-02-10T07:38:34.781305+0000","last_undegraded":"2024-02-10T07:38:34.781305+0000","last_fullsized":"2024-02-10T07:38:34.781305+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:33.771236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:27.368799+0000","last_clean_scrub_stamp":"2024-02-10T07:37:33.771236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:50.051502+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":325,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.781070+0000","last_change":"2024-02-10T07:36:32.597715+0000","last_active":"2024-02-10T07:38:34.781070+0000","last_peered":"2024-02-10T07:38:34.781070+0000","last_clean":"2024-02-10T07:38:34.781070+0000","last_became_active":"2024-02-10T07:31:51.321237+0000","last_became_peered":"2024-02-10T07:31:51.321237+0000","last_unstale":"2024-02-10T07:38:34.781070+0000","last_undegraded":"2024-02-10T07:38:34.781070+0000","last_fullsized":"2024-02-10T07:38:34.781070+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:32.597591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:36:32.597591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":336,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783227+0000","last_change":"2024-02-10T07:35:34.823951+0000","last_active":"2024-02-10T07:38:34.783227+0000","last_peered":"2024-02-10T07:38:34.783227+0000","last_clean":"2024-02-10T07:38:34.783227+0000","last_became_active":"2024-02-10T07:31:53.294914+0000","last_became_peered":"2024-02-10T07:31:53.294914+0000","last_unstale":"2024-02-10T07:38:34.783227+0000","last_undegraded":"2024-02-10T07:38:34.783227+0000","last_fullsized":"2024-02-10T07:38:34.783227+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:35:34.823683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:35:34.823683+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":315,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786959+0000","last_change":"2024-02-10T07:36:47.829905+0000","last_active":"2024-02-10T07:38:34.786959+0000","last_peered":"2024-02-10T07:38:34.786959+0000","last_clean":"2024-02-10T07:38:34.786959+0000","last_became_active":"2024-02-10T07:36:33.751352+0000","last_became_peered":"2024-02-10T07:36:33.751352+0000","last_unstale":"2024-02-10T07:38:34.786959+0000","last_undegraded":"2024-02-10T07:38:34.786959+0000","last_fullsized":"2024-02-10T07:38:34.786959+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:36:47.829690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:36:47.829690+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":418,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783662+0000","last_change":"2024-02-10T07:37:47.826041+0000","last_active":"2024-02-10T07:38:34.783662+0000","last_peered":"2024-02-10T07:38:34.783662+0000","last_clean":"2024-02-10T07:38:34.783662+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:38:34.783662+0000","last_undegraded":"2024-02-10T07:38:34.783662+0000","last_fullsized":"2024-02-10T07:38:34.783662+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:37:47.825964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:47.825964+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:07.048168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":324,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786297+0000","last_change":"2024-02-10T07:38:23.669717+0000","last_active":"2024-02-10T07:38:34.786297+0000","last_peered":"2024-02-10T07:38:34.786297+0000","last_clean":"2024-02-10T07:38:34.786297+0000","last_became_active":"2024-02-10T07:31:51.282077+0000","last_became_peered":"2024-02-10T07:31:51.282077+0000","last_unstale":"2024-02-10T07:38:34.786297+0000","last_undegraded":"2024-02-10T07:38:34.786297+0000","last_fullsized":"2024-02-10T07:38:34.786297+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:38:23.669587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:23.669587+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:24.629043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":336,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786579+0000","last_change":"2024-02-10T07:37:42.514388+0000","last_active":"2024-02-10T07:38:34.786579+0000","last_peered":"2024-02-10T07:38:34.786579+0000","last_clean":"2024-02-10T07:38:34.786579+0000","last_became_active":"2024-02-10T07:31:53.292936+0000","last_became_peered":"2024-02-10T07:31:53.292936+0000","last_unstale":"2024-02-10T07:38:34.786579+0000","last_undegraded":"2024-02-10T07:38:34.786579+0000","last_fullsized":"2024-02-10T07:38:34.786579+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:37:42.512560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:42.512560+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:48.083349+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":320,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.784712+0000","last_change":"2024-02-10T07:38:26.689435+0000","last_active":"2024-02-10T07:38:34.784712+0000","last_peered":"2024-02-10T07:38:34.784712+0000","last_clean":"2024-02-10T07:38:34.784712+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:38:34.784712+0000","last_undegraded":"2024-02-10T07:38:34.784712+0000","last_fullsized":"2024-02-10T07:38:34.784712+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:26.688895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:26.688895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:48.373320+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":309,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786818+0000","last_change":"2024-02-10T07:36:25.205049+0000","last_active":"2024-02-10T07:38:34.786818+0000","last_peered":"2024-02-10T07:38:34.786818+0000","last_clean":"2024-02-10T07:38:34.786818+0000","last_became_active":"2024-02-10T07:31:55.297595+0000","last_became_peered":"2024-02-10T07:31:55.297595+0000","last_unstale":"2024-02-10T07:38:34.786818+0000","last_undegraded":"2024-02-10T07:38:34.786818+0000","last_fullsized":"2024-02-10T07:38:34.786818+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:25.204859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:36:25.204859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":325,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783212+0000","last_change":"2024-02-10T07:38:30.226232+0000","last_active":"2024-02-10T07:38:34.783212+0000","last_peered":"2024-02-10T07:38:34.783212+0000","last_clean":"2024-02-10T07:38:34.783212+0000","last_became_active":"2024-02-10T07:31:51.285405+0000","last_became_peered":"2024-02-10T07:31:51.285405+0000","last_unstale":"2024-02-10T07:38:34.783212+0000","last_undegraded":"2024-02-10T07:38:34.783212+0000","last_fullsized":"2024-02-10T07:38:34.783212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:30.226109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:30.226109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:53.772743+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":349,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783522+0000","last_change":"2024-02-10T07:37:23.942006+0000","last_active":"2024-02-10T07:38:34.783522+0000","last_peered":"2024-02-10T07:38:34.783522+0000","last_clean":"2024-02-10T07:38:34.783522+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:38:34.783522+0000","last_undegraded":"2024-02-10T07:38:34.783522+0000","last_fullsized":"2024-02-10T07:38:34.783522+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:37:23.941920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:23.941920+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:48.928838+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":305,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786468+0000","last_change":"2024-02-10T07:35:46.544505+0000","last_active":"2024-02-10T07:38:34.786468+0000","last_peered":"2024-02-10T07:38:34.786468+0000","last_clean":"2024-02-10T07:38:34.786468+0000","last_became_active":"2024-02-10T07:31:57.306776+0000","last_became_peered":"2024-02-10T07:31:57.306776+0000","last_unstale":"2024-02-10T07:38:34.786468+0000","last_undegraded":"2024-02-10T07:38:34.786468+0000","last_fullsized":"2024-02-10T07:38:34.786468+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:35:46.544418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:44.251492+0000","last_clean_scrub_stamp":"2024-02-10T07:35:46.544418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":316,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.782871+0000","last_change":"2024-02-10T07:37:35.840572+0000","last_active":"2024-02-10T07:38:34.782871+0000","last_peered":"2024-02-10T07:38:34.782871+0000","last_clean":"2024-02-10T07:38:34.782871+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:38:34.782871+0000","last_undegraded":"2024-02-10T07:38:34.782871+0000","last_fullsized":"2024-02-10T07:38:34.782871+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:35.840498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:35.840498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:37.091945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":333,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.791603+0000","last_change":"2024-02-10T07:37:24.433655+0000","last_active":"2024-02-10T07:38:34.791603+0000","last_peered":"2024-02-10T07:38:34.791603+0000","last_clean":"2024-02-10T07:38:34.791603+0000","last_became_active":"2024-02-10T07:36:33.760213+0000","last_became_peered":"2024-02-10T07:36:33.760213+0000","last_unstale":"2024-02-10T07:38:34.791603+0000","last_undegraded":"2024-02-10T07:38:34.791603+0000","last_fullsized":"2024-02-10T07:38:34.791603+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:24.433491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:24.433491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:48.295437+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"95'6","reported_seq":321,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783302+0000","last_change":"2024-02-10T07:37:22.000087+0000","last_active":"2024-02-10T07:38:34.783302+0000","last_peered":"2024-02-10T07:38:34.783302+0000","last_clean":"2024-02-10T07:38:34.783302+0000","last_became_active":"2024-02-10T07:31:53.295520+0000","last_became_peered":"2024-02-10T07:31:53.295520+0000","last_unstale":"2024-02-10T07:38:34.783302+0000","last_undegraded":"2024-02-10T07:38:34.783302+0000","last_fullsized":"2024-02-10T07:38:34.783302+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-02-10T07:37:21.999818+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:37:21.999818+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:35.647908+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":305,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.792302+0000","last_change":"2024-02-10T07:36:25.155652+0000","last_active":"2024-02-10T07:38:34.792302+0000","last_peered":"2024-02-10T07:38:34.792302+0000","last_clean":"2024-02-10T07:38:34.792302+0000","last_became_active":"2024-02-10T07:31:57.315738+0000","last_became_peered":"2024-02-10T07:31:57.315738+0000","last_unstale":"2024-02-10T07:38:34.792302+0000","last_undegraded":"2024-02-10T07:38:34.792302+0000","last_fullsized":"2024-02-10T07:38:34.792302+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:25.155547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:36:25.155547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":317,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.787564+0000","last_change":"2024-02-10T07:38:31.441903+0000","last_active":"2024-02-10T07:38:34.787564+0000","last_peered":"2024-02-10T07:38:34.787564+0000","last_clean":"2024-02-10T07:38:34.787564+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:38:34.787564+0000","last_undegraded":"2024-02-10T07:38:34.787564+0000","last_fullsized":"2024-02-10T07:38:34.787564+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:31.441795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:31.441795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:52.264706+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":249,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785008+0000","last_change":"2024-02-10T07:37:40.838525+0000","last_active":"2024-02-10T07:38:34.785008+0000","last_peered":"2024-02-10T07:38:34.785008+0000","last_clean":"2024-02-10T07:38:34.785008+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:38:34.785008+0000","last_undegraded":"2024-02-10T07:38:34.785008+0000","last_fullsized":"2024-02-10T07:38:34.785008+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:40.838446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:37:40.838446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:02.539369+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":340,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783889+0000","last_change":"2024-02-10T07:37:47.175823+0000","last_active":"2024-02-10T07:38:34.783889+0000","last_peered":"2024-02-10T07:38:34.783889+0000","last_clean":"2024-02-10T07:38:34.783889+0000","last_became_active":"2024-02-10T07:31:53.287823+0000","last_became_peered":"2024-02-10T07:31:53.287823+0000","last_unstale":"2024-02-10T07:38:34.783889+0000","last_undegraded":"2024-02-10T07:38:34.783889+0000","last_fullsized":"2024-02-10T07:38:34.783889+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:37:47.175681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:47.175681+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:03.100368+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":313,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783703+0000","last_change":"2024-02-10T07:38:25.610309+0000","last_active":"2024-02-10T07:38:34.783703+0000","last_peered":"2024-02-10T07:38:34.783703+0000","last_clean":"2024-02-10T07:38:34.783703+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:38:34.783703+0000","last_undegraded":"2024-02-10T07:38:34.783703+0000","last_fullsized":"2024-02-10T07:38:34.783703+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:25.610132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:25.610132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:46.400503+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":317,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.792031+0000","last_change":"2024-02-10T07:38:25.154667+0000","last_active":"2024-02-10T07:38:34.792031+0000","last_peered":"2024-02-10T07:38:34.792031+0000","last_clean":"2024-02-10T07:38:34.792031+0000","last_became_active":"2024-02-10T07:31:55.296670+0000","last_became_peered":"2024-02-10T07:31:55.296670+0000","last_unstale":"2024-02-10T07:38:34.792031+0000","last_undegraded":"2024-02-10T07:38:34.792031+0000","last_fullsized":"2024-02-10T07:38:34.792031+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:25.154556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:25.154556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:45.636749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":325,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.787058+0000","last_change":"2024-02-10T07:37:24.647379+0000","last_active":"2024-02-10T07:38:34.787058+0000","last_peered":"2024-02-10T07:38:34.787058+0000","last_clean":"2024-02-10T07:38:34.787058+0000","last_became_active":"2024-02-10T07:31:51.288165+0000","last_became_peered":"2024-02-10T07:31:51.288165+0000","last_unstale":"2024-02-10T07:38:34.787058+0000","last_undegraded":"2024-02-10T07:38:34.787058+0000","last_fullsized":"2024-02-10T07:38:34.787058+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:24.647260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:24.647260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":105,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785947+0000","last_change":"2024-02-10T07:38:20.668585+0000","last_active":"2024-02-10T07:38:34.785947+0000","last_peered":"2024-02-10T07:38:34.785947+0000","last_clean":"2024-02-10T07:38:34.785947+0000","last_became_active":"2024-02-10T07:37:06.187551+0000","last_became_peered":"2024-02-10T07:37:06.187551+0000","last_unstale":"2024-02-10T07:38:34.785947+0000","last_undegraded":"2024-02-10T07:38:34.785947+0000","last_fullsized":"2024-02-10T07:38:34.785947+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:38:20.668498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:20.668498+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:21.641880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":314,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.782871+0000","last_change":"2024-02-10T07:37:20.031409+0000","last_active":"2024-02-10T07:38:34.782871+0000","last_peered":"2024-02-10T07:38:34.782871+0000","last_clean":"2024-02-10T07:38:34.782871+0000","last_became_active":"2024-02-10T07:36:33.750030+0000","last_became_peered":"2024-02-10T07:36:33.750030+0000","last_unstale":"2024-02-10T07:38:34.782871+0000","last_undegraded":"2024-02-10T07:38:34.782871+0000","last_fullsized":"2024-02-10T07:38:34.782871+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:20.031295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:20.031295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:47.770521+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":324,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786495+0000","last_change":"2024-02-10T07:38:24.627406+0000","last_active":"2024-02-10T07:38:34.786495+0000","last_peered":"2024-02-10T07:38:34.786495+0000","last_clean":"2024-02-10T07:38:34.786495+0000","last_became_active":"2024-02-10T07:36:33.752418+0000","last_became_peered":"2024-02-10T07:36:33.752418+0000","last_unstale":"2024-02-10T07:38:34.786495+0000","last_undegraded":"2024-02-10T07:38:34.786495+0000","last_fullsized":"2024-02-10T07:38:34.786495+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:24.627321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:24.627321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:46.364817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":325,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.781201+0000","last_change":"2024-02-10T07:37:40.763596+0000","last_active":"2024-02-10T07:38:34.781201+0000","last_peered":"2024-02-10T07:38:34.781201+0000","last_clean":"2024-02-10T07:38:34.781201+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:38:34.781201+0000","last_undegraded":"2024-02-10T07:38:34.781201+0000","last_fullsized":"2024-02-10T07:38:34.781201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:40.763490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:40.763490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:48.249998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"120'16","reported_seq":363,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.781108+0000","last_change":"2024-02-10T07:36:55.866629+0000","last_active":"2024-02-10T07:38:34.781108+0000","last_peered":"2024-02-10T07:38:34.781108+0000","last_clean":"2024-02-10T07:38:34.781108+0000","last_became_active":"2024-02-10T07:32:18.389564+0000","last_became_peered":"2024-02-10T07:32:18.389564+0000","last_unstale":"2024-02-10T07:38:34.781108+0000","last_undegraded":"2024-02-10T07:38:34.781108+0000","last_fullsized":"2024-02-10T07:38:34.781108+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'16","last_scrub_stamp":"2024-02-10T07:36:55.866526+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:36:55.866526+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":312,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786306+0000","last_change":"2024-02-10T07:38:24.598544+0000","last_active":"2024-02-10T07:38:34.786306+0000","last_peered":"2024-02-10T07:38:34.786306+0000","last_clean":"2024-02-10T07:38:34.786306+0000","last_became_active":"2024-02-10T07:31:57.318009+0000","last_became_peered":"2024-02-10T07:31:57.318009+0000","last_unstale":"2024-02-10T07:38:34.786306+0000","last_undegraded":"2024-02-10T07:38:34.786306+0000","last_fullsized":"2024-02-10T07:38:34.786306+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:24.598439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:38:24.598439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:50.791442+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"158'10","reported_seq":325,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:36.446608+0000","last_change":"2024-02-10T07:37:49.768451+0000","last_active":"2024-02-10T07:38:36.446608+0000","last_peered":"2024-02-10T07:38:36.446608+0000","last_clean":"2024-02-10T07:38:36.446608+0000","last_became_active":"2024-02-10T07:36:33.774773+0000","last_became_peered":"2024-02-10T07:36:33.774773+0000","last_unstale":"2024-02-10T07:38:36.446608+0000","last_undegraded":"2024-02-10T07:38:36.446608+0000","last_fullsized":"2024-02-10T07:38:36.446608+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:37:49.768327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:49.768327+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:13.077119+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":325,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786707+0000","last_change":"2024-02-10T07:36:52.846442+0000","last_active":"2024-02-10T07:38:34.786707+0000","last_peered":"2024-02-10T07:38:34.786707+0000","last_clean":"2024-02-10T07:38:34.786707+0000","last_became_active":"2024-02-10T07:31:51.320644+0000","last_became_peered":"2024-02-10T07:31:51.320644+0000","last_unstale":"2024-02-10T07:38:34.786707+0000","last_undegraded":"2024-02-10T07:38:34.786707+0000","last_fullsized":"2024-02-10T07:38:34.786707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:52.846359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:36:52.846359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"96'15","reported_seq":151,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785487+0000","last_change":"2024-02-10T07:37:26.933801+0000","last_active":"2024-02-10T07:38:34.785487+0000","last_peered":"2024-02-10T07:38:34.785487+0000","last_clean":"2024-02-10T07:38:34.785487+0000","last_became_active":"2024-02-10T07:36:33.752317+0000","last_became_peered":"2024-02-10T07:36:33.752317+0000","last_unstale":"2024-02-10T07:38:34.785487+0000","last_undegraded":"2024-02-10T07:38:34.785487+0000","last_fullsized":"2024-02-10T07:38:34.785487+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'15","last_scrub_stamp":"2024-02-10T07:37:26.933710+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:37:26.933710+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:42.442878+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":314,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.781504+0000","last_change":"2024-02-10T07:36:42.822988+0000","last_active":"2024-02-10T07:38:34.781504+0000","last_peered":"2024-02-10T07:38:34.781504+0000","last_clean":"2024-02-10T07:38:34.781504+0000","last_became_active":"2024-02-10T07:36:33.766051+0000","last_became_peered":"2024-02-10T07:36:33.766051+0000","last_unstale":"2024-02-10T07:38:34.781504+0000","last_undegraded":"2024-02-10T07:38:34.781504+0000","last_fullsized":"2024-02-10T07:38:34.781504+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:42.822880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:36:42.822880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":399,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783523+0000","last_change":"2024-02-10T07:36:30.059099+0000","last_active":"2024-02-10T07:38:34.783523+0000","last_peered":"2024-02-10T07:38:34.783523+0000","last_clean":"2024-02-10T07:38:34.783523+0000","last_became_active":"2024-02-10T07:31:55.294594+0000","last_became_peered":"2024-02-10T07:31:55.294594+0000","last_unstale":"2024-02-10T07:38:34.783523+0000","last_undegraded":"2024-02-10T07:38:34.783523+0000","last_fullsized":"2024-02-10T07:38:34.783523+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:36:30.058979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:36:30.058979+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":324,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785713+0000","last_change":"2024-02-10T07:37:43.502899+0000","last_active":"2024-02-10T07:38:34.785713+0000","last_peered":"2024-02-10T07:38:34.785713+0000","last_clean":"2024-02-10T07:38:34.785713+0000","last_became_active":"2024-02-10T07:31:51.282353+0000","last_became_peered":"2024-02-10T07:31:51.282353+0000","last_unstale":"2024-02-10T07:38:34.785713+0000","last_undegraded":"2024-02-10T07:38:34.785713+0000","last_fullsized":"2024-02-10T07:38:34.785713+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:43.502790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:43.502790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:05.395150+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":289,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785508+0000","last_change":"2024-02-10T07:38:25.146074+0000","last_active":"2024-02-10T07:38:34.785508+0000","last_peered":"2024-02-10T07:38:34.785508+0000","last_clean":"2024-02-10T07:38:34.785508+0000","last_became_active":"2024-02-10T07:36:33.769894+0000","last_became_peered":"2024-02-10T07:36:33.769894+0000","last_unstale":"2024-02-10T07:38:34.785508+0000","last_undegraded":"2024-02-10T07:38:34.785508+0000","last_fullsized":"2024-02-10T07:38:34.785508+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:38:25.145962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:25.145962+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:44.000441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":320,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.789812+0000","last_change":"2024-02-10T07:38:31.696484+0000","last_active":"2024-02-10T07:38:34.789812+0000","last_peered":"2024-02-10T07:38:34.789812+0000","last_clean":"2024-02-10T07:38:34.789812+0000","last_became_active":"2024-02-10T07:31:57.308615+0000","last_became_peered":"2024-02-10T07:31:57.308615+0000","last_unstale":"2024-02-10T07:38:34.789812+0000","last_undegraded":"2024-02-10T07:38:34.789812+0000","last_fullsized":"2024-02-10T07:38:34.789812+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:31.696375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:38:31.696375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:45.014841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":309,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785890+0000","last_change":"2024-02-10T07:36:32.034358+0000","last_active":"2024-02-10T07:38:34.785890+0000","last_peered":"2024-02-10T07:38:34.785890+0000","last_clean":"2024-02-10T07:38:34.785890+0000","last_became_active":"2024-02-10T07:31:55.297431+0000","last_became_peered":"2024-02-10T07:31:55.297431+0000","last_unstale":"2024-02-10T07:38:34.785890+0000","last_undegraded":"2024-02-10T07:38:34.785890+0000","last_fullsized":"2024-02-10T07:38:34.785890+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:32.034106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:36:32.034106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":325,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786475+0000","last_change":"2024-02-10T07:36:44.862446+0000","last_active":"2024-02-10T07:38:34.786475+0000","last_peered":"2024-02-10T07:38:34.786475+0000","last_clean":"2024-02-10T07:38:34.786475+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:38:34.786475+0000","last_undegraded":"2024-02-10T07:38:34.786475+0000","last_fullsized":"2024-02-10T07:38:34.786475+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:44.862337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:36:44.862337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"89'13","reported_seq":340,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786669+0000","last_change":"2024-02-10T07:36:52.280659+0000","last_active":"2024-02-10T07:38:34.786669+0000","last_peered":"2024-02-10T07:38:34.786669+0000","last_clean":"2024-02-10T07:38:34.786669+0000","last_became_active":"2024-02-10T07:31:53.295353+0000","last_became_peered":"2024-02-10T07:31:53.295353+0000","last_unstale":"2024-02-10T07:38:34.786669+0000","last_undegraded":"2024-02-10T07:38:34.786669+0000","last_fullsized":"2024-02-10T07:38:34.786669+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-10T07:36:52.280569+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:36:52.280569+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":305,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786918+0000","last_change":"2024-02-10T07:36:54.820672+0000","last_active":"2024-02-10T07:38:34.786918+0000","last_peered":"2024-02-10T07:38:34.786918+0000","last_clean":"2024-02-10T07:38:34.786918+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:38:34.786918+0000","last_undegraded":"2024-02-10T07:38:34.786918+0000","last_fullsized":"2024-02-10T07:38:34.786918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:54.820598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:36:54.820598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":273,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785093+0000","last_change":"2024-02-10T07:37:17.963214+0000","last_active":"2024-02-10T07:38:34.785093+0000","last_peered":"2024-02-10T07:38:34.785093+0000","last_clean":"2024-02-10T07:38:34.785093+0000","last_became_active":"2024-02-10T07:36:33.752051+0000","last_became_peered":"2024-02-10T07:36:33.752051+0000","last_unstale":"2024-02-10T07:38:34.785093+0000","last_undegraded":"2024-02-10T07:38:34.785093+0000","last_fullsized":"2024-02-10T07:38:34.785093+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:17.963100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:17.963100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:40.236814+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":333,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783186+0000","last_change":"2024-02-10T07:37:42.790522+0000","last_active":"2024-02-10T07:38:34.783186+0000","last_peered":"2024-02-10T07:38:34.783186+0000","last_clean":"2024-02-10T07:38:34.783186+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:38:34.783186+0000","last_undegraded":"2024-02-10T07:38:34.783186+0000","last_fullsized":"2024-02-10T07:38:34.783186+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:42.790447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:42.790447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:06.830654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":322,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785713+0000","last_change":"2024-02-10T07:37:30.985038+0000","last_active":"2024-02-10T07:38:34.785713+0000","last_peered":"2024-02-10T07:38:34.785713+0000","last_clean":"2024-02-10T07:38:34.785713+0000","last_became_active":"2024-02-10T07:31:53.296831+0000","last_became_peered":"2024-02-10T07:31:53.296831+0000","last_unstale":"2024-02-10T07:38:34.785713+0000","last_undegraded":"2024-02-10T07:38:34.785713+0000","last_fullsized":"2024-02-10T07:38:34.785713+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-10T07:37:30.984906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:30.984906+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:00.502651+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"211'2175","reported_seq":2207,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:41.737637+0000","last_change":"2024-02-10T07:38:14.779128+0000","last_active":"2024-02-10T07:38:41.737637+0000","last_peered":"2024-02-10T07:38:41.737637+0000","last_clean":"2024-02-10T07:38:41.737637+0000","last_became_active":"2024-02-10T07:38:14.778672+0000","last_became_peered":"2024-02-10T07:38:14.778672+0000","last_unstale":"2024-02-10T07:38:41.737637+0000","last_undegraded":"2024-02-10T07:38:41.737637+0000","last_fullsized":"2024-02-10T07:38:41.737637+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":2175,"log_dups_size":0,"ondisk_log_size":2175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:14.448083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142540800,"num_objects":2175,"num_object_clones":0,"num_object_copies":6525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2175,"num_write_kb":139200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":278,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785187+0000","last_change":"2024-02-10T07:37:18.959136+0000","last_active":"2024-02-10T07:38:34.785187+0000","last_peered":"2024-02-10T07:38:34.785187+0000","last_clean":"2024-02-10T07:38:34.785187+0000","last_became_active":"2024-02-10T07:36:33.763431+0000","last_became_peered":"2024-02-10T07:36:33.763431+0000","last_unstale":"2024-02-10T07:38:34.785187+0000","last_undegraded":"2024-02-10T07:38:34.785187+0000","last_fullsized":"2024-02-10T07:38:34.785187+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:18.959019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:18.959019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":338,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.787305+0000","last_change":"2024-02-10T07:37:49.683118+0000","last_active":"2024-02-10T07:38:34.787305+0000","last_peered":"2024-02-10T07:38:34.787305+0000","last_clean":"2024-02-10T07:38:34.787305+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:38:34.787305+0000","last_undegraded":"2024-02-10T07:38:34.787305+0000","last_fullsized":"2024-02-10T07:38:34.787305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-10T07:37:49.682987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:49.682987+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:02.993173+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":269,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.784682+0000","last_change":"2024-02-10T07:37:29.907162+0000","last_active":"2024-02-10T07:38:34.784682+0000","last_peered":"2024-02-10T07:38:34.784682+0000","last_clean":"2024-02-10T07:38:34.784682+0000","last_became_active":"2024-02-10T07:36:33.754134+0000","last_became_peered":"2024-02-10T07:36:33.754134+0000","last_unstale":"2024-02-10T07:38:34.784682+0000","last_undegraded":"2024-02-10T07:38:34.784682+0000","last_fullsized":"2024-02-10T07:38:34.784682+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:37:29.907084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:29.907084+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:53.307426+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":408,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786626+0000","last_change":"2024-02-10T07:38:27.441013+0000","last_active":"2024-02-10T07:38:34.786626+0000","last_peered":"2024-02-10T07:38:34.786626+0000","last_clean":"2024-02-10T07:38:34.786626+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:38:34.786626+0000","last_undegraded":"2024-02-10T07:38:34.786626+0000","last_fullsized":"2024-02-10T07:38:34.786626+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:38:27.440874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:27.440874+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:29.086690+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"211'1784","reported_seq":1816,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:37.747723+0000","last_change":"2024-02-10T07:38:15.020533+0000","last_active":"2024-02-10T07:38:37.747723+0000","last_peered":"2024-02-10T07:38:37.747723+0000","last_clean":"2024-02-10T07:38:37.747723+0000","last_became_active":"2024-02-10T07:38:15.020222+0000","last_became_peered":"2024-02-10T07:38:15.020222+0000","last_unstale":"2024-02-10T07:38:37.747723+0000","last_undegraded":"2024-02-10T07:38:37.747723+0000","last_fullsized":"2024-02-10T07:38:37.747723+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":1784,"log_dups_size":0,"ondisk_log_size":1784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:15.690629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116916224,"num_objects":1784,"num_object_clones":0,"num_object_copies":5352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1784,"num_write_kb":114176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":305,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.792213+0000","last_change":"2024-02-10T07:37:19.036835+0000","last_active":"2024-02-10T07:38:34.792213+0000","last_peered":"2024-02-10T07:38:34.792213+0000","last_clean":"2024-02-10T07:38:34.792213+0000","last_became_active":"2024-02-10T07:31:57.314872+0000","last_became_peered":"2024-02-10T07:31:57.314872+0000","last_unstale":"2024-02-10T07:38:34.792213+0000","last_undegraded":"2024-02-10T07:38:34.792213+0000","last_fullsized":"2024-02-10T07:38:34.792213+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:19.036588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:19.036588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":326,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:35.497614+0000","last_change":"2024-02-10T07:38:35.497614+0000","last_active":"2024-02-10T07:38:35.497614+0000","last_peered":"2024-02-10T07:38:35.497614+0000","last_clean":"2024-02-10T07:38:35.497614+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:38:35.497614+0000","last_undegraded":"2024-02-10T07:38:35.497614+0000","last_fullsized":"2024-02-10T07:38:35.497614+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:35.497479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:38:35.497479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:03.165678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":363,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.780901+0000","last_change":"2024-02-10T07:37:44.688729+0000","last_active":"2024-02-10T07:38:34.780901+0000","last_peered":"2024-02-10T07:38:34.780901+0000","last_clean":"2024-02-10T07:38:34.780901+0000","last_became_active":"2024-02-10T07:32:18.389409+0000","last_became_peered":"2024-02-10T07:32:18.389409+0000","last_unstale":"2024-02-10T07:38:34.780901+0000","last_undegraded":"2024-02-10T07:38:34.780901+0000","last_fullsized":"2024-02-10T07:38:34.780901+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:37:44.688627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:44.688627+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:51.858934+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":325,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:36.701108+0000","last_change":"2024-02-10T07:38:36.701149+0000","last_active":"2024-02-10T07:38:36.701108+0000","last_peered":"2024-02-10T07:38:36.701108+0000","last_clean":"2024-02-10T07:38:36.701108+0000","last_became_active":"2024-02-10T07:31:55.303059+0000","last_became_peered":"2024-02-10T07:31:55.303059+0000","last_unstale":"2024-02-10T07:38:36.701108+0000","last_undegraded":"2024-02-10T07:38:36.701108+0000","last_fullsized":"2024-02-10T07:38:36.701108+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:36.701019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:36.701019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:05.304703+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"211'1769","reported_seq":1801,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:38.303446+0000","last_change":"2024-02-10T07:38:15.020510+0000","last_active":"2024-02-10T07:38:38.303446+0000","last_peered":"2024-02-10T07:38:38.303446+0000","last_clean":"2024-02-10T07:38:38.303446+0000","last_became_active":"2024-02-10T07:38:15.020166+0000","last_became_peered":"2024-02-10T07:38:15.020166+0000","last_unstale":"2024-02-10T07:38:38.303446+0000","last_undegraded":"2024-02-10T07:38:38.303446+0000","last_fullsized":"2024-02-10T07:38:38.303446+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":1769,"log_dups_size":0,"ondisk_log_size":1769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:26.981914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115933184,"num_objects":1769,"num_object_clones":0,"num_object_copies":5307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1769,"num_write_kb":113216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":314,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.787144+0000","last_change":"2024-02-10T07:36:46.899550+0000","last_active":"2024-02-10T07:38:34.787144+0000","last_peered":"2024-02-10T07:38:34.787144+0000","last_clean":"2024-02-10T07:38:34.787144+0000","last_became_active":"2024-02-10T07:31:57.304971+0000","last_became_peered":"2024-02-10T07:31:57.304971+0000","last_unstale":"2024-02-10T07:38:34.787144+0000","last_undegraded":"2024-02-10T07:38:34.787144+0000","last_fullsized":"2024-02-10T07:38:34.787144+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:46.899444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:36:46.899444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":333,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:37.699505+0000","last_change":"2024-02-10T07:38:37.699541+0000","last_active":"2024-02-10T07:38:37.699505+0000","last_peered":"2024-02-10T07:38:37.699505+0000","last_clean":"2024-02-10T07:38:37.699505+0000","last_became_active":"2024-02-10T07:36:33.754602+0000","last_became_peered":"2024-02-10T07:36:33.754602+0000","last_unstale":"2024-02-10T07:38:37.699505+0000","last_undegraded":"2024-02-10T07:38:37.699505+0000","last_fullsized":"2024-02-10T07:38:37.699505+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:37.699423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:37.699423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:06.556224+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":335,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.787410+0000","last_change":"2024-02-10T07:37:22.620665+0000","last_active":"2024-02-10T07:38:34.787410+0000","last_peered":"2024-02-10T07:38:34.787410+0000","last_clean":"2024-02-10T07:38:34.787410+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:38:34.787410+0000","last_undegraded":"2024-02-10T07:38:34.787410+0000","last_fullsized":"2024-02-10T07:38:34.787410+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:22.620597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:22.620597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:52.465985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":360,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.782806+0000","last_change":"2024-02-10T07:38:25.855922+0000","last_active":"2024-02-10T07:38:34.782806+0000","last_peered":"2024-02-10T07:38:34.782806+0000","last_clean":"2024-02-10T07:38:34.782806+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:38:34.782806+0000","last_undegraded":"2024-02-10T07:38:34.782806+0000","last_fullsized":"2024-02-10T07:38:34.782806+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-10T07:38:25.855813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:25.855813+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:44.243702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"211'1795","reported_seq":1827,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:37.967636+0000","last_change":"2024-02-10T07:38:14.779431+0000","last_active":"2024-02-10T07:38:37.967636+0000","last_peered":"2024-02-10T07:38:37.967636+0000","last_clean":"2024-02-10T07:38:37.967636+0000","last_became_active":"2024-02-10T07:38:14.779127+0000","last_became_peered":"2024-02-10T07:38:14.779127+0000","last_unstale":"2024-02-10T07:38:37.967636+0000","last_undegraded":"2024-02-10T07:38:37.967636+0000","last_fullsized":"2024-02-10T07:38:37.967636+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":1795,"log_dups_size":0,"ondisk_log_size":1795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:24.308291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117637120,"num_objects":1795,"num_object_clones":0,"num_object_copies":5385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1795,"num_write_kb":114880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":313,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.784327+0000","last_change":"2024-02-10T07:37:48.592946+0000","last_active":"2024-02-10T07:38:34.784327+0000","last_peered":"2024-02-10T07:38:34.784327+0000","last_clean":"2024-02-10T07:38:34.784327+0000","last_became_active":"2024-02-10T07:31:57.313394+0000","last_became_peered":"2024-02-10T07:31:57.313394+0000","last_unstale":"2024-02-10T07:38:34.784327+0000","last_undegraded":"2024-02-10T07:38:34.784327+0000","last_fullsized":"2024-02-10T07:38:34.784327+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:48.592835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:48.592835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:55.346477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":317,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.788708+0000","last_change":"2024-02-10T07:38:29.689095+0000","last_active":"2024-02-10T07:38:34.788708+0000","last_peered":"2024-02-10T07:38:34.788708+0000","last_clean":"2024-02-10T07:38:34.788708+0000","last_became_active":"2024-02-10T07:31:55.298622+0000","last_became_peered":"2024-02-10T07:31:55.298622+0000","last_unstale":"2024-02-10T07:38:34.788708+0000","last_undegraded":"2024-02-10T07:38:34.788708+0000","last_fullsized":"2024-02-10T07:38:34.788708+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:29.688966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:29.688966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:35.050177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":281,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785375+0000","last_change":"2024-02-10T07:38:26.195581+0000","last_active":"2024-02-10T07:38:34.785375+0000","last_peered":"2024-02-10T07:38:34.785375+0000","last_clean":"2024-02-10T07:38:34.785375+0000","last_became_active":"2024-02-10T07:36:33.762155+0000","last_became_peered":"2024-02-10T07:36:33.762155+0000","last_unstale":"2024-02-10T07:38:34.785375+0000","last_undegraded":"2024-02-10T07:38:34.785375+0000","last_fullsized":"2024-02-10T07:38:34.785375+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:26.195471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:26.195471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:31.570471+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":336,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.784774+0000","last_change":"2024-02-10T07:38:30.603907+0000","last_active":"2024-02-10T07:38:34.784774+0000","last_peered":"2024-02-10T07:38:34.784774+0000","last_clean":"2024-02-10T07:38:34.784774+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:38:34.784774+0000","last_undegraded":"2024-02-10T07:38:34.784774+0000","last_fullsized":"2024-02-10T07:38:34.784774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:38:30.602734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:30.602734+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:43.285464+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"211'2059","reported_seq":2091,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:41.751235+0000","last_change":"2024-02-10T07:38:15.019596+0000","last_active":"2024-02-10T07:38:41.751235+0000","last_peered":"2024-02-10T07:38:41.751235+0000","last_clean":"2024-02-10T07:38:41.751235+0000","last_became_active":"2024-02-10T07:38:15.019052+0000","last_became_peered":"2024-02-10T07:38:15.019052+0000","last_unstale":"2024-02-10T07:38:41.751235+0000","last_undegraded":"2024-02-10T07:38:41.751235+0000","last_fullsized":"2024-02-10T07:38:41.751235+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":2059,"log_dups_size":0,"ondisk_log_size":2059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:17.302916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134938624,"num_objects":2059,"num_object_clones":0,"num_object_copies":6177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2059,"num_write_kb":131776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":306,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783708+0000","last_change":"2024-02-10T07:37:39.095527+0000","last_active":"2024-02-10T07:38:34.783708+0000","last_peered":"2024-02-10T07:38:34.783708+0000","last_clean":"2024-02-10T07:38:34.783708+0000","last_became_active":"2024-02-10T07:31:57.313854+0000","last_became_peered":"2024-02-10T07:31:57.313854+0000","last_unstale":"2024-02-10T07:38:34.783708+0000","last_undegraded":"2024-02-10T07:38:34.783708+0000","last_fullsized":"2024-02-10T07:38:34.783708+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:37:39.095411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:08.117301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":265,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785005+0000","last_change":"2024-02-10T07:37:43.754879+0000","last_active":"2024-02-10T07:38:34.785005+0000","last_peered":"2024-02-10T07:38:34.785005+0000","last_clean":"2024-02-10T07:38:34.785005+0000","last_became_active":"2024-02-10T07:36:33.759656+0000","last_became_peered":"2024-02-10T07:36:33.759656+0000","last_unstale":"2024-02-10T07:38:34.785005+0000","last_undegraded":"2024-02-10T07:38:34.785005+0000","last_fullsized":"2024-02-10T07:38:34.785005+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:37:43.754618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:37:43.754618+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:52.815153+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":326,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.780958+0000","last_change":"2024-02-10T07:38:26.562972+0000","last_active":"2024-02-10T07:38:34.780958+0000","last_peered":"2024-02-10T07:38:34.780958+0000","last_clean":"2024-02-10T07:38:34.780958+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:38:34.780958+0000","last_undegraded":"2024-02-10T07:38:34.780958+0000","last_fullsized":"2024-02-10T07:38:34.780958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:26.562864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:26.562864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:52.775819+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":450,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:37.949001+0000","last_change":"2024-02-10T07:38:37.949081+0000","last_active":"2024-02-10T07:38:37.949001+0000","last_peered":"2024-02-10T07:38:37.949001+0000","last_clean":"2024-02-10T07:38:37.949001+0000","last_became_active":"2024-02-10T07:31:53.301863+0000","last_became_peered":"2024-02-10T07:31:53.301863+0000","last_unstale":"2024-02-10T07:38:37.949001+0000","last_undegraded":"2024-02-10T07:38:37.949001+0000","last_fullsized":"2024-02-10T07:38:37.949001+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-02-10T07:38:37.948929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:37.948929+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:07.680973+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":317,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.791124+0000","last_change":"2024-02-10T07:37:49.614365+0000","last_active":"2024-02-10T07:38:34.791124+0000","last_peered":"2024-02-10T07:38:34.791124+0000","last_clean":"2024-02-10T07:38:34.791124+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:38:34.791124+0000","last_undegraded":"2024-02-10T07:38:34.791124+0000","last_fullsized":"2024-02-10T07:38:34.791124+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:49.614249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:49.614249+0000","last_clean_scrub_stamp":"2024-02-10T07:37:49.614249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:56.011439+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"211'1886","reported_seq":1916,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:38.607516+0000","last_change":"2024-02-10T07:38:14.780713+0000","last_active":"2024-02-10T07:38:38.607516+0000","last_peered":"2024-02-10T07:38:38.607516+0000","last_clean":"2024-02-10T07:38:38.607516+0000","last_became_active":"2024-02-10T07:38:14.780419+0000","last_became_peered":"2024-02-10T07:38:14.780419+0000","last_unstale":"2024-02-10T07:38:38.607516+0000","last_undegraded":"2024-02-10T07:38:38.607516+0000","last_fullsized":"2024-02-10T07:38:38.607516+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":1886,"log_dups_size":0,"ondisk_log_size":1886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:35.222475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123600896,"num_objects":1886,"num_object_clones":0,"num_object_copies":5658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1886,"num_write_kb":120704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":314,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.787358+0000","last_change":"2024-02-10T07:37:40.674423+0000","last_active":"2024-02-10T07:38:34.787358+0000","last_peered":"2024-02-10T07:38:34.787358+0000","last_clean":"2024-02-10T07:38:34.787358+0000","last_became_active":"2024-02-10T07:31:57.299958+0000","last_became_peered":"2024-02-10T07:31:57.299958+0000","last_unstale":"2024-02-10T07:38:34.787358+0000","last_undegraded":"2024-02-10T07:38:34.787358+0000","last_fullsized":"2024-02-10T07:38:34.787358+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:40.674326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:40.674326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:05.194267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":324,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785566+0000","last_change":"2024-02-10T07:37:38.688852+0000","last_active":"2024-02-10T07:38:34.785566+0000","last_peered":"2024-02-10T07:38:34.785566+0000","last_clean":"2024-02-10T07:38:34.785566+0000","last_became_active":"2024-02-10T07:31:51.320864+0000","last_became_peered":"2024-02-10T07:31:51.320864+0000","last_unstale":"2024-02-10T07:38:34.785566+0000","last_undegraded":"2024-02-10T07:38:34.785566+0000","last_fullsized":"2024-02-10T07:38:34.785566+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:38.688738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:38.688738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:02.698589+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":334,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.788277+0000","last_change":"2024-02-10T07:37:46.833081+0000","last_active":"2024-02-10T07:38:34.788277+0000","last_peered":"2024-02-10T07:38:34.788277+0000","last_clean":"2024-02-10T07:38:34.788277+0000","last_became_active":"2024-02-10T07:31:50.281517+0000","last_became_peered":"2024-02-10T07:31:50.281517+0000","last_unstale":"2024-02-10T07:38:34.788277+0000","last_undegraded":"2024-02-10T07:38:34.788277+0000","last_fullsized":"2024-02-10T07:38:34.788277+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:46.832758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:37:46.832758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:00.227603+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":107,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.787613+0000","last_change":"2024-02-10T07:38:25.597385+0000","last_active":"2024-02-10T07:38:34.787613+0000","last_peered":"2024-02-10T07:38:34.787613+0000","last_clean":"2024-02-10T07:38:34.787613+0000","last_became_active":"2024-02-10T07:37:06.196787+0000","last_became_peered":"2024-02-10T07:37:06.196787+0000","last_unstale":"2024-02-10T07:38:34.787613+0000","last_undegraded":"2024-02-10T07:38:34.787613+0000","last_fullsized":"2024-02-10T07:38:34.787613+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-10T07:38:25.597236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:25.597236+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:41.209357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":326,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.791170+0000","last_change":"2024-02-10T07:37:48.818550+0000","last_active":"2024-02-10T07:38:34.791170+0000","last_peered":"2024-02-10T07:38:34.791170+0000","last_clean":"2024-02-10T07:38:34.791170+0000","last_became_active":"2024-02-10T07:36:33.751551+0000","last_became_peered":"2024-02-10T07:36:33.751551+0000","last_unstale":"2024-02-10T07:38:34.791170+0000","last_undegraded":"2024-02-10T07:38:34.791170+0000","last_fullsized":"2024-02-10T07:38:34.791170+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:48.818436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:48.818436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:10.693261+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"211'1806","reported_seq":1838,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:37.752029+0000","last_change":"2024-02-10T07:38:14.779982+0000","last_active":"2024-02-10T07:38:37.752029+0000","last_peered":"2024-02-10T07:38:37.752029+0000","last_clean":"2024-02-10T07:38:37.752029+0000","last_became_active":"2024-02-10T07:38:14.779523+0000","last_became_peered":"2024-02-10T07:38:14.779523+0000","last_unstale":"2024-02-10T07:38:37.752029+0000","last_undegraded":"2024-02-10T07:38:37.752029+0000","last_fullsized":"2024-02-10T07:38:37.752029+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":1806,"log_dups_size":0,"ondisk_log_size":1806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:35.074709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118358016,"num_objects":1806,"num_object_clones":0,"num_object_copies":5418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1806,"num_write_kb":115584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":278,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785382+0000","last_change":"2024-02-10T07:38:33.256517+0000","last_active":"2024-02-10T07:38:34.785382+0000","last_peered":"2024-02-10T07:38:34.785382+0000","last_clean":"2024-02-10T07:38:34.785382+0000","last_became_active":"2024-02-10T07:36:33.769763+0000","last_became_peered":"2024-02-10T07:36:33.769763+0000","last_unstale":"2024-02-10T07:38:34.785382+0000","last_undegraded":"2024-02-10T07:38:34.785382+0000","last_fullsized":"2024-02-10T07:38:34.785382+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:38:33.256420+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:33:41.150454+0000","last_clean_scrub_stamp":"2024-02-10T07:38:33.256420+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:41.430744+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":571,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.787285+0000","last_change":"2024-02-10T07:37:39.652563+0000","last_active":"2024-02-10T07:38:34.787285+0000","last_peered":"2024-02-10T07:38:34.787285+0000","last_clean":"2024-02-10T07:38:34.787285+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:38:34.787285+0000","last_undegraded":"2024-02-10T07:38:34.787285+0000","last_fullsized":"2024-02-10T07:38:34.787285+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:37:39.652478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:37:39.652478+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:45.396751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.782798+0000","last_change":"2024-02-10T07:36:57.022219+0000","last_active":"2024-02-10T07:38:34.782798+0000","last_peered":"2024-02-10T07:38:34.782798+0000","last_clean":"2024-02-10T07:38:34.782798+0000","last_became_active":"2024-02-10T07:36:33.752703+0000","last_became_peered":"2024-02-10T07:36:33.752703+0000","last_unstale":"2024-02-10T07:38:34.782798+0000","last_undegraded":"2024-02-10T07:38:34.782798+0000","last_fullsized":"2024-02-10T07:38:34.782798+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:57.022108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:36:57.022108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"89'24","reported_seq":441,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783262+0000","last_change":"2024-02-10T07:38:32.878953+0000","last_active":"2024-02-10T07:38:34.783262+0000","last_peered":"2024-02-10T07:38:34.783262+0000","last_clean":"2024-02-10T07:38:34.783262+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:38:34.783262+0000","last_undegraded":"2024-02-10T07:38:34.783262+0000","last_fullsized":"2024-02-10T07:38:34.783262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'24","last_scrub_stamp":"2024-02-10T07:38:32.878819+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:38:32.878819+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:01.404818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":319,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.787057+0000","last_change":"2024-02-10T07:36:41.813349+0000","last_active":"2024-02-10T07:38:34.787057+0000","last_peered":"2024-02-10T07:38:34.787057+0000","last_clean":"2024-02-10T07:38:34.787057+0000","last_became_active":"2024-02-10T07:36:33.752711+0000","last_became_peered":"2024-02-10T07:36:33.752711+0000","last_unstale":"2024-02-10T07:38:34.787057+0000","last_undegraded":"2024-02-10T07:38:34.787057+0000","last_fullsized":"2024-02-10T07:38:34.787057+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:41.813240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:36:41.813240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"211'1856","reported_seq":1888,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:37.951300+0000","last_change":"2024-02-10T07:38:14.780182+0000","last_active":"2024-02-10T07:38:37.951300+0000","last_peered":"2024-02-10T07:38:37.951300+0000","last_clean":"2024-02-10T07:38:37.951300+0000","last_became_active":"2024-02-10T07:38:14.779819+0000","last_became_peered":"2024-02-10T07:38:14.779819+0000","last_unstale":"2024-02-10T07:38:37.951300+0000","last_undegraded":"2024-02-10T07:38:37.951300+0000","last_fullsized":"2024-02-10T07:38:37.951300+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":1856,"log_dups_size":0,"ondisk_log_size":1856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:31.688480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":1856,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1856,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":95,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783227+0000","last_change":"2024-02-10T07:37:22.536358+0000","last_active":"2024-02-10T07:38:34.783227+0000","last_peered":"2024-02-10T07:38:34.783227+0000","last_clean":"2024-02-10T07:38:34.783227+0000","last_became_active":"2024-02-10T07:37:06.202738+0000","last_became_peered":"2024-02-10T07:37:06.202738+0000","last_unstale":"2024-02-10T07:38:34.783227+0000","last_undegraded":"2024-02-10T07:38:34.783227+0000","last_fullsized":"2024-02-10T07:38:34.783227+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:22.536223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:22.536223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:46.096601+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":325,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:37.574571+0000","last_change":"2024-02-10T07:38:37.574614+0000","last_active":"2024-02-10T07:38:37.574571+0000","last_peered":"2024-02-10T07:38:37.574571+0000","last_clean":"2024-02-10T07:38:37.574571+0000","last_became_active":"2024-02-10T07:31:51.286076+0000","last_became_peered":"2024-02-10T07:31:51.286076+0000","last_unstale":"2024-02-10T07:38:37.574571+0000","last_undegraded":"2024-02-10T07:38:37.574571+0000","last_fullsized":"2024-02-10T07:38:37.574571+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:37.574311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:37.574311+0000","last_clean_scrub_stamp":"2024-02-10T07:38:37.574311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:55.614745+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":739,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:36.716119+0000","last_change":"2024-02-10T07:38:21.504325+0000","last_active":"2024-02-10T07:38:36.716119+0000","last_peered":"2024-02-10T07:38:36.716119+0000","last_clean":"2024-02-10T07:38:36.716119+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:38:36.716119+0000","last_undegraded":"2024-02-10T07:38:36.716119+0000","last_fullsized":"2024-02-10T07:38:36.716119+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:38:21.504219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:38:21.504219+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:40.281918+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":401,"num_read_kb":396,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":339,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:35.477101+0000","last_change":"2024-02-10T07:38:35.477150+0000","last_active":"2024-02-10T07:38:35.477101+0000","last_peered":"2024-02-10T07:38:35.477101+0000","last_clean":"2024-02-10T07:38:35.477101+0000","last_became_active":"2024-02-10T07:31:53.300687+0000","last_became_peered":"2024-02-10T07:31:53.300687+0000","last_unstale":"2024-02-10T07:38:35.477101+0000","last_undegraded":"2024-02-10T07:38:35.477101+0000","last_fullsized":"2024-02-10T07:38:35.477101+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:38:35.476708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:35.476708+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:01.667916+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":272,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785623+0000","last_change":"2024-02-10T07:38:31.220461+0000","last_active":"2024-02-10T07:38:34.785623+0000","last_peered":"2024-02-10T07:38:34.785623+0000","last_clean":"2024-02-10T07:38:34.785623+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:38:34.785623+0000","last_undegraded":"2024-02-10T07:38:34.785623+0000","last_fullsized":"2024-02-10T07:38:34.785623+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:31.220325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:31.220325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:36.968611+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"211'1904","reported_seq":1936,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:38.606085+0000","last_change":"2024-02-10T07:38:14.780763+0000","last_active":"2024-02-10T07:38:38.606085+0000","last_peered":"2024-02-10T07:38:38.606085+0000","last_clean":"2024-02-10T07:38:38.606085+0000","last_became_active":"2024-02-10T07:38:14.780467+0000","last_became_peered":"2024-02-10T07:38:14.780467+0000","last_unstale":"2024-02-10T07:38:38.606085+0000","last_undegraded":"2024-02-10T07:38:38.606085+0000","last_fullsized":"2024-02-10T07:38:38.606085+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":1904,"log_dups_size":0,"ondisk_log_size":1904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:42.719171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124780544,"num_objects":1904,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1904,"num_write_kb":121856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":304,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786029+0000","last_change":"2024-02-10T07:35:44.445551+0000","last_active":"2024-02-10T07:38:34.786029+0000","last_peered":"2024-02-10T07:38:34.786029+0000","last_clean":"2024-02-10T07:38:34.786029+0000","last_became_active":"2024-02-10T07:31:57.307615+0000","last_became_peered":"2024-02-10T07:31:57.307615+0000","last_unstale":"2024-02-10T07:38:34.786029+0000","last_undegraded":"2024-02-10T07:38:34.786029+0000","last_fullsized":"2024-02-10T07:38:34.786029+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:35:44.445433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:35:44.445433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":333,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:36.734263+0000","last_change":"2024-02-10T07:38:36.734296+0000","last_active":"2024-02-10T07:38:36.734263+0000","last_peered":"2024-02-10T07:38:36.734263+0000","last_clean":"2024-02-10T07:38:36.734263+0000","last_became_active":"2024-02-10T07:36:33.750867+0000","last_became_peered":"2024-02-10T07:36:33.750867+0000","last_unstale":"2024-02-10T07:38:36.734263+0000","last_undegraded":"2024-02-10T07:38:36.734263+0000","last_fullsized":"2024-02-10T07:38:36.734263+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:38:36.734185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:51.560679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":366,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:36.492427+0000","last_change":"2024-02-10T07:38:36.492494+0000","last_active":"2024-02-10T07:38:36.492427+0000","last_peered":"2024-02-10T07:38:36.492427+0000","last_clean":"2024-02-10T07:38:36.492427+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:38:36.492427+0000","last_undegraded":"2024-02-10T07:38:36.492427+0000","last_fullsized":"2024-02-10T07:38:36.492427+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-10T07:38:36.492339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:36.492339+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:54.191943+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":326,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.782811+0000","last_change":"2024-02-10T07:37:22.987910+0000","last_active":"2024-02-10T07:38:34.782811+0000","last_peered":"2024-02-10T07:38:34.782811+0000","last_clean":"2024-02-10T07:38:34.782811+0000","last_became_active":"2024-02-10T07:36:33.765998+0000","last_became_peered":"2024-02-10T07:36:33.765998+0000","last_unstale":"2024-02-10T07:38:34.782811+0000","last_undegraded":"2024-02-10T07:38:34.782811+0000","last_fullsized":"2024-02-10T07:38:34.782811+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:22.987796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:22.987796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:41.037525+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"211'1761","reported_seq":1793,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:38.289626+0000","last_change":"2024-02-10T07:38:14.781247+0000","last_active":"2024-02-10T07:38:38.289626+0000","last_peered":"2024-02-10T07:38:38.289626+0000","last_clean":"2024-02-10T07:38:38.289626+0000","last_became_active":"2024-02-10T07:38:14.780910+0000","last_became_peered":"2024-02-10T07:38:14.780910+0000","last_unstale":"2024-02-10T07:38:38.289626+0000","last_undegraded":"2024-02-10T07:38:38.289626+0000","last_fullsized":"2024-02-10T07:38:38.289626+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":1761,"log_dups_size":0,"ondisk_log_size":1761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:15.581390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115408896,"num_objects":1761,"num_object_clones":0,"num_object_copies":5283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1761,"num_write_kb":112704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":200,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783303+0000","last_change":"2024-02-10T07:38:27.898967+0000","last_active":"2024-02-10T07:38:34.783303+0000","last_peered":"2024-02-10T07:38:34.783303+0000","last_clean":"2024-02-10T07:38:34.783303+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:38:34.783303+0000","last_undegraded":"2024-02-10T07:38:34.783303+0000","last_fullsized":"2024-02-10T07:38:34.783303+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:27.898770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:27.898770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:47.783371+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":294,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:39.220662+0000","last_change":"2024-02-10T07:38:39.220690+0000","last_active":"2024-02-10T07:38:39.220662+0000","last_peered":"2024-02-10T07:38:39.220662+0000","last_clean":"2024-02-10T07:38:39.220662+0000","last_became_active":"2024-02-10T07:36:33.779820+0000","last_became_peered":"2024-02-10T07:36:33.779820+0000","last_unstale":"2024-02-10T07:38:39.220662+0000","last_undegraded":"2024-02-10T07:38:39.220662+0000","last_fullsized":"2024-02-10T07:38:39.220662+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:39.220580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:39.220580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:50.640335+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":107,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786406+0000","last_change":"2024-02-10T07:37:41.661838+0000","last_active":"2024-02-10T07:38:34.786406+0000","last_peered":"2024-02-10T07:38:34.786406+0000","last_clean":"2024-02-10T07:38:34.786406+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:38:34.786406+0000","last_undegraded":"2024-02-10T07:38:34.786406+0000","last_fullsized":"2024-02-10T07:38:34.786406+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:37:41.661685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:41.661685+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:11.608613+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"158'10","reported_seq":417,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.784910+0000","last_change":"2024-02-10T07:38:31.556370+0000","last_active":"2024-02-10T07:38:34.784910+0000","last_peered":"2024-02-10T07:38:34.784910+0000","last_clean":"2024-02-10T07:38:34.784910+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:38:34.784910+0000","last_undegraded":"2024-02-10T07:38:34.784910+0000","last_fullsized":"2024-02-10T07:38:34.784910+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:38:31.556290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:31.556290+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:38.821295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"211'2090","reported_seq":2122,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:41.766961+0000","last_change":"2024-02-10T07:38:15.018575+0000","last_active":"2024-02-10T07:38:41.766961+0000","last_peered":"2024-02-10T07:38:41.766961+0000","last_clean":"2024-02-10T07:38:41.766961+0000","last_became_active":"2024-02-10T07:38:15.018262+0000","last_became_peered":"2024-02-10T07:38:15.018262+0000","last_unstale":"2024-02-10T07:38:41.766961+0000","last_undegraded":"2024-02-10T07:38:41.766961+0000","last_fullsized":"2024-02-10T07:38:41.766961+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:32.512344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136970240,"num_objects":2090,"num_object_clones":0,"num_object_copies":6270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2090,"num_write_kb":133760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":306,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.787014+0000","last_change":"2024-02-10T07:36:32.882898+0000","last_active":"2024-02-10T07:38:34.787014+0000","last_peered":"2024-02-10T07:38:34.787014+0000","last_clean":"2024-02-10T07:38:34.787014+0000","last_became_active":"2024-02-10T07:31:57.305620+0000","last_became_peered":"2024-02-10T07:31:57.305620+0000","last_unstale":"2024-02-10T07:38:34.787014+0000","last_undegraded":"2024-02-10T07:38:34.787014+0000","last_fullsized":"2024-02-10T07:38:34.787014+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:32.882824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:36:32.882824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":326,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786487+0000","last_change":"2024-02-10T07:37:23.482179+0000","last_active":"2024-02-10T07:38:34.786487+0000","last_peered":"2024-02-10T07:38:34.786487+0000","last_clean":"2024-02-10T07:38:34.786487+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:38:34.786487+0000","last_undegraded":"2024-02-10T07:38:34.786487+0000","last_fullsized":"2024-02-10T07:38:34.786487+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:37:23.482025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:23.482025+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":344,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.780746+0000","last_change":"2024-02-10T07:36:40.794069+0000","last_active":"2024-02-10T07:38:34.780746+0000","last_peered":"2024-02-10T07:38:34.780746+0000","last_clean":"2024-02-10T07:38:34.780746+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:38:34.780746+0000","last_undegraded":"2024-02-10T07:38:34.780746+0000","last_fullsized":"2024-02-10T07:38:34.780746+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-02-10T07:36:40.793949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:36:40.793949+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":408,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783609+0000","last_change":"2024-02-10T07:37:49.163617+0000","last_active":"2024-02-10T07:38:34.783609+0000","last_peered":"2024-02-10T07:38:34.783609+0000","last_clean":"2024-02-10T07:38:34.783609+0000","last_became_active":"2024-02-10T07:31:55.302863+0000","last_became_peered":"2024-02-10T07:31:55.302863+0000","last_unstale":"2024-02-10T07:38:34.783609+0000","last_undegraded":"2024-02-10T07:38:34.783609+0000","last_fullsized":"2024-02-10T07:38:34.783609+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:37:49.163478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:49.163478+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:02.762326+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"211'1812","reported_seq":1844,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:38.300560+0000","last_change":"2024-02-10T07:38:15.018995+0000","last_active":"2024-02-10T07:38:38.300560+0000","last_peered":"2024-02-10T07:38:38.300560+0000","last_clean":"2024-02-10T07:38:38.300560+0000","last_became_active":"2024-02-10T07:38:15.018665+0000","last_became_peered":"2024-02-10T07:38:15.018665+0000","last_unstale":"2024-02-10T07:38:38.300560+0000","last_undegraded":"2024-02-10T07:38:38.300560+0000","last_fullsized":"2024-02-10T07:38:38.300560+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":1812,"log_dups_size":0,"ondisk_log_size":1812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:29.858726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118751232,"num_objects":1812,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1812,"num_write_kb":115968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":306,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.780808+0000","last_change":"2024-02-10T07:36:48.752664+0000","last_active":"2024-02-10T07:38:34.780808+0000","last_peered":"2024-02-10T07:38:34.780808+0000","last_clean":"2024-02-10T07:38:34.780808+0000","last_became_active":"2024-02-10T07:31:57.316047+0000","last_became_peered":"2024-02-10T07:31:57.316047+0000","last_unstale":"2024-02-10T07:38:34.780808+0000","last_undegraded":"2024-02-10T07:38:34.780808+0000","last_fullsized":"2024-02-10T07:38:34.780808+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:36:48.752552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":318,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785424+0000","last_change":"2024-02-10T07:35:50.749720+0000","last_active":"2024-02-10T07:38:34.785424+0000","last_peered":"2024-02-10T07:38:34.785424+0000","last_clean":"2024-02-10T07:38:34.785424+0000","last_became_active":"2024-02-10T07:31:51.285333+0000","last_became_peered":"2024-02-10T07:31:51.285333+0000","last_unstale":"2024-02-10T07:38:34.785424+0000","last_undegraded":"2024-02-10T07:38:34.785424+0000","last_fullsized":"2024-02-10T07:38:34.785424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:35:50.749524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:35:50.749524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":346,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783474+0000","last_change":"2024-02-10T07:36:28.084850+0000","last_active":"2024-02-10T07:38:34.783474+0000","last_peered":"2024-02-10T07:38:34.783474+0000","last_clean":"2024-02-10T07:38:34.783474+0000","last_became_active":"2024-02-10T07:31:53.290157+0000","last_became_peered":"2024-02-10T07:31:53.290157+0000","last_unstale":"2024-02-10T07:38:34.783474+0000","last_undegraded":"2024-02-10T07:38:34.783474+0000","last_fullsized":"2024-02-10T07:38:34.783474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-10T07:36:28.084678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:36:28.084678+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":318,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.780500+0000","last_change":"2024-02-10T07:37:19.822653+0000","last_active":"2024-02-10T07:38:34.780500+0000","last_peered":"2024-02-10T07:38:34.780500+0000","last_clean":"2024-02-10T07:38:34.780500+0000","last_became_active":"2024-02-10T07:31:55.295900+0000","last_became_peered":"2024-02-10T07:31:55.295900+0000","last_unstale":"2024-02-10T07:38:34.780500+0000","last_undegraded":"2024-02-10T07:38:34.780500+0000","last_fullsized":"2024-02-10T07:38:34.780500+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:19.822560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:19.822560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"211'1698","reported_seq":1730,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:37.965539+0000","last_change":"2024-02-10T07:38:14.781045+0000","last_active":"2024-02-10T07:38:37.965539+0000","last_peered":"2024-02-10T07:38:37.965539+0000","last_clean":"2024-02-10T07:38:37.965539+0000","last_became_active":"2024-02-10T07:38:14.780765+0000","last_became_peered":"2024-02-10T07:38:14.780765+0000","last_unstale":"2024-02-10T07:38:37.965539+0000","last_undegraded":"2024-02-10T07:38:37.965539+0000","last_fullsized":"2024-02-10T07:38:37.965539+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":1698,"log_dups_size":0,"ondisk_log_size":1698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:25.625058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111280128,"num_objects":1698,"num_object_clones":0,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1698,"num_write_kb":108672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":271,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785258+0000","last_change":"2024-02-10T07:38:27.180747+0000","last_active":"2024-02-10T07:38:34.785258+0000","last_peered":"2024-02-10T07:38:34.785258+0000","last_clean":"2024-02-10T07:38:34.785258+0000","last_became_active":"2024-02-10T07:36:33.772094+0000","last_became_peered":"2024-02-10T07:36:33.772094+0000","last_unstale":"2024-02-10T07:38:34.785258+0000","last_undegraded":"2024-02-10T07:38:34.785258+0000","last_fullsized":"2024-02-10T07:38:34.785258+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:27.180619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:27.180619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:39.283016+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"158'10","reported_seq":409,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.781538+0000","last_change":"2024-02-10T07:36:43.830494+0000","last_active":"2024-02-10T07:38:34.781538+0000","last_peered":"2024-02-10T07:38:34.781538+0000","last_clean":"2024-02-10T07:38:34.781538+0000","last_became_active":"2024-02-10T07:36:33.755111+0000","last_became_peered":"2024-02-10T07:36:33.755111+0000","last_unstale":"2024-02-10T07:38:34.781538+0000","last_undegraded":"2024-02-10T07:38:34.781538+0000","last_fullsized":"2024-02-10T07:38:34.781538+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:36:43.830379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:36:43.830379+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":318,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783392+0000","last_change":"2024-02-10T07:37:25.985327+0000","last_active":"2024-02-10T07:38:34.783392+0000","last_peered":"2024-02-10T07:38:34.783392+0000","last_clean":"2024-02-10T07:38:34.783392+0000","last_became_active":"2024-02-10T07:31:51.282137+0000","last_became_peered":"2024-02-10T07:31:51.282137+0000","last_unstale":"2024-02-10T07:38:34.783392+0000","last_undegraded":"2024-02-10T07:38:34.783392+0000","last_fullsized":"2024-02-10T07:38:34.783392+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:25.985209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:25.985209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":344,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.786763+0000","last_change":"2024-02-10T07:36:58.580010+0000","last_active":"2024-02-10T07:38:34.786763+0000","last_peered":"2024-02-10T07:38:34.786763+0000","last_clean":"2024-02-10T07:38:34.786763+0000","last_became_active":"2024-02-10T07:31:53.294218+0000","last_became_peered":"2024-02-10T07:31:53.294218+0000","last_unstale":"2024-02-10T07:38:34.786763+0000","last_undegraded":"2024-02-10T07:38:34.786763+0000","last_fullsized":"2024-02-10T07:38:34.786763+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:36:58.579868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:36:58.579868+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"211'1782","reported_seq":1814,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:37.730862+0000","last_change":"2024-02-10T07:38:14.780614+0000","last_active":"2024-02-10T07:38:37.730862+0000","last_peered":"2024-02-10T07:38:37.730862+0000","last_clean":"2024-02-10T07:38:37.730862+0000","last_became_active":"2024-02-10T07:38:14.780264+0000","last_became_peered":"2024-02-10T07:38:14.780264+0000","last_unstale":"2024-02-10T07:38:37.730862+0000","last_undegraded":"2024-02-10T07:38:37.730862+0000","last_fullsized":"2024-02-10T07:38:37.730862+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":1782,"log_dups_size":0,"ondisk_log_size":1782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:25.475891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116785152,"num_objects":1782,"num_object_clones":0,"num_object_copies":5346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1782,"num_write_kb":114048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":314,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783284+0000","last_change":"2024-02-10T07:36:33.754484+0000","last_active":"2024-02-10T07:38:34.783284+0000","last_peered":"2024-02-10T07:38:34.783284+0000","last_clean":"2024-02-10T07:38:34.783284+0000","last_became_active":"2024-02-10T07:36:33.753921+0000","last_became_peered":"2024-02-10T07:36:33.753921+0000","last_unstale":"2024-02-10T07:38:34.783284+0000","last_undegraded":"2024-02-10T07:38:34.783284+0000","last_fullsized":"2024-02-10T07:38:34.783284+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:35:45.929352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:35:45.929352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":245,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.785411+0000","last_change":"2024-02-10T07:37:32.921439+0000","last_active":"2024-02-10T07:38:34.785411+0000","last_peered":"2024-02-10T07:38:34.785411+0000","last_clean":"2024-02-10T07:38:34.785411+0000","last_became_active":"2024-02-10T07:36:33.759566+0000","last_became_peered":"2024-02-10T07:36:33.759566+0000","last_unstale":"2024-02-10T07:38:34.785411+0000","last_undegraded":"2024-02-10T07:38:34.785411+0000","last_fullsized":"2024-02-10T07:38:34.785411+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:32.921308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:32.921308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:38.000659+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":326,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.780586+0000","last_change":"2024-02-10T07:37:36.765881+0000","last_active":"2024-02-10T07:38:34.780586+0000","last_peered":"2024-02-10T07:38:34.780586+0000","last_clean":"2024-02-10T07:38:34.780586+0000","last_became_active":"2024-02-10T07:31:51.320715+0000","last_became_peered":"2024-02-10T07:31:51.320715+0000","last_unstale":"2024-02-10T07:38:34.780586+0000","last_undegraded":"2024-02-10T07:38:34.780586+0000","last_fullsized":"2024-02-10T07:38:34.780586+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:37:36.765729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:55.408005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"89'16","reported_seq":381,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.781332+0000","last_change":"2024-02-10T07:37:31.825488+0000","last_active":"2024-02-10T07:38:34.781332+0000","last_peered":"2024-02-10T07:38:34.781332+0000","last_clean":"2024-02-10T07:38:34.781332+0000","last_became_active":"2024-02-10T07:37:06.193386+0000","last_became_peered":"2024-02-10T07:37:06.193386+0000","last_unstale":"2024-02-10T07:38:34.781332+0000","last_undegraded":"2024-02-10T07:38:34.781332+0000","last_fullsized":"2024-02-10T07:38:34.781332+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'16","last_scrub_stamp":"2024-02-10T07:37:31.825342+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:37:31.825342+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:00.309213+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"211'1829","reported_seq":1861,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:37.959490+0000","last_change":"2024-02-10T07:38:14.782039+0000","last_active":"2024-02-10T07:38:37.959490+0000","last_peered":"2024-02-10T07:38:37.959490+0000","last_clean":"2024-02-10T07:38:37.959490+0000","last_became_active":"2024-02-10T07:38:14.781754+0000","last_became_peered":"2024-02-10T07:38:14.781754+0000","last_unstale":"2024-02-10T07:38:37.959490+0000","last_undegraded":"2024-02-10T07:38:37.959490+0000","last_fullsized":"2024-02-10T07:38:37.959490+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":1829,"log_dups_size":0,"ondisk_log_size":1829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:16.541925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119865344,"num_objects":1829,"num_object_clones":0,"num_object_copies":5487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1829,"num_write_kb":117056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":312,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.789169+0000","last_change":"2024-02-10T07:37:21.672164+0000","last_active":"2024-02-10T07:38:34.789169+0000","last_peered":"2024-02-10T07:38:34.789169+0000","last_clean":"2024-02-10T07:38:34.789169+0000","last_became_active":"2024-02-10T07:31:57.311427+0000","last_became_peered":"2024-02-10T07:31:57.311427+0000","last_unstale":"2024-02-10T07:38:34.789169+0000","last_undegraded":"2024-02-10T07:38:34.789169+0000","last_fullsized":"2024-02-10T07:38:34.789169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:21.671876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:37:21.671876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:51.057686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":315,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.783671+0000","last_change":"2024-02-10T07:38:33.755591+0000","last_active":"2024-02-10T07:38:34.783671+0000","last_peered":"2024-02-10T07:38:34.783671+0000","last_clean":"2024-02-10T07:38:34.783671+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:38:34.783671+0000","last_undegraded":"2024-02-10T07:38:34.783671+0000","last_fullsized":"2024-02-10T07:38:34.783671+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:33.755481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:33.755481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:54.785308+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":333,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.782907+0000","last_change":"2024-02-10T07:38:21.809441+0000","last_active":"2024-02-10T07:38:34.782907+0000","last_peered":"2024-02-10T07:38:34.782907+0000","last_clean":"2024-02-10T07:38:34.782907+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:38:34.782907+0000","last_undegraded":"2024-02-10T07:38:34.782907+0000","last_fullsized":"2024-02-10T07:38:34.782907+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:21.809350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:56.900501+0000","last_clean_scrub_stamp":"2024-02-10T07:38:21.809350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:43.755545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"89'25","reported_seq":357,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.781893+0000","last_change":"2024-02-10T07:37:43.726839+0000","last_active":"2024-02-10T07:38:34.781893+0000","last_peered":"2024-02-10T07:38:34.781893+0000","last_clean":"2024-02-10T07:38:34.781893+0000","last_became_active":"2024-02-10T07:31:53.292277+0000","last_became_peered":"2024-02-10T07:31:53.292277+0000","last_unstale":"2024-02-10T07:38:34.781893+0000","last_undegraded":"2024-02-10T07:38:34.781893+0000","last_fullsized":"2024-02-10T07:38:34.781893+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'25","last_scrub_stamp":"2024-02-10T07:37:43.726688+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:33:20.251980+0000","last_clean_scrub_stamp":"2024-02-10T07:37:43.726688+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:50.183430+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"211'1835","reported_seq":1863,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:38.601696+0000","last_change":"2024-02-10T07:38:14.779934+0000","last_active":"2024-02-10T07:38:38.601696+0000","last_peered":"2024-02-10T07:38:38.601696+0000","last_clean":"2024-02-10T07:38:38.601696+0000","last_became_active":"2024-02-10T07:38:14.779651+0000","last_became_peered":"2024-02-10T07:38:14.779651+0000","last_unstale":"2024-02-10T07:38:38.601696+0000","last_undegraded":"2024-02-10T07:38:38.601696+0000","last_fullsized":"2024-02-10T07:38:38.601696+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":1835,"log_dups_size":0,"ondisk_log_size":1835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:42.849966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120258560,"num_objects":1835,"num_object_clones":0,"num_object_copies":5505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1835,"num_write_kb":117440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":313,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.781587+0000","last_change":"2024-02-10T07:38:29.654433+0000","last_active":"2024-02-10T07:38:34.781587+0000","last_peered":"2024-02-10T07:38:34.781587+0000","last_clean":"2024-02-10T07:38:34.781587+0000","last_became_active":"2024-02-10T07:31:57.308608+0000","last_became_peered":"2024-02-10T07:31:57.308608+0000","last_unstale":"2024-02-10T07:38:34.781587+0000","last_undegraded":"2024-02-10T07:38:34.781587+0000","last_fullsized":"2024-02-10T07:38:34.781587+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:29.654310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:29.654310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:30.590763+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"158'10","reported_seq":419,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.796531+0000","last_change":"2024-02-10T07:37:45.789514+0000","last_active":"2024-02-10T07:38:34.796531+0000","last_peered":"2024-02-10T07:38:34.796531+0000","last_clean":"2024-02-10T07:38:34.796531+0000","last_became_active":"2024-02-10T07:36:33.776454+0000","last_became_peered":"2024-02-10T07:36:33.776454+0000","last_unstale":"2024-02-10T07:38:34.796531+0000","last_undegraded":"2024-02-10T07:38:34.796531+0000","last_fullsized":"2024-02-10T07:38:34.796531+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:37:45.789402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:45.789402+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:53.584470+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":318,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.791983+0000","last_change":"2024-02-10T07:36:54.961833+0000","last_active":"2024-02-10T07:38:34.791983+0000","last_peered":"2024-02-10T07:38:34.791983+0000","last_clean":"2024-02-10T07:38:34.791983+0000","last_became_active":"2024-02-10T07:31:51.288197+0000","last_became_peered":"2024-02-10T07:31:51.288197+0000","last_unstale":"2024-02-10T07:38:34.791983+0000","last_undegraded":"2024-02-10T07:38:34.791983+0000","last_fullsized":"2024-02-10T07:38:34.791983+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:54.961713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:36:54.961713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":336,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-02-10T07:38:34.781287+0000","last_change":"2024-02-10T07:37:48.735421+0000","last_active":"2024-02-10T07:38:34.781287+0000","last_peered":"2024-02-10T07:38:34.781287+0000","last_clean":"2024-02-10T07:38:34.781287+0000","last_became_active":"2024-02-10T07:31:53.287984+0000","last_became_peered":"2024-02-10T07:31:53.287984+0000","last_unstale":"2024-02-10T07:38:34.781287+0000","last_undegraded":"2024-02-10T07:38:34.781287+0000","last_fullsized":"2024-02-10T07:38:34.781287+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:37:48.735290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:48.735290+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:54.941049+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1955659776,"num_objects":29841,"num_object_clones":0,"num_object_copies":89523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29841,"num_write_kb":1909824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5833162752,"data_stored":5833162752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29841,"ondisk_log_size":29841,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":401,"num_read_kb":396,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":437,"ondisk_log_size":437,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037846,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":760356,"kb_used_data":552860,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":92984796,"statfs":{"total":95995035648,"available":95216431104,"internally_reserved":0,"allocated":566128640,"data_stored":565752478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":212466122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:37:43 2024","interfaces":[{"interface":"back","average":{"1min":1.901,"5min":2.27,"15min":1.1140000000000001},"min":{"1min":0.20799999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.0490000000000004,"5min":52.591999999999999,"15min":52.591999999999999},"last":0.216},{"interface":"front","average":{"1min":1.208,"5min":1.8740000000000001,"15min":1.008},"min":{"1min":0.28399999999999997,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":5.5670000000000002,"5min":52.494,"15min":52.494},"last":0.45700000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:37:43 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.1859999999999999,"15min":0.753},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":5.7009999999999996,"5min":19.43,"15min":19.43},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.304,"15min":0.79800000000000004},"min":{"1min":0.30199999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":5.5339999999999998,"5min":19.460999999999999,"15min":19.460999999999999},"last":1.9810000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:37:43 2024","interfaces":[{"interface":"back","average":{"1min":1.732,"5min":1.4430000000000001,"15min":0.85099999999999998},"min":{"1min":0.29399999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.2530000000000001,"5min":19.710000000000001,"15min":19.710000000000001},"last":1.403},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.581,"15min":0.91000000000000003},"min":{"1min":0.376,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":6.5970000000000004,"5min":19.643000000000001,"15min":19.643000000000001},"last":0.40600000000000003}]},{"osd":3,"last update":"Sat Feb 10 07:37:43 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":2.3140000000000001,"15min":1.1479999999999999},"min":{"1min":0.28399999999999997,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.6440000000000001,"5min":64.120999999999995,"15min":64.120999999999995},"last":0.438},{"interface":"front","average":{"1min":1.696,"5min":2.645,"15min":1.2729999999999999},"min":{"1min":0.308,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.0899999999999999,"5min":64.400000000000006,"15min":64.400000000000006},"last":0.33900000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:37:43 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.48299999999999998,"15min":0.49299999999999999},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.49299999999999999,"15min":0.50900000000000001},"min":{"1min":0.315,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.35099999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:38:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.69599999999999995,"15min":0.71799999999999997},"min":{"1min":0.371,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1719999999999999,"5min":3.3969999999999998,"15min":3.3969999999999998},"last":0.372},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.70399999999999996,"15min":0.71499999999999997},"min":{"1min":0.41199999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.4610000000000001,"5min":2.141,"15min":2.141},"last":0.41899999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:37:43 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.33500000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":2.2250000000000001,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.032,"5min":0.63600000000000001,"15min":0.58499999999999996},"min":{"1min":0.36599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":6.2169999999999996,"5min":6.2169999999999996,"15min":6.2169999999999996},"last":0.39600000000000002}]}]},{"osd":6,"up_from":50,"seq":214748364891,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":673688,"kb_used_data":530060,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93071464,"statfs":{"total":95995035648,"available":95305179136,"internally_reserved":0,"allocated":542781440,"data_stored":542421009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:38:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":1.8540000000000001,"15min":0.98799999999999999},"min":{"1min":0.246,"5min":0.217,"15min":0.217},"max":{"1min":2.9900000000000002,"5min":63.405999999999999,"15min":63.405999999999999},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.89200000000000002,"5min":2.4940000000000002,"15min":1.198},"min":{"1min":0.38,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":6.0590000000000002,"5min":67.748000000000005,"15min":67.748000000000005},"last":0.378}]},{"osd":1,"last update":"Sat Feb 10 07:38:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.878,"15min":0.66300000000000003},"min":{"1min":0.35499999999999998,"5min":0.255,"15min":0.255},"max":{"1min":2.508,"5min":8.3940000000000001,"15min":8.3940000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.92600000000000005,"15min":0.68799999999999994},"min":{"1min":0.34000000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.8220000000000001,"5min":9.6120000000000001,"15min":9.6120000000000001},"last":0.48899999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:38:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.3620000000000001,"15min":1.1759999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.1269999999999998,"5min":65.278999999999996,"15min":65.278999999999996},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":2.5920000000000001,"15min":1.244},"min":{"1min":0.29399999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.0550000000000002,"5min":65.265000000000001,"15min":65.265000000000001},"last":0.46200000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:38:25 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":2.0009999999999999,"15min":1.778},"min":{"1min":0.34699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":2.3879999999999999,"5min":62.835999999999999,"15min":62.835999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.5720000000000001,"15min":0.93400000000000005},"min":{"1min":0.39600000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.8570000000000002,"5min":63.878999999999998,"15min":63.878999999999998},"last":0.61899999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:38:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57799999999999996,"15min":0.50700000000000001},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.6419999999999999,"5min":7.7839999999999998,"15min":7.7839999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.52100000000000002,"15min":0.495},"min":{"1min":0.29399999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":0.77400000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:38:35 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.71699999999999997,"15min":0.745},"min":{"1min":0.35299999999999998,"5min":0.312,"15min":0.312},"max":{"1min":1.028,"5min":4.3760000000000003,"15min":4.3760000000000003},"last":0.433},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.6659999999999999,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":3.0059999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:37:47 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.67000000000000004,"15min":0.57199999999999995},"min":{"1min":0.38500000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.4609999999999999,"5min":7.859,"15min":7.859},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.64500000000000002,"15min":0.59199999999999997},"min":{"1min":0.39400000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4470000000000001,"5min":2.4470000000000001,"15min":2.4470000000000001},"last":0.73999999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":976572,"kb_used_data":805172,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":92768580,"statfs":{"total":95995035648,"available":94995025920,"internally_reserved":0,"allocated":824496128,"data_stored":824095140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":175503814},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:38:24 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":2.661,"15min":1.123},"min":{"1min":0.255,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":114.27,"5min":114.27,"15min":114.27},"last":0.92400000000000004},{"interface":"front","average":{"1min":7.6749999999999998,"5min":3.2040000000000002,"15min":1.3180000000000001},"min":{"1min":0.253,"5min":0.22500000000000001,"15min":0.22},"max":{"1min":119.499,"5min":119.499,"15min":119.499},"last":1.4330000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:37:45 2024","interfaces":[{"interface":"back","average":{"1min":2.2810000000000001,"5min":1.3260000000000001,"15min":0.71099999999999997},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":8.9209999999999994,"5min":8.9209999999999994,"15min":8.9209999999999994},"last":0.76600000000000001},{"interface":"front","average":{"1min":2.2490000000000001,"5min":1.4159999999999999,"15min":0.753},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23699999999999999},"max":{"1min":7.2229999999999999,"5min":7.2229999999999999,"15min":7.2229999999999999},"last":0.98499999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:38:20 2024","interfaces":[{"interface":"back","average":{"1min":1.8879999999999999,"5min":1.698,"15min":0.84899999999999998},"min":{"1min":0.22,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":1.228},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.518,"15min":0.84599999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":6.1050000000000004,"5min":11.698,"15min":11.698},"last":0.96399999999999997}]},{"osd":4,"last update":"Sat Feb 10 07:38:37 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":1.105,"15min":1.0249999999999999},"min":{"1min":0.33800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.0619999999999998,"5min":8.9730000000000008,"15min":10.292},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.109,"15min":0.755},"min":{"1min":0.32300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":7.8079999999999998,"5min":7.8079999999999998,"15min":7.8079999999999998},"last":0.54200000000000004}]},{"osd":5,"last update":"Sat Feb 10 07:38:37 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":1.5289999999999999,"15min":1.6579999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.0059999999999998,"5min":9.0410000000000004,"15min":9.0410000000000004},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.4750000000000001,"15min":1.5609999999999999},"min":{"1min":0.46500000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":5.1459999999999999,"5min":5.2069999999999999,"15min":5.2069999999999999},"last":0.751}]},{"osd":6,"last update":"Sat Feb 10 07:38:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.1799999999999999,"15min":0.81100000000000005},"min":{"1min":0.36499999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":11.244999999999999,"5min":11.244999999999999,"15min":11.244999999999999},"last":0.629},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.21,"15min":0.86199999999999999},"min":{"1min":0.41199999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.8050000000000002,"5min":9.0090000000000003,"15min":9.0090000000000003},"last":0.79800000000000004}]},{"osd":7,"last update":"Sat Feb 10 07:37:50 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.216,"15min":0.82999999999999996},"min":{"1min":0.42399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":5.4809999999999999,"5min":6.6360000000000001,"15min":6.6360000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.248,"15min":0.83599999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":11.212,"5min":11.212,"15min":11.212},"last":0.57999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":886316,"kb_used_data":670500,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92858836,"statfs":{"total":95995035648,"available":95087448064,"internally_reserved":0,"allocated":686592000,"data_stored":686208263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:38:20 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.7170000000000001,"15min":0.80200000000000005},"min":{"1min":0.216,"5min":0.192,"15min":0.192},"max":{"1min":7.6989999999999998,"5min":16.709,"15min":16.709},"last":14.593999999999999},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.742,"15min":0.82699999999999996},"min":{"1min":0.26800000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.4589999999999996,"5min":9.7170000000000005,"15min":9.7170000000000005},"last":13.867000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:37:51 2024","interfaces":[{"interface":"back","average":{"1min":2.8460000000000001,"5min":1.8620000000000001,"15min":0.879},"min":{"1min":0.57099999999999995,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":12.002000000000001,"5min":12.002000000000001,"15min":12.002000000000001},"last":1.345},{"interface":"front","average":{"1min":2.2709999999999999,"5min":1.726,"15min":0.85999999999999999},"min":{"1min":0.499,"5min":0.248,"15min":0.23799999999999999},"max":{"1min":4.4870000000000001,"5min":9.0540000000000003,"15min":9.0540000000000003},"last":1.3680000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:38:09 2024","interfaces":[{"interface":"back","average":{"1min":3.7789999999999999,"5min":2.347,"15min":1.0760000000000001},"min":{"1min":0.32600000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":27.242000000000001,"5min":27.242000000000001,"15min":27.242000000000001},"last":1.391},{"interface":"front","average":{"1min":1.843,"5min":2.181,"15min":1.042},"min":{"1min":0.34200000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.3979999999999997,"5min":12.885,"15min":12.885},"last":1.45}]},{"osd":4,"last update":"Sat Feb 10 07:38:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.2569999999999999,"15min":0.77800000000000002},"min":{"1min":0.372,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":5.4260000000000002,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":0.372},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.5700000000000001,"15min":0.95999999999999996},"min":{"1min":0.44600000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.3879999999999999,"5min":11.867000000000001,"15min":19.219000000000001},"last":1.512}]},{"osd":5,"last update":"Sat Feb 10 07:38:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":3.0099999999999998,"15min":3.3140000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":5.468,"5min":33.289999999999999,"15min":33.289999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":1.0169999999999999,"5min":3.6549999999999998,"15min":4.0949999999999998},"min":{"1min":0.51000000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.4119999999999999,"5min":33.430999999999997,"15min":33.430999999999997},"last":1.478}]},{"osd":6,"last update":"Sat Feb 10 07:38:20 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.4450000000000001,"15min":0.97399999999999998},"min":{"1min":0.44,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":4.3550000000000004,"5min":6.4749999999999996,"15min":19.16},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.3919999999999999,"15min":0.88300000000000001},"min":{"1min":0.433,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":5.5049999999999999,"5min":7.5119999999999996,"15min":7.5119999999999996},"last":0.40300000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:37:51 2024","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":1.607,"15min":1.5609999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":11.912000000000001,"5min":11.912000000000001,"15min":19.178000000000001},"last":1.4399999999999999},{"interface":"front","average":{"1min":2.2429999999999999,"5min":1.4730000000000001,"15min":0.92600000000000005},"min":{"1min":0.44700000000000001,"5min":0.376,"15min":0.376},"max":{"1min":4.431,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":1.6160000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411439,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":913820,"kb_used_data":688276,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":92831332,"statfs":{"total":95995035648,"available":95059283968,"internally_reserved":0,"allocated":704794624,"data_stored":704426487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":230947269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:37:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4409999999999998,"5min":3.2330000000000001,"15min":1.331},"min":{"1min":0.47099999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":21.984000000000002,"5min":90.046999999999997,"15min":90.046999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":2.351,"5min":3.1259999999999999,"15min":1.321},"min":{"1min":0.35299999999999998,"5min":0.27700000000000002,"15min":0.25700000000000001},"max":{"1min":10.867000000000001,"5min":90.016000000000005,"15min":90.016000000000005},"last":0.41999999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:37:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.7,"15min":0.85199999999999998},"min":{"1min":0.54700000000000004,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":3.2930000000000001,"5min":12.382,"15min":12.382},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.7929999999999999,"5min":1.7090000000000001,"15min":0.85399999999999998},"min":{"1min":0.371,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":3.6389999999999998,"5min":9.5630000000000006,"15min":9.5630000000000006},"last":1.0169999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:38:18 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":2.0419999999999998,"15min":1.034},"min":{"1min":0.30399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.7530000000000001,"5min":14.525,"15min":14.525},"last":0.443},{"interface":"front","average":{"1min":1.095,"5min":2.1440000000000001,"15min":1.079},"min":{"1min":0.30199999999999999,"5min":0.30099999999999999,"15min":0.29699999999999999},"max":{"1min":2.7250000000000001,"5min":11.739000000000001,"15min":11.739000000000001},"last":1.1439999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:38:37 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.6399999999999999,"15min":0.96599999999999997},"min":{"1min":0.42899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.2010000000000001,"5min":33.634999999999998,"15min":33.634999999999998},"last":1.089},{"interface":"front","average":{"1min":0.80300000000000005,"5min":1.8009999999999999,"15min":1.0409999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":2.4609999999999999,"5min":33.554000000000002,"15min":33.554000000000002},"last":1.1839999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:38:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.161,"15min":1.1870000000000001},"min":{"1min":0.35899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.9429999999999996,"5min":5.0579999999999998,"15min":5.0579999999999998},"last":4.9429999999999996},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.5349999999999999,"15min":1.613},"min":{"1min":0.51400000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.4740000000000002,"5min":9.4280000000000008,"15min":9.4280000000000008},"last":4.4740000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":1.456,"15min":1.0660000000000001},"min":{"1min":0.44800000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.3079999999999998,"5min":9.3230000000000004,"15min":9.3230000000000004},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.3859999999999999,"15min":0.89100000000000001},"min":{"1min":0.376,"5min":0.315,"15min":0.315},"max":{"1min":2.4609999999999999,"5min":8.1899999999999995,"15min":8.1899999999999995},"last":0.77200000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:37:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4299999999999999,"15min":0.91800000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":7.0979999999999999,"5min":8.2189999999999994,"15min":8.2189999999999994},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.496,"5min":1.415,"15min":0.93000000000000005},"min":{"1min":0.52100000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.665,"5min":9.2859999999999996,"15min":9.2859999999999996},"last":2.024}]}]},{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":960308,"kb_used_data":700076,"kb_used_omap":1,"kb_used_meta":260222,"kb_avail":92784844,"statfs":{"total":95995035648,"available":95011680256,"internally_reserved":0,"allocated":716877824,"data_stored":716484587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":266467783},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.8340000000000001,"5min":2.181,"15min":0.97799999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.8079999999999998,"5min":25.277000000000001,"15min":25.277000000000001},"last":1.052},{"interface":"front","average":{"1min":1.4770000000000001,"5min":2.1520000000000001,"15min":0.995},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.18099999999999999},"max":{"1min":8.9570000000000007,"5min":28.390000000000001,"15min":28.390000000000001},"last":3.9140000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.7470000000000001,"15min":0.86199999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23899999999999999},"max":{"1min":10.279,"5min":13.076000000000001,"15min":13.076000000000001},"last":3.7869999999999999},{"interface":"front","average":{"1min":1.9159999999999999,"5min":2.0219999999999998,"15min":0.96599999999999997},"min":{"1min":0.29399999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":7.3860000000000001,"5min":12.179,"15min":12.179},"last":2.4009999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:38:19 2024","interfaces":[{"interface":"back","average":{"1min":2.2309999999999999,"5min":3.3359999999999999,"15min":1.429},"min":{"1min":0.372,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.5490000000000004,"5min":163.441,"15min":163.441},"last":1.238},{"interface":"front","average":{"1min":2.1099999999999999,"5min":3.7130000000000001,"15min":1.526},"min":{"1min":0.34100000000000003,"5min":0.26100000000000001,"15min":0.218},"max":{"1min":8.8599999999999994,"5min":166.82400000000001,"15min":166.82400000000001},"last":1.1279999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:37:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3660000000000001,"15min":0.83299999999999996},"min":{"1min":0.38600000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":8.8300000000000001,"5min":8.8300000000000001,"15min":8.8300000000000001},"last":3.859},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.351,"15min":0.85999999999999999},"min":{"1min":0.29999999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":7.2999999999999998,"5min":7.3390000000000004,"15min":7.3390000000000004},"last":1.1910000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:38:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.1319999999999999,"15min":1.077},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"last":3.8279999999999998},{"interface":"front","average":{"1min":1.46,"5min":1.24,"15min":1.204},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":7.234,"5min":7.234,"15min":7.234},"last":1.153}]},{"osd":6,"last update":"Sat Feb 10 07:38:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.8220000000000001,"15min":1.056},"min":{"1min":0.40500000000000003,"5min":0.38,"15min":0.38},"max":{"1min":7.3520000000000003,"5min":32.048999999999999,"15min":32.048999999999999},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.675,"15min":0.95999999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":7.5099999999999998,"5min":25.937000000000001,"15min":25.937000000000001},"last":0.33600000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:37:54 2024","interfaces":[{"interface":"back","average":{"1min":2.484,"5min":1.6930000000000001,"15min":0.96299999999999997},"min":{"1min":0.41399999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":25.273,"5min":25.273,"15min":25.273},"last":3.8860000000000001},{"interface":"front","average":{"1min":2.7749999999999999,"5min":1.7390000000000001,"15min":1.0269999999999999},"min":{"1min":0.45700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":27.75,"5min":27.75,"15min":27.75},"last":1.331}]}]},{"osd":4,"up_from":32,"seq":137438953573,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":753392,"kb_used_data":563944,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":92991760,"statfs":{"total":95995035648,"available":95223562240,"internally_reserved":0,"allocated":577478656,"data_stored":577089627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:38:36 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":2.5419999999999998,"15min":1.258},"min":{"1min":0.26600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.0579999999999998,"5min":58.142000000000003,"15min":58.142000000000003},"last":1.5960000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":2.4079999999999999,"15min":1.169},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.1880000000000002,"5min":59.042000000000002,"15min":59.042000000000002},"last":0.40300000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:38:36 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.4359999999999999,"15min":0.84299999999999997},"min":{"1min":0.253,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.784,"5min":49.777999999999999,"15min":49.777999999999999},"last":1.5660000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.4239999999999999,"15min":0.84999999999999998},"min":{"1min":0.307,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.0649999999999999,"5min":49.765999999999998,"15min":49.765999999999998},"last":0.76600000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:38:36 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":2.0070000000000001,"15min":1.028},"min":{"1min":0.29499999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.96,"5min":50.162999999999997,"15min":50.162999999999997},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.503,"5min":1.8300000000000001,"15min":1.008},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.431,"5min":50.453000000000003,"15min":50.453000000000003},"last":0.436}]},{"osd":3,"last update":"Sat Feb 10 07:38:36 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":2.1539999999999999,"15min":1.1060000000000001},"min":{"1min":0.32200000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":7.016,"5min":54.366,"15min":54.366},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.628,"5min":2.0390000000000001,"15min":1.073},"min":{"1min":0.34000000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.776,"5min":53.039000000000001,"15min":53.039000000000001},"last":1.631}]},{"osd":5,"last update":"Sat Feb 10 07:38:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.49299999999999999,"15min":0.48199999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.46999999999999997,"15min":0.46000000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.1639999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":1.7669999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.51600000000000001,"15min":0.46100000000000002},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.84099999999999997,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.53000000000000003,"15min":0.46500000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.80700000000000005,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.82899999999999996}]},{"osd":7,"last update":"Sat Feb 10 07:37:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55600000000000005,"15min":0.52000000000000002},"min":{"1min":0.35599999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61899999999999999,"5min":1.98,"15min":1.98},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57099999999999995,"15min":0.52500000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.3340000000000001,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.5189999999999999}]}]},{"osd":5,"up_from":156,"seq":670014898203,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1179292,"kb_used_data":1060244,"kb_used_omap":4,"kb_used_meta":119035,"kb_avail":92565860,"statfs":{"total":95995035648,"available":94787440640,"internally_reserved":0,"allocated":1085689856,"data_stored":1085321901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":121892514},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.5049999999999999,"15min":1.597},"min":{"1min":0.308,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.6580000000000004,"5min":8.734,"15min":8.734},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.603,"15min":1.7709999999999999},"min":{"1min":0.26400000000000001,"5min":0.246,"15min":0.246},"max":{"1min":2.0049999999999999,"5min":8.1999999999999993,"15min":8.1999999999999993},"last":0.66200000000000003}]},{"osd":1,"last update":"Sat Feb 10 07:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":3.0089999999999999,"15min":3.415},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.9410000000000001,"5min":47.139000000000003,"15min":47.139000000000003},"last":1.327},{"interface":"front","average":{"1min":0.68700000000000006,"5min":3.1269999999999998,"15min":3.5339999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.3839999999999999,"5min":47.328000000000003,"15min":47.328000000000003},"last":0.57099999999999995}]},{"osd":2,"last update":"Sat Feb 10 07:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.728,"15min":1.8819999999999999},"min":{"1min":0.33600000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.8999999999999999,"5min":11.601000000000001,"15min":11.601000000000001},"last":1.004},{"interface":"front","average":{"1min":0.90100000000000002,"5min":1.841,"15min":1.9970000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.9119999999999999,"5min":11.618,"15min":11.618},"last":1.0169999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.4399999999999999,"15min":1.5509999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.4300000000000002,"5min":8.3870000000000005,"15min":8.3870000000000005},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.746,"15min":1.889},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":2.7000000000000002,"5min":8.3209999999999997,"15min":8.3209999999999997},"last":0.68700000000000006}]},{"osd":4,"last update":"Sat Feb 10 07:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.51700000000000002,"15min":0.52600000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.70199999999999996,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.318},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.46600000000000003,"15min":0.45900000000000002},"min":{"1min":0.29899999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.70799999999999996,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.58999999999999997}]},{"osd":6,"last update":"Sat Feb 10 07:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.67400000000000004,"15min":0.69799999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81399999999999995,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.626}]},{"osd":7,"last update":"Sat Feb 10 07:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35399999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.68899999999999995,"5min":0.877,"15min":0.877},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.32700000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.91000000000000003,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.64000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":713883648,"data_stored":713883648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":826277888,"data_stored":826277888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":722665472,"data_stored":722665472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":728236032,"data_stored":728236032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":583204864,"data_stored":583204864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1109262336,"data_stored":1109262336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":576520192,"data_stored":576520192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":573112320,"data_stored":573112320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:38:45.127 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:38:45.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:45.128 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:45.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:45.133 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: 5.1b scrub starts 2024-02-10T07:38:45.133 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: 5.1b scrub ok 2024-02-10T07:38:45.134 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: 4.16 scrub starts 2024-02-10T07:38:45.134 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: 4.16 scrub ok 2024-02-10T07:38:45.134 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: 4.1 scrub starts 2024-02-10T07:38:45.134 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: 4.1 scrub ok 2024-02-10T07:38:45.134 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: pgmap v319: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 123 MiB/s wr, 1.98k op/s; 12 B/s, 0 objects/s recovering 2024-02-10T07:38:45.134 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:38:45.134 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:38:45.134 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: 6.1 scrub starts 2024-02-10T07:38:45.135 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: 6.1 scrub ok 2024-02-10T07:38:45.135 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: osdmap e213: 8 total, 8 up, 8 in 2024-02-10T07:38:45.135 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: 4.1d scrub starts 2024-02-10T07:38:45.135 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: 4.1d scrub ok 2024-02-10T07:38:45.135 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: 3.6 scrub starts 2024-02-10T07:38:45.135 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[40743]: 3.6 scrub ok 2024-02-10T07:38:45.136 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: 5.1b scrub starts 2024-02-10T07:38:45.136 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: 5.1b scrub ok 2024-02-10T07:38:45.136 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: 4.16 scrub starts 2024-02-10T07:38:45.136 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: 4.16 scrub ok 2024-02-10T07:38:45.136 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: 4.1 scrub starts 2024-02-10T07:38:45.136 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: 4.1 scrub ok 2024-02-10T07:38:45.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: pgmap v319: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 123 MiB/s wr, 1.98k op/s; 12 B/s, 0 objects/s recovering 2024-02-10T07:38:45.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:38:45.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:38:45.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: 6.1 scrub starts 2024-02-10T07:38:45.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: 6.1 scrub ok 2024-02-10T07:38:45.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: osdmap e213: 8 total, 8 up, 8 in 2024-02-10T07:38:45.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: 4.1d scrub starts 2024-02-10T07:38:45.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: 4.1d scrub ok 2024-02-10T07:38:45.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: 3.6 scrub starts 2024-02-10T07:38:45.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:45 smithi008 ceph-mon[29816]: 3.6 scrub ok 2024-02-10T07:38:45.230 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:45.261 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:45.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:45.363 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: 5.1b scrub starts 2024-02-10T07:38:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: 5.1b scrub ok 2024-02-10T07:38:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: 4.16 scrub starts 2024-02-10T07:38:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: 4.16 scrub ok 2024-02-10T07:38:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: 4.1 scrub starts 2024-02-10T07:38:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: 4.1 scrub ok 2024-02-10T07:38:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: pgmap v319: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 123 MiB/s wr, 1.98k op/s; 12 B/s, 0 objects/s recovering 2024-02-10T07:38:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:38:45.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:38:45.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: 6.1 scrub starts 2024-02-10T07:38:45.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: 6.1 scrub ok 2024-02-10T07:38:45.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: osdmap e213: 8 total, 8 up, 8 in 2024-02-10T07:38:45.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: 4.1d scrub starts 2024-02-10T07:38:45.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: 4.1d scrub ok 2024-02-10T07:38:45.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: 3.6 scrub starts 2024-02-10T07:38:45.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:45 smithi170 ceph-mon[33394]: 3.6 scrub ok 2024-02-10T07:38:45.395 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:45.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:45.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:45.497 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:45.537 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:45.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:45.633 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 19 16 40122 40106 131.891 117.312 0.00906596 0.00757704 2024-02-10T07:38:45.640 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:45.679 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:45.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:45.781 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:45.820 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:45.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:45.860 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:45.860 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:45.919 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-10T07:38:45.921 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:46.011 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:46.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:46.112 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:46.140 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:46.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:46.242 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:46.261 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:46 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:38:46.261 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:46 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:38:46.263 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:46 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:38:46.263 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:46 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:38:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:46.293 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:46.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:46 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:38:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:46 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:38:46.394 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:46.427 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:46.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:46.529 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:46.574 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:46.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:46.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:46.633 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:38:46.634340+0000 min lat: 0.00127253 max lat: 0.194119 avg lat: 0.00765188 2024-02-10T07:38:46.634 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:38:46.634 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 20 16 41818 41802 130.596 106 0.00718683 0.00765188 2024-02-10T07:38:46.676 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:46.711 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:46.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:46.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:46.812 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:46.852 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:46.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:46.954 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:46.980 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:46.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:46.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:47.082 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:47.121 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:47.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:47.223 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:47.263 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:47.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:47.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:47.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:47 smithi008 ceph-mon[29816]: 6.1a scrub starts 2024-02-10T07:38:47.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:47 smithi008 ceph-mon[29816]: 6.1a scrub ok 2024-02-10T07:38:47.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:47 smithi008 ceph-mon[29816]: pgmap v321: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-02-10T07:38:47.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:47 smithi008 ceph-mon[29816]: 4.f scrub starts 2024-02-10T07:38:47.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:47 smithi008 ceph-mon[29816]: 4.f scrub ok 2024-02-10T07:38:47.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:47 smithi008 ceph-mon[29816]: osdmap e214: 8 total, 8 up, 8 in 2024-02-10T07:38:47.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:47 smithi008 ceph-mon[29816]: osdmap e215: 8 total, 8 up, 8 in 2024-02-10T07:38:47.310 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:47 smithi008 ceph-mon[40743]: 6.1a scrub starts 2024-02-10T07:38:47.310 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:47 smithi008 ceph-mon[40743]: 6.1a scrub ok 2024-02-10T07:38:47.310 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:47 smithi008 ceph-mon[40743]: pgmap v321: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-02-10T07:38:47.310 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:47 smithi008 ceph-mon[40743]: 4.f scrub starts 2024-02-10T07:38:47.310 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:47 smithi008 ceph-mon[40743]: 4.f scrub ok 2024-02-10T07:38:47.310 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:47 smithi008 ceph-mon[40743]: osdmap e214: 8 total, 8 up, 8 in 2024-02-10T07:38:47.310 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:47 smithi008 ceph-mon[40743]: osdmap e215: 8 total, 8 up, 8 in 2024-02-10T07:38:47.365 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:47 smithi170 ceph-mon[33394]: 6.1a scrub starts 2024-02-10T07:38:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:47 smithi170 ceph-mon[33394]: 6.1a scrub ok 2024-02-10T07:38:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:47 smithi170 ceph-mon[33394]: pgmap v321: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.69k op/s 2024-02-10T07:38:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:47 smithi170 ceph-mon[33394]: 4.f scrub starts 2024-02-10T07:38:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:47 smithi170 ceph-mon[33394]: 4.f scrub ok 2024-02-10T07:38:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:47 smithi170 ceph-mon[33394]: osdmap e214: 8 total, 8 up, 8 in 2024-02-10T07:38:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:47 smithi170 ceph-mon[33394]: osdmap e215: 8 total, 8 up, 8 in 2024-02-10T07:38:47.404 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:47.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:47.506 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:47.543 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:47.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:47.633 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 21 16 43438 43422 129.199 101.25 0.00385601 0.00773413 2024-02-10T07:38:47.645 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:47.677 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:47.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:47.778 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:47.821 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:47.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:47.922 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:47.957 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:47.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:47.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:48.059 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:48.100 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:48.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:48.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:48.202 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:48.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:48.238 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:48.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:48.339 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:48.372 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:48.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:48.474 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:48.515 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:48.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:48.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:48.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:48 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:48.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:48 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:48.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:48 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:48.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:48 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:48.617 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:48 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:38:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:48 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:38:48.634 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 22 16 45301 45285 128.615 116.438 0.00950665 0.00776877 2024-02-10T07:38:48.656 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:48.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:48.758 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:48.802 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:48.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:48.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:48.904 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:48.938 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:48.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:48.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:49.039 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:49.067 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:49.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:49.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:49.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:38:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:38:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:38:49.169 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:49.197 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:49.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:49.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:49.299 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:49.349 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:49.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:49.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:49.451 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:49.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:49.493 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:49.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:49.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:49 smithi008 ceph-mon[29816]: 6.2 scrub starts 2024-02-10T07:38:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:49 smithi008 ceph-mon[29816]: 6.2 scrub ok 2024-02-10T07:38:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:49 smithi008 ceph-mon[29816]: pgmap v324: 148 pgs: 148 active+clean; 2.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-10T07:38:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:49 smithi008 ceph-mon[29816]: 4.1e deep-scrub starts 2024-02-10T07:38:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:49 smithi008 ceph-mon[40743]: 6.2 scrub starts 2024-02-10T07:38:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:49 smithi008 ceph-mon[40743]: 6.2 scrub ok 2024-02-10T07:38:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:49 smithi008 ceph-mon[40743]: pgmap v324: 148 pgs: 148 active+clean; 2.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-10T07:38:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:49 smithi008 ceph-mon[40743]: 4.1e deep-scrub starts 2024-02-10T07:38:49.594 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:49.630 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:49.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:49.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:49.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:49 smithi170 ceph-mon[33394]: 6.2 scrub starts 2024-02-10T07:38:49.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:49 smithi170 ceph-mon[33394]: 6.2 scrub ok 2024-02-10T07:38:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:49 smithi170 ceph-mon[33394]: pgmap v324: 148 pgs: 148 active+clean; 2.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-10T07:38:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:49 smithi170 ceph-mon[33394]: 4.1e deep-scrub starts 2024-02-10T07:38:49.634 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 23 15 46951 46936 127.509 103.188 0.0100285 0.00783755 2024-02-10T07:38:49.732 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:49.766 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:49.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:49.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:49.777 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:49.868 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:49.921 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:49.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:50.022 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:50.053 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:50.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:50.156 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:50.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:50.197 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:50.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:50.298 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:50.338 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:50.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:50.440 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:50.471 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:50.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:50.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:50.573 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:50.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:50 smithi008 ceph-mon[29816]: 4.1e deep-scrub ok 2024-02-10T07:38:50.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:50 smithi008 ceph-mon[29816]: 5.e scrub starts 2024-02-10T07:38:50.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:50 smithi008 ceph-mon[29816]: 5.e scrub ok 2024-02-10T07:38:50.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:50 smithi008 ceph-mon[29816]: 3.1e scrub starts 2024-02-10T07:38:50.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:50 smithi008 ceph-mon[29816]: 3.1e scrub ok 2024-02-10T07:38:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:50 smithi008 ceph-mon[40743]: 4.1e deep-scrub ok 2024-02-10T07:38:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:50 smithi008 ceph-mon[40743]: 5.e scrub starts 2024-02-10T07:38:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:50 smithi008 ceph-mon[40743]: 5.e scrub ok 2024-02-10T07:38:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:50 smithi008 ceph-mon[40743]: 3.1e scrub starts 2024-02-10T07:38:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:50 smithi008 ceph-mon[40743]: 3.1e scrub ok 2024-02-10T07:38:50.603 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:50.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:50 smithi170 ceph-mon[33394]: 4.1e deep-scrub ok 2024-02-10T07:38:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:50 smithi170 ceph-mon[33394]: 5.e scrub starts 2024-02-10T07:38:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:50 smithi170 ceph-mon[33394]: 5.e scrub ok 2024-02-10T07:38:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:50 smithi170 ceph-mon[33394]: 3.1e scrub starts 2024-02-10T07:38:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:50 smithi170 ceph-mon[33394]: 3.1e scrub ok 2024-02-10T07:38:50.634 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 24 16 48550 48534 126.358 99.875 0.00802153 0.00790892 2024-02-10T07:38:50.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:50.737 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:50.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:50.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:50.838 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:50.868 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:50.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:50.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:50.970 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:51.002 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:51.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:51.104 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:51.135 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:51.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:51.237 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:51.270 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:51.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:51.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:51.365 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:51 smithi008 ceph-mon[29816]: pgmap v325: 148 pgs: 148 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-02-10T07:38:51.365 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:51 smithi008 ceph-mon[29816]: 3.7 deep-scrub starts 2024-02-10T07:38:51.365 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:51 smithi008 ceph-mon[29816]: 3.7 deep-scrub ok 2024-02-10T07:38:51.365 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:51 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3764925250' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:38:51.365 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:51 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:38:51.365 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:51 smithi008 ceph-mon[40743]: pgmap v325: 148 pgs: 148 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-02-10T07:38:51.365 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:51 smithi008 ceph-mon[40743]: 3.7 deep-scrub starts 2024-02-10T07:38:51.365 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:51 smithi008 ceph-mon[40743]: 3.7 deep-scrub ok 2024-02-10T07:38:51.365 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:51 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3764925250' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:38:51.365 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:51 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:38:51.372 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:51.373 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:38:51.422 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:51.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:51.524 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:51.560 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:51.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:51.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:51.575 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:38:51.619 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:38:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:51 smithi170 ceph-mon[33394]: pgmap v325: 148 pgs: 148 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-02-10T07:38:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:51 smithi170 ceph-mon[33394]: 3.7 deep-scrub starts 2024-02-10T07:38:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:51 smithi170 ceph-mon[33394]: 3.7 deep-scrub ok 2024-02-10T07:38:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:51 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3764925250' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:38:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:51 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:38:51.634 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 25 16 50043 50027 125.036 93.3125 0.00577347 0.00799198 2024-02-10T07:38:51.663 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:51.664 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:38:51.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:51.693 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:51.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:51.706 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:38:51.744 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:38:51.787 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:38:51.795 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:51.834 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:38:51.835 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:51.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:51.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:51.871 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:38:51.875 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:38:51.876 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:38:51.876 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:38:51.876 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:38:51.876 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:38:51.876 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:38:51.876 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:38:51.876 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:38:51.938 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:51.977 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:51.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:52.079 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:52.113 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:52.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:52.215 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:52.246 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:52.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:52.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:52.348 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:52.388 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:52.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:52.490 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:52.521 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:52.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[29816]: 6.17 scrub starts 2024-02-10T07:38:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[29816]: 6.17 scrub ok 2024-02-10T07:38:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:38:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:38:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[29816]: osdmap e216: 8 total, 8 up, 8 in 2024-02-10T07:38:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1784650256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:38:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[29816]: osdmap e217: 8 total, 8 up, 8 in 2024-02-10T07:38:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[29816]: pgmap v328: 148 pgs: 148 active+clean; 2.8 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 887 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-02-10T07:38:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[40743]: 6.17 scrub starts 2024-02-10T07:38:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[40743]: 6.17 scrub ok 2024-02-10T07:38:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:38:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:38:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[40743]: osdmap e216: 8 total, 8 up, 8 in 2024-02-10T07:38:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1784650256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:38:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[40743]: osdmap e217: 8 total, 8 up, 8 in 2024-02-10T07:38:52.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:52 smithi008 ceph-mon[40743]: pgmap v328: 148 pgs: 148 active+clean; 2.8 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 887 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-02-10T07:38:52.623 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:52 smithi170 ceph-mon[33394]: 6.17 scrub starts 2024-02-10T07:38:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:52 smithi170 ceph-mon[33394]: 6.17 scrub ok 2024-02-10T07:38:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:52 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:38:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:52 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:38:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:52 smithi170 ceph-mon[33394]: osdmap e216: 8 total, 8 up, 8 in 2024-02-10T07:38:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:52 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1784650256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:38:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:52 smithi170 ceph-mon[33394]: osdmap e217: 8 total, 8 up, 8 in 2024-02-10T07:38:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:52 smithi170 ceph-mon[33394]: pgmap v328: 148 pgs: 148 active+clean; 2.8 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 887 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-02-10T07:38:52.634 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 26 16 51547 51531 123.842 94 0.00344535 0.00806964 2024-02-10T07:38:52.653 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:52.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:52.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:52.755 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:52.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:52.795 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:52.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:52.896 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:52.930 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:52.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:52.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:53.032 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:53.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:53.075 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:53.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:53.177 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:53.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:53.211 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:53.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:53.312 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:53.348 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:53.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:53.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:53.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:38:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:38:53.450 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:53.505 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:53.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:53.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:53.607 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:53.634 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 27 15 53196 53181 123.074 103.125 0.00627851 0.00812041 2024-02-10T07:38:53.644 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:53.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:53.746 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:53.779 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:53.786 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:53.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:53.888 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:53.928 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:53.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:54.031 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:54.054 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:53 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:54.054 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:53 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:54.063 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:54.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:54.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:53 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:38:54.164 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:54.203 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:38:54.205 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:54.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:54.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:54.309 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:54.342 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:54.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:54.444 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:54.477 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:54.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:54.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:54.579 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:54.612 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:54.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:54.634 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 28 16 54601 54585 121.813 87.75 0.00347034 0.00820248 2024-02-10T07:38:54.714 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:54.754 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:54.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:54.857 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:54.904 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:54.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:54.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:55.006 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:55.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:55.051 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:55.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:55.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:54 smithi008 ceph-mon[29816]: pgmap v329: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 147 MiB/s wr, 2.36k op/s 2024-02-10T07:38:55.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:54 smithi008 ceph-mon[40743]: pgmap v329: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 147 MiB/s wr, 2.36k op/s 2024-02-10T07:38:55.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:54 smithi170 ceph-mon[33394]: pgmap v329: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 147 MiB/s wr, 2.36k op/s 2024-02-10T07:38:55.152 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:55.207 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:55.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:55.308 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:55.343 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:55.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:55.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:55.445 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:55.455 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:38:55.455 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:38:55.455 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-02-10T07:38:55.455 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-02-10T07:38:55.455 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:38:55.474 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:55.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:55.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:55.576 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:55.614 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:55.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:55.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:55.635 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 29 16 56050 56034 120.735 90.5625 0.0112331 0.00827815 2024-02-10T07:38:55.716 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:55.745 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:55.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:55.825 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:38:55.847 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:55.920 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:55.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:56.021 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:56.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:56.075 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:56.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:56.176 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:56.214 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:56.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:56.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:56.316 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:56.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:56.364 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:56.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:56.465 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:56.498 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:56.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:56.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:56.599 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:56.634 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 30 16 57526 57510 119.785 92.25 0.0107659 0.00834248 2024-02-10T07:38:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:56.639 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:56.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:56.740 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:56.782 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:56.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:56.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:56.874 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-10T07:38:56.884 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:56.966 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:56.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:56.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:57.067 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:57.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:56 smithi008 ceph-mon[29816]: pgmap v330: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-02-10T07:38:57.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:56 smithi008 ceph-mon[40743]: pgmap v330: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-02-10T07:38:57.101 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:57.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:57.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:57.203 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:57.230 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:57.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:57.331 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:57.377 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:57.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:57.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:57.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:56 smithi170 ceph-mon[33394]: pgmap v330: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-02-10T07:38:57.480 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:57.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:57.512 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:57.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:57.557 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:57.613 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:57.634 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:38:57.635 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 31 16 58828 58812 118.544 81.375 0.00494046 0.00842945 2024-02-10T07:38:57.640 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:57.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:57.742 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:38:57.793 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:57.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:57.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:38:57.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:38:57.923 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:57.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:38:57.951 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:38:57.965 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:38:57.985 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:57 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:38:57.986 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:57 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:38:57.986 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:57 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:38:57.986 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:57 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:38:58.026 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:38:58.062 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:58.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:38:58.164 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:58.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:58.205 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:58.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:58.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:57 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:38:58.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:57 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:38:58.306 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:58.337 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:58.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:58.439 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:58.477 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:58.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:58.579 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:58.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:58.611 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:58.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:58.636 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 32 16 60220 60204 117.558 87 0.00645635 0.00850154 2024-02-10T07:38:58.712 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:58.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:58.751 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:58.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:58.852 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:58.890 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:58.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:58.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:58.992 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:59.032 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:59.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:59.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:38:59.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:38:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:38:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:38:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:58 smithi008 ceph-mon[29816]: pgmap v331: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-02-10T07:38:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:58 smithi008 ceph-mon[29816]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:38:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:58 smithi008 ceph-mon[29816]: osdmap e218: 8 total, 8 up, 8 in 2024-02-10T07:38:59.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:58 smithi008 ceph-mon[40743]: pgmap v331: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-02-10T07:38:59.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:58 smithi008 ceph-mon[40743]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:38:59.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:58 smithi008 ceph-mon[40743]: osdmap e218: 8 total, 8 up, 8 in 2024-02-10T07:38:59.134 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:59.170 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:59.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:59.272 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:38:59.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:59.308 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:59.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:38:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:58 smithi170 ceph-mon[33394]: pgmap v331: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-02-10T07:38:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:58 smithi170 ceph-mon[33394]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:38:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:58 smithi170 ceph-mon[33394]: osdmap e218: 8 total, 8 up, 8 in 2024-02-10T07:38:59.409 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:38:59.443 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:59.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:59.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:38:59.545 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:59.581 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:59.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:59.636 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 33 16 61767 61751 116.925 96.6875 0.00602882 0.00854702 2024-02-10T07:38:59.683 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:38:59.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:59.715 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:38:59.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:38:59.816 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:38:59.852 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:59.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:59.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:38:59.954 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:38:59.995 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:38:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:38:59.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:00.097 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:00.138 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:00.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:00.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:00.206 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:59 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:39:00.206 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:59 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:39:00.206 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:38:59 smithi008 ceph-mon[29816]: osdmap e219: 8 total, 8 up, 8 in 2024-02-10T07:39:00.207 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:59 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:39:00.207 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:59 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:39:00.207 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:38:59 smithi008 ceph-mon[40743]: osdmap e219: 8 total, 8 up, 8 in 2024-02-10T07:39:00.239 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:00.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:00.276 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:00.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:00.378 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:59 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:39:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:59 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:39:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:38:59 smithi170 ceph-mon[33394]: osdmap e219: 8 total, 8 up, 8 in 2024-02-10T07:39:00.411 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:00.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:00.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:00.513 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:00.547 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:00.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:00.632 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:00.636 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 34 16 63254 63238 116.219 92.9375 0.0178269 0.00859904 2024-02-10T07:39:00.649 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:00.677 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:00.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:00.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:00.694 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-02-10T07:38:57.902127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3591379292,"num_objects":54989,"num_object_clones":0,"num_object_copies":164967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54989,"num_whiteouts":0,"num_read":1350,"num_read_kb":1196,"num_write":55406,"num_write_kb":3509085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":1772204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55450,"ondisk_log_size":55450,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12239820,"kb_used_data":10536444,"kb_used_omap":15,"kb_used_meta":1703280,"kb_avail":737721396,"statfs":{"total":767960285184,"available":755426709504,"internally_reserved":0,"allocated":10789318656,"data_stored":10786080836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15602,"internal_metadata":1744159502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,3,3,4,8,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":709820416,"num_objects":10831,"num_object_clones":0,"num_object_copies":32493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10831,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10831,"num_write_kb":693184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.554741"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":334,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.774599+0000","last_change":"2024-02-10T07:38:45.701833+0000","last_active":"2024-02-10T07:38:51.774599+0000","last_peered":"2024-02-10T07:38:51.774599+0000","last_clean":"2024-02-10T07:38:51.774599+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:38:51.774599+0000","last_undegraded":"2024-02-10T07:38:51.774599+0000","last_fullsized":"2024-02-10T07:38:51.774599+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:45.701628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:45.701628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:46.337219+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":336,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.786649+0000","last_change":"2024-02-10T07:38:34.709543+0000","last_active":"2024-02-10T07:38:51.786649+0000","last_peered":"2024-02-10T07:38:51.786649+0000","last_clean":"2024-02-10T07:38:51.786649+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:38:51.786649+0000","last_undegraded":"2024-02-10T07:38:51.786649+0000","last_fullsized":"2024-02-10T07:38:51.786649+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:34.709422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:34.709422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:41.456368+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":352,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.781172+0000","last_change":"2024-02-10T07:38:29.585901+0000","last_active":"2024-02-10T07:38:51.781172+0000","last_peered":"2024-02-10T07:38:51.781172+0000","last_clean":"2024-02-10T07:38:51.781172+0000","last_became_active":"2024-02-10T07:36:33.760016+0000","last_became_peered":"2024-02-10T07:36:33.760016+0000","last_unstale":"2024-02-10T07:38:51.781172+0000","last_undegraded":"2024-02-10T07:38:51.781172+0000","last_fullsized":"2024-02-10T07:38:51.781172+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:38:29.585764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:29.585764+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:50.749980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":365,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.774255+0000","last_change":"2024-02-10T07:38:34.797625+0000","last_active":"2024-02-10T07:38:51.774255+0000","last_peered":"2024-02-10T07:38:51.774255+0000","last_clean":"2024-02-10T07:38:51.774255+0000","last_became_active":"2024-02-10T07:38:34.790379+0000","last_became_peered":"2024-02-10T07:38:34.790379+0000","last_unstale":"2024-02-10T07:38:51.774255+0000","last_undegraded":"2024-02-10T07:38:51.774255+0000","last_fullsized":"2024-02-10T07:38:51.774255+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:37:41.734697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:41.734697+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":325,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779985+0000","last_change":"2024-02-10T07:38:24.207588+0000","last_active":"2024-02-10T07:38:51.779985+0000","last_peered":"2024-02-10T07:38:51.779985+0000","last_clean":"2024-02-10T07:38:51.779985+0000","last_became_active":"2024-02-10T07:31:57.300056+0000","last_became_peered":"2024-02-10T07:31:57.300056+0000","last_unstale":"2024-02-10T07:38:51.779985+0000","last_undegraded":"2024-02-10T07:38:51.779985+0000","last_fullsized":"2024-02-10T07:38:51.779985+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:24.207501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:24.207501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:40.217247+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":329,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.774198+0000","last_change":"2024-02-10T07:37:29.763145+0000","last_active":"2024-02-10T07:38:51.774198+0000","last_peered":"2024-02-10T07:38:51.774198+0000","last_clean":"2024-02-10T07:38:51.774198+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:38:51.774198+0000","last_undegraded":"2024-02-10T07:38:51.774198+0000","last_fullsized":"2024-02-10T07:38:51.774198+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:29.763023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:29.763023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779005+0000","last_change":"2024-02-10T07:38:49.306882+0000","last_active":"2024-02-10T07:38:51.779005+0000","last_peered":"2024-02-10T07:38:51.779005+0000","last_clean":"2024-02-10T07:38:51.779005+0000","last_became_active":"2024-02-10T07:31:51.323331+0000","last_became_peered":"2024-02-10T07:31:51.323331+0000","last_unstale":"2024-02-10T07:38:51.779005+0000","last_undegraded":"2024-02-10T07:38:51.779005+0000","last_fullsized":"2024-02-10T07:38:51.779005+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:49.306769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:38:49.306769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:55.488849+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":358,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779471+0000","last_change":"2024-02-10T07:37:32.922146+0000","last_active":"2024-02-10T07:38:51.779471+0000","last_peered":"2024-02-10T07:38:51.779471+0000","last_clean":"2024-02-10T07:38:51.779471+0000","last_became_active":"2024-02-10T07:31:53.286957+0000","last_became_peered":"2024-02-10T07:31:53.286957+0000","last_unstale":"2024-02-10T07:38:51.779471+0000","last_undegraded":"2024-02-10T07:38:51.779471+0000","last_fullsized":"2024-02-10T07:38:51.779471+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:37:32.921838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:32.921838+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:02.913956+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":324,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.776629+0000","last_change":"2024-02-10T07:38:32.551419+0000","last_active":"2024-02-10T07:38:51.776629+0000","last_peered":"2024-02-10T07:38:51.776629+0000","last_clean":"2024-02-10T07:38:51.776629+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:38:51.776629+0000","last_undegraded":"2024-02-10T07:38:51.776629+0000","last_fullsized":"2024-02-10T07:38:51.776629+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:32.551293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:32.551293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:51.987665+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":297,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780113+0000","last_change":"2024-02-10T07:38:43.221155+0000","last_active":"2024-02-10T07:38:51.780113+0000","last_peered":"2024-02-10T07:38:51.780113+0000","last_clean":"2024-02-10T07:38:51.780113+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:38:51.780113+0000","last_undegraded":"2024-02-10T07:38:51.780113+0000","last_fullsized":"2024-02-10T07:38:51.780113+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:43.221050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:43.221050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:09.732828+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":278,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779866+0000","last_change":"2024-02-10T07:38:24.115694+0000","last_active":"2024-02-10T07:38:51.779866+0000","last_peered":"2024-02-10T07:38:51.779866+0000","last_clean":"2024-02-10T07:38:51.779866+0000","last_became_active":"2024-02-10T07:36:33.771340+0000","last_became_peered":"2024-02-10T07:36:33.771340+0000","last_unstale":"2024-02-10T07:38:51.779866+0000","last_undegraded":"2024-02-10T07:38:51.779866+0000","last_fullsized":"2024-02-10T07:38:51.779866+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:38:24.115550+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:34:49.349360+0000","last_clean_scrub_stamp":"2024-02-10T07:38:24.115550+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:50.858815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":349,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.775238+0000","last_change":"2024-02-10T07:37:49.609721+0000","last_active":"2024-02-10T07:38:51.775238+0000","last_peered":"2024-02-10T07:38:51.775238+0000","last_clean":"2024-02-10T07:38:51.775238+0000","last_became_active":"2024-02-10T07:31:53.287858+0000","last_became_peered":"2024-02-10T07:31:53.287858+0000","last_unstale":"2024-02-10T07:38:51.775238+0000","last_undegraded":"2024-02-10T07:38:51.775238+0000","last_fullsized":"2024-02-10T07:38:51.775238+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:37:49.609631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:49.609631+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:01.288487+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":290,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779392+0000","last_change":"2024-02-10T07:37:21.968926+0000","last_active":"2024-02-10T07:38:51.779392+0000","last_peered":"2024-02-10T07:38:51.779392+0000","last_clean":"2024-02-10T07:38:51.779392+0000","last_became_active":"2024-02-10T07:36:33.757073+0000","last_became_peered":"2024-02-10T07:36:33.757073+0000","last_unstale":"2024-02-10T07:38:51.779392+0000","last_undegraded":"2024-02-10T07:38:51.779392+0000","last_fullsized":"2024-02-10T07:38:51.779392+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:21.968825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:21.968825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:59.155372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":329,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.783595+0000","last_change":"2024-02-10T07:38:28.455565+0000","last_active":"2024-02-10T07:38:51.783595+0000","last_peered":"2024-02-10T07:38:51.783595+0000","last_clean":"2024-02-10T07:38:51.783595+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:38:51.783595+0000","last_undegraded":"2024-02-10T07:38:51.783595+0000","last_fullsized":"2024-02-10T07:38:51.783595+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:28.455461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:28.455461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:37.658639+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":301,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779781+0000","last_change":"2024-02-10T07:38:35.189838+0000","last_active":"2024-02-10T07:38:51.779781+0000","last_peered":"2024-02-10T07:38:51.779781+0000","last_clean":"2024-02-10T07:38:51.779781+0000","last_became_active":"2024-02-10T07:36:33.767970+0000","last_became_peered":"2024-02-10T07:36:33.767970+0000","last_unstale":"2024-02-10T07:38:51.779781+0000","last_undegraded":"2024-02-10T07:38:51.779781+0000","last_fullsized":"2024-02-10T07:38:51.779781+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:35.189720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:35.189720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:01.666908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":343,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.775712+0000","last_change":"2024-02-10T07:38:37.597887+0000","last_active":"2024-02-10T07:38:51.775712+0000","last_peered":"2024-02-10T07:38:51.775712+0000","last_clean":"2024-02-10T07:38:51.775712+0000","last_became_active":"2024-02-10T07:31:53.292417+0000","last_became_peered":"2024-02-10T07:31:53.292417+0000","last_unstale":"2024-02-10T07:38:51.775712+0000","last_undegraded":"2024-02-10T07:38:51.775712+0000","last_fullsized":"2024-02-10T07:38:51.775712+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-10T07:38:37.597762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:37.597762+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:59.682048+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":335,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.775224+0000","last_change":"2024-02-10T07:37:28.737774+0000","last_active":"2024-02-10T07:38:51.775224+0000","last_peered":"2024-02-10T07:38:51.775224+0000","last_clean":"2024-02-10T07:38:51.775224+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:38:51.775224+0000","last_undegraded":"2024-02-10T07:38:51.775224+0000","last_fullsized":"2024-02-10T07:38:51.775224+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:28.737622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:28.737622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":327,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.785734+0000","last_change":"2024-02-10T07:38:29.698707+0000","last_active":"2024-02-10T07:38:51.785734+0000","last_peered":"2024-02-10T07:38:51.785734+0000","last_clean":"2024-02-10T07:38:51.785734+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:38:51.785734+0000","last_undegraded":"2024-02-10T07:38:51.785734+0000","last_fullsized":"2024-02-10T07:38:51.785734+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:29.698581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:29.698581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:47.178090+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":336,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.777667+0000","last_change":"2024-02-10T07:38:38.528786+0000","last_active":"2024-02-10T07:38:51.777667+0000","last_peered":"2024-02-10T07:38:51.777667+0000","last_clean":"2024-02-10T07:38:51.777667+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:38:51.777667+0000","last_undegraded":"2024-02-10T07:38:51.777667+0000","last_fullsized":"2024-02-10T07:38:51.777667+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:38.528710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:38:38.528710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:48.796543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"89'22","reported_seq":364,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.782372+0000","last_change":"2024-02-10T07:38:24.669466+0000","last_active":"2024-02-10T07:38:51.782372+0000","last_peered":"2024-02-10T07:38:51.782372+0000","last_clean":"2024-02-10T07:38:51.782372+0000","last_became_active":"2024-02-10T07:31:53.295527+0000","last_became_peered":"2024-02-10T07:31:53.295527+0000","last_unstale":"2024-02-10T07:38:51.782372+0000","last_undegraded":"2024-02-10T07:38:51.782372+0000","last_fullsized":"2024-02-10T07:38:51.782372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'22","last_scrub_stamp":"2024-02-10T07:38:24.669327+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:38:24.669327+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:54.083623+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":327,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780325+0000","last_change":"2024-02-10T07:38:22.233892+0000","last_active":"2024-02-10T07:38:51.780325+0000","last_peered":"2024-02-10T07:38:51.780325+0000","last_clean":"2024-02-10T07:38:51.780325+0000","last_became_active":"2024-02-10T07:36:33.754341+0000","last_became_peered":"2024-02-10T07:36:33.754341+0000","last_unstale":"2024-02-10T07:38:51.780325+0000","last_undegraded":"2024-02-10T07:38:51.780325+0000","last_fullsized":"2024-02-10T07:38:51.780325+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:22.233765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:22.233765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:34.922632+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":329,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.774935+0000","last_change":"2024-02-10T07:37:33.771341+0000","last_active":"2024-02-10T07:38:51.774935+0000","last_peered":"2024-02-10T07:38:51.774935+0000","last_clean":"2024-02-10T07:38:51.774935+0000","last_became_active":"2024-02-10T07:31:55.298474+0000","last_became_peered":"2024-02-10T07:31:55.298474+0000","last_unstale":"2024-02-10T07:38:51.774935+0000","last_undegraded":"2024-02-10T07:38:51.774935+0000","last_fullsized":"2024-02-10T07:38:51.774935+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:33.771236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:27.368799+0000","last_clean_scrub_stamp":"2024-02-10T07:37:33.771236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":345,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.775322+0000","last_change":"2024-02-10T07:38:42.693951+0000","last_active":"2024-02-10T07:38:51.775322+0000","last_peered":"2024-02-10T07:38:51.775322+0000","last_clean":"2024-02-10T07:38:51.775322+0000","last_became_active":"2024-02-10T07:31:51.321237+0000","last_became_peered":"2024-02-10T07:31:51.321237+0000","last_unstale":"2024-02-10T07:38:51.775322+0000","last_undegraded":"2024-02-10T07:38:51.775322+0000","last_fullsized":"2024-02-10T07:38:51.775322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:42.693827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:42.693827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:46.456969+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":356,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.778964+0000","last_change":"2024-02-10T07:38:44.274459+0000","last_active":"2024-02-10T07:38:51.778964+0000","last_peered":"2024-02-10T07:38:51.778964+0000","last_clean":"2024-02-10T07:38:51.778964+0000","last_became_active":"2024-02-10T07:31:53.294914+0000","last_became_peered":"2024-02-10T07:31:53.294914+0000","last_unstale":"2024-02-10T07:38:51.778964+0000","last_undegraded":"2024-02-10T07:38:51.778964+0000","last_fullsized":"2024-02-10T07:38:51.778964+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:38:44.274331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:44.274331+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:59.882292+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":327,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.784585+0000","last_change":"2024-02-10T07:36:47.829905+0000","last_active":"2024-02-10T07:38:51.784585+0000","last_peered":"2024-02-10T07:38:51.784585+0000","last_clean":"2024-02-10T07:38:51.784585+0000","last_became_active":"2024-02-10T07:36:33.751352+0000","last_became_peered":"2024-02-10T07:36:33.751352+0000","last_unstale":"2024-02-10T07:38:51.784585+0000","last_undegraded":"2024-02-10T07:38:51.784585+0000","last_fullsized":"2024-02-10T07:38:51.784585+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:36:47.829690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:36:47.829690+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":434,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.778941+0000","last_change":"2024-02-10T07:37:47.826041+0000","last_active":"2024-02-10T07:38:51.778941+0000","last_peered":"2024-02-10T07:38:51.778941+0000","last_clean":"2024-02-10T07:38:51.778941+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:38:51.778941+0000","last_undegraded":"2024-02-10T07:38:51.778941+0000","last_fullsized":"2024-02-10T07:38:51.778941+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:37:47.825964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:47.825964+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:07.048168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":336,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.786765+0000","last_change":"2024-02-10T07:38:23.669717+0000","last_active":"2024-02-10T07:38:51.786765+0000","last_peered":"2024-02-10T07:38:51.786765+0000","last_clean":"2024-02-10T07:38:51.786765+0000","last_became_active":"2024-02-10T07:31:51.282077+0000","last_became_peered":"2024-02-10T07:31:51.282077+0000","last_unstale":"2024-02-10T07:38:51.786765+0000","last_undegraded":"2024-02-10T07:38:51.786765+0000","last_fullsized":"2024-02-10T07:38:51.786765+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:38:23.669587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:23.669587+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:24.629043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"215'15","reported_seq":361,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.776581+0000","last_change":"2024-02-10T07:38:48.605358+0000","last_active":"2024-02-10T07:38:51.776581+0000","last_peered":"2024-02-10T07:38:51.776581+0000","last_clean":"2024-02-10T07:38:51.776581+0000","last_became_active":"2024-02-10T07:31:53.292936+0000","last_became_peered":"2024-02-10T07:31:53.292936+0000","last_unstale":"2024-02-10T07:38:51.776581+0000","last_undegraded":"2024-02-10T07:38:51.776581+0000","last_fullsized":"2024-02-10T07:38:51.776581+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'15","last_scrub_stamp":"2024-02-10T07:38:48.605224+0000","last_deep_scrub":"215'15","last_deep_scrub_stamp":"2024-02-10T07:38:48.605224+0000","last_clean_scrub_stamp":"2024-02-10T07:38:48.605224+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:16.926968+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":332,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.785827+0000","last_change":"2024-02-10T07:38:26.689435+0000","last_active":"2024-02-10T07:38:51.785827+0000","last_peered":"2024-02-10T07:38:51.785827+0000","last_clean":"2024-02-10T07:38:51.785827+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:38:51.785827+0000","last_undegraded":"2024-02-10T07:38:51.785827+0000","last_fullsized":"2024-02-10T07:38:51.785827+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:26.688895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:26.688895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:48.373320+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":329,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.776822+0000","last_change":"2024-02-10T07:38:39.569860+0000","last_active":"2024-02-10T07:38:51.776822+0000","last_peered":"2024-02-10T07:38:51.776822+0000","last_clean":"2024-02-10T07:38:51.776822+0000","last_became_active":"2024-02-10T07:31:55.297595+0000","last_became_peered":"2024-02-10T07:31:55.297595+0000","last_unstale":"2024-02-10T07:38:51.776822+0000","last_undegraded":"2024-02-10T07:38:51.776822+0000","last_fullsized":"2024-02-10T07:38:51.776822+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:39.569633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:39.569633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:55.253316+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":337,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.782960+0000","last_change":"2024-02-10T07:38:30.226232+0000","last_active":"2024-02-10T07:38:51.782960+0000","last_peered":"2024-02-10T07:38:51.782960+0000","last_clean":"2024-02-10T07:38:51.782960+0000","last_became_active":"2024-02-10T07:31:51.285405+0000","last_became_peered":"2024-02-10T07:31:51.285405+0000","last_unstale":"2024-02-10T07:38:51.782960+0000","last_undegraded":"2024-02-10T07:38:51.782960+0000","last_fullsized":"2024-02-10T07:38:51.782960+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:30.226109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:30.226109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:53.772743+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":361,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.778816+0000","last_change":"2024-02-10T07:37:23.942006+0000","last_active":"2024-02-10T07:38:51.778816+0000","last_peered":"2024-02-10T07:38:51.778816+0000","last_clean":"2024-02-10T07:38:51.778816+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:38:51.778816+0000","last_undegraded":"2024-02-10T07:38:51.778816+0000","last_fullsized":"2024-02-10T07:38:51.778816+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:37:23.941920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:23.941920+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":326,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.783573+0000","last_change":"2024-02-10T07:38:38.475429+0000","last_active":"2024-02-10T07:38:51.783573+0000","last_peered":"2024-02-10T07:38:51.783573+0000","last_clean":"2024-02-10T07:38:51.783573+0000","last_became_active":"2024-02-10T07:31:57.306776+0000","last_became_peered":"2024-02-10T07:31:57.306776+0000","last_unstale":"2024-02-10T07:38:51.783573+0000","last_undegraded":"2024-02-10T07:38:51.783573+0000","last_fullsized":"2024-02-10T07:38:51.783573+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:38:38.475358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:41.411734+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":337,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.778242+0000","last_change":"2024-02-10T07:38:39.864410+0000","last_active":"2024-02-10T07:38:51.778242+0000","last_peered":"2024-02-10T07:38:51.778242+0000","last_clean":"2024-02-10T07:38:51.778242+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:38:51.778242+0000","last_undegraded":"2024-02-10T07:38:51.778242+0000","last_fullsized":"2024-02-10T07:38:51.778242+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:39.864301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:39.864301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:03.775465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":345,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.781361+0000","last_change":"2024-02-10T07:37:24.433655+0000","last_active":"2024-02-10T07:38:51.781361+0000","last_peered":"2024-02-10T07:38:51.781361+0000","last_clean":"2024-02-10T07:38:51.781361+0000","last_became_active":"2024-02-10T07:36:33.760213+0000","last_became_peered":"2024-02-10T07:36:33.760213+0000","last_unstale":"2024-02-10T07:38:51.781361+0000","last_undegraded":"2024-02-10T07:38:51.781361+0000","last_fullsized":"2024-02-10T07:38:51.781361+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:24.433491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:24.433491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:56.578495+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"95'6","reported_seq":334,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.783058+0000","last_change":"2024-02-10T07:37:22.000087+0000","last_active":"2024-02-10T07:38:51.783058+0000","last_peered":"2024-02-10T07:38:51.783058+0000","last_clean":"2024-02-10T07:38:51.783058+0000","last_became_active":"2024-02-10T07:31:53.295520+0000","last_became_peered":"2024-02-10T07:31:53.295520+0000","last_unstale":"2024-02-10T07:38:51.783058+0000","last_undegraded":"2024-02-10T07:38:51.783058+0000","last_fullsized":"2024-02-10T07:38:51.783058+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'6","last_scrub_stamp":"2024-02-10T07:37:21.999818+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:37:21.999818+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":317,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780149+0000","last_change":"2024-02-10T07:36:25.155652+0000","last_active":"2024-02-10T07:38:51.780149+0000","last_peered":"2024-02-10T07:38:51.780149+0000","last_clean":"2024-02-10T07:38:51.780149+0000","last_became_active":"2024-02-10T07:31:57.315738+0000","last_became_peered":"2024-02-10T07:31:57.315738+0000","last_unstale":"2024-02-10T07:38:51.780149+0000","last_undegraded":"2024-02-10T07:38:51.780149+0000","last_fullsized":"2024-02-10T07:38:51.780149+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:25.155547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:36:25.155547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":330,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.783870+0000","last_change":"2024-02-10T07:38:31.441903+0000","last_active":"2024-02-10T07:38:51.783870+0000","last_peered":"2024-02-10T07:38:51.783870+0000","last_clean":"2024-02-10T07:38:51.783870+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:38:51.783870+0000","last_undegraded":"2024-02-10T07:38:51.783870+0000","last_fullsized":"2024-02-10T07:38:51.783870+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:31.441795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:31.441795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:52.264706+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":262,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779642+0000","last_change":"2024-02-10T07:37:40.838525+0000","last_active":"2024-02-10T07:38:51.779642+0000","last_peered":"2024-02-10T07:38:51.779642+0000","last_clean":"2024-02-10T07:38:51.779642+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:38:51.779642+0000","last_undegraded":"2024-02-10T07:38:51.779642+0000","last_fullsized":"2024-02-10T07:38:51.779642+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:40.838446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:37:40.838446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:02.539369+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":353,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779421+0000","last_change":"2024-02-10T07:37:47.175823+0000","last_active":"2024-02-10T07:38:51.779421+0000","last_peered":"2024-02-10T07:38:51.779421+0000","last_clean":"2024-02-10T07:38:51.779421+0000","last_became_active":"2024-02-10T07:31:53.287823+0000","last_became_peered":"2024-02-10T07:31:53.287823+0000","last_unstale":"2024-02-10T07:38:51.779421+0000","last_undegraded":"2024-02-10T07:38:51.779421+0000","last_fullsized":"2024-02-10T07:38:51.779421+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-10T07:37:47.175681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:47.175681+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:03.100368+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":325,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.776224+0000","last_change":"2024-02-10T07:38:25.610309+0000","last_active":"2024-02-10T07:38:51.776224+0000","last_peered":"2024-02-10T07:38:51.776224+0000","last_clean":"2024-02-10T07:38:51.776224+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:38:51.776224+0000","last_undegraded":"2024-02-10T07:38:51.776224+0000","last_fullsized":"2024-02-10T07:38:51.776224+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:25.610132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:25.610132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:46.400503+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":329,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779909+0000","last_change":"2024-02-10T07:38:25.154667+0000","last_active":"2024-02-10T07:38:51.779909+0000","last_peered":"2024-02-10T07:38:51.779909+0000","last_clean":"2024-02-10T07:38:51.779909+0000","last_became_active":"2024-02-10T07:31:55.296670+0000","last_became_peered":"2024-02-10T07:31:55.296670+0000","last_unstale":"2024-02-10T07:38:51.779909+0000","last_undegraded":"2024-02-10T07:38:51.779909+0000","last_fullsized":"2024-02-10T07:38:51.779909+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:25.154556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:25.154556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:45.636749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":337,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.786958+0000","last_change":"2024-02-10T07:37:24.647379+0000","last_active":"2024-02-10T07:38:51.786958+0000","last_peered":"2024-02-10T07:38:51.786958+0000","last_clean":"2024-02-10T07:38:51.786958+0000","last_became_active":"2024-02-10T07:31:51.288165+0000","last_became_peered":"2024-02-10T07:31:51.288165+0000","last_unstale":"2024-02-10T07:38:51.786958+0000","last_undegraded":"2024-02-10T07:38:51.786958+0000","last_fullsized":"2024-02-10T07:38:51.786958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:24.647260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:24.647260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":117,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.784816+0000","last_change":"2024-02-10T07:38:20.668585+0000","last_active":"2024-02-10T07:38:51.784816+0000","last_peered":"2024-02-10T07:38:51.784816+0000","last_clean":"2024-02-10T07:38:51.784816+0000","last_became_active":"2024-02-10T07:37:06.187551+0000","last_became_peered":"2024-02-10T07:37:06.187551+0000","last_unstale":"2024-02-10T07:38:51.784816+0000","last_undegraded":"2024-02-10T07:38:51.784816+0000","last_fullsized":"2024-02-10T07:38:51.784816+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:38:20.668498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:20.668498+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:21.641880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":326,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779080+0000","last_change":"2024-02-10T07:37:20.031409+0000","last_active":"2024-02-10T07:38:51.779080+0000","last_peered":"2024-02-10T07:38:51.779080+0000","last_clean":"2024-02-10T07:38:51.779080+0000","last_became_active":"2024-02-10T07:36:33.750030+0000","last_became_peered":"2024-02-10T07:36:33.750030+0000","last_unstale":"2024-02-10T07:38:51.779080+0000","last_undegraded":"2024-02-10T07:38:51.779080+0000","last_fullsized":"2024-02-10T07:38:51.779080+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:20.031295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:20.031295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":336,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.787391+0000","last_change":"2024-02-10T07:38:24.627406+0000","last_active":"2024-02-10T07:38:51.787391+0000","last_peered":"2024-02-10T07:38:51.787391+0000","last_clean":"2024-02-10T07:38:51.787391+0000","last_became_active":"2024-02-10T07:36:33.752418+0000","last_became_peered":"2024-02-10T07:36:33.752418+0000","last_unstale":"2024-02-10T07:38:51.787391+0000","last_undegraded":"2024-02-10T07:38:51.787391+0000","last_fullsized":"2024-02-10T07:38:51.787391+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:24.627321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:24.627321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:46.364817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.774675+0000","last_change":"2024-02-10T07:37:40.763596+0000","last_active":"2024-02-10T07:38:51.774675+0000","last_peered":"2024-02-10T07:38:51.774675+0000","last_clean":"2024-02-10T07:38:51.774675+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:38:51.774675+0000","last_undegraded":"2024-02-10T07:38:51.774675+0000","last_fullsized":"2024-02-10T07:38:51.774675+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:40.763490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:40.763490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"120'16","reported_seq":376,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.775659+0000","last_change":"2024-02-10T07:36:55.866629+0000","last_active":"2024-02-10T07:38:51.775659+0000","last_peered":"2024-02-10T07:38:51.775659+0000","last_clean":"2024-02-10T07:38:51.775659+0000","last_became_active":"2024-02-10T07:32:18.389564+0000","last_became_peered":"2024-02-10T07:32:18.389564+0000","last_unstale":"2024-02-10T07:38:51.775659+0000","last_undegraded":"2024-02-10T07:38:51.775659+0000","last_fullsized":"2024-02-10T07:38:51.775659+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'16","last_scrub_stamp":"2024-02-10T07:36:55.866526+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:36:55.866526+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":324,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.776265+0000","last_change":"2024-02-10T07:38:24.598544+0000","last_active":"2024-02-10T07:38:51.776265+0000","last_peered":"2024-02-10T07:38:51.776265+0000","last_clean":"2024-02-10T07:38:51.776265+0000","last_became_active":"2024-02-10T07:31:57.318009+0000","last_became_peered":"2024-02-10T07:31:57.318009+0000","last_unstale":"2024-02-10T07:38:51.776265+0000","last_undegraded":"2024-02-10T07:38:51.776265+0000","last_fullsized":"2024-02-10T07:38:51.776265+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:24.598439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:38:24.598439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:50.791442+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"158'10","reported_seq":341,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:56.453492+0000","last_change":"2024-02-10T07:37:49.768451+0000","last_active":"2024-02-10T07:38:56.453492+0000","last_peered":"2024-02-10T07:38:56.453492+0000","last_clean":"2024-02-10T07:38:56.453492+0000","last_became_active":"2024-02-10T07:36:33.774773+0000","last_became_peered":"2024-02-10T07:36:33.774773+0000","last_unstale":"2024-02-10T07:38:56.453492+0000","last_undegraded":"2024-02-10T07:38:56.453492+0000","last_fullsized":"2024-02-10T07:38:56.453492+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:37:49.768327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:49.768327+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:13.077119+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.784417+0000","last_change":"2024-02-10T07:36:52.846442+0000","last_active":"2024-02-10T07:38:51.784417+0000","last_peered":"2024-02-10T07:38:51.784417+0000","last_clean":"2024-02-10T07:38:51.784417+0000","last_became_active":"2024-02-10T07:31:51.320644+0000","last_became_peered":"2024-02-10T07:31:51.320644+0000","last_unstale":"2024-02-10T07:38:51.784417+0000","last_undegraded":"2024-02-10T07:38:51.784417+0000","last_fullsized":"2024-02-10T07:38:51.784417+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:52.846359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:36:52.846359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"96'15","reported_seq":164,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779599+0000","last_change":"2024-02-10T07:37:26.933801+0000","last_active":"2024-02-10T07:38:51.779599+0000","last_peered":"2024-02-10T07:38:51.779599+0000","last_clean":"2024-02-10T07:38:51.779599+0000","last_became_active":"2024-02-10T07:36:33.752317+0000","last_became_peered":"2024-02-10T07:36:33.752317+0000","last_unstale":"2024-02-10T07:38:51.779599+0000","last_undegraded":"2024-02-10T07:38:51.779599+0000","last_fullsized":"2024-02-10T07:38:51.779599+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'15","last_scrub_stamp":"2024-02-10T07:37:26.933710+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:37:26.933710+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":334,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.775352+0000","last_change":"2024-02-10T07:38:50.625812+0000","last_active":"2024-02-10T07:38:51.775352+0000","last_peered":"2024-02-10T07:38:51.775352+0000","last_clean":"2024-02-10T07:38:51.775352+0000","last_became_active":"2024-02-10T07:36:33.766051+0000","last_became_peered":"2024-02-10T07:36:33.766051+0000","last_unstale":"2024-02-10T07:38:51.775352+0000","last_undegraded":"2024-02-10T07:38:51.775352+0000","last_fullsized":"2024-02-10T07:38:51.775352+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:50.625690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:50.625690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:08.542150+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":414,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779688+0000","last_change":"2024-02-10T07:36:30.059099+0000","last_active":"2024-02-10T07:38:51.779688+0000","last_peered":"2024-02-10T07:38:51.779688+0000","last_clean":"2024-02-10T07:38:51.779688+0000","last_became_active":"2024-02-10T07:31:55.294594+0000","last_became_peered":"2024-02-10T07:31:55.294594+0000","last_unstale":"2024-02-10T07:38:51.779688+0000","last_undegraded":"2024-02-10T07:38:51.779688+0000","last_fullsized":"2024-02-10T07:38:51.779688+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:36:30.058979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:36:30.058979+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":337,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780903+0000","last_change":"2024-02-10T07:37:43.502899+0000","last_active":"2024-02-10T07:38:51.780903+0000","last_peered":"2024-02-10T07:38:51.780903+0000","last_clean":"2024-02-10T07:38:51.780903+0000","last_became_active":"2024-02-10T07:31:51.282353+0000","last_became_peered":"2024-02-10T07:31:51.282353+0000","last_unstale":"2024-02-10T07:38:51.780903+0000","last_undegraded":"2024-02-10T07:38:51.780903+0000","last_fullsized":"2024-02-10T07:38:51.780903+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:43.502790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:43.502790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:05.395150+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":302,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780278+0000","last_change":"2024-02-10T07:38:25.146074+0000","last_active":"2024-02-10T07:38:51.780278+0000","last_peered":"2024-02-10T07:38:51.780278+0000","last_clean":"2024-02-10T07:38:51.780278+0000","last_became_active":"2024-02-10T07:36:33.769894+0000","last_became_peered":"2024-02-10T07:36:33.769894+0000","last_unstale":"2024-02-10T07:38:51.780278+0000","last_undegraded":"2024-02-10T07:38:51.780278+0000","last_fullsized":"2024-02-10T07:38:51.780278+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:38:25.145962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:25.145962+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:44.000441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":332,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.782687+0000","last_change":"2024-02-10T07:38:31.696484+0000","last_active":"2024-02-10T07:38:51.782687+0000","last_peered":"2024-02-10T07:38:51.782687+0000","last_clean":"2024-02-10T07:38:51.782687+0000","last_became_active":"2024-02-10T07:31:57.308615+0000","last_became_peered":"2024-02-10T07:31:57.308615+0000","last_unstale":"2024-02-10T07:38:51.782687+0000","last_undegraded":"2024-02-10T07:38:51.782687+0000","last_fullsized":"2024-02-10T07:38:51.782687+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:31.696375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:38:31.696375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:45.014841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":321,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780770+0000","last_change":"2024-02-10T07:36:32.034358+0000","last_active":"2024-02-10T07:38:51.780770+0000","last_peered":"2024-02-10T07:38:51.780770+0000","last_clean":"2024-02-10T07:38:51.780770+0000","last_became_active":"2024-02-10T07:31:55.297431+0000","last_became_peered":"2024-02-10T07:31:55.297431+0000","last_unstale":"2024-02-10T07:38:51.780770+0000","last_undegraded":"2024-02-10T07:38:51.780770+0000","last_fullsized":"2024-02-10T07:38:51.780770+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:32.034106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:36:32.034106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.783865+0000","last_change":"2024-02-10T07:36:44.862446+0000","last_active":"2024-02-10T07:38:51.783865+0000","last_peered":"2024-02-10T07:38:51.783865+0000","last_clean":"2024-02-10T07:38:51.783865+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:38:51.783865+0000","last_undegraded":"2024-02-10T07:38:51.783865+0000","last_fullsized":"2024-02-10T07:38:51.783865+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:44.862337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:36:44.862337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"89'13","reported_seq":360,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.776678+0000","last_change":"2024-02-10T07:38:43.612645+0000","last_active":"2024-02-10T07:38:51.776678+0000","last_peered":"2024-02-10T07:38:51.776678+0000","last_clean":"2024-02-10T07:38:51.776678+0000","last_became_active":"2024-02-10T07:31:53.295353+0000","last_became_peered":"2024-02-10T07:31:53.295353+0000","last_unstale":"2024-02-10T07:38:51.776678+0000","last_undegraded":"2024-02-10T07:38:51.776678+0000","last_fullsized":"2024-02-10T07:38:51.776678+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'13","last_scrub_stamp":"2024-02-10T07:38:43.612524+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:38:43.612524+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:50.429000+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":317,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.784524+0000","last_change":"2024-02-10T07:36:54.820672+0000","last_active":"2024-02-10T07:38:51.784524+0000","last_peered":"2024-02-10T07:38:51.784524+0000","last_clean":"2024-02-10T07:38:51.784524+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:38:51.784524+0000","last_undegraded":"2024-02-10T07:38:51.784524+0000","last_fullsized":"2024-02-10T07:38:51.784524+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:54.820598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:36:54.820598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":286,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779750+0000","last_change":"2024-02-10T07:37:17.963214+0000","last_active":"2024-02-10T07:38:51.779750+0000","last_peered":"2024-02-10T07:38:51.779750+0000","last_clean":"2024-02-10T07:38:51.779750+0000","last_became_active":"2024-02-10T07:36:33.752051+0000","last_became_peered":"2024-02-10T07:36:33.752051+0000","last_unstale":"2024-02-10T07:38:51.779750+0000","last_undegraded":"2024-02-10T07:38:51.779750+0000","last_fullsized":"2024-02-10T07:38:51.779750+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:17.963100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:17.963100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:59.155428+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":346,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.778205+0000","last_change":"2024-02-10T07:37:42.790522+0000","last_active":"2024-02-10T07:38:51.778205+0000","last_peered":"2024-02-10T07:38:51.778205+0000","last_clean":"2024-02-10T07:38:51.778205+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:38:51.778205+0000","last_undegraded":"2024-02-10T07:38:51.778205+0000","last_fullsized":"2024-02-10T07:38:51.778205+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:42.790447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:42.790447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:06.830654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":334,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780608+0000","last_change":"2024-02-10T07:37:30.985038+0000","last_active":"2024-02-10T07:38:51.780608+0000","last_peered":"2024-02-10T07:38:51.780608+0000","last_clean":"2024-02-10T07:38:51.780608+0000","last_became_active":"2024-02-10T07:31:53.296831+0000","last_became_peered":"2024-02-10T07:31:53.296831+0000","last_unstale":"2024-02-10T07:38:51.780608+0000","last_undegraded":"2024-02-10T07:38:51.780608+0000","last_fullsized":"2024-02-10T07:38:51.780608+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-10T07:37:30.984906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:30.984906+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:00.502651+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"217'3660","reported_seq":3704,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:56.770820+0000","last_change":"2024-02-10T07:38:14.779128+0000","last_active":"2024-02-10T07:38:56.770820+0000","last_peered":"2024-02-10T07:38:56.770820+0000","last_clean":"2024-02-10T07:38:56.770820+0000","last_became_active":"2024-02-10T07:38:14.778672+0000","last_became_peered":"2024-02-10T07:38:14.778672+0000","last_unstale":"2024-02-10T07:38:56.770820+0000","last_undegraded":"2024-02-10T07:38:56.770820+0000","last_fullsized":"2024-02-10T07:38:56.770820+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3660,"log_dups_size":0,"ondisk_log_size":3660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:14.448083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239861760,"num_objects":3660,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3660,"num_write_kb":234240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":290,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779803+0000","last_change":"2024-02-10T07:37:18.959136+0000","last_active":"2024-02-10T07:38:51.779803+0000","last_peered":"2024-02-10T07:38:51.779803+0000","last_clean":"2024-02-10T07:38:51.779803+0000","last_became_active":"2024-02-10T07:36:33.763431+0000","last_became_peered":"2024-02-10T07:36:33.763431+0000","last_unstale":"2024-02-10T07:38:51.779803+0000","last_undegraded":"2024-02-10T07:38:51.779803+0000","last_fullsized":"2024-02-10T07:38:51.779803+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:18.959019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:18.959019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":350,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.784681+0000","last_change":"2024-02-10T07:37:49.683118+0000","last_active":"2024-02-10T07:38:51.784681+0000","last_peered":"2024-02-10T07:38:51.784681+0000","last_clean":"2024-02-10T07:38:51.784681+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:38:51.784681+0000","last_undegraded":"2024-02-10T07:38:51.784681+0000","last_fullsized":"2024-02-10T07:38:51.784681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-10T07:37:49.682987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:49.682987+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:02.993173+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":281,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779471+0000","last_change":"2024-02-10T07:37:29.907162+0000","last_active":"2024-02-10T07:38:51.779471+0000","last_peered":"2024-02-10T07:38:51.779471+0000","last_clean":"2024-02-10T07:38:51.779471+0000","last_became_active":"2024-02-10T07:36:33.754134+0000","last_became_peered":"2024-02-10T07:36:33.754134+0000","last_unstale":"2024-02-10T07:38:51.779471+0000","last_undegraded":"2024-02-10T07:38:51.779471+0000","last_fullsized":"2024-02-10T07:38:51.779471+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:37:29.907084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:29.907084+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:53.307426+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":424,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.784038+0000","last_change":"2024-02-10T07:38:27.441013+0000","last_active":"2024-02-10T07:38:51.784038+0000","last_peered":"2024-02-10T07:38:51.784038+0000","last_clean":"2024-02-10T07:38:51.784038+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:38:51.784038+0000","last_undegraded":"2024-02-10T07:38:51.784038+0000","last_fullsized":"2024-02-10T07:38:51.784038+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:38:27.440874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:27.440874+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:29.086690+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"217'3691","reported_seq":3735,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:57.742033+0000","last_change":"2024-02-10T07:38:15.020533+0000","last_active":"2024-02-10T07:38:57.742033+0000","last_peered":"2024-02-10T07:38:57.742033+0000","last_clean":"2024-02-10T07:38:57.742033+0000","last_became_active":"2024-02-10T07:38:15.020222+0000","last_became_peered":"2024-02-10T07:38:15.020222+0000","last_unstale":"2024-02-10T07:38:57.742033+0000","last_undegraded":"2024-02-10T07:38:57.742033+0000","last_fullsized":"2024-02-10T07:38:57.742033+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3691,"log_dups_size":0,"ondisk_log_size":3691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:15.690629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241893376,"num_objects":3691,"num_object_clones":0,"num_object_copies":11073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3691,"num_write_kb":236224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":318,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780064+0000","last_change":"2024-02-10T07:37:19.036835+0000","last_active":"2024-02-10T07:38:51.780064+0000","last_peered":"2024-02-10T07:38:51.780064+0000","last_clean":"2024-02-10T07:38:51.780064+0000","last_became_active":"2024-02-10T07:31:57.314872+0000","last_became_peered":"2024-02-10T07:31:57.314872+0000","last_unstale":"2024-02-10T07:38:51.780064+0000","last_undegraded":"2024-02-10T07:38:51.780064+0000","last_fullsized":"2024-02-10T07:38:51.780064+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:19.036588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:19.036588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.785124+0000","last_change":"2024-02-10T07:38:35.497614+0000","last_active":"2024-02-10T07:38:51.785124+0000","last_peered":"2024-02-10T07:38:51.785124+0000","last_clean":"2024-02-10T07:38:51.785124+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:38:51.785124+0000","last_undegraded":"2024-02-10T07:38:51.785124+0000","last_fullsized":"2024-02-10T07:38:51.785124+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:35.497479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:38:35.497479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:03.165678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":375,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.774875+0000","last_change":"2024-02-10T07:37:44.688729+0000","last_active":"2024-02-10T07:38:51.774875+0000","last_peered":"2024-02-10T07:38:51.774875+0000","last_clean":"2024-02-10T07:38:51.774875+0000","last_became_active":"2024-02-10T07:32:18.389409+0000","last_became_peered":"2024-02-10T07:32:18.389409+0000","last_unstale":"2024-02-10T07:38:51.774875+0000","last_undegraded":"2024-02-10T07:38:51.774875+0000","last_fullsized":"2024-02-10T07:38:51.774875+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:37:44.688627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:44.688627+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:51.858934+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":337,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780505+0000","last_change":"2024-02-10T07:38:36.701149+0000","last_active":"2024-02-10T07:38:51.780505+0000","last_peered":"2024-02-10T07:38:51.780505+0000","last_clean":"2024-02-10T07:38:51.780505+0000","last_became_active":"2024-02-10T07:31:55.303059+0000","last_became_peered":"2024-02-10T07:31:55.303059+0000","last_unstale":"2024-02-10T07:38:51.780505+0000","last_undegraded":"2024-02-10T07:38:51.780505+0000","last_fullsized":"2024-02-10T07:38:51.780505+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:36.701019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:36.701019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:05.304703+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"217'3249","reported_seq":3293,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:53.325907+0000","last_change":"2024-02-10T07:38:15.020510+0000","last_active":"2024-02-10T07:38:53.325907+0000","last_peered":"2024-02-10T07:38:53.325907+0000","last_clean":"2024-02-10T07:38:53.325907+0000","last_became_active":"2024-02-10T07:38:15.020166+0000","last_became_peered":"2024-02-10T07:38:15.020166+0000","last_unstale":"2024-02-10T07:38:53.325907+0000","last_undegraded":"2024-02-10T07:38:53.325907+0000","last_fullsized":"2024-02-10T07:38:53.325907+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3249,"log_dups_size":0,"ondisk_log_size":3249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:26.981914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212926464,"num_objects":3249,"num_object_clones":0,"num_object_copies":9747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3249,"num_write_kb":207936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":326,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.783575+0000","last_change":"2024-02-10T07:36:46.899550+0000","last_active":"2024-02-10T07:38:51.783575+0000","last_peered":"2024-02-10T07:38:51.783575+0000","last_clean":"2024-02-10T07:38:51.783575+0000","last_became_active":"2024-02-10T07:31:57.304971+0000","last_became_peered":"2024-02-10T07:31:57.304971+0000","last_unstale":"2024-02-10T07:38:51.783575+0000","last_undegraded":"2024-02-10T07:38:51.783575+0000","last_fullsized":"2024-02-10T07:38:51.783575+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:46.899444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:36:46.899444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":346,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780764+0000","last_change":"2024-02-10T07:38:37.699541+0000","last_active":"2024-02-10T07:38:51.780764+0000","last_peered":"2024-02-10T07:38:51.780764+0000","last_clean":"2024-02-10T07:38:51.780764+0000","last_became_active":"2024-02-10T07:36:33.754602+0000","last_became_peered":"2024-02-10T07:36:33.754602+0000","last_unstale":"2024-02-10T07:38:51.780764+0000","last_undegraded":"2024-02-10T07:38:51.780764+0000","last_fullsized":"2024-02-10T07:38:51.780764+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:37.699423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:37.699423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:06.556224+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":347,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.785054+0000","last_change":"2024-02-10T07:37:22.620665+0000","last_active":"2024-02-10T07:38:51.785054+0000","last_peered":"2024-02-10T07:38:51.785054+0000","last_clean":"2024-02-10T07:38:51.785054+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:38:51.785054+0000","last_undegraded":"2024-02-10T07:38:51.785054+0000","last_fullsized":"2024-02-10T07:38:51.785054+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:22.620597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:22.620597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:52.465985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":372,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.778862+0000","last_change":"2024-02-10T07:38:25.855922+0000","last_active":"2024-02-10T07:38:51.778862+0000","last_peered":"2024-02-10T07:38:51.778862+0000","last_clean":"2024-02-10T07:38:51.778862+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:38:51.778862+0000","last_undegraded":"2024-02-10T07:38:51.778862+0000","last_fullsized":"2024-02-10T07:38:51.778862+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-10T07:38:25.855813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:25.855813+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:44.243702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"217'3300","reported_seq":3344,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:53.046521+0000","last_change":"2024-02-10T07:38:14.779431+0000","last_active":"2024-02-10T07:38:53.046521+0000","last_peered":"2024-02-10T07:38:53.046521+0000","last_clean":"2024-02-10T07:38:53.046521+0000","last_became_active":"2024-02-10T07:38:14.779127+0000","last_became_peered":"2024-02-10T07:38:14.779127+0000","last_unstale":"2024-02-10T07:38:53.046521+0000","last_undegraded":"2024-02-10T07:38:53.046521+0000","last_fullsized":"2024-02-10T07:38:53.046521+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3300,"log_dups_size":0,"ondisk_log_size":3300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:24.308291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216268800,"num_objects":3300,"num_object_clones":0,"num_object_copies":9900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3300,"num_write_kb":211200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":325,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.776267+0000","last_change":"2024-02-10T07:37:48.592946+0000","last_active":"2024-02-10T07:38:51.776267+0000","last_peered":"2024-02-10T07:38:51.776267+0000","last_clean":"2024-02-10T07:38:51.776267+0000","last_became_active":"2024-02-10T07:31:57.313394+0000","last_became_peered":"2024-02-10T07:31:57.313394+0000","last_unstale":"2024-02-10T07:38:51.776267+0000","last_undegraded":"2024-02-10T07:38:51.776267+0000","last_fullsized":"2024-02-10T07:38:51.776267+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:48.592835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:48.592835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:55.346477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":329,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780732+0000","last_change":"2024-02-10T07:38:29.689095+0000","last_active":"2024-02-10T07:38:51.780732+0000","last_peered":"2024-02-10T07:38:51.780732+0000","last_clean":"2024-02-10T07:38:51.780732+0000","last_became_active":"2024-02-10T07:31:55.298622+0000","last_became_peered":"2024-02-10T07:31:55.298622+0000","last_unstale":"2024-02-10T07:38:51.780732+0000","last_undegraded":"2024-02-10T07:38:51.780732+0000","last_fullsized":"2024-02-10T07:38:51.780732+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:29.688966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:29.688966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:35.050177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":294,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779378+0000","last_change":"2024-02-10T07:38:26.195581+0000","last_active":"2024-02-10T07:38:51.779378+0000","last_peered":"2024-02-10T07:38:51.779378+0000","last_clean":"2024-02-10T07:38:51.779378+0000","last_became_active":"2024-02-10T07:36:33.762155+0000","last_became_peered":"2024-02-10T07:36:33.762155+0000","last_unstale":"2024-02-10T07:38:51.779378+0000","last_undegraded":"2024-02-10T07:38:51.779378+0000","last_fullsized":"2024-02-10T07:38:51.779378+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:26.195471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:26.195471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:31.570471+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":349,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780913+0000","last_change":"2024-02-10T07:38:30.603907+0000","last_active":"2024-02-10T07:38:51.780913+0000","last_peered":"2024-02-10T07:38:51.780913+0000","last_clean":"2024-02-10T07:38:51.780913+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:38:51.780913+0000","last_undegraded":"2024-02-10T07:38:51.780913+0000","last_fullsized":"2024-02-10T07:38:51.780913+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-10T07:38:30.602734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:30.602734+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:43.285464+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"217'3512","reported_seq":3556,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:56.742809+0000","last_change":"2024-02-10T07:38:15.019596+0000","last_active":"2024-02-10T07:38:56.742809+0000","last_peered":"2024-02-10T07:38:56.742809+0000","last_clean":"2024-02-10T07:38:56.742809+0000","last_became_active":"2024-02-10T07:38:15.019052+0000","last_became_peered":"2024-02-10T07:38:15.019052+0000","last_unstale":"2024-02-10T07:38:56.742809+0000","last_undegraded":"2024-02-10T07:38:56.742809+0000","last_fullsized":"2024-02-10T07:38:56.742809+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3512,"log_dups_size":0,"ondisk_log_size":3512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:17.302916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230162432,"num_objects":3512,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3512,"num_write_kb":224768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":318,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779242+0000","last_change":"2024-02-10T07:37:39.095527+0000","last_active":"2024-02-10T07:38:51.779242+0000","last_peered":"2024-02-10T07:38:51.779242+0000","last_clean":"2024-02-10T07:38:51.779242+0000","last_became_active":"2024-02-10T07:31:57.313854+0000","last_became_peered":"2024-02-10T07:31:57.313854+0000","last_unstale":"2024-02-10T07:38:51.779242+0000","last_undegraded":"2024-02-10T07:38:51.779242+0000","last_fullsized":"2024-02-10T07:38:51.779242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:37:39.095411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:08.117301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":277,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779656+0000","last_change":"2024-02-10T07:37:43.754879+0000","last_active":"2024-02-10T07:38:51.779656+0000","last_peered":"2024-02-10T07:38:51.779656+0000","last_clean":"2024-02-10T07:38:51.779656+0000","last_became_active":"2024-02-10T07:36:33.759656+0000","last_became_peered":"2024-02-10T07:36:33.759656+0000","last_unstale":"2024-02-10T07:38:51.779656+0000","last_undegraded":"2024-02-10T07:38:51.779656+0000","last_fullsized":"2024-02-10T07:38:51.779656+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:37:43.754618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:37:43.754618+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:52.815153+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.775544+0000","last_change":"2024-02-10T07:38:26.562972+0000","last_active":"2024-02-10T07:38:51.775544+0000","last_peered":"2024-02-10T07:38:51.775544+0000","last_clean":"2024-02-10T07:38:51.775544+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:38:51.775544+0000","last_undegraded":"2024-02-10T07:38:51.775544+0000","last_fullsized":"2024-02-10T07:38:51.775544+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:26.562864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:26.562864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:52.775819+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":465,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.786064+0000","last_change":"2024-02-10T07:38:37.949081+0000","last_active":"2024-02-10T07:38:51.786064+0000","last_peered":"2024-02-10T07:38:51.786064+0000","last_clean":"2024-02-10T07:38:51.786064+0000","last_became_active":"2024-02-10T07:31:53.301863+0000","last_became_peered":"2024-02-10T07:31:53.301863+0000","last_unstale":"2024-02-10T07:38:51.786064+0000","last_undegraded":"2024-02-10T07:38:51.786064+0000","last_fullsized":"2024-02-10T07:38:51.786064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-02-10T07:38:37.948929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:37.948929+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:07.680973+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":329,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.776052+0000","last_change":"2024-02-10T07:37:49.614365+0000","last_active":"2024-02-10T07:38:51.776052+0000","last_peered":"2024-02-10T07:38:51.776052+0000","last_clean":"2024-02-10T07:38:51.776052+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:38:51.776052+0000","last_undegraded":"2024-02-10T07:38:51.776052+0000","last_fullsized":"2024-02-10T07:38:51.776052+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:49.614249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:49.614249+0000","last_clean_scrub_stamp":"2024-02-10T07:37:49.614249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:56.011439+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"217'3344","reported_seq":3386,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:53.629098+0000","last_change":"2024-02-10T07:38:14.780713+0000","last_active":"2024-02-10T07:38:53.629098+0000","last_peered":"2024-02-10T07:38:53.629098+0000","last_clean":"2024-02-10T07:38:53.629098+0000","last_became_active":"2024-02-10T07:38:14.780419+0000","last_became_peered":"2024-02-10T07:38:14.780419+0000","last_unstale":"2024-02-10T07:38:53.629098+0000","last_undegraded":"2024-02-10T07:38:53.629098+0000","last_fullsized":"2024-02-10T07:38:53.629098+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3344,"log_dups_size":0,"ondisk_log_size":3344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:35.222475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219152384,"num_objects":3344,"num_object_clones":0,"num_object_copies":10032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3344,"num_write_kb":214016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":326,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.785200+0000","last_change":"2024-02-10T07:37:40.674423+0000","last_active":"2024-02-10T07:38:51.785200+0000","last_peered":"2024-02-10T07:38:51.785200+0000","last_clean":"2024-02-10T07:38:51.785200+0000","last_became_active":"2024-02-10T07:31:57.299958+0000","last_became_peered":"2024-02-10T07:31:57.299958+0000","last_unstale":"2024-02-10T07:38:51.785200+0000","last_undegraded":"2024-02-10T07:38:51.785200+0000","last_fullsized":"2024-02-10T07:38:51.785200+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:40.674326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:40.674326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:05.194267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":336,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.785657+0000","last_change":"2024-02-10T07:37:38.688852+0000","last_active":"2024-02-10T07:38:51.785657+0000","last_peered":"2024-02-10T07:38:51.785657+0000","last_clean":"2024-02-10T07:38:51.785657+0000","last_became_active":"2024-02-10T07:31:51.320864+0000","last_became_peered":"2024-02-10T07:31:51.320864+0000","last_unstale":"2024-02-10T07:38:51.785657+0000","last_undegraded":"2024-02-10T07:38:51.785657+0000","last_fullsized":"2024-02-10T07:38:51.785657+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:38.688738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:38.688738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:02.698589+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":346,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.781727+0000","last_change":"2024-02-10T07:37:46.833081+0000","last_active":"2024-02-10T07:38:51.781727+0000","last_peered":"2024-02-10T07:38:51.781727+0000","last_clean":"2024-02-10T07:38:51.781727+0000","last_became_active":"2024-02-10T07:31:50.281517+0000","last_became_peered":"2024-02-10T07:31:50.281517+0000","last_unstale":"2024-02-10T07:38:51.781727+0000","last_undegraded":"2024-02-10T07:38:51.781727+0000","last_fullsized":"2024-02-10T07:38:51.781727+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:46.832758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:37:46.832758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:00.227603+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":119,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.788229+0000","last_change":"2024-02-10T07:38:25.597385+0000","last_active":"2024-02-10T07:38:51.788229+0000","last_peered":"2024-02-10T07:38:51.788229+0000","last_clean":"2024-02-10T07:38:51.788229+0000","last_became_active":"2024-02-10T07:37:06.196787+0000","last_became_peered":"2024-02-10T07:37:06.196787+0000","last_unstale":"2024-02-10T07:38:51.788229+0000","last_undegraded":"2024-02-10T07:38:51.788229+0000","last_fullsized":"2024-02-10T07:38:51.788229+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-10T07:38:25.597236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:25.597236+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:41.209357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.781914+0000","last_change":"2024-02-10T07:37:48.818550+0000","last_active":"2024-02-10T07:38:51.781914+0000","last_peered":"2024-02-10T07:38:51.781914+0000","last_clean":"2024-02-10T07:38:51.781914+0000","last_became_active":"2024-02-10T07:36:33.751551+0000","last_became_peered":"2024-02-10T07:36:33.751551+0000","last_unstale":"2024-02-10T07:38:51.781914+0000","last_undegraded":"2024-02-10T07:38:51.781914+0000","last_fullsized":"2024-02-10T07:38:51.781914+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:48.818436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:48.818436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:10.693261+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"217'3661","reported_seq":3705,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:57.757475+0000","last_change":"2024-02-10T07:38:14.779982+0000","last_active":"2024-02-10T07:38:57.757475+0000","last_peered":"2024-02-10T07:38:57.757475+0000","last_clean":"2024-02-10T07:38:57.757475+0000","last_became_active":"2024-02-10T07:38:14.779523+0000","last_became_peered":"2024-02-10T07:38:14.779523+0000","last_unstale":"2024-02-10T07:38:57.757475+0000","last_undegraded":"2024-02-10T07:38:57.757475+0000","last_fullsized":"2024-02-10T07:38:57.757475+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:35.074709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239927296,"num_objects":3661,"num_object_clones":0,"num_object_copies":10983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3661,"num_write_kb":234304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":290,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779719+0000","last_change":"2024-02-10T07:38:33.256517+0000","last_active":"2024-02-10T07:38:51.779719+0000","last_peered":"2024-02-10T07:38:51.779719+0000","last_clean":"2024-02-10T07:38:51.779719+0000","last_became_active":"2024-02-10T07:36:33.769763+0000","last_became_peered":"2024-02-10T07:36:33.769763+0000","last_unstale":"2024-02-10T07:38:51.779719+0000","last_undegraded":"2024-02-10T07:38:51.779719+0000","last_fullsized":"2024-02-10T07:38:51.779719+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:38:33.256420+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:33:41.150454+0000","last_clean_scrub_stamp":"2024-02-10T07:38:33.256420+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:41.430744+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":583,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.783664+0000","last_change":"2024-02-10T07:37:39.652563+0000","last_active":"2024-02-10T07:38:51.783664+0000","last_peered":"2024-02-10T07:38:51.783664+0000","last_clean":"2024-02-10T07:38:51.783664+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:38:51.783664+0000","last_undegraded":"2024-02-10T07:38:51.783664+0000","last_fullsized":"2024-02-10T07:38:51.783664+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:37:39.652478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:37:39.652478+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":339,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.778507+0000","last_change":"2024-02-10T07:36:57.022219+0000","last_active":"2024-02-10T07:38:51.778507+0000","last_peered":"2024-02-10T07:38:51.778507+0000","last_clean":"2024-02-10T07:38:51.778507+0000","last_became_active":"2024-02-10T07:36:33.752703+0000","last_became_peered":"2024-02-10T07:36:33.752703+0000","last_unstale":"2024-02-10T07:38:51.778507+0000","last_undegraded":"2024-02-10T07:38:51.778507+0000","last_fullsized":"2024-02-10T07:38:51.778507+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:57.022108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:36:57.022108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"89'24","reported_seq":457,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779236+0000","last_change":"2024-02-10T07:38:32.878953+0000","last_active":"2024-02-10T07:38:51.779236+0000","last_peered":"2024-02-10T07:38:51.779236+0000","last_clean":"2024-02-10T07:38:51.779236+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:38:51.779236+0000","last_undegraded":"2024-02-10T07:38:51.779236+0000","last_fullsized":"2024-02-10T07:38:51.779236+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'24","last_scrub_stamp":"2024-02-10T07:38:32.878819+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:38:32.878819+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:01.404818+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":339,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.784689+0000","last_change":"2024-02-10T07:38:41.490159+0000","last_active":"2024-02-10T07:38:51.784689+0000","last_peered":"2024-02-10T07:38:51.784689+0000","last_clean":"2024-02-10T07:38:51.784689+0000","last_became_active":"2024-02-10T07:36:33.752711+0000","last_became_peered":"2024-02-10T07:36:33.752711+0000","last_unstale":"2024-02-10T07:38:51.784689+0000","last_undegraded":"2024-02-10T07:38:51.784689+0000","last_fullsized":"2024-02-10T07:38:51.784689+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:41.490080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:41.490080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:05.272773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"217'3368","reported_seq":3412,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:53.016717+0000","last_change":"2024-02-10T07:38:14.780182+0000","last_active":"2024-02-10T07:38:53.016717+0000","last_peered":"2024-02-10T07:38:53.016717+0000","last_clean":"2024-02-10T07:38:53.016717+0000","last_became_active":"2024-02-10T07:38:14.779819+0000","last_became_peered":"2024-02-10T07:38:14.779819+0000","last_unstale":"2024-02-10T07:38:53.016717+0000","last_undegraded":"2024-02-10T07:38:53.016717+0000","last_fullsized":"2024-02-10T07:38:53.016717+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3368,"log_dups_size":0,"ondisk_log_size":3368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:31.688480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220725248,"num_objects":3368,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3368,"num_write_kb":215552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":107,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.784823+0000","last_change":"2024-02-10T07:37:22.536358+0000","last_active":"2024-02-10T07:38:51.784823+0000","last_peered":"2024-02-10T07:38:51.784823+0000","last_clean":"2024-02-10T07:38:51.784823+0000","last_became_active":"2024-02-10T07:37:06.202738+0000","last_became_peered":"2024-02-10T07:37:06.202738+0000","last_unstale":"2024-02-10T07:38:51.784823+0000","last_undegraded":"2024-02-10T07:38:51.784823+0000","last_fullsized":"2024-02-10T07:38:51.784823+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:22.536223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:37:22.536223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":337,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.776023+0000","last_change":"2024-02-10T07:38:37.574614+0000","last_active":"2024-02-10T07:38:51.776023+0000","last_peered":"2024-02-10T07:38:51.776023+0000","last_clean":"2024-02-10T07:38:51.776023+0000","last_became_active":"2024-02-10T07:31:51.286076+0000","last_became_peered":"2024-02-10T07:31:51.286076+0000","last_unstale":"2024-02-10T07:38:51.776023+0000","last_undegraded":"2024-02-10T07:38:51.776023+0000","last_fullsized":"2024-02-10T07:38:51.776023+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:37.574311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:37.574311+0000","last_clean_scrub_stamp":"2024-02-10T07:38:37.574311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:55.614745+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":771,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:56.745118+0000","last_change":"2024-02-10T07:38:21.504325+0000","last_active":"2024-02-10T07:38:56.745118+0000","last_peered":"2024-02-10T07:38:56.745118+0000","last_clean":"2024-02-10T07:38:56.745118+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:38:56.745118+0000","last_undegraded":"2024-02-10T07:38:56.745118+0000","last_fullsized":"2024-02-10T07:38:56.745118+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:38:21.504219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:38:21.504219+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:40.281918+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":421,"num_read_kb":416,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":351,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.776359+0000","last_change":"2024-02-10T07:38:35.477150+0000","last_active":"2024-02-10T07:38:51.776359+0000","last_peered":"2024-02-10T07:38:51.776359+0000","last_clean":"2024-02-10T07:38:51.776359+0000","last_became_active":"2024-02-10T07:31:53.300687+0000","last_became_peered":"2024-02-10T07:31:53.300687+0000","last_unstale":"2024-02-10T07:38:51.776359+0000","last_undegraded":"2024-02-10T07:38:51.776359+0000","last_fullsized":"2024-02-10T07:38:51.776359+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-10T07:38:35.476708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:35.476708+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:01.667916+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":284,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780185+0000","last_change":"2024-02-10T07:38:31.220461+0000","last_active":"2024-02-10T07:38:51.780185+0000","last_peered":"2024-02-10T07:38:51.780185+0000","last_clean":"2024-02-10T07:38:51.780185+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:38:51.780185+0000","last_undegraded":"2024-02-10T07:38:51.780185+0000","last_fullsized":"2024-02-10T07:38:51.780185+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:31.220325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:31.220325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:36.968611+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"217'3371","reported_seq":3415,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:53.602501+0000","last_change":"2024-02-10T07:38:14.780763+0000","last_active":"2024-02-10T07:38:53.602501+0000","last_peered":"2024-02-10T07:38:53.602501+0000","last_clean":"2024-02-10T07:38:53.602501+0000","last_became_active":"2024-02-10T07:38:14.780467+0000","last_became_peered":"2024-02-10T07:38:14.780467+0000","last_unstale":"2024-02-10T07:38:53.602501+0000","last_undegraded":"2024-02-10T07:38:53.602501+0000","last_fullsized":"2024-02-10T07:38:53.602501+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3371,"log_dups_size":0,"ondisk_log_size":3371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:42.719171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220921856,"num_objects":3371,"num_object_clones":0,"num_object_copies":10113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3371,"num_write_kb":215744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":324,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.786570+0000","last_change":"2024-02-10T07:38:43.874651+0000","last_active":"2024-02-10T07:38:51.786570+0000","last_peered":"2024-02-10T07:38:51.786570+0000","last_clean":"2024-02-10T07:38:51.786570+0000","last_became_active":"2024-02-10T07:31:57.307615+0000","last_became_peered":"2024-02-10T07:31:57.307615+0000","last_unstale":"2024-02-10T07:38:51.786570+0000","last_undegraded":"2024-02-10T07:38:51.786570+0000","last_fullsized":"2024-02-10T07:38:51.786570+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:43.874184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:43.874184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:47.939719+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":345,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.785982+0000","last_change":"2024-02-10T07:38:36.734296+0000","last_active":"2024-02-10T07:38:51.785982+0000","last_peered":"2024-02-10T07:38:51.785982+0000","last_clean":"2024-02-10T07:38:51.785982+0000","last_became_active":"2024-02-10T07:36:33.750867+0000","last_became_peered":"2024-02-10T07:36:33.750867+0000","last_unstale":"2024-02-10T07:38:51.785982+0000","last_undegraded":"2024-02-10T07:38:51.785982+0000","last_fullsized":"2024-02-10T07:38:51.785982+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:38:36.734185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:51.560679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":378,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.777620+0000","last_change":"2024-02-10T07:38:36.492494+0000","last_active":"2024-02-10T07:38:51.777620+0000","last_peered":"2024-02-10T07:38:51.777620+0000","last_clean":"2024-02-10T07:38:51.777620+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:38:51.777620+0000","last_undegraded":"2024-02-10T07:38:51.777620+0000","last_fullsized":"2024-02-10T07:38:51.777620+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-10T07:38:36.492339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:36.492339+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:54.191943+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":346,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.778481+0000","last_change":"2024-02-10T07:38:42.894255+0000","last_active":"2024-02-10T07:38:51.778481+0000","last_peered":"2024-02-10T07:38:51.778481+0000","last_clean":"2024-02-10T07:38:51.778481+0000","last_became_active":"2024-02-10T07:36:33.765998+0000","last_became_peered":"2024-02-10T07:36:33.765998+0000","last_unstale":"2024-02-10T07:38:51.778481+0000","last_undegraded":"2024-02-10T07:38:51.778481+0000","last_fullsized":"2024-02-10T07:38:51.778481+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:42.894130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:42.894130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:47.055312+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"217'3213","reported_seq":3257,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:53.326058+0000","last_change":"2024-02-10T07:38:14.781247+0000","last_active":"2024-02-10T07:38:53.326058+0000","last_peered":"2024-02-10T07:38:53.326058+0000","last_clean":"2024-02-10T07:38:53.326058+0000","last_became_active":"2024-02-10T07:38:14.780910+0000","last_became_peered":"2024-02-10T07:38:14.780910+0000","last_unstale":"2024-02-10T07:38:53.326058+0000","last_undegraded":"2024-02-10T07:38:53.326058+0000","last_fullsized":"2024-02-10T07:38:53.326058+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3213,"log_dups_size":0,"ondisk_log_size":3213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:15.581390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210567168,"num_objects":3213,"num_object_clones":0,"num_object_copies":9639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3213,"num_write_kb":205632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":212,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.777914+0000","last_change":"2024-02-10T07:38:27.898967+0000","last_active":"2024-02-10T07:38:51.777914+0000","last_peered":"2024-02-10T07:38:51.777914+0000","last_clean":"2024-02-10T07:38:51.777914+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:38:51.777914+0000","last_undegraded":"2024-02-10T07:38:51.777914+0000","last_fullsized":"2024-02-10T07:38:51.777914+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:27.898770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:27.898770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:47.783371+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":306,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779970+0000","last_change":"2024-02-10T07:38:39.220690+0000","last_active":"2024-02-10T07:38:51.779970+0000","last_peered":"2024-02-10T07:38:51.779970+0000","last_clean":"2024-02-10T07:38:51.779970+0000","last_became_active":"2024-02-10T07:36:33.779820+0000","last_became_peered":"2024-02-10T07:36:33.779820+0000","last_unstale":"2024-02-10T07:38:51.779970+0000","last_undegraded":"2024-02-10T07:38:51.779970+0000","last_fullsized":"2024-02-10T07:38:51.779970+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:39.220580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:39.220580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:50.640335+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":119,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.787284+0000","last_change":"2024-02-10T07:37:41.661838+0000","last_active":"2024-02-10T07:38:51.787284+0000","last_peered":"2024-02-10T07:38:51.787284+0000","last_clean":"2024-02-10T07:38:51.787284+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:38:51.787284+0000","last_undegraded":"2024-02-10T07:38:51.787284+0000","last_fullsized":"2024-02-10T07:38:51.787284+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:37:41.661685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:41.661685+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:11.608613+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"158'10","reported_seq":432,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780937+0000","last_change":"2024-02-10T07:38:31.556370+0000","last_active":"2024-02-10T07:38:51.780937+0000","last_peered":"2024-02-10T07:38:51.780937+0000","last_clean":"2024-02-10T07:38:51.780937+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:38:51.780937+0000","last_undegraded":"2024-02-10T07:38:51.780937+0000","last_fullsized":"2024-02-10T07:38:51.780937+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:38:31.556290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:31.556290+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:38.821295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"217'3554","reported_seq":3598,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:56.768595+0000","last_change":"2024-02-10T07:38:15.018575+0000","last_active":"2024-02-10T07:38:56.768595+0000","last_peered":"2024-02-10T07:38:56.768595+0000","last_clean":"2024-02-10T07:38:56.768595+0000","last_became_active":"2024-02-10T07:38:15.018262+0000","last_became_peered":"2024-02-10T07:38:15.018262+0000","last_unstale":"2024-02-10T07:38:56.768595+0000","last_undegraded":"2024-02-10T07:38:56.768595+0000","last_fullsized":"2024-02-10T07:38:56.768595+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3554,"log_dups_size":0,"ondisk_log_size":3554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:32.512344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232914944,"num_objects":3554,"num_object_clones":0,"num_object_copies":10662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3554,"num_write_kb":227456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":326,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.784567+0000","last_change":"2024-02-10T07:38:39.523379+0000","last_active":"2024-02-10T07:38:51.784567+0000","last_peered":"2024-02-10T07:38:51.784567+0000","last_clean":"2024-02-10T07:38:51.784567+0000","last_became_active":"2024-02-10T07:31:57.305620+0000","last_became_peered":"2024-02-10T07:31:57.305620+0000","last_unstale":"2024-02-10T07:38:51.784567+0000","last_undegraded":"2024-02-10T07:38:51.784567+0000","last_fullsized":"2024-02-10T07:38:51.784567+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:39.523260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:39.523260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:43.730930+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":346,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.776449+0000","last_change":"2024-02-10T07:38:44.560403+0000","last_active":"2024-02-10T07:38:51.776449+0000","last_peered":"2024-02-10T07:38:51.776449+0000","last_clean":"2024-02-10T07:38:51.776449+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:38:51.776449+0000","last_undegraded":"2024-02-10T07:38:51.776449+0000","last_fullsized":"2024-02-10T07:38:51.776449+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:38:44.559964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:38:44.559964+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:56.785153+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":364,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.774679+0000","last_change":"2024-02-10T07:38:43.684979+0000","last_active":"2024-02-10T07:38:51.774679+0000","last_peered":"2024-02-10T07:38:51.774679+0000","last_clean":"2024-02-10T07:38:51.774679+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:38:51.774679+0000","last_undegraded":"2024-02-10T07:38:51.774679+0000","last_fullsized":"2024-02-10T07:38:51.774679+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-02-10T07:38:43.684626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:43.684626+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:45.563642+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":423,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.783153+0000","last_change":"2024-02-10T07:37:49.163617+0000","last_active":"2024-02-10T07:38:51.783153+0000","last_peered":"2024-02-10T07:38:51.783153+0000","last_clean":"2024-02-10T07:38:51.783153+0000","last_became_active":"2024-02-10T07:31:55.302863+0000","last_became_peered":"2024-02-10T07:31:55.302863+0000","last_unstale":"2024-02-10T07:38:51.783153+0000","last_undegraded":"2024-02-10T07:38:51.783153+0000","last_fullsized":"2024-02-10T07:38:51.783153+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:37:49.163478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:49.163478+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:02.762326+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"217'3307","reported_seq":3350,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:53.276415+0000","last_change":"2024-02-10T07:38:15.018995+0000","last_active":"2024-02-10T07:38:53.276415+0000","last_peered":"2024-02-10T07:38:53.276415+0000","last_clean":"2024-02-10T07:38:53.276415+0000","last_became_active":"2024-02-10T07:38:15.018665+0000","last_became_peered":"2024-02-10T07:38:15.018665+0000","last_unstale":"2024-02-10T07:38:53.276415+0000","last_undegraded":"2024-02-10T07:38:53.276415+0000","last_fullsized":"2024-02-10T07:38:53.276415+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3307,"log_dups_size":0,"ondisk_log_size":3307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:29.858726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216727552,"num_objects":3307,"num_object_clones":0,"num_object_copies":9921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3307,"num_write_kb":211648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":326,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.774684+0000","last_change":"2024-02-10T07:38:47.661824+0000","last_active":"2024-02-10T07:38:51.774684+0000","last_peered":"2024-02-10T07:38:51.774684+0000","last_clean":"2024-02-10T07:38:51.774684+0000","last_became_active":"2024-02-10T07:31:57.316047+0000","last_became_peered":"2024-02-10T07:31:57.316047+0000","last_unstale":"2024-02-10T07:38:51.774684+0000","last_undegraded":"2024-02-10T07:38:51.774684+0000","last_fullsized":"2024-02-10T07:38:51.774684+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:47.661678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:38:47.661678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:47.673583+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780460+0000","last_change":"2024-02-10T07:38:50.299436+0000","last_active":"2024-02-10T07:38:51.780460+0000","last_peered":"2024-02-10T07:38:51.780460+0000","last_clean":"2024-02-10T07:38:51.780460+0000","last_became_active":"2024-02-10T07:31:51.285333+0000","last_became_peered":"2024-02-10T07:31:51.285333+0000","last_unstale":"2024-02-10T07:38:51.780460+0000","last_undegraded":"2024-02-10T07:38:51.780460+0000","last_fullsized":"2024-02-10T07:38:51.780460+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:38:50.299314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:52.415567+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":358,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.783147+0000","last_change":"2024-02-10T07:36:28.084850+0000","last_active":"2024-02-10T07:38:51.783147+0000","last_peered":"2024-02-10T07:38:51.783147+0000","last_clean":"2024-02-10T07:38:51.783147+0000","last_became_active":"2024-02-10T07:31:53.290157+0000","last_became_peered":"2024-02-10T07:31:53.290157+0000","last_unstale":"2024-02-10T07:38:51.783147+0000","last_undegraded":"2024-02-10T07:38:51.783147+0000","last_fullsized":"2024-02-10T07:38:51.783147+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-10T07:36:28.084678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:36:28.084678+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":330,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.774444+0000","last_change":"2024-02-10T07:37:19.822653+0000","last_active":"2024-02-10T07:38:51.774444+0000","last_peered":"2024-02-10T07:38:51.774444+0000","last_clean":"2024-02-10T07:38:51.774444+0000","last_became_active":"2024-02-10T07:31:55.295900+0000","last_became_peered":"2024-02-10T07:31:55.295900+0000","last_unstale":"2024-02-10T07:38:51.774444+0000","last_undegraded":"2024-02-10T07:38:51.774444+0000","last_fullsized":"2024-02-10T07:38:51.774444+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:19.822560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:19.822560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"217'3237","reported_seq":3281,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:53.035091+0000","last_change":"2024-02-10T07:38:14.781045+0000","last_active":"2024-02-10T07:38:53.035091+0000","last_peered":"2024-02-10T07:38:53.035091+0000","last_clean":"2024-02-10T07:38:53.035091+0000","last_became_active":"2024-02-10T07:38:14.780765+0000","last_became_peered":"2024-02-10T07:38:14.780765+0000","last_unstale":"2024-02-10T07:38:53.035091+0000","last_undegraded":"2024-02-10T07:38:53.035091+0000","last_fullsized":"2024-02-10T07:38:53.035091+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3237,"log_dups_size":0,"ondisk_log_size":3237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:25.625058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212140032,"num_objects":3237,"num_object_clones":0,"num_object_copies":9711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3237,"num_write_kb":207168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":284,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.780032+0000","last_change":"2024-02-10T07:38:27.180747+0000","last_active":"2024-02-10T07:38:51.780032+0000","last_peered":"2024-02-10T07:38:51.780032+0000","last_clean":"2024-02-10T07:38:51.780032+0000","last_became_active":"2024-02-10T07:36:33.772094+0000","last_became_peered":"2024-02-10T07:36:33.772094+0000","last_unstale":"2024-02-10T07:38:51.780032+0000","last_undegraded":"2024-02-10T07:38:51.780032+0000","last_fullsized":"2024-02-10T07:38:51.780032+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:27.180619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:27.180619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:39.283016+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"158'10","reported_seq":434,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.775371+0000","last_change":"2024-02-10T07:38:48.639883+0000","last_active":"2024-02-10T07:38:51.775371+0000","last_peered":"2024-02-10T07:38:51.775371+0000","last_clean":"2024-02-10T07:38:51.775371+0000","last_became_active":"2024-02-10T07:36:33.755111+0000","last_became_peered":"2024-02-10T07:36:33.755111+0000","last_unstale":"2024-02-10T07:38:51.775371+0000","last_undegraded":"2024-02-10T07:38:51.775371+0000","last_fullsized":"2024-02-10T07:38:51.775371+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:38:48.639748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:48.639748+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:40:04.786925+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":330,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.783039+0000","last_change":"2024-02-10T07:37:25.985327+0000","last_active":"2024-02-10T07:38:51.783039+0000","last_peered":"2024-02-10T07:38:51.783039+0000","last_clean":"2024-02-10T07:38:51.783039+0000","last_became_active":"2024-02-10T07:31:51.282137+0000","last_became_peered":"2024-02-10T07:31:51.282137+0000","last_unstale":"2024-02-10T07:38:51.783039+0000","last_undegraded":"2024-02-10T07:38:51.783039+0000","last_fullsized":"2024-02-10T07:38:51.783039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:25.985209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:37:25.985209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":365,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.787884+0000","last_change":"2024-02-10T07:38:45.925920+0000","last_active":"2024-02-10T07:38:51.787884+0000","last_peered":"2024-02-10T07:38:51.787884+0000","last_clean":"2024-02-10T07:38:51.787884+0000","last_became_active":"2024-02-10T07:31:53.294218+0000","last_became_peered":"2024-02-10T07:31:53.294218+0000","last_unstale":"2024-02-10T07:38:51.787884+0000","last_undegraded":"2024-02-10T07:38:51.787884+0000","last_fullsized":"2024-02-10T07:38:51.787884+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-10T07:38:45.925741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:38:45.925741+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:57.489821+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"217'3719","reported_seq":3763,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:57.690326+0000","last_change":"2024-02-10T07:38:14.780614+0000","last_active":"2024-02-10T07:38:57.690326+0000","last_peered":"2024-02-10T07:38:57.690326+0000","last_clean":"2024-02-10T07:38:57.690326+0000","last_became_active":"2024-02-10T07:38:14.780264+0000","last_became_peered":"2024-02-10T07:38:14.780264+0000","last_unstale":"2024-02-10T07:38:57.690326+0000","last_undegraded":"2024-02-10T07:38:57.690326+0000","last_fullsized":"2024-02-10T07:38:57.690326+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3719,"log_dups_size":0,"ondisk_log_size":3719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:25.475891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243728384,"num_objects":3719,"num_object_clones":0,"num_object_copies":11157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3719,"num_write_kb":238016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":335,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779481+0000","last_change":"2024-02-10T07:38:39.231730+0000","last_active":"2024-02-10T07:38:51.779481+0000","last_peered":"2024-02-10T07:38:51.779481+0000","last_clean":"2024-02-10T07:38:51.779481+0000","last_became_active":"2024-02-10T07:36:33.753921+0000","last_became_peered":"2024-02-10T07:36:33.753921+0000","last_unstale":"2024-02-10T07:38:51.779481+0000","last_undegraded":"2024-02-10T07:38:51.779481+0000","last_fullsized":"2024-02-10T07:38:51.779481+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_clean_scrub_stamp":"2024-02-10T07:38:39.231605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:57.827005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":258,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779908+0000","last_change":"2024-02-10T07:37:32.921439+0000","last_active":"2024-02-10T07:38:51.779908+0000","last_peered":"2024-02-10T07:38:51.779908+0000","last_clean":"2024-02-10T07:38:51.779908+0000","last_became_active":"2024-02-10T07:36:33.759566+0000","last_became_peered":"2024-02-10T07:36:33.759566+0000","last_unstale":"2024-02-10T07:38:51.779908+0000","last_undegraded":"2024-02-10T07:38:51.779908+0000","last_fullsized":"2024-02-10T07:38:51.779908+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:32.921308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:32.921308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":338,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.774532+0000","last_change":"2024-02-10T07:37:36.765881+0000","last_active":"2024-02-10T07:38:51.774532+0000","last_peered":"2024-02-10T07:38:51.774532+0000","last_clean":"2024-02-10T07:38:51.774532+0000","last_became_active":"2024-02-10T07:31:51.320715+0000","last_became_peered":"2024-02-10T07:31:51.320715+0000","last_unstale":"2024-02-10T07:38:51.774532+0000","last_undegraded":"2024-02-10T07:38:51.774532+0000","last_fullsized":"2024-02-10T07:38:51.774532+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:37:36.765729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:55.408005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"89'16","reported_seq":394,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.775133+0000","last_change":"2024-02-10T07:37:31.825488+0000","last_active":"2024-02-10T07:38:51.775133+0000","last_peered":"2024-02-10T07:38:51.775133+0000","last_clean":"2024-02-10T07:38:51.775133+0000","last_became_active":"2024-02-10T07:37:06.193386+0000","last_became_peered":"2024-02-10T07:37:06.193386+0000","last_unstale":"2024-02-10T07:38:51.775133+0000","last_undegraded":"2024-02-10T07:38:51.775133+0000","last_fullsized":"2024-02-10T07:38:51.775133+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'16","last_scrub_stamp":"2024-02-10T07:37:31.825342+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:37:31.825342+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:00.309213+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"217'3318","reported_seq":3362,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:53.038948+0000","last_change":"2024-02-10T07:38:14.782039+0000","last_active":"2024-02-10T07:38:53.038948+0000","last_peered":"2024-02-10T07:38:53.038948+0000","last_clean":"2024-02-10T07:38:53.038948+0000","last_became_active":"2024-02-10T07:38:14.781754+0000","last_became_peered":"2024-02-10T07:38:14.781754+0000","last_unstale":"2024-02-10T07:38:53.038948+0000","last_undegraded":"2024-02-10T07:38:53.038948+0000","last_fullsized":"2024-02-10T07:38:53.038948+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3318,"log_dups_size":0,"ondisk_log_size":3318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:16.541925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217448448,"num_objects":3318,"num_object_clones":0,"num_object_copies":9954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3318,"num_write_kb":212352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":325,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.783305+0000","last_change":"2024-02-10T07:37:21.672164+0000","last_active":"2024-02-10T07:38:51.783305+0000","last_peered":"2024-02-10T07:38:51.783305+0000","last_clean":"2024-02-10T07:38:51.783305+0000","last_became_active":"2024-02-10T07:31:57.311427+0000","last_became_peered":"2024-02-10T07:31:57.311427+0000","last_unstale":"2024-02-10T07:38:51.783305+0000","last_undegraded":"2024-02-10T07:38:51.783305+0000","last_fullsized":"2024-02-10T07:38:51.783305+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:37:21.671876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:37:21.671876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":328,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.786387+0000","last_change":"2024-02-10T07:38:33.755591+0000","last_active":"2024-02-10T07:38:51.786387+0000","last_peered":"2024-02-10T07:38:51.786387+0000","last_clean":"2024-02-10T07:38:51.786387+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:38:51.786387+0000","last_undegraded":"2024-02-10T07:38:51.786387+0000","last_fullsized":"2024-02-10T07:38:51.786387+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:33.755481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:38:33.755481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:54.785308+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":345,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.778561+0000","last_change":"2024-02-10T07:38:21.809441+0000","last_active":"2024-02-10T07:38:51.778561+0000","last_peered":"2024-02-10T07:38:51.778561+0000","last_clean":"2024-02-10T07:38:51.778561+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:38:51.778561+0000","last_undegraded":"2024-02-10T07:38:51.778561+0000","last_fullsized":"2024-02-10T07:38:51.778561+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:21.809350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:56.900501+0000","last_clean_scrub_stamp":"2024-02-10T07:38:21.809350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:43.755545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"89'25","reported_seq":370,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.775110+0000","last_change":"2024-02-10T07:37:43.726839+0000","last_active":"2024-02-10T07:38:51.775110+0000","last_peered":"2024-02-10T07:38:51.775110+0000","last_clean":"2024-02-10T07:38:51.775110+0000","last_became_active":"2024-02-10T07:31:53.292277+0000","last_became_peered":"2024-02-10T07:31:53.292277+0000","last_unstale":"2024-02-10T07:38:51.775110+0000","last_undegraded":"2024-02-10T07:38:51.775110+0000","last_fullsized":"2024-02-10T07:38:51.775110+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'25","last_scrub_stamp":"2024-02-10T07:37:43.726688+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:33:20.251980+0000","last_clean_scrub_stamp":"2024-02-10T07:37:43.726688+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"217'3287","reported_seq":3331,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:53.605871+0000","last_change":"2024-02-10T07:38:14.779934+0000","last_active":"2024-02-10T07:38:53.605871+0000","last_peered":"2024-02-10T07:38:53.605871+0000","last_clean":"2024-02-10T07:38:53.605871+0000","last_became_active":"2024-02-10T07:38:14.779651+0000","last_became_peered":"2024-02-10T07:38:14.779651+0000","last_unstale":"2024-02-10T07:38:53.605871+0000","last_undegraded":"2024-02-10T07:38:53.605871+0000","last_fullsized":"2024-02-10T07:38:53.605871+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:13.762172+0000","last_clean_scrub_stamp":"2024-02-10T07:38:13.762172+0000","objects_scrubbed":0,"log_size":3287,"log_dups_size":0,"ondisk_log_size":3287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:42.849966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215416832,"num_objects":3287,"num_object_clones":0,"num_object_copies":9861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3287,"num_write_kb":210368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":326,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.775332+0000","last_change":"2024-02-10T07:38:29.654433+0000","last_active":"2024-02-10T07:38:51.775332+0000","last_peered":"2024-02-10T07:38:51.775332+0000","last_clean":"2024-02-10T07:38:51.775332+0000","last_became_active":"2024-02-10T07:31:57.308608+0000","last_became_peered":"2024-02-10T07:31:57.308608+0000","last_unstale":"2024-02-10T07:38:51.775332+0000","last_undegraded":"2024-02-10T07:38:51.775332+0000","last_fullsized":"2024-02-10T07:38:51.775332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:38:29.654310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:38:29.654310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:39:30.590763+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"158'10","reported_seq":435,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.783026+0000","last_change":"2024-02-10T07:37:45.789514+0000","last_active":"2024-02-10T07:38:51.783026+0000","last_peered":"2024-02-10T07:38:51.783026+0000","last_clean":"2024-02-10T07:38:51.783026+0000","last_became_active":"2024-02-10T07:36:33.776454+0000","last_became_peered":"2024-02-10T07:36:33.776454+0000","last_unstale":"2024-02-10T07:38:51.783026+0000","last_undegraded":"2024-02-10T07:38:51.783026+0000","last_fullsized":"2024-02-10T07:38:51.783026+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:37:45.789402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:37:45.789402+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:53.584470+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":330,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.779831+0000","last_change":"2024-02-10T07:36:54.961833+0000","last_active":"2024-02-10T07:38:51.779831+0000","last_peered":"2024-02-10T07:38:51.779831+0000","last_clean":"2024-02-10T07:38:51.779831+0000","last_became_active":"2024-02-10T07:31:51.288197+0000","last_became_peered":"2024-02-10T07:31:51.288197+0000","last_unstale":"2024-02-10T07:38:51.779831+0000","last_undegraded":"2024-02-10T07:38:51.779831+0000","last_fullsized":"2024-02-10T07:38:51.779831+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:36:54.961713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:36:54.961713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":349,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-02-10T07:38:51.774935+0000","last_change":"2024-02-10T07:37:48.735421+0000","last_active":"2024-02-10T07:38:51.774935+0000","last_peered":"2024-02-10T07:38:51.774935+0000","last_clean":"2024-02-10T07:38:51.774935+0000","last_became_active":"2024-02-10T07:31:53.287984+0000","last_became_peered":"2024-02-10T07:31:53.287984+0000","last_unstale":"2024-02-10T07:38:51.774935+0000","last_undegraded":"2024-02-10T07:38:51.774935+0000","last_fullsized":"2024-02-10T07:38:51.774935+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-10T07:37:48.735290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:37:48.735290+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:38:54.941049+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3590782976,"num_objects":54791,"num_object_clones":0,"num_object_copies":164373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54791,"num_write_kb":3506624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10906370048,"data_stored":10906370048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54791,"ondisk_log_size":54791,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":421,"num_read_kb":416,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":693,"num_read_kb":450,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":442,"ondisk_log_size":442,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037850,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1397932,"kb_used_data":1168228,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92347220,"statfs":{"total":95995035648,"available":94563553280,"internally_reserved":0,"allocated":1196265472,"data_stored":1195864542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":235207114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:38:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":2.3210000000000002,"15min":1.1319999999999999},"min":{"1min":0.216,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.5169999999999999,"5min":52.591999999999999,"15min":52.591999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.8999999999999999,"15min":1.0129999999999999},"min":{"1min":0.27700000000000002,"5min":0.25,"15min":0.23100000000000001},"max":{"1min":1.429,"5min":52.494,"15min":52.494},"last":0.51500000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:38:44 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.2549999999999999,"15min":0.77600000000000002},"min":{"1min":0.35899999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.1909999999999998,"5min":19.43,"15min":19.43},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":1.377,"15min":0.82299999999999995},"min":{"1min":0.48199999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.7320000000000002,"5min":19.460999999999999,"15min":19.460999999999999},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Feb 10 07:38:44 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.4830000000000001,"15min":0.86399999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.405,"5min":19.710000000000001,"15min":19.710000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.875,"5min":1.6459999999999999,"15min":0.93000000000000005},"min":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":4.5830000000000002,"5min":19.643000000000001,"15min":19.643000000000001},"last":0.66000000000000003}]},{"osd":3,"last update":"Sat Feb 10 07:38:44 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":2.3780000000000001,"15min":1.169},"min":{"1min":0.32200000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.3479999999999999,"5min":64.120999999999995,"15min":64.120999999999995},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.077,"5min":2.742,"15min":1.306},"min":{"1min":0.33900000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.399,"5min":64.400000000000006,"15min":64.400000000000006},"last":0.68999999999999995}]},{"osd":4,"last update":"Sat Feb 10 07:38:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.501,"15min":0.496},"min":{"1min":0.29899999999999999,"5min":0.252,"15min":0.252},"max":{"1min":1.2250000000000001,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":0.441},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.51900000000000002,"15min":0.51700000000000002},"min":{"1min":0.34300000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.488,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.36399999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:38:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.69599999999999995,"15min":0.71799999999999997},"min":{"1min":0.371,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1719999999999999,"5min":3.3969999999999998,"15min":3.3969999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.70399999999999996,"15min":0.71499999999999997},"min":{"1min":0.41199999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.4610000000000001,"5min":2.141,"15min":2.141},"last":0.97799999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:38:44 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.61099999999999999,"15min":0.59599999999999997},"min":{"1min":0.48399999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.387,"5min":2.2250000000000001,"15min":2.2250000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65000000000000002,"15min":0.59099999999999997},"min":{"1min":0.39600000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":1.2749999999999999,"5min":6.2169999999999996,"15min":6.2169999999999996},"last":0.56699999999999995}]}]},{"osd":6,"up_from":50,"seq":214748364895,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1340060,"kb_used_data":1156180,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92405092,"statfs":{"total":95995035648,"available":94622814208,"internally_reserved":0,"allocated":1183928320,"data_stored":1183543121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":188283334},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:38:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":1.8540000000000001,"15min":0.98799999999999999},"min":{"1min":0.246,"5min":0.217,"15min":0.217},"max":{"1min":2.9900000000000002,"5min":63.405999999999999,"15min":63.405999999999999},"last":3.528},{"interface":"front","average":{"1min":0.89200000000000002,"5min":2.4940000000000002,"15min":1.198},"min":{"1min":0.38,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":6.0590000000000002,"5min":67.748000000000005,"15min":67.748000000000005},"last":0.73499999999999999}]},{"osd":1,"last update":"Sat Feb 10 07:38:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.878,"15min":0.66300000000000003},"min":{"1min":0.35499999999999998,"5min":0.255,"15min":0.255},"max":{"1min":2.508,"5min":8.3940000000000001,"15min":8.3940000000000001},"last":2.1000000000000001},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.92600000000000005,"15min":0.68799999999999994},"min":{"1min":0.34000000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.8220000000000001,"5min":9.6120000000000001,"15min":9.6120000000000001},"last":4.3440000000000003}]},{"osd":2,"last update":"Sat Feb 10 07:38:25 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.3620000000000001,"15min":1.1759999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.1269999999999998,"5min":65.278999999999996,"15min":65.278999999999996},"last":3.4129999999999998},{"interface":"front","average":{"1min":0.72299999999999998,"5min":2.5920000000000001,"15min":1.244},"min":{"1min":0.29399999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.0550000000000002,"5min":65.265000000000001,"15min":65.265000000000001},"last":2.2450000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:38:25 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":2.0009999999999999,"15min":1.778},"min":{"1min":0.34699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":2.3879999999999999,"5min":62.835999999999999,"15min":62.835999999999999},"last":6.2610000000000001},{"interface":"front","average":{"1min":0.84699999999999998,"5min":1.5720000000000001,"15min":0.93400000000000005},"min":{"1min":0.39600000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.8570000000000002,"5min":63.878999999999998,"15min":63.878999999999998},"last":3.3519999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:38:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57799999999999996,"15min":0.50700000000000001},"min":{"1min":0.27900000000000003,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.6419999999999999,"5min":7.7839999999999998,"15min":7.7839999999999998},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.52100000000000002,"15min":0.495},"min":{"1min":0.29399999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.3559999999999999,"5min":2.3559999999999999,"15min":2.3559999999999999},"last":3.5019999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:38:35 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.71699999999999997,"15min":0.745},"min":{"1min":0.35299999999999998,"5min":0.312,"15min":0.312},"max":{"1min":1.028,"5min":4.3760000000000003,"15min":4.3760000000000003},"last":0.314},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.6659999999999999,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":2.536}]},{"osd":7,"last update":"Sat Feb 10 07:38:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.66400000000000003,"15min":0.57299999999999995},"min":{"1min":0.35399999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70999999999999996,"5min":7.859,"15min":7.859},"last":1.3959999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65000000000000002,"15min":0.59599999999999997},"min":{"1min":0.38100000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.1519999999999999,"5min":2.4470000000000001,"15min":2.4470000000000001},"last":2.129}]}]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1665028,"kb_used_data":1471420,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92080124,"statfs":{"total":95995035648,"available":94290046976,"internally_reserved":0,"allocated":1506734080,"data_stored":1506308324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":198244806},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:38:24 2024","interfaces":[{"interface":"back","average":{"1min":7.6020000000000003,"5min":2.661,"15min":1.123},"min":{"1min":0.255,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":114.27,"5min":114.27,"15min":114.27},"last":3.806},{"interface":"front","average":{"1min":7.6749999999999998,"5min":3.2040000000000002,"15min":1.3180000000000001},"min":{"1min":0.253,"5min":0.22500000000000001,"15min":0.22},"max":{"1min":119.499,"5min":119.499,"15min":119.499},"last":1.4470000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:38:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":1.3720000000000001,"15min":0.72699999999999998},"min":{"1min":0.23899999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.629,"5min":8.9209999999999994,"15min":8.9209999999999994},"last":1.075},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.569,"15min":0.80300000000000005},"min":{"1min":0.28000000000000003,"5min":0.23999999999999999,"15min":0.23699999999999999},"max":{"1min":4.2839999999999998,"5min":7.2229999999999999,"15min":7.2229999999999999},"last":1.7509999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:38:20 2024","interfaces":[{"interface":"back","average":{"1min":1.8879999999999999,"5min":1.698,"15min":0.84899999999999998},"min":{"1min":0.22,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":1.8080000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.518,"15min":0.84599999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":6.1050000000000004,"5min":11.698,"15min":11.698},"last":0.996}]},{"osd":4,"last update":"Sat Feb 10 07:38:37 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":1.105,"15min":1.0249999999999999},"min":{"1min":0.33800000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.0619999999999998,"5min":8.9730000000000008,"15min":10.292},"last":1.099},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.109,"15min":0.755},"min":{"1min":0.32300000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":7.8079999999999998,"5min":7.8079999999999998,"15min":7.8079999999999998},"last":0.94399999999999995}]},{"osd":5,"last update":"Sat Feb 10 07:38:37 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":1.5289999999999999,"15min":1.6579999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":2.0059999999999998,"5min":9.0410000000000004,"15min":9.0410000000000004},"last":1.042},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.4750000000000001,"15min":1.5609999999999999},"min":{"1min":0.46500000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":5.1459999999999999,"5min":5.2069999999999999,"15min":5.2069999999999999},"last":0.52900000000000003}]},{"osd":6,"last update":"Sat Feb 10 07:38:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.1799999999999999,"15min":0.81100000000000005},"min":{"1min":0.36499999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":11.244999999999999,"5min":11.244999999999999,"15min":11.244999999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.21,"15min":0.86199999999999999},"min":{"1min":0.41199999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":3.8050000000000002,"5min":9.0090000000000003,"15min":9.0090000000000003},"last":1.129}]},{"osd":7,"last update":"Sat Feb 10 07:38:50 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.3320000000000001,"15min":0.86399999999999999},"min":{"1min":0.48299999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":4.9050000000000002,"5min":6.6360000000000001,"15min":6.6360000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.3,"15min":0.85099999999999998},"min":{"1min":0.51800000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.4319999999999999,"5min":11.212,"15min":11.212},"last":1.0169999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738493,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1504188,"kb_used_data":1234220,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92240964,"statfs":{"total":95995035648,"available":94454747136,"internally_reserved":0,"allocated":1263841280,"data_stored":1263432775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:38:20 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.7170000000000001,"15min":0.80200000000000005},"min":{"1min":0.216,"5min":0.192,"15min":0.192},"max":{"1min":7.6989999999999998,"5min":16.709,"15min":16.709},"last":7.5090000000000003},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.742,"15min":0.82699999999999996},"min":{"1min":0.26800000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.4589999999999996,"5min":9.7170000000000005,"15min":9.7170000000000005},"last":0.89000000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:37:51 2024","interfaces":[{"interface":"back","average":{"1min":2.8460000000000001,"5min":1.8620000000000001,"15min":0.879},"min":{"1min":0.57099999999999995,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":12.002000000000001,"5min":12.002000000000001,"15min":12.002000000000001},"last":2.6160000000000001},{"interface":"front","average":{"1min":2.2709999999999999,"5min":1.726,"15min":0.85999999999999999},"min":{"1min":0.499,"5min":0.248,"15min":0.23799999999999999},"max":{"1min":4.4870000000000001,"5min":9.0540000000000003,"15min":9.0540000000000003},"last":0.86399999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:38:09 2024","interfaces":[{"interface":"back","average":{"1min":3.7789999999999999,"5min":2.347,"15min":1.0760000000000001},"min":{"1min":0.32600000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":27.242000000000001,"5min":27.242000000000001,"15min":27.242000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.843,"5min":2.181,"15min":1.042},"min":{"1min":0.34200000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.3979999999999997,"5min":12.885,"15min":12.885},"last":0.77700000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:38:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.2569999999999999,"15min":0.77800000000000002},"min":{"1min":0.372,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":5.4260000000000002,"5min":5.4260000000000002,"15min":5.4260000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.5700000000000001,"15min":0.95999999999999996},"min":{"1min":0.44600000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.3879999999999999,"5min":11.867000000000001,"15min":19.219000000000001},"last":2.669}]},{"osd":5,"last update":"Sat Feb 10 07:38:37 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":3.0099999999999998,"15min":3.3140000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":5.468,"5min":33.289999999999999,"15min":33.289999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.0169999999999999,"5min":3.6549999999999998,"15min":4.0949999999999998},"min":{"1min":0.51000000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.4119999999999999,"5min":33.430999999999997,"15min":33.430999999999997},"last":2.7040000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:38:20 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.4450000000000001,"15min":0.97399999999999998},"min":{"1min":0.44,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":4.3550000000000004,"5min":6.4749999999999996,"15min":19.16},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.423,"5min":1.3919999999999999,"15min":0.88300000000000001},"min":{"1min":0.433,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":5.5049999999999999,"5min":7.5119999999999996,"15min":7.5119999999999996},"last":7.3170000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:37:51 2024","interfaces":[{"interface":"back","average":{"1min":2.3719999999999999,"5min":1.607,"15min":1.5609999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":11.912000000000001,"5min":11.912000000000001,"15min":19.178000000000001},"last":2.738},{"interface":"front","average":{"1min":2.2429999999999999,"5min":1.4730000000000001,"15min":0.92600000000000005},"min":{"1min":0.44700000000000001,"5min":0.376,"15min":0.376},"max":{"1min":4.431,"5min":8.5700000000000003,"15min":8.5700000000000003},"last":0.91600000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411442,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1494252,"kb_used_data":1250652,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":92250900,"statfs":{"total":95995035648,"available":94464921600,"internally_reserved":0,"allocated":1280667648,"data_stored":1280274743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:37:50 2024","interfaces":[{"interface":"back","average":{"1min":3.4409999999999998,"5min":3.2330000000000001,"15min":1.331},"min":{"1min":0.47099999999999997,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":21.984000000000002,"5min":90.046999999999997,"15min":90.046999999999997},"last":0.248},{"interface":"front","average":{"1min":2.351,"5min":3.1259999999999999,"15min":1.321},"min":{"1min":0.35299999999999998,"5min":0.27700000000000002,"15min":0.25700000000000001},"max":{"1min":10.867000000000001,"5min":90.016000000000005,"15min":90.016000000000005},"last":0.72399999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:37:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":1.7,"15min":0.85199999999999998},"min":{"1min":0.54700000000000004,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":3.2930000000000001,"5min":12.382,"15min":12.382},"last":4.548},{"interface":"front","average":{"1min":1.7929999999999999,"5min":1.7090000000000001,"15min":0.85399999999999998},"min":{"1min":0.371,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":3.6389999999999998,"5min":9.5630000000000006,"15min":9.5630000000000006},"last":1.2689999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:38:18 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":2.0419999999999998,"15min":1.034},"min":{"1min":0.30399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.7530000000000001,"5min":14.525,"15min":14.525},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.095,"5min":2.1440000000000001,"15min":1.079},"min":{"1min":0.30199999999999999,"5min":0.30099999999999999,"15min":0.29699999999999999},"max":{"1min":2.7250000000000001,"5min":11.739000000000001,"15min":11.739000000000001},"last":3.6099999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:38:37 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.6399999999999999,"15min":0.96599999999999997},"min":{"1min":0.42899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.2010000000000001,"5min":33.634999999999998,"15min":33.634999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.80300000000000005,"5min":1.8009999999999999,"15min":1.0409999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":2.4609999999999999,"5min":33.554000000000002,"15min":33.554000000000002},"last":1.234}]},{"osd":5,"last update":"Sat Feb 10 07:38:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.161,"15min":1.1870000000000001},"min":{"1min":0.35899999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":4.9429999999999996,"5min":5.0579999999999998,"15min":5.0579999999999998},"last":4.274},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.5349999999999999,"15min":1.613},"min":{"1min":0.51400000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.4740000000000002,"5min":9.4280000000000008,"15min":9.4280000000000008},"last":1.829}]},{"osd":6,"last update":"Sat Feb 10 07:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":1.456,"15min":1.0660000000000001},"min":{"1min":0.44800000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.3079999999999998,"5min":9.3230000000000004,"15min":9.3230000000000004},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.3859999999999999,"15min":0.89100000000000001},"min":{"1min":0.376,"5min":0.315,"15min":0.315},"max":{"1min":2.4609999999999999,"5min":8.1899999999999995,"15min":8.1899999999999995},"last":0.77700000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:38:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.464,"15min":0.92900000000000005},"min":{"1min":0.42499999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":2.5169999999999999,"5min":8.2189999999999994,"15min":8.2189999999999994},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":1.446,"15min":0.94099999999999995},"min":{"1min":0.52800000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.024,"5min":9.2859999999999996,"15min":9.2859999999999996},"last":0.74399999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1541116,"kb_used_data":1266996,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92204036,"statfs":{"total":95995035648,"available":94416932864,"internally_reserved":0,"allocated":1297403904,"data_stored":1296985899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":280689095},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.8340000000000001,"5min":2.181,"15min":0.97799999999999998},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.8079999999999998,"5min":25.277000000000001,"15min":25.277000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.4770000000000001,"5min":2.1520000000000001,"15min":0.995},"min":{"1min":0.25600000000000001,"5min":0.222,"15min":0.18099999999999999},"max":{"1min":8.9570000000000007,"5min":28.390000000000001,"15min":28.390000000000001},"last":0.54200000000000004}]},{"osd":1,"last update":"Sat Feb 10 07:38:19 2024","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":1.7470000000000001,"15min":0.86199999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23899999999999999},"max":{"1min":10.279,"5min":13.076000000000001,"15min":13.076000000000001},"last":2.9409999999999998},{"interface":"front","average":{"1min":1.9159999999999999,"5min":2.0219999999999998,"15min":0.96599999999999997},"min":{"1min":0.29399999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":7.3860000000000001,"5min":12.179,"15min":12.179},"last":3.0009999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:38:19 2024","interfaces":[{"interface":"back","average":{"1min":2.2309999999999999,"5min":3.3359999999999999,"15min":1.429},"min":{"1min":0.372,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.5490000000000004,"5min":163.441,"15min":163.441},"last":1.306},{"interface":"front","average":{"1min":2.1099999999999999,"5min":3.7130000000000001,"15min":1.526},"min":{"1min":0.34100000000000003,"5min":0.26100000000000001,"15min":0.218},"max":{"1min":8.8599999999999994,"5min":166.82400000000001,"15min":166.82400000000001},"last":0.84399999999999997}]},{"osd":4,"last update":"Sat Feb 10 07:38:42 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.4159999999999999,"15min":0.85099999999999998},"min":{"1min":0.47099999999999997,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.859,"5min":8.8300000000000001,"15min":8.8300000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.516,"15min":0.91200000000000003},"min":{"1min":0.443,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.9009999999999998,"5min":7.3390000000000004,"15min":7.3390000000000004},"last":1.113}]},{"osd":5,"last update":"Sat Feb 10 07:38:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.1319999999999999,"15min":1.077},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":8.9179999999999993,"5min":8.9179999999999993,"15min":8.9179999999999993},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.24,"15min":1.204},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":7.234,"5min":7.234,"15min":7.234},"last":1.2250000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:38:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.8220000000000001,"15min":1.056},"min":{"1min":0.40500000000000003,"5min":0.38,"15min":0.38},"max":{"1min":7.3520000000000003,"5min":32.048999999999999,"15min":32.048999999999999},"last":1.8149999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.675,"15min":0.95999999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":7.5099999999999998,"5min":25.937000000000001,"15min":25.937000000000001},"last":0.57399999999999995}]},{"osd":7,"last update":"Sat Feb 10 07:37:54 2024","interfaces":[{"interface":"back","average":{"1min":2.484,"5min":1.6930000000000001,"15min":0.96299999999999997},"min":{"1min":0.41399999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":25.273,"5min":25.273,"15min":25.273},"last":0.51500000000000001},{"interface":"front","average":{"1min":2.7749999999999999,"5min":1.7390000000000001,"15min":1.0269999999999999},"min":{"1min":0.45700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":27.75,"5min":27.75,"15min":27.75},"last":1.262}]}]},{"osd":4,"up_from":32,"seq":137438953577,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1366648,"kb_used_data":1173040,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92378504,"statfs":{"total":95995035648,"available":94595588096,"internally_reserved":0,"allocated":1201192960,"data_stored":1200779163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:38:36 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":2.5419999999999998,"15min":1.258},"min":{"1min":0.26600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.0579999999999998,"5min":58.142000000000003,"15min":58.142000000000003},"last":0.247},{"interface":"front","average":{"1min":0.75600000000000001,"5min":2.4079999999999999,"15min":1.169},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.1880000000000002,"5min":59.042000000000002,"15min":59.042000000000002},"last":0.34399999999999997}]},{"osd":1,"last update":"Sat Feb 10 07:38:36 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.4359999999999999,"15min":0.84299999999999997},"min":{"1min":0.253,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.784,"5min":49.777999999999999,"15min":49.777999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.4239999999999999,"15min":0.84999999999999998},"min":{"1min":0.307,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.0649999999999999,"5min":49.765999999999998,"15min":49.765999999999998},"last":0.47099999999999997}]},{"osd":2,"last update":"Sat Feb 10 07:38:36 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":2.0070000000000001,"15min":1.028},"min":{"1min":0.29499999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.96,"5min":50.162999999999997,"15min":50.162999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.503,"5min":1.8300000000000001,"15min":1.008},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.431,"5min":50.453000000000003,"15min":50.453000000000003},"last":4.242}]},{"osd":3,"last update":"Sat Feb 10 07:38:36 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":2.1539999999999999,"15min":1.1060000000000001},"min":{"1min":0.32200000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":7.016,"5min":54.366,"15min":54.366},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.628,"5min":2.0390000000000001,"15min":1.073},"min":{"1min":0.34000000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.776,"5min":53.039000000000001,"15min":53.039000000000001},"last":1.0009999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:38:36 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.49299999999999999,"15min":0.48199999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.46999999999999997,"15min":0.46000000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.1639999999999999,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.309}]},{"osd":6,"last update":"Sat Feb 10 07:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.51600000000000001,"15min":0.46100000000000002},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.84099999999999997,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.53000000000000003,"15min":0.46500000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.80700000000000005,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.51400000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:38:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56000000000000005,"15min":0.52400000000000002},"min":{"1min":0.34499999999999997,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.86399999999999999,"5min":1.98,"15min":1.98},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.65300000000000002,"15min":0.55300000000000005},"min":{"1min":0.37,"5min":0.318,"15min":0.318},"max":{"1min":7.0359999999999996,"5min":7.0359999999999996,"15min":7.0359999999999996},"last":0.436}]}]},{"osd":5,"up_from":156,"seq":670014898206,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1930596,"kb_used_data":1815708,"kb_used_omap":4,"kb_used_meta":114875,"kb_avail":91814556,"statfs":{"total":95995035648,"available":94018105344,"internally_reserved":0,"allocated":1859284992,"data_stored":1858892269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":117632674},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.5049999999999999,"15min":1.597},"min":{"1min":0.308,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.6580000000000004,"5min":8.734,"15min":8.734},"last":1.474},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.603,"15min":1.7709999999999999},"min":{"1min":0.26400000000000001,"5min":0.246,"15min":0.246},"max":{"1min":2.0049999999999999,"5min":8.1999999999999993,"15min":8.1999999999999993},"last":2.165}]},{"osd":1,"last update":"Sat Feb 10 07:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":3.0089999999999999,"15min":3.415},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.9410000000000001,"5min":47.139000000000003,"15min":47.139000000000003},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.68700000000000006,"5min":3.1269999999999998,"15min":3.5339999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.3839999999999999,"5min":47.328000000000003,"15min":47.328000000000003},"last":0.52900000000000003}]},{"osd":2,"last update":"Sat Feb 10 07:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.728,"15min":1.8819999999999999},"min":{"1min":0.33600000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.8999999999999999,"5min":11.601000000000001,"15min":11.601000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":1.841,"15min":1.9970000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.9119999999999999,"5min":11.618,"15min":11.618},"last":0.58499999999999996}]},{"osd":3,"last update":"Sat Feb 10 07:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.4399999999999999,"15min":1.5509999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.4300000000000002,"5min":8.3870000000000005,"15min":8.3870000000000005},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.88700000000000001,"5min":1.746,"15min":1.889},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":2.7000000000000002,"5min":8.3209999999999997,"15min":8.3209999999999997},"last":0.83899999999999997}]},{"osd":4,"last update":"Sat Feb 10 07:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.51700000000000002,"15min":0.52600000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.70199999999999996,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.46600000000000003,"15min":0.45900000000000002},"min":{"1min":0.29899999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.70799999999999996,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.56699999999999995}]},{"osd":6,"last update":"Sat Feb 10 07:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.81699999999999995,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.67400000000000004,"15min":0.69799999999999995},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81399999999999995,"5min":4.2839999999999998,"15min":4.2839999999999998},"last":0.74199999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35399999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.68899999999999995,"5min":0.877,"15min":0.877},"last":1.657},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.60299999999999998,"15min":0.60899999999999999},"min":{"1min":0.32700000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.91000000000000003,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.55300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1292042240,"data_stored":1292042240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1511718912,"data_stored":1511718912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1299185664,"data_stored":1299185664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1305870336,"data_stored":1305870336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1207435264,"data_stored":1207435264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1891172352,"data_stored":1891172352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1202847744,"data_stored":1202847744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1196097536,"data_stored":1196097536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:39:00.701 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-10T07:39:00.701 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:39:00.701 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-02-10T07:39:00.702 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-02-10T07:39:00.778 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:00.808 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:00.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:00.909 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:00.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:00.956 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:00.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:01.057 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:01.135 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:01.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:01.237 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:01.254 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:00 smithi008 ceph-mon[29816]: pgmap v334: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-02-10T07:39:01.254 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:00 smithi008 ceph-mon[40743]: pgmap v334: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-02-10T07:39:01.299 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:01.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:01.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:00 smithi170 ceph-mon[33394]: pgmap v334: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-02-10T07:39:01.401 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:01.448 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:01.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:01.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:01.550 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:01.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:01.584 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:01.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:01.636 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 35 16 64579 64563 115.265 82.8125 0.0204561 0.0086701 2024-02-10T07:39:01.684 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:01.727 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:01.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:01.829 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:01.861 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:01.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:01.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:01.963 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:01.995 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:01.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:01.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:02.097 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:02.135 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:02.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:02.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:02.240 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:02.271 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:02.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:02.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:02.275 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:01 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:39:02.275 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:01 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:39:02.276 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:01 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:39:02.276 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:01 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:39:02.372 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:02.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:01 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:39:02.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:01 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:39:02.414 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:02.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:02.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:02.512 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:02.514 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:39:02.516 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:02.544 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:02.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:02.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:02.558 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:39:02.605 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:39:02.636 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 36 16 65797 65781 114.177 76.125 0.00695033 0.00875301 2024-02-10T07:39:02.645 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:02.658 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:39:02.673 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:02.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:02.701 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:39:02.746 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:39:02.774 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:02.785 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:39:02.809 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:02.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:02.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:02.827 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:39:02.835 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:39:02.835 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:39:02.835 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:39:02.835 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:39:02.835 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:39:02.835 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:39:02.835 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:39:02.835 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:39:02.911 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:02.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:02.949 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:02.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:03.050 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:03.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:02 smithi008 ceph-mon[29816]: pgmap v335: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-10T07:39:03.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:02 smithi008 ceph-mon[29816]: osdmap e220: 8 total, 8 up, 8 in 2024-02-10T07:39:03.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:02 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:39:03.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:02 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:03.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:02 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4244012344' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:03.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:02 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2931432667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:39:03.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:02 smithi008 ceph-mon[40743]: pgmap v335: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-10T07:39:03.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:02 smithi008 ceph-mon[40743]: osdmap e220: 8 total, 8 up, 8 in 2024-02-10T07:39:03.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:02 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:39:03.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:02 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:03.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:02 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4244012344' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:03.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:02 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2931432667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:39:03.088 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:03.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:03.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:03.190 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:03.241 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:03.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:03.343 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:03.347 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:39:03.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:03.377 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:03.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:03.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:39:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:39:03.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:02 smithi170 ceph-mon[33394]: pgmap v335: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-10T07:39:03.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:02 smithi170 ceph-mon[33394]: osdmap e220: 8 total, 8 up, 8 in 2024-02-10T07:39:03.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:02 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:39:03.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:02 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:03.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:02 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4244012344' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:03.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:02 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2931432667' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:39:03.478 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:03.516 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:03.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:03.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:03.618 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:03.636 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 37 16 67148 67132 113.372 84.4375 0.0158454 0.00881578 2024-02-10T07:39:03.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:03.661 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:03.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:03.762 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:03.806 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:03.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:03.908 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:03.947 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:03.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:04.049 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:04.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:04.086 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:04.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:04.187 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:04.216 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:03 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:04.216 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:03 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:39:04.216 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:03 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2931432667' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:39:04.216 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:03 smithi008 ceph-mon[29816]: osdmap e221: 8 total, 8 up, 8 in 2024-02-10T07:39:04.216 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:03 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:04.216 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:03 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:39:04.216 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:03 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2931432667' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:39:04.217 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:03 smithi008 ceph-mon[40743]: osdmap e221: 8 total, 8 up, 8 in 2024-02-10T07:39:04.235 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:04.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:04.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:04.337 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:04.375 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:04.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:04.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:03 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:03 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:39:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:03 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2931432667' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:39:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:03 smithi170 ceph-mon[33394]: osdmap e221: 8 total, 8 up, 8 in 2024-02-10T07:39:04.476 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:04.516 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:04.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:04.618 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:04.637 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 38 16 68574 68558 112.733 89.125 0.00933414 0.00886587 2024-02-10T07:39:04.657 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:04.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:04.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:04.759 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:04.799 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:04.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:04.900 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:04.939 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:04.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:05.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:05.078 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:05.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:05.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:05.180 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:05.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:05.212 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:05.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:05.313 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:05.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:04 smithi008 ceph-mon[29816]: pgmap v338: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 691 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-02-10T07:39:05.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:04 smithi008 ceph-mon[40743]: pgmap v338: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 691 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-02-10T07:39:05.350 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:05.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:04 smithi170 ceph-mon[33394]: pgmap v338: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 691 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-02-10T07:39:05.451 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:05.484 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:05.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:05.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:05.586 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:05.628 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:05.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:05.637 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 39 16 69846 69830 111.881 79.5 0.0138007 0.00893329 2024-02-10T07:39:05.730 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:05.764 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:05.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:05.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:05.866 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:05.902 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:05.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:06.004 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:06.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:06.041 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:06.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:06.143 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:06.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:06.186 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:06.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:06.218 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:39:06.287 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:06.319 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:06.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:06.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:06.421 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:06.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:06.475 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:06.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:06.577 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:06.619 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:06.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:06.637 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:39:06.638250+0000 min lat: 0.00127253 max lat: 0.194119 avg lat: 0.00900587 2024-02-10T07:39:06.637 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:39:06.637 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 40 16 71059 71043 110.979 75.8125 0.0048527 0.00900587 2024-02-10T07:39:06.720 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:06.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:06.753 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:06.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:06.854 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:06.891 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:06.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:06.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:06.993 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:07.028 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:07.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:07.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:06 smithi008 ceph-mon[29816]: pgmap v339: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 592 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-02-10T07:39:07.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:06 smithi008 ceph-mon[40743]: pgmap v339: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 592 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-02-10T07:39:07.128 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:07.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:07.162 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:07.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:07.264 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:07.296 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:07.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:07.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:06 smithi170 ceph-mon[33394]: pgmap v339: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 592 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-02-10T07:39:07.398 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:07.433 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:07.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:07.534 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:07.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:07.569 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:07.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:07.637 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 41 16 72412 72396 110.335 84.5625 0.00730392 0.00905864 2024-02-10T07:39:07.670 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:07.697 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:07.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:07.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:07.800 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:07.834 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:07.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:07.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:07.936 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:07.978 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:07.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:08.079 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:08.114 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:08.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:08.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:08.216 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:08.248 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:08.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:08.350 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:08.388 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:08.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:08.490 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:08.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:08.533 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:08.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:08.634 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:08.637 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 42 16 73699 73683 109.623 80.4375 0.0142202 0.00911702 2024-02-10T07:39:08.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:08.675 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:08.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:08.776 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:08.813 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:08.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:08.914 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:08.981 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:08.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:09.039 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:39:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:39:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:39:09.039 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:08 smithi008 ceph-mon[29816]: pgmap v340: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-10T07:39:09.039 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:08 smithi008 ceph-mon[40743]: pgmap v340: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-10T07:39:09.083 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:09.126 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:09.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:09.227 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:09.265 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:09.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:09.367 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:09.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:08 smithi170 ceph-mon[33394]: pgmap v340: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-10T07:39:09.404 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:09.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:09.506 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:09.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:09.539 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:09.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:09.637 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 43 16 74977 74961 108.931 79.875 0.00633716 0.00917475 2024-02-10T07:39:09.640 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:09.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:09.678 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:09.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:09.779 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:09.817 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:09.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:09.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:09.829 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:39:09.829 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:39:09.919 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:09.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:09.961 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:09.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:10.062 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:10.113 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:10.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:10.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:10.215 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:10.254 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:10.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:10.356 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:10.393 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:10.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:10.495 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:10.544 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:10.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:10.638 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 44 16 76214 76198 108.212 77.3125 0.0182615 0.00923557 2024-02-10T07:39:10.645 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:10.687 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:10.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:10.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:10.715 INFO:teuthology.orchestra.run.smithi008.stderr:no valid command found; 10 closest matches: 2024-02-10T07:39:10.716 INFO:teuthology.orchestra.run.smithi008.stderr:0 2024-02-10T07:39:10.716 INFO:teuthology.orchestra.run.smithi008.stderr:1 2024-02-10T07:39:10.716 INFO:teuthology.orchestra.run.smithi008.stderr:2 2024-02-10T07:39:10.716 INFO:teuthology.orchestra.run.smithi008.stderr:abort 2024-02-10T07:39:10.716 INFO:teuthology.orchestra.run.smithi008.stderr:assert 2024-02-10T07:39:10.716 INFO:teuthology.orchestra.run.smithi008.stderr:bench [] [] [] [] 2024-02-10T07:39:10.716 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs debug_inject_read_zeros 2024-02-10T07:39:10.716 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs files list 2024-02-10T07:39:10.716 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs stats 2024-02-10T07:39:10.716 INFO:teuthology.orchestra.run.smithi008.stderr:bluestore allocator dump block 2024-02-10T07:39:10.716 INFO:teuthology.orchestra.run.smithi008.stderr:Error EINVAL: invalid command 2024-02-10T07:39:10.789 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:10.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:10.832 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:10.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:10.933 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:10.966 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:39:10.973 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:10.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:11.006 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:10 smithi008 ceph-mon[29816]: pgmap v341: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-02-10T07:39:11.007 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:10 smithi008 ceph-mon[40743]: pgmap v341: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-02-10T07:39:11.074 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:11.134 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:11.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:11.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:11.174 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-10T07:39:11.236 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:11.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:11.264 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:11.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:11.365 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:11.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:10 smithi170 ceph-mon[33394]: pgmap v341: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-02-10T07:39:11.418 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:11.442 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:11.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:11.543 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:11.589 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:11.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:11.639 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 45 16 77382 77366 107.426 73 0.0144223 0.00930382 2024-02-10T07:39:11.691 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:11.723 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:11.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:11.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:11.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:11.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:11.861 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:11.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-10T07:39:11.868 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-10T07:39:11.868 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-02-10T07:39:11.930 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:11.961 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:12.026 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:12.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:12.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:12.128 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:12.196 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:12.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:12.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:12.298 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:12.331 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:12.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:12.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:12.433 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:12.465 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:12.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:12.567 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:12.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:12.600 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:12.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:12.639 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 46 16 78480 78464 106.583 68.625 0.0125789 0.00937758 2024-02-10T07:39:12.701 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:12.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:12.748 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:12.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:12.849 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:12.892 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:12.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:12.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:12.995 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:13.029 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:13.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:13.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:13.051 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:12 smithi008 ceph-mon[29816]: pgmap v342: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 931 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-10T07:39:13.052 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:12 smithi008 ceph-mon[40743]: pgmap v342: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 931 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-10T07:39:13.058 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:12 smithi170 ceph-mon[33394]: pgmap v342: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 931 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-10T07:39:13.132 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:13.166 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:13.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:13.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:13.267 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:13.295 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:13.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:13.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:13.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:39:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:39:13.397 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:13.424 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:13.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:13.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:13.526 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:13.570 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:13.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:13.639 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 47 16 79562 79546 105.754 67.625 0.00513954 0.00945112 2024-02-10T07:39:13.671 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:13.717 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:13.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:13.818 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:13.857 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:13.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:13.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:13.959 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:13.997 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:13.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:13.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:14.052 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:13 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:14.052 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:13 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:14.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:14.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:13 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:14.134 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:14.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:14.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:14.236 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:14.268 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:14.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:14.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:14.370 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:14.417 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:14.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:14.518 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:14.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:14.553 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:14.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:14.639 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 48 16 80637 80621 104.95 67.1875 0.00985459 0.00952319 2024-02-10T07:39:14.654 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:14.687 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:14.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:14.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:14.788 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:14.841 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:14.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:14.942 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:14.984 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:14.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:15.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:14 smithi008 ceph-mon[29816]: pgmap v343: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 975 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-02-10T07:39:15.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:14 smithi008 ceph-mon[40743]: pgmap v343: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 975 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-02-10T07:39:15.086 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:15.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:15.116 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:15.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:15.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:14 smithi170 ceph-mon[33394]: pgmap v343: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 975 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-02-10T07:39:15.217 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:15.254 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:15.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:15.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:15.356 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:15.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:15.397 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:15.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:15.498 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:15.534 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:15.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:15.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:15.635 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:15.641 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 49 16 81838 81822 104.341 75.0625 0.00674437 0.00957735 2024-02-10T07:39:15.645 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:15.701 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:15.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:15.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:15.803 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:15.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:15.837 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:15.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:15.938 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:15.972 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:15.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:16.074 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:16.114 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:16.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:16.216 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:16.254 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:16.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:16.355 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:16.389 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:16.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:16.490 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:16.531 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:16.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:16.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:16.632 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:16.640 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 50 16 82939 82923 103.63 68.8125 0.0138008 0.0096447 2024-02-10T07:39:16.667 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:16.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:16.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:16.806 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:16.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:16.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:16.907 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:39:16.907 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:16.953 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:16.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:16.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:17.055 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:17.087 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:17.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:17.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:17.189 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:17.232 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:17.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:16 smithi008 ceph-mon[29816]: pgmap v344: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-10T07:39:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:16 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4194327700' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:39:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:16 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:39:17.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:16 smithi008 ceph-mon[40743]: pgmap v344: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-10T07:39:17.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:16 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4194327700' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:39:17.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:16 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:39:17.335 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:17.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:17.370 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:17.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:17.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:16 smithi170 ceph-mon[33394]: pgmap v344: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-10T07:39:17.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:16 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4194327700' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:39:17.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:16 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:39:17.472 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:17.570 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:17.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:17.640 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 51 15 84171 84156 103.109 77.0625 0.0147107 0.00969366 2024-02-10T07:39:17.652 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:17.672 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:17.709 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:17.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:17.811 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:17.847 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:17.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:17.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:17.949 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:18.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:18.005 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:18.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:18.021 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:17 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:39:18.021 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:17 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:39:18.022 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:17 smithi008 ceph-mon[29816]: osdmap e222: 8 total, 8 up, 8 in 2024-02-10T07:39:18.022 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:17 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:18.022 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:17 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:39:18.022 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:17 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:39:18.022 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:17 smithi008 ceph-mon[40743]: osdmap e222: 8 total, 8 up, 8 in 2024-02-10T07:39:18.022 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:17 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:18.107 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:18.148 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:18.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:18.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:18.250 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:18.289 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:18.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:18.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:18.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:17 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:39:18.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:17 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:39:18.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:17 smithi170 ceph-mon[33394]: osdmap e222: 8 total, 8 up, 8 in 2024-02-10T07:39:18.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:17 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:18.391 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:18.431 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:18.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:18.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:18.533 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:18.568 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:18.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:18.640 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 52 16 85397 85381 102.598 76.5625 0.0213088 0.00974212 2024-02-10T07:39:18.670 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:18.708 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:18.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:18.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:18.809 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:18.848 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:18.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:18.950 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:18.990 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:18.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:18.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:19.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:39:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:39:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-mon[29816]: 4.1f scrub starts 2024-02-10T07:39:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-mon[29816]: 4.1f scrub ok 2024-02-10T07:39:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-mon[29816]: 6.a scrub starts 2024-02-10T07:39:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-mon[29816]: 6.a scrub ok 2024-02-10T07:39:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-mon[29816]: 5.6 scrub starts 2024-02-10T07:39:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-mon[29816]: 5.6 scrub ok 2024-02-10T07:39:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-mon[29816]: pgmap v346: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-10T07:39:19.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-mon[40743]: 4.1f scrub starts 2024-02-10T07:39:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-mon[40743]: 4.1f scrub ok 2024-02-10T07:39:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-mon[40743]: 6.a scrub starts 2024-02-10T07:39:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-mon[40743]: 6.a scrub ok 2024-02-10T07:39:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-mon[40743]: 5.6 scrub starts 2024-02-10T07:39:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-mon[40743]: 5.6 scrub ok 2024-02-10T07:39:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:18 smithi008 ceph-mon[40743]: pgmap v346: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-10T07:39:19.091 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:19.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:19.141 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:19.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:19.244 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:19.274 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:19.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:19.375 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:18 smithi170 ceph-mon[33394]: 4.1f scrub starts 2024-02-10T07:39:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:18 smithi170 ceph-mon[33394]: 4.1f scrub ok 2024-02-10T07:39:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:18 smithi170 ceph-mon[33394]: 6.a scrub starts 2024-02-10T07:39:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:18 smithi170 ceph-mon[33394]: 6.a scrub ok 2024-02-10T07:39:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:18 smithi170 ceph-mon[33394]: 5.6 scrub starts 2024-02-10T07:39:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:18 smithi170 ceph-mon[33394]: 5.6 scrub ok 2024-02-10T07:39:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:18 smithi170 ceph-mon[33394]: pgmap v346: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-10T07:39:19.417 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:19.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:19.519 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:19.563 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:19.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:19.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:19.640 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 53 15 86554 86539 102.028 72.375 0.00759983 0.00979596 2024-02-10T07:39:19.665 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:19.694 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:19.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:19.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:19.796 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:19.827 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:19.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:19.928 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:19.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:19.960 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:19.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:20.061 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:20.100 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:20.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:20.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:20.202 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:20.247 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:20.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:20.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:19 smithi008 ceph-mon[29816]: 8.1 scrub starts 2024-02-10T07:39:20.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:19 smithi008 ceph-mon[29816]: 3.17 scrub starts 2024-02-10T07:39:20.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:19 smithi008 ceph-mon[29816]: 3.17 scrub ok 2024-02-10T07:39:20.316 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:19 smithi008 ceph-mon[40743]: 8.1 scrub starts 2024-02-10T07:39:20.316 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:19 smithi008 ceph-mon[40743]: 3.17 scrub starts 2024-02-10T07:39:20.317 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:19 smithi008 ceph-mon[40743]: 3.17 scrub ok 2024-02-10T07:39:20.349 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:20.379 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:20.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:19 smithi170 ceph-mon[33394]: 8.1 scrub starts 2024-02-10T07:39:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:19 smithi170 ceph-mon[33394]: 3.17 scrub starts 2024-02-10T07:39:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:19 smithi170 ceph-mon[33394]: 3.17 scrub ok 2024-02-10T07:39:20.481 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:20.512 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:20.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:20.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:20.614 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:20.640 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 54 16 87673 87657 101.433 69.875 0.0183776 0.00985347 2024-02-10T07:39:20.691 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:20.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:20.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:20.793 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:20.843 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:20.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:20.936 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:39:20.945 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:21.062 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:21.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:21.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:21.159 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:20 smithi008 ceph-mon[40743]: pgmap v347: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-10T07:39:21.160 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:20 smithi008 ceph-mon[29816]: pgmap v347: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-10T07:39:21.164 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:21.225 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:21.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:21.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:21.251 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:39:21.251 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:39:21.269 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:39:21.307 INFO:teuthology.orchestra.run.smithi008.stderr:no valid command found; 10 closest matches: 2024-02-10T07:39:21.307 INFO:teuthology.orchestra.run.smithi008.stderr:0 2024-02-10T07:39:21.307 INFO:teuthology.orchestra.run.smithi008.stderr:1 2024-02-10T07:39:21.307 INFO:teuthology.orchestra.run.smithi008.stderr:2 2024-02-10T07:39:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:abort 2024-02-10T07:39:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:assert 2024-02-10T07:39:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:bench [] [] [] [] 2024-02-10T07:39:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs debug_inject_read_zeros 2024-02-10T07:39:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs files list 2024-02-10T07:39:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs stats 2024-02-10T07:39:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:bluestore allocator dump block 2024-02-10T07:39:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:Error EINVAL: invalid command 2024-02-10T07:39:21.312 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:39:21.328 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:21.352 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:39:21.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:21.377 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:21.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:21.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:20 smithi170 ceph-mon[33394]: pgmap v347: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-10T07:39:21.395 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:39:21.433 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:39:21.475 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:39:21.478 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:21.479 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:39:21.479 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:39:21.479 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:39:21.479 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:39:21.479 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:39:21.479 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:39:21.479 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:39:21.479 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:39:21.521 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:21.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:21.623 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:21.640 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 55 16 88631 88615 100.677 59.875 0.00939501 0.00992786 2024-02-10T07:39:21.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:21.666 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:21.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:21.766 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:21.804 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:21.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:21.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:21.906 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:21.940 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:21.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:21.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:22.041 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:22.070 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[29816]: 3.10 scrub starts 2024-02-10T07:39:22.070 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[29816]: 3.10 scrub ok 2024-02-10T07:39:22.070 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[29816]: 6.19 scrub starts 2024-02-10T07:39:22.070 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[29816]: 6.19 scrub ok 2024-02-10T07:39:22.070 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[29816]: 6.8 scrub starts 2024-02-10T07:39:22.070 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[29816]: 6.8 scrub ok 2024-02-10T07:39:22.070 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3596118528' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:22.070 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[29816]: 8.1 scrub ok 2024-02-10T07:39:22.070 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[29816]: 5.14 scrub starts 2024-02-10T07:39:22.070 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[29816]: 5.14 scrub ok 2024-02-10T07:39:22.070 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[29816]: 6.9 scrub starts 2024-02-10T07:39:22.071 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[29816]: 6.9 scrub ok 2024-02-10T07:39:22.071 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[29816]: osdmap e223: 8 total, 8 up, 8 in 2024-02-10T07:39:22.071 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[40743]: 3.10 scrub starts 2024-02-10T07:39:22.071 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[40743]: 3.10 scrub ok 2024-02-10T07:39:22.071 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[40743]: 6.19 scrub starts 2024-02-10T07:39:22.071 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[40743]: 6.19 scrub ok 2024-02-10T07:39:22.071 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[40743]: 6.8 scrub starts 2024-02-10T07:39:22.071 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[40743]: 6.8 scrub ok 2024-02-10T07:39:22.071 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3596118528' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:22.071 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[40743]: 8.1 scrub ok 2024-02-10T07:39:22.071 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[40743]: 5.14 scrub starts 2024-02-10T07:39:22.071 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[40743]: 5.14 scrub ok 2024-02-10T07:39:22.072 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[40743]: 6.9 scrub starts 2024-02-10T07:39:22.072 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[40743]: 6.9 scrub ok 2024-02-10T07:39:22.072 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:21 smithi008 ceph-mon[40743]: osdmap e223: 8 total, 8 up, 8 in 2024-02-10T07:39:22.079 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:22.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:22.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:22.181 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:22.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:22.228 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:22.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:22.329 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:22.364 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:22.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:22.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:21 smithi170 ceph-mon[33394]: 3.10 scrub starts 2024-02-10T07:39:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:21 smithi170 ceph-mon[33394]: 3.10 scrub ok 2024-02-10T07:39:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:21 smithi170 ceph-mon[33394]: 6.19 scrub starts 2024-02-10T07:39:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:21 smithi170 ceph-mon[33394]: 6.19 scrub ok 2024-02-10T07:39:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:21 smithi170 ceph-mon[33394]: 6.8 scrub starts 2024-02-10T07:39:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:21 smithi170 ceph-mon[33394]: 6.8 scrub ok 2024-02-10T07:39:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:21 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3596118528' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:21 smithi170 ceph-mon[33394]: 8.1 scrub ok 2024-02-10T07:39:22.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:21 smithi170 ceph-mon[33394]: 5.14 scrub starts 2024-02-10T07:39:22.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:21 smithi170 ceph-mon[33394]: 5.14 scrub ok 2024-02-10T07:39:22.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:21 smithi170 ceph-mon[33394]: 6.9 scrub starts 2024-02-10T07:39:22.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:21 smithi170 ceph-mon[33394]: 6.9 scrub ok 2024-02-10T07:39:22.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:21 smithi170 ceph-mon[33394]: osdmap e223: 8 total, 8 up, 8 in 2024-02-10T07:39:22.466 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:22.501 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:22.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:22.603 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:22.630 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:22.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:22.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:22.640 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 56 15 89841 89826 100.231 75.6875 0.00476406 0.00997118 2024-02-10T07:39:22.731 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:22.767 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:22.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:22.869 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:22.916 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:22.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:23.018 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:23.051 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:23.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[29816]: 5.1f scrub starts 2024-02-10T07:39:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[29816]: 5.1f scrub ok 2024-02-10T07:39:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[29816]: 4.13 scrub starts 2024-02-10T07:39:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[29816]: 4.13 scrub ok 2024-02-10T07:39:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[29816]: 3.11 scrub starts 2024-02-10T07:39:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[29816]: 3.11 scrub ok 2024-02-10T07:39:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[29816]: pgmap v349: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 849 op/s 2024-02-10T07:39:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[29816]: 6.4 scrub starts 2024-02-10T07:39:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[29816]: 6.4 scrub ok 2024-02-10T07:39:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[29816]: 5.5 scrub starts 2024-02-10T07:39:23.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[29816]: 5.5 scrub ok 2024-02-10T07:39:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[40743]: 5.1f scrub starts 2024-02-10T07:39:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[40743]: 5.1f scrub ok 2024-02-10T07:39:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[40743]: 4.13 scrub starts 2024-02-10T07:39:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[40743]: 4.13 scrub ok 2024-02-10T07:39:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[40743]: 3.11 scrub starts 2024-02-10T07:39:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[40743]: 3.11 scrub ok 2024-02-10T07:39:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[40743]: pgmap v349: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 849 op/s 2024-02-10T07:39:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[40743]: 6.4 scrub starts 2024-02-10T07:39:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[40743]: 6.4 scrub ok 2024-02-10T07:39:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[40743]: 5.5 scrub starts 2024-02-10T07:39:23.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:22 smithi008 ceph-mon[40743]: 5.5 scrub ok 2024-02-10T07:39:23.153 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:23.185 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:23.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:23.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:23.240 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:39:23.240 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:39:23.244 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.244 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15087.0:90492 8.f 8:f6e42f9a:::benchmark_data_smithi008_109566_object90491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-02-10T07:39:23.244 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:39:23.219806+0000", 2024-02-10T07:39:23.244 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.016971252999999999, 2024-02-10T07:39:23.244 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.017014026000000002, 2024-02-10T07:39:23.244 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15087", 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 90492 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.219806+0000", 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:23.245 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.219806+0000", 2024-02-10T07:39:23.246 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.246 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.246 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:23.246 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.219807+0000", 2024-02-10T07:39:23.246 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2160000000000001e-06 2024-02-10T07:39:23.246 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.246 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:23.246 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.219830+0000", 2024-02-10T07:39:23.246 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3501000000000001e-05 2024-02-10T07:39:23.246 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.246 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.219839+0000", 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0180000000000007e-06 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.219847+0000", 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3579999999999997e-06 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.224991+0000", 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0051448450000000003 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.247 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.225150+0000", 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015842699999999999 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.225300+0000", 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000150381 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.236563+0000", 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011262350000000001 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:23.248 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.15087.0:90495 8.7 e223/200 8:ed1a9b7f:::benchmark_data_smithi008_109566_object90494:head v 223'5605, mlcod=223'5602)", 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:39:23.224574+0000", 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.012202453, 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012346062, 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.224574+0000", 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.224574+0000", 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.224580+0000", 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6300000000000003e-06 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.224608+0000", 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7532999999999999e-05 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.224615+0000", 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2649999999999999e-06 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.224620+0000", 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4090000000000002e-06 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.228394+0000", 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0037734169999999998 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.228412+0000", 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8326999999999999e-05 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15087.0:90504 8.0 8:035272c2:::benchmark_data_smithi008_109566_object90503:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:39:23.229263+0000", 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0075133539999999999, 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0077271350000000004, 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:39:23.252 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15087", 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 90504 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.229263+0000", 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.229263+0000", 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.229265+0000", 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.339e-06 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.229286+0000", 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1036999999999999e-05 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.229297+0000", 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1005000000000001e-05 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.229304+0000", 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2099999999999996e-06 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.229359+0000", 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4539000000000001e-05 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.229481+0000", 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012274700000000001 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.255 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.256 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,5", 2024-02-10T07:39:23.256 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.229637+0000", 2024-02-10T07:39:23.256 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000155826 2024-02-10T07:39:23.256 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.256 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:23.256 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.256 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.256 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.256 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.15087.0:90499 8.8 e223/200 8:1ff59ded:::benchmark_data_smithi008_109566_object90498:head v 223'5563, mlcod=223'5561)", 2024-02-10T07:39:23.256 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:39:23.231197+0000", 2024-02-10T07:39:23.256 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0055796650000000001, 2024-02-10T07:39:23.256 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0058697330000000002, 2024-02-10T07:39:23.256 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:23.256 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:39:23.257 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:23.257 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.257 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:23.257 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.231197+0000", 2024-02-10T07:39:23.257 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.258 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:23.258 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.231197+0000", 2024-02-10T07:39:23.258 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.258 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.258 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:23.258 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.231198+0000", 2024-02-10T07:39:23.258 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6299999999999993e-07 2024-02-10T07:39:23.258 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.258 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:23.258 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.231242+0000", 2024-02-10T07:39:23.258 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4236999999999997e-05 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.231254+0000", 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1436e-05 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.231261+0000", 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6310000000000003e-06 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.232550+0000", 2024-02-10T07:39:23.259 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0012881279999999999 2024-02-10T07:39:23.260 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.260 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.260 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:23.260 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.232569+0000", 2024-02-10T07:39:23.260 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9080999999999999e-05 2024-02-10T07:39:23.260 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.260 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:23.260 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.260 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.260 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.260 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15087.0:90508 8.0 8:02b1f4ba:::benchmark_data_smithi008_109566_object90507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-02-10T07:39:23.260 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:39:23.235017+0000", 2024-02-10T07:39:23.260 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.001759322, 2024-02-10T07:39:23.260 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022556099999999999, 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15087", 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 90508 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235017+0000", 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:23.261 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235017+0000", 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235019+0000", 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4330000000000001e-06 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235041+0000", 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2245000000000001e-05 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235053+0000", 2024-02-10T07:39:23.262 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1895e-05 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235062+0000", 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6160000000000002e-06 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235241+0000", 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017951100000000001 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235411+0000", 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000170209 2024-02-10T07:39:23.263 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.264 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.264 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,5", 2024-02-10T07:39:23.264 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235579+0000", 2024-02-10T07:39:23.264 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016756599999999999 2024-02-10T07:39:23.264 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.264 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:23.264 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.264 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.264 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.15087.0:90506 8.8 e223/200 8:190b6ea1:::benchmark_data_smithi008_109566_object90505:head v 223'5564, mlcod=223'5561)", 2024-02-10T07:39:23.264 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:39:23.235104+0000", 2024-02-10T07:39:23.264 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.001672953, 2024-02-10T07:39:23.264 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002255661, 2024-02-10T07:39:23.264 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235104+0000", 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235104+0000", 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:23.265 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235105+0000", 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0800000000000004e-07 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235160+0000", 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5028999999999998e-05 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235168+0000", 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6419999999999996e-06 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235175+0000", 2024-02-10T07:39:23.266 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4609999999999996e-06 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235422+0000", 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00024776000000000003 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235445+0000", 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3034999999999999e-05 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.267 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15087.0:90486 8.f 8:fbc6629a:::benchmark_data_smithi008_109566_object90485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:39:23.214133+0000", 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.022644002999999999, 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.023300120000000001, 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15087", 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 90486 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.214133+0000", 2024-02-10T07:39:23.268 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.214133+0000", 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.214134+0000", 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.099e-06 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.214155+0000", 2024-02-10T07:39:23.269 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1512999999999999e-05 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.214167+0000", 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1676e-05 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.214175+0000", 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0229999999999992e-06 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.220351+0000", 2024-02-10T07:39:23.270 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0061762789999999998 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.220485+0000", 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000133187 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.220630+0000", 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014552200000000001 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.236486+0000", 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015856377000000001 2024-02-10T07:39:23.271 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.272 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.272 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:23.272 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.236525+0000", 2024-02-10T07:39:23.272 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8512e-05 2024-02-10T07:39:23.272 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.272 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.272 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:23.272 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.236902+0000", 2024-02-10T07:39:23.272 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00037730499999999998 2024-02-10T07:39:23.272 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.272 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.272 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:23.272 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.237174+0000", 2024-02-10T07:39:23.272 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00027172999999999998 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop_reply(client.15087.0:90504 8.0 e223/200)", 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:39:23.235914+0000", 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.00086316300000000002, 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001624288, 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235914+0000", 2024-02-10T07:39:23.273 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.274 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.274 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:23.274 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235914+0000", 2024-02-10T07:39:23.274 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.274 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.274 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:23.274 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235914+0000", 2024-02-10T07:39:23.274 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7199999999999998e-07 2024-02-10T07:39:23.274 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.274 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:23.274 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235919+0000", 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5020000000000003e-06 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235930+0000", 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0813e-05 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.235937+0000", 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2359999999999998e-06 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.275 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop_reply(client.15087.0:90492 8.f e223/200)", 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:39:23.237003+0000", 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4294967295.9997735, 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00058208900000000004, 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.237003+0000", 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:23.276 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.237003+0000", 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.237004+0000", 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0999999999999997e-07 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.237009+0000", 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6999999999999999e-06 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.237020+0000", 2024-02-10T07:39:23.277 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0791999999999999e-05 2024-02-10T07:39:23.278 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:23.278 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:23.278 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:23.278 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:39:23.237027+0000", 2024-02-10T07:39:23.278 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.267e-06 2024-02-10T07:39:23.278 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.278 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:23.278 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.278 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:23.278 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:39:23.278 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 9 2024-02-10T07:39:23.278 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:39:23.288 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:23.330 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:23.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:23.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:39:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:39:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:22 smithi170 ceph-mon[33394]: 5.1f scrub starts 2024-02-10T07:39:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:22 smithi170 ceph-mon[33394]: 5.1f scrub ok 2024-02-10T07:39:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:22 smithi170 ceph-mon[33394]: 4.13 scrub starts 2024-02-10T07:39:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:22 smithi170 ceph-mon[33394]: 4.13 scrub ok 2024-02-10T07:39:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:22 smithi170 ceph-mon[33394]: 3.11 scrub starts 2024-02-10T07:39:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:22 smithi170 ceph-mon[33394]: 3.11 scrub ok 2024-02-10T07:39:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:22 smithi170 ceph-mon[33394]: pgmap v349: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 849 op/s 2024-02-10T07:39:23.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:22 smithi170 ceph-mon[33394]: 6.4 scrub starts 2024-02-10T07:39:23.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:22 smithi170 ceph-mon[33394]: 6.4 scrub ok 2024-02-10T07:39:23.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:22 smithi170 ceph-mon[33394]: 5.5 scrub starts 2024-02-10T07:39:23.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:22 smithi170 ceph-mon[33394]: 5.5 scrub ok 2024-02-10T07:39:23.432 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:23.472 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:23.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:23.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:23.574 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:23.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:23.614 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:23.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:23.640 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 57 16 90987 90971 99.7277 71.5625 0.0134799 0.0100225 2024-02-10T07:39:23.716 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:23.748 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:23.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:23.850 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:23.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:23.886 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:23.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:23.987 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:24.025 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:24.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:24.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:24.127 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:24.172 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:24.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:24.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:24.190 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:23 smithi008 ceph-mon[29816]: 5.f scrub starts 2024-02-10T07:39:24.190 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:23 smithi008 ceph-mon[29816]: 5.f scrub ok 2024-02-10T07:39:24.190 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:23 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:24.190 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:23 smithi008 ceph-mon[40743]: 5.f scrub starts 2024-02-10T07:39:24.190 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:23 smithi008 ceph-mon[40743]: 5.f scrub ok 2024-02-10T07:39:24.190 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:23 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:24.191 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-10T07:39:24.274 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:24.322 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:24.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:23 smithi170 ceph-mon[33394]: 5.f scrub starts 2024-02-10T07:39:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:23 smithi170 ceph-mon[33394]: 5.f scrub ok 2024-02-10T07:39:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:23 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:24.424 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:24.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:24.455 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:24.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:24.538 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:39:24.557 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:24.591 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:24.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:24.640 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 58 16 92135 92119 99.2453 71.75 0.0275376 0.0100705 2024-02-10T07:39:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-10T07:39:24.645 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-10T07:39:24.645 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-02-10T07:39:24.676 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:24.692 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:24.774 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:24.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:24.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:24.777 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:24.876 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:24.907 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:24.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:24.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:25.009 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:25.052 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:25.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:25.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:25.154 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:25.170 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[29816]: 5.16 scrub starts 2024-02-10T07:39:25.170 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[29816]: 5.16 scrub ok 2024-02-10T07:39:25.170 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[29816]: 5.1 scrub starts 2024-02-10T07:39:25.170 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[29816]: 5.1 scrub ok 2024-02-10T07:39:25.171 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[29816]: 6.1c scrub starts 2024-02-10T07:39:25.171 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[29816]: 6.1c scrub ok 2024-02-10T07:39:25.171 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[29816]: pgmap v350: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-10T07:39:25.171 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[29816]: 6.13 scrub starts 2024-02-10T07:39:25.171 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[29816]: 6.13 scrub ok 2024-02-10T07:39:25.171 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[40743]: 5.16 scrub starts 2024-02-10T07:39:25.171 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[40743]: 5.16 scrub ok 2024-02-10T07:39:25.171 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[40743]: 5.1 scrub starts 2024-02-10T07:39:25.171 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[40743]: 5.1 scrub ok 2024-02-10T07:39:25.171 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[40743]: 6.1c scrub starts 2024-02-10T07:39:25.171 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[40743]: 6.1c scrub ok 2024-02-10T07:39:25.171 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[40743]: pgmap v350: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-10T07:39:25.171 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[40743]: 6.13 scrub starts 2024-02-10T07:39:25.172 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:24 smithi008 ceph-mon[40743]: 6.13 scrub ok 2024-02-10T07:39:25.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:25.187 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:25.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:25.289 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:25.325 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:25.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:24 smithi170 ceph-mon[33394]: 5.16 scrub starts 2024-02-10T07:39:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:24 smithi170 ceph-mon[33394]: 5.16 scrub ok 2024-02-10T07:39:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:24 smithi170 ceph-mon[33394]: 5.1 scrub starts 2024-02-10T07:39:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:24 smithi170 ceph-mon[33394]: 5.1 scrub ok 2024-02-10T07:39:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:24 smithi170 ceph-mon[33394]: 6.1c scrub starts 2024-02-10T07:39:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:24 smithi170 ceph-mon[33394]: 6.1c scrub ok 2024-02-10T07:39:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:24 smithi170 ceph-mon[33394]: pgmap v350: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-10T07:39:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:24 smithi170 ceph-mon[33394]: 6.13 scrub starts 2024-02-10T07:39:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:24 smithi170 ceph-mon[33394]: 6.13 scrub ok 2024-02-10T07:39:25.428 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:25.459 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:25.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:25.561 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:25.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:25.615 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:25.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:25.641 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 59 16 93201 93185 98.6924 66.625 0.0089152 0.010128 2024-02-10T07:39:25.717 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:25.777 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:25.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:25.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:25.878 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:25.925 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:25.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:26.026 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:26.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:26.065 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:26.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:26.166 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:26.203 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:26.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:26.305 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:26.323 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:25 smithi008 ceph-mon[40743]: 4.14 scrub starts 2024-02-10T07:39:26.323 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:25 smithi008 ceph-mon[40743]: 4.14 scrub ok 2024-02-10T07:39:26.323 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:25 smithi008 ceph-mon[40743]: 3.0 scrub starts 2024-02-10T07:39:26.323 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:25 smithi008 ceph-mon[40743]: 3.0 scrub ok 2024-02-10T07:39:26.323 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:25 smithi008 ceph-mon[40743]: 3.12 scrub starts 2024-02-10T07:39:26.323 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:25 smithi008 ceph-mon[40743]: 3.12 scrub ok 2024-02-10T07:39:26.324 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:25 smithi008 ceph-mon[29816]: 4.14 scrub starts 2024-02-10T07:39:26.324 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:25 smithi008 ceph-mon[29816]: 4.14 scrub ok 2024-02-10T07:39:26.324 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:25 smithi008 ceph-mon[29816]: 3.0 scrub starts 2024-02-10T07:39:26.324 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:25 smithi008 ceph-mon[29816]: 3.0 scrub ok 2024-02-10T07:39:26.324 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:25 smithi008 ceph-mon[29816]: 3.12 scrub starts 2024-02-10T07:39:26.324 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:25 smithi008 ceph-mon[29816]: 3.12 scrub ok 2024-02-10T07:39:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:26.350 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:26.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:25 smithi170 ceph-mon[33394]: 4.14 scrub starts 2024-02-10T07:39:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:25 smithi170 ceph-mon[33394]: 4.14 scrub ok 2024-02-10T07:39:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:25 smithi170 ceph-mon[33394]: 3.0 scrub starts 2024-02-10T07:39:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:25 smithi170 ceph-mon[33394]: 3.0 scrub ok 2024-02-10T07:39:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:25 smithi170 ceph-mon[33394]: 3.12 scrub starts 2024-02-10T07:39:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:25 smithi170 ceph-mon[33394]: 3.12 scrub ok 2024-02-10T07:39:26.452 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:26.504 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:26.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:26.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:26.606 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:26.641 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:39:26.641351+0000 min lat: 0.00127253 max lat: 0.194119 avg lat: 0.010176 2024-02-10T07:39:26.641 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:39:26.641 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 60 16 94325 94309 98.2183 70.25 0.0157152 0.010176 2024-02-10T07:39:26.643 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:26.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:26.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:26.745 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:26.788 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:26.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:26.789 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:26.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:26.790 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:26.890 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:26.933 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:26.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:27.035 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:27.070 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:27.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:27.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:27.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:27.210 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:27.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:27.311 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:27.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:26 smithi008 ceph-mon[29816]: 2.2 scrub starts 2024-02-10T07:39:27.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:26 smithi008 ceph-mon[29816]: 2.2 scrub ok 2024-02-10T07:39:27.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:26 smithi008 ceph-mon[29816]: pgmap v351: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-10T07:39:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:26 smithi008 ceph-mon[40743]: 2.2 scrub starts 2024-02-10T07:39:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:26 smithi008 ceph-mon[40743]: 2.2 scrub ok 2024-02-10T07:39:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:26 smithi008 ceph-mon[40743]: pgmap v351: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-10T07:39:27.350 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:27.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:27.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:26 smithi170 ceph-mon[33394]: 2.2 scrub starts 2024-02-10T07:39:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:26 smithi170 ceph-mon[33394]: 2.2 scrub ok 2024-02-10T07:39:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:26 smithi170 ceph-mon[33394]: pgmap v351: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-10T07:39:27.452 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:27.489 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:27.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:27.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:27.590 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:27.639 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:27.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:27.641 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 61 16 95429 95413 97.7388 69 0.0120656 0.0102262 2024-02-10T07:39:27.742 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:27.773 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:27.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:27.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:27.875 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:27.908 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:27.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:27.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:28.009 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:28.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:28.061 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:28.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:28.162 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:28.206 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:28.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:28.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:27 smithi170 ceph-mon[33394]: 6.f scrub starts 2024-02-10T07:39:28.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:27 smithi170 ceph-mon[33394]: 6.f scrub ok 2024-02-10T07:39:28.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:27 smithi170 ceph-mon[33394]: 8.5 scrub starts 2024-02-10T07:39:28.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:27 smithi170 ceph-mon[33394]: 4.2 scrub starts 2024-02-10T07:39:28.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:27 smithi170 ceph-mon[33394]: 4.2 scrub ok 2024-02-10T07:39:28.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:27 smithi170 ceph-mon[33394]: 8.e scrub starts 2024-02-10T07:39:28.307 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:28.331 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:28.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:27 smithi008 ceph-mon[29816]: 6.f scrub starts 2024-02-10T07:39:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:27 smithi008 ceph-mon[29816]: 6.f scrub ok 2024-02-10T07:39:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:27 smithi008 ceph-mon[29816]: 8.5 scrub starts 2024-02-10T07:39:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:27 smithi008 ceph-mon[29816]: 4.2 scrub starts 2024-02-10T07:39:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:27 smithi008 ceph-mon[29816]: 4.2 scrub ok 2024-02-10T07:39:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:27 smithi008 ceph-mon[29816]: 8.e scrub starts 2024-02-10T07:39:28.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:27 smithi008 ceph-mon[40743]: 6.f scrub starts 2024-02-10T07:39:28.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:27 smithi008 ceph-mon[40743]: 6.f scrub ok 2024-02-10T07:39:28.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:27 smithi008 ceph-mon[40743]: 8.5 scrub starts 2024-02-10T07:39:28.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:27 smithi008 ceph-mon[40743]: 4.2 scrub starts 2024-02-10T07:39:28.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:27 smithi008 ceph-mon[40743]: 4.2 scrub ok 2024-02-10T07:39:28.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:27 smithi008 ceph-mon[40743]: 8.e scrub starts 2024-02-10T07:39:28.432 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:28.465 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:28.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:28.567 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:28.592 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:28.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:28.641 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 62 16 96383 96367 97.124 59.625 0.0152081 0.0102914 2024-02-10T07:39:28.694 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:28.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:28.724 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:28.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:28.826 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:28.862 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:28.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:28.964 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:28.983 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:39:29.009 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:29.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:29.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-mon[29816]: 4.8 scrub starts 2024-02-10T07:39:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-mon[29816]: 4.8 scrub ok 2024-02-10T07:39:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-mon[29816]: pgmap v352: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-02-10T07:39:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-mon[29816]: 5.17 scrub starts 2024-02-10T07:39:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-mon[29816]: 5.17 scrub ok 2024-02-10T07:39:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/185367041' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:39:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-mon[29816]: 8.6 scrub starts 2024-02-10T07:39:29.083 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:39:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:39:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-mon[40743]: 4.8 scrub starts 2024-02-10T07:39:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-mon[40743]: 4.8 scrub ok 2024-02-10T07:39:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-mon[40743]: pgmap v352: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-02-10T07:39:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-mon[40743]: 5.17 scrub starts 2024-02-10T07:39:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-mon[40743]: 5.17 scrub ok 2024-02-10T07:39:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/185367041' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:39:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:28 smithi008 ceph-mon[40743]: 8.6 scrub starts 2024-02-10T07:39:29.112 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:29.147 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:29.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:29.248 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:29.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:29.294 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:29.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:28 smithi170 ceph-mon[33394]: 4.8 scrub starts 2024-02-10T07:39:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:28 smithi170 ceph-mon[33394]: 4.8 scrub ok 2024-02-10T07:39:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:28 smithi170 ceph-mon[33394]: pgmap v352: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-02-10T07:39:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:28 smithi170 ceph-mon[33394]: 5.17 scrub starts 2024-02-10T07:39:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:28 smithi170 ceph-mon[33394]: 5.17 scrub ok 2024-02-10T07:39:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:28 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/185367041' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:39:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:28 smithi170 ceph-mon[33394]: 8.6 scrub starts 2024-02-10T07:39:29.395 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:29.433 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:29.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:29.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:29.535 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:29.571 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:29.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:29.641 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 63 16 97304 97288 96.4961 57.5625 0.00483259 0.0103583 2024-02-10T07:39:29.672 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:29.708 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:29.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:29.810 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:29.859 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:29.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:29.961 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:29.990 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:29.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:29.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:30.092 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:30.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:30.124 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:30.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:30.226 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:30.265 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:30.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:30.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:30.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[29816]: 3.16 scrub starts 2024-02-10T07:39:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[29816]: 8.5 scrub ok 2024-02-10T07:39:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[29816]: 3.16 scrub ok 2024-02-10T07:39:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[29816]: 5.18 scrub starts 2024-02-10T07:39:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[29816]: 5.18 scrub ok 2024-02-10T07:39:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:39:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:39:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/185367041' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:39:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[29816]: osdmap e224: 8 total, 8 up, 8 in 2024-02-10T07:39:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[40743]: 3.16 scrub starts 2024-02-10T07:39:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[40743]: 8.5 scrub ok 2024-02-10T07:39:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[40743]: 3.16 scrub ok 2024-02-10T07:39:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[40743]: 5.18 scrub starts 2024-02-10T07:39:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[40743]: 5.18 scrub ok 2024-02-10T07:39:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:39:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:39:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/185367041' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:39:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:29 smithi008 ceph-mon[40743]: osdmap e224: 8 total, 8 up, 8 in 2024-02-10T07:39:30.367 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:29 smithi170 ceph-mon[33394]: 3.16 scrub starts 2024-02-10T07:39:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:29 smithi170 ceph-mon[33394]: 8.5 scrub ok 2024-02-10T07:39:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:29 smithi170 ceph-mon[33394]: 3.16 scrub ok 2024-02-10T07:39:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:29 smithi170 ceph-mon[33394]: 5.18 scrub starts 2024-02-10T07:39:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:29 smithi170 ceph-mon[33394]: 5.18 scrub ok 2024-02-10T07:39:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:29 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:39:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:29 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:39:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:29 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/185367041' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:39:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:29 smithi170 ceph-mon[33394]: osdmap e224: 8 total, 8 up, 8 in 2024-02-10T07:39:30.403 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:30.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:30.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:30.505 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:30.537 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:30.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:30.639 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:30.641 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 64 16 98360 98344 96.0195 66 0.00787011 0.0104095 2024-02-10T07:39:30.670 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:30.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:30.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:30.771 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:30.804 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:30.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:30.906 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:30.940 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:30.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:30.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:31.042 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:31.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:31.074 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:31.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:31.175 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:31.216 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:31.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:31.266 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:31 smithi008 ceph-mon[29816]: 5.15 scrub starts 2024-02-10T07:39:31.267 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:31 smithi008 ceph-mon[29816]: 5.15 scrub ok 2024-02-10T07:39:31.267 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:31 smithi008 ceph-mon[29816]: 5.d scrub starts 2024-02-10T07:39:31.267 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:31 smithi008 ceph-mon[29816]: 5.d scrub ok 2024-02-10T07:39:31.267 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:31 smithi008 ceph-mon[29816]: pgmap v354: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-10T07:39:31.267 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:31 smithi008 ceph-mon[29816]: 8.c scrub starts 2024-02-10T07:39:31.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:31 smithi008 ceph-mon[40743]: 5.15 scrub starts 2024-02-10T07:39:31.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:31 smithi008 ceph-mon[40743]: 5.15 scrub ok 2024-02-10T07:39:31.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:31 smithi008 ceph-mon[40743]: 5.d scrub starts 2024-02-10T07:39:31.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:31 smithi008 ceph-mon[40743]: 5.d scrub ok 2024-02-10T07:39:31.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:31 smithi008 ceph-mon[40743]: pgmap v354: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-10T07:39:31.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:31 smithi008 ceph-mon[40743]: 8.c scrub starts 2024-02-10T07:39:31.318 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:31.355 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:31.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:31.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:31 smithi170 ceph-mon[33394]: 5.15 scrub starts 2024-02-10T07:39:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:31 smithi170 ceph-mon[33394]: 5.15 scrub ok 2024-02-10T07:39:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:31 smithi170 ceph-mon[33394]: 5.d scrub starts 2024-02-10T07:39:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:31 smithi170 ceph-mon[33394]: 5.d scrub ok 2024-02-10T07:39:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:31 smithi170 ceph-mon[33394]: pgmap v354: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-10T07:39:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:31 smithi170 ceph-mon[33394]: 8.c scrub starts 2024-02-10T07:39:31.458 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:31.501 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:31.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:31.604 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:31.640 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:31.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:31.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:31.641 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 65 15 99394 99379 95.5375 64.6875 0.0111313 0.0104617 2024-02-10T07:39:31.742 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:31.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:31.774 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:31.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:31.875 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:31.927 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:31.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:32.029 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:32.057 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:32.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:32.159 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:32.207 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:32.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:32.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:32.309 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:32 smithi008 ceph-mon[29816]: 8.e scrub ok 2024-02-10T07:39:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:32 smithi008 ceph-mon[29816]: osdmap e225: 8 total, 8 up, 8 in 2024-02-10T07:39:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:32 smithi008 ceph-mon[40743]: 8.e scrub ok 2024-02-10T07:39:32.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:32 smithi008 ceph-mon[40743]: osdmap e225: 8 total, 8 up, 8 in 2024-02-10T07:39:32.351 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:32.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:32.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:32 smithi170 ceph-mon[33394]: 8.e scrub ok 2024-02-10T07:39:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:32 smithi170 ceph-mon[33394]: osdmap e225: 8 total, 8 up, 8 in 2024-02-10T07:39:32.452 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:32.500 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:32.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:32.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:32.602 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:32.642 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 66 16 100422 100406 95.0616 64.1875 0.00859001 0.0105139 2024-02-10T07:39:32.647 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:32.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:32.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:32.749 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:32.785 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:32.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:32.887 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:32.922 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:32.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:32.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:33.024 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:33.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:33.061 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:33.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:33.161 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:33.200 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:33.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:33.302 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:33.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[29816]: 6.15 scrub starts 2024-02-10T07:39:33.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[29816]: 6.15 scrub ok 2024-02-10T07:39:33.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[29816]: pgmap v356: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s 2024-02-10T07:39:33.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[29816]: 8.6 scrub ok 2024-02-10T07:39:33.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:39:33.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:33.318 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[29816]: 6.1e scrub starts 2024-02-10T07:39:33.318 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[29816]: 6.1e scrub ok 2024-02-10T07:39:33.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[40743]: 6.15 scrub starts 2024-02-10T07:39:33.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[40743]: 6.15 scrub ok 2024-02-10T07:39:33.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[40743]: pgmap v356: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s 2024-02-10T07:39:33.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[40743]: 8.6 scrub ok 2024-02-10T07:39:33.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:39:33.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:33.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[40743]: 6.1e scrub starts 2024-02-10T07:39:33.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:33 smithi008 ceph-mon[40743]: 6.1e scrub ok 2024-02-10T07:39:33.335 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:33.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:33.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:33.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:39:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:39:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:33 smithi170 ceph-mon[33394]: 6.15 scrub starts 2024-02-10T07:39:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:33 smithi170 ceph-mon[33394]: 6.15 scrub ok 2024-02-10T07:39:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:33 smithi170 ceph-mon[33394]: pgmap v356: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 815 op/s 2024-02-10T07:39:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:33 smithi170 ceph-mon[33394]: 8.6 scrub ok 2024-02-10T07:39:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:39:33.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:33.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:33 smithi170 ceph-mon[33394]: 6.1e scrub starts 2024-02-10T07:39:33.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:33 smithi170 ceph-mon[33394]: 6.1e scrub ok 2024-02-10T07:39:33.437 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:33.440 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:39:33.440 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:39:33.440 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-02-10T07:39:33.440 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-02-10T07:39:33.440 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:39:33.466 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:33.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:33.568 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:33.601 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:33.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:33.642 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 67 16 101481 101465 94.6306 66.1875 0.00432872 0.0105616 2024-02-10T07:39:33.703 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:33.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:33.746 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:33.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:33.847 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:33.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:33.878 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:33.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:33.980 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:34.022 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:34.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:34.124 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:34.159 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:34.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:34.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:34.261 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[29816]: 3.13 scrub starts 2024-02-10T07:39:34.261 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[29816]: 3.13 scrub ok 2024-02-10T07:39:34.261 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[29816]: 4.12 deep-scrub starts 2024-02-10T07:39:34.261 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[29816]: 4.12 deep-scrub ok 2024-02-10T07:39:34.261 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[29816]: 8.c scrub ok 2024-02-10T07:39:34.261 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:34.261 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[29816]: 4.0 scrub starts 2024-02-10T07:39:34.261 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[29816]: 4.0 scrub ok 2024-02-10T07:39:34.262 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[40743]: 3.13 scrub starts 2024-02-10T07:39:34.262 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[40743]: 3.13 scrub ok 2024-02-10T07:39:34.262 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[40743]: 4.12 deep-scrub starts 2024-02-10T07:39:34.262 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[40743]: 4.12 deep-scrub ok 2024-02-10T07:39:34.262 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[40743]: 8.c scrub ok 2024-02-10T07:39:34.262 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:34.262 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[40743]: 4.0 scrub starts 2024-02-10T07:39:34.262 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:34 smithi008 ceph-mon[40743]: 4.0 scrub ok 2024-02-10T07:39:34.262 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:34.299 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:34.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:34.308 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-10T07:39:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:34 smithi170 ceph-mon[33394]: 3.13 scrub starts 2024-02-10T07:39:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:34 smithi170 ceph-mon[33394]: 3.13 scrub ok 2024-02-10T07:39:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:34 smithi170 ceph-mon[33394]: 4.12 deep-scrub starts 2024-02-10T07:39:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:34 smithi170 ceph-mon[33394]: 4.12 deep-scrub ok 2024-02-10T07:39:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:34 smithi170 ceph-mon[33394]: 8.c scrub ok 2024-02-10T07:39:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:34 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:34 smithi170 ceph-mon[33394]: 4.0 scrub starts 2024-02-10T07:39:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:34 smithi170 ceph-mon[33394]: 4.0 scrub ok 2024-02-10T07:39:34.400 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:34.430 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:34.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:34.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:34.532 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:34.562 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:34.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:34.642 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 68 16 102616 102600 94.2819 70.9375 0.0132725 0.0106016 2024-02-10T07:39:34.644 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:39:34.664 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:34.683 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:39:34.693 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:34.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:34.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:34.729 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:39:34.769 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:39:34.795 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:34.795 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:39:34.816 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:39:34.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:34.822 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:34.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:34.857 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:39:34.898 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:39:34.902 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:39:34.902 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:39:34.902 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:39:34.903 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:39:34.903 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:39:34.903 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:39:34.903 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:39:34.903 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:39:34.923 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:34.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:34.966 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:34.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:35.027 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:35 smithi008 ceph-mon[40743]: pgmap v357: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-02-10T07:39:35.067 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:35.099 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:35.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:35.108 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:39:35.201 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:35.256 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:35.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:35.285 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:35 smithi008 ceph-mon[29816]: pgmap v357: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-02-10T07:39:35.285 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:35 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1298656914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:35.287 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:35 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1298656914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:35.358 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:35 smithi170 ceph-mon[33394]: pgmap v357: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-02-10T07:39:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:35 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1298656914' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:35.425 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:35.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:35.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:35.527 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:35.561 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:35.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:35.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:35.642 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 69 16 103642 103626 93.8449 64.125 0.00897149 0.0106509 2024-02-10T07:39:35.663 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:35.701 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:35.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:35.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:35.803 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:35.847 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:35.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:35.949 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:35.983 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:35.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:36.085 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:36.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:36.126 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:36.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:36.227 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:36.270 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:36.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:36.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:36 smithi008 ceph-mon[29816]: 3.15 scrub starts 2024-02-10T07:39:36.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:36 smithi008 ceph-mon[29816]: 3.15 scrub ok 2024-02-10T07:39:36.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:36 smithi008 ceph-mon[29816]: 8.7 scrub starts 2024-02-10T07:39:36.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:36 smithi008 ceph-mon[29816]: 3.14 scrub starts 2024-02-10T07:39:36.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:36 smithi008 ceph-mon[29816]: 3.14 scrub ok 2024-02-10T07:39:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:36 smithi008 ceph-mon[40743]: 3.15 scrub starts 2024-02-10T07:39:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:36 smithi008 ceph-mon[40743]: 3.15 scrub ok 2024-02-10T07:39:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:36 smithi008 ceph-mon[40743]: 8.7 scrub starts 2024-02-10T07:39:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:36 smithi008 ceph-mon[40743]: 3.14 scrub starts 2024-02-10T07:39:36.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:36 smithi008 ceph-mon[40743]: 3.14 scrub ok 2024-02-10T07:39:36.373 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:36 smithi170 ceph-mon[33394]: 3.15 scrub starts 2024-02-10T07:39:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:36 smithi170 ceph-mon[33394]: 3.15 scrub ok 2024-02-10T07:39:36.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:36 smithi170 ceph-mon[33394]: 8.7 scrub starts 2024-02-10T07:39:36.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:36 smithi170 ceph-mon[33394]: 3.14 scrub starts 2024-02-10T07:39:36.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:36 smithi170 ceph-mon[33394]: 3.14 scrub ok 2024-02-10T07:39:36.399 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:36.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:36.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:36.501 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:36.547 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:36.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:36.642 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 70 16 104725 104709 93.4711 67.6875 0.0244437 0.0106934 2024-02-10T07:39:36.648 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:36.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:36.680 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:36.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:36.781 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:36.814 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:36.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:36.915 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:36.953 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:36.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:36.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:37.055 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:37.094 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:37.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:37.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:37.196 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:37.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:37.233 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:37.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:37.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:37 smithi008 ceph-mon[29816]: pgmap v358: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-10T07:39:37.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:37 smithi008 ceph-mon[29816]: 3.2 scrub starts 2024-02-10T07:39:37.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:37 smithi008 ceph-mon[29816]: 3.2 scrub ok 2024-02-10T07:39:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:37 smithi008 ceph-mon[40743]: pgmap v358: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-10T07:39:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:37 smithi008 ceph-mon[40743]: 3.2 scrub starts 2024-02-10T07:39:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:37 smithi008 ceph-mon[40743]: 3.2 scrub ok 2024-02-10T07:39:37.334 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:37.368 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:37.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:37.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:37 smithi170 ceph-mon[33394]: pgmap v358: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-10T07:39:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:37 smithi170 ceph-mon[33394]: 3.2 scrub starts 2024-02-10T07:39:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:37 smithi170 ceph-mon[33394]: 3.2 scrub ok 2024-02-10T07:39:37.470 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:37.493 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:37.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:37.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:37.594 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:37.627 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:37.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:37.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:37.642 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 71 15 105652 105637 92.9715 58 0.00818663 0.0107506 2024-02-10T07:39:37.729 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:37.763 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:37.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:37.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:37.865 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:37.901 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:37.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:37.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:38.003 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:38.029 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:39:38.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:38.045 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:38.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:38.146 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:38.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:38.202 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:38.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:38.297 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[29816]: 5.1c scrub starts 2024-02-10T07:39:38.297 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[29816]: 5.1c scrub ok 2024-02-10T07:39:38.298 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[29816]: 1.0 scrub starts 2024-02-10T07:39:38.298 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[29816]: 1.0 scrub ok 2024-02-10T07:39:38.298 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[29816]: 3.19 scrub starts 2024-02-10T07:39:38.298 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[29816]: 3.19 scrub ok 2024-02-10T07:39:38.298 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:39:38.298 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:39:38.298 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[29816]: 8.7 scrub ok 2024-02-10T07:39:38.298 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[40743]: 5.1c scrub starts 2024-02-10T07:39:38.298 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[40743]: 5.1c scrub ok 2024-02-10T07:39:38.298 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[40743]: 1.0 scrub starts 2024-02-10T07:39:38.298 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[40743]: 1.0 scrub ok 2024-02-10T07:39:38.299 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[40743]: 3.19 scrub starts 2024-02-10T07:39:38.299 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[40743]: 3.19 scrub ok 2024-02-10T07:39:38.299 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:39:38.299 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:39:38.299 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-mon[40743]: 8.7 scrub ok 2024-02-10T07:39:38.299 INFO:teuthology.orchestra.run.smithi008.stderr:no valid command found; 10 closest matches: 2024-02-10T07:39:38.299 INFO:teuthology.orchestra.run.smithi008.stderr:0 2024-02-10T07:39:38.299 INFO:teuthology.orchestra.run.smithi008.stderr:1 2024-02-10T07:39:38.299 INFO:teuthology.orchestra.run.smithi008.stderr:2 2024-02-10T07:39:38.299 INFO:teuthology.orchestra.run.smithi008.stderr:abort 2024-02-10T07:39:38.299 INFO:teuthology.orchestra.run.smithi008.stderr:assert 2024-02-10T07:39:38.299 INFO:teuthology.orchestra.run.smithi008.stderr:bench [] [] [] [] 2024-02-10T07:39:38.300 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs debug_inject_read_zeros 2024-02-10T07:39:38.300 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs files list 2024-02-10T07:39:38.300 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs stats 2024-02-10T07:39:38.300 INFO:teuthology.orchestra.run.smithi008.stderr:bluestore allocator dump block 2024-02-10T07:39:38.300 INFO:teuthology.orchestra.run.smithi008.stderr:Error EINVAL: invalid command 2024-02-10T07:39:38.302 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:38.353 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:38.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:38.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:38 smithi170 ceph-mon[33394]: 5.1c scrub starts 2024-02-10T07:39:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:38 smithi170 ceph-mon[33394]: 5.1c scrub ok 2024-02-10T07:39:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:38 smithi170 ceph-mon[33394]: 1.0 scrub starts 2024-02-10T07:39:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:38 smithi170 ceph-mon[33394]: 1.0 scrub ok 2024-02-10T07:39:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:38 smithi170 ceph-mon[33394]: 3.19 scrub starts 2024-02-10T07:39:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:38 smithi170 ceph-mon[33394]: 3.19 scrub ok 2024-02-10T07:39:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:38 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:39:38.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:38 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:39:38.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:38 smithi170 ceph-mon[33394]: 8.7 scrub ok 2024-02-10T07:39:38.455 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:38.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:38.482 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:38.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:38.584 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:38.622 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:38.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:38.642 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 72 16 106569 106553 92.4754 57.25 0.00948682 0.0108088 2024-02-10T07:39:38.724 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:38.766 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:38.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:38.868 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:38.904 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:38.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:39.006 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:39.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:39.044 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:39.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:39.051 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:39:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:39:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:39:39.145 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:39.179 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:39.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:39.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:39.281 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:39.322 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:39.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:39.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[29816]: 8.4 scrub starts 2024-02-10T07:39:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[29816]: 3.d scrub starts 2024-02-10T07:39:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[29816]: 3.d scrub ok 2024-02-10T07:39:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[29816]: pgmap v359: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 50 MiB/s wr, 797 op/s 2024-02-10T07:39:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[29816]: 8.9 scrub starts 2024-02-10T07:39:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[29816]: osdmap e226: 8 total, 8 up, 8 in 2024-02-10T07:39:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[29816]: 2.1 scrub starts 2024-02-10T07:39:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[29816]: 2.1 scrub ok 2024-02-10T07:39:39.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[40743]: 8.4 scrub starts 2024-02-10T07:39:39.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[40743]: 3.d scrub starts 2024-02-10T07:39:39.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[40743]: 3.d scrub ok 2024-02-10T07:39:39.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[40743]: pgmap v359: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 50 MiB/s wr, 797 op/s 2024-02-10T07:39:39.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[40743]: 8.9 scrub starts 2024-02-10T07:39:39.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[40743]: osdmap e226: 8 total, 8 up, 8 in 2024-02-10T07:39:39.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[40743]: 2.1 scrub starts 2024-02-10T07:39:39.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:39 smithi008 ceph-mon[40743]: 2.1 scrub ok 2024-02-10T07:39:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:39 smithi170 ceph-mon[33394]: 8.4 scrub starts 2024-02-10T07:39:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:39 smithi170 ceph-mon[33394]: 3.d scrub starts 2024-02-10T07:39:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:39 smithi170 ceph-mon[33394]: 3.d scrub ok 2024-02-10T07:39:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:39 smithi170 ceph-mon[33394]: pgmap v359: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 50 MiB/s wr, 797 op/s 2024-02-10T07:39:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:39 smithi170 ceph-mon[33394]: 8.9 scrub starts 2024-02-10T07:39:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:39 smithi170 ceph-mon[33394]: osdmap e226: 8 total, 8 up, 8 in 2024-02-10T07:39:39.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:39 smithi170 ceph-mon[33394]: 2.1 scrub starts 2024-02-10T07:39:39.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:39 smithi170 ceph-mon[33394]: 2.1 scrub ok 2024-02-10T07:39:39.424 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:39.465 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:39.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:39.566 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:39.604 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:39.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:39.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:39.642 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 73 16 107557 107541 92.0545 61.75 0.00897507 0.0108579 2024-02-10T07:39:39.707 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:39.732 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:39.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:39.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:39.834 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:39.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:39.873 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:39.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:39.974 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:40.010 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:40.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:40.111 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:40.170 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:40.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:40.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:40.272 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:40.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:40.311 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:40.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:40.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:40 smithi008 ceph-mon[29816]: 8.8 scrub starts 2024-02-10T07:39:40.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:40 smithi008 ceph-mon[29816]: 8.3 scrub starts 2024-02-10T07:39:40.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:40 smithi008 ceph-mon[29816]: 3.b scrub starts 2024-02-10T07:39:40.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:40 smithi008 ceph-mon[29816]: 3.b scrub ok 2024-02-10T07:39:40.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:40 smithi008 ceph-mon[40743]: 8.8 scrub starts 2024-02-10T07:39:40.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:40 smithi008 ceph-mon[40743]: 8.3 scrub starts 2024-02-10T07:39:40.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:40 smithi008 ceph-mon[40743]: 3.b scrub starts 2024-02-10T07:39:40.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:40 smithi008 ceph-mon[40743]: 3.b scrub ok 2024-02-10T07:39:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:40 smithi170 ceph-mon[33394]: 8.8 scrub starts 2024-02-10T07:39:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:40 smithi170 ceph-mon[33394]: 8.3 scrub starts 2024-02-10T07:39:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:40 smithi170 ceph-mon[33394]: 3.b scrub starts 2024-02-10T07:39:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:40 smithi170 ceph-mon[33394]: 3.b scrub ok 2024-02-10T07:39:40.412 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:40.448 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:40.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:40.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:40.550 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:40.589 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:40.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:40.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:40.645 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 74 16 108547 108531 91.6435 61.875 0.0100155 0.0109069 2024-02-10T07:39:40.655 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-10T07:39:40.656 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-10T07:39:40.656 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-02-10T07:39:40.691 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:40.719 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:40.735 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:40.751 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:40.792 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:40.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:40.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:40.894 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:40.951 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:40.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:41.053 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:41.093 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:41.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:41.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:41.194 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:41.208 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:41 smithi008 ceph-mon[29816]: 8.4 scrub ok 2024-02-10T07:39:41.208 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:41 smithi008 ceph-mon[29816]: pgmap v361: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-02-10T07:39:41.208 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:41 smithi008 ceph-mon[29816]: 8.9 scrub ok 2024-02-10T07:39:41.208 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:41 smithi008 ceph-mon[29816]: 3.8 scrub starts 2024-02-10T07:39:41.208 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:41 smithi008 ceph-mon[29816]: 3.8 scrub ok 2024-02-10T07:39:41.208 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:41 smithi008 ceph-mon[40743]: 8.4 scrub ok 2024-02-10T07:39:41.208 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:41 smithi008 ceph-mon[40743]: pgmap v361: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-02-10T07:39:41.209 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:41 smithi008 ceph-mon[40743]: 8.9 scrub ok 2024-02-10T07:39:41.209 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:41 smithi008 ceph-mon[40743]: 3.8 scrub starts 2024-02-10T07:39:41.209 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:41 smithi008 ceph-mon[40743]: 3.8 scrub ok 2024-02-10T07:39:41.228 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:41.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:41.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:41.330 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:41 smithi170 ceph-mon[33394]: 8.4 scrub ok 2024-02-10T07:39:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:41 smithi170 ceph-mon[33394]: pgmap v361: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-02-10T07:39:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:41 smithi170 ceph-mon[33394]: 8.9 scrub ok 2024-02-10T07:39:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:41 smithi170 ceph-mon[33394]: 3.8 scrub starts 2024-02-10T07:39:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:41 smithi170 ceph-mon[33394]: 3.8 scrub ok 2024-02-10T07:39:41.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:41.400 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:41.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:41.501 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:41.531 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:41.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:41.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:41.633 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:41.645 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 75 16 109508 109492 91.222 60.0625 0.0162366 0.0109571 2024-02-10T07:39:41.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:41.724 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:41.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:41.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:41.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:41.855 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:41.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:41.956 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:41.990 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:41.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:41.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:42.100 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:42.132 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:42.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:42.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:42.234 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:42.328 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:42.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:42.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:42.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[29816]: 4.d scrub starts 2024-02-10T07:39:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[29816]: 4.d scrub ok 2024-02-10T07:39:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[29816]: 3.c scrub starts 2024-02-10T07:39:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[29816]: 3.c scrub ok 2024-02-10T07:39:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[29816]: 8.8 scrub ok 2024-02-10T07:39:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[29816]: 5.a scrub starts 2024-02-10T07:39:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[29816]: 5.a scrub ok 2024-02-10T07:39:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[29816]: 3.f scrub starts 2024-02-10T07:39:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[29816]: 3.f scrub ok 2024-02-10T07:39:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[29816]: 8.3 scrub ok 2024-02-10T07:39:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[29816]: 6.b scrub starts 2024-02-10T07:39:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[29816]: 6.b scrub ok 2024-02-10T07:39:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[29816]: osdmap e227: 8 total, 8 up, 8 in 2024-02-10T07:39:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[40743]: 4.d scrub starts 2024-02-10T07:39:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[40743]: 4.d scrub ok 2024-02-10T07:39:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[40743]: 3.c scrub starts 2024-02-10T07:39:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[40743]: 3.c scrub ok 2024-02-10T07:39:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[40743]: 8.8 scrub ok 2024-02-10T07:39:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[40743]: 5.a scrub starts 2024-02-10T07:39:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[40743]: 5.a scrub ok 2024-02-10T07:39:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[40743]: 3.f scrub starts 2024-02-10T07:39:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[40743]: 3.f scrub ok 2024-02-10T07:39:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[40743]: 8.3 scrub ok 2024-02-10T07:39:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[40743]: 6.b scrub starts 2024-02-10T07:39:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[40743]: 6.b scrub ok 2024-02-10T07:39:42.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:42 smithi008 ceph-mon[40743]: osdmap e227: 8 total, 8 up, 8 in 2024-02-10T07:39:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:42 smithi170 ceph-mon[33394]: 4.d scrub starts 2024-02-10T07:39:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:42 smithi170 ceph-mon[33394]: 4.d scrub ok 2024-02-10T07:39:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:42 smithi170 ceph-mon[33394]: 3.c scrub starts 2024-02-10T07:39:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:42 smithi170 ceph-mon[33394]: 3.c scrub ok 2024-02-10T07:39:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:42 smithi170 ceph-mon[33394]: 8.8 scrub ok 2024-02-10T07:39:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:42 smithi170 ceph-mon[33394]: 5.a scrub starts 2024-02-10T07:39:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:42 smithi170 ceph-mon[33394]: 5.a scrub ok 2024-02-10T07:39:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:42 smithi170 ceph-mon[33394]: 3.f scrub starts 2024-02-10T07:39:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:42 smithi170 ceph-mon[33394]: 3.f scrub ok 2024-02-10T07:39:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:42 smithi170 ceph-mon[33394]: 8.3 scrub ok 2024-02-10T07:39:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:42 smithi170 ceph-mon[33394]: 6.b scrub starts 2024-02-10T07:39:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:42 smithi170 ceph-mon[33394]: 6.b scrub ok 2024-02-10T07:39:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:42 smithi170 ceph-mon[33394]: osdmap e227: 8 total, 8 up, 8 in 2024-02-10T07:39:42.430 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:42.466 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:42.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:42.575 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:42.614 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:42.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:42.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 76 16 110514 110498 90.849 62.875 0.0109297 0.0110018 2024-02-10T07:39:42.716 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:42.748 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:42.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:42.849 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:42.924 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:42.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:43.025 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:43.069 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:43.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:43.183 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:43.220 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:43.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:43.289 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:43 smithi008 ceph-mon[29816]: 5.3 scrub starts 2024-02-10T07:39:43.290 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:43 smithi008 ceph-mon[29816]: 5.3 scrub ok 2024-02-10T07:39:43.290 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:43 smithi008 ceph-mon[29816]: pgmap v363: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 762 op/s 2024-02-10T07:39:43.290 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:43 smithi008 ceph-mon[29816]: 8.a scrub starts 2024-02-10T07:39:43.290 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:43 smithi008 ceph-mon[29816]: 4.10 scrub starts 2024-02-10T07:39:43.290 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:43 smithi008 ceph-mon[29816]: 4.10 scrub ok 2024-02-10T07:39:43.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:43 smithi008 ceph-mon[40743]: 5.3 scrub starts 2024-02-10T07:39:43.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:43 smithi008 ceph-mon[40743]: 5.3 scrub ok 2024-02-10T07:39:43.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:43 smithi008 ceph-mon[40743]: pgmap v363: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 762 op/s 2024-02-10T07:39:43.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:43 smithi008 ceph-mon[40743]: 8.a scrub starts 2024-02-10T07:39:43.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:43 smithi008 ceph-mon[40743]: 4.10 scrub starts 2024-02-10T07:39:43.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:43 smithi008 ceph-mon[40743]: 4.10 scrub ok 2024-02-10T07:39:43.322 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:43.367 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:43.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:43.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:43.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:39:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:39:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:43 smithi170 ceph-mon[33394]: 5.3 scrub starts 2024-02-10T07:39:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:43 smithi170 ceph-mon[33394]: 5.3 scrub ok 2024-02-10T07:39:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:43 smithi170 ceph-mon[33394]: pgmap v363: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 762 op/s 2024-02-10T07:39:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:43 smithi170 ceph-mon[33394]: 8.a scrub starts 2024-02-10T07:39:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:43 smithi170 ceph-mon[33394]: 4.10 scrub starts 2024-02-10T07:39:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:43 smithi170 ceph-mon[33394]: 4.10 scrub ok 2024-02-10T07:39:43.470 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:43.515 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:43.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:43.617 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:43.632 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:39:43.633 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:39:43.646 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:39:43.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 77 16 111309 111293 90.3145 49.6875 0.0208418 0.0110672 2024-02-10T07:39:43.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:43.664 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:43.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:43.695 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:39:43.741 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:39:43.765 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:43.791 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:39:43.820 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:43.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:43.838 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:39:43.887 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:39:43.895 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:39:43.895 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:39:43.895 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:39:43.895 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:39:43.895 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:39:43.896 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:39:43.896 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:39:43.896 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:39:43.922 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:43.966 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:43.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:43.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:44.067 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:44.130 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:44.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:44.231 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:44.312 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:44.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:44 smithi170 ceph-mon[33394]: 4.18 scrub starts 2024-02-10T07:39:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:44 smithi170 ceph-mon[33394]: 4.18 scrub ok 2024-02-10T07:39:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:44 smithi170 ceph-mon[33394]: 6.5 deep-scrub starts 2024-02-10T07:39:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:44 smithi170 ceph-mon[33394]: 6.5 deep-scrub ok 2024-02-10T07:39:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:44 smithi170 ceph-mon[33394]: 4.7 scrub starts 2024-02-10T07:39:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:44 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:44 smithi170 ceph-mon[33394]: 4.7 scrub ok 2024-02-10T07:39:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:44 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:39:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:44 smithi170 ceph-mon[33394]: 6.11 scrub starts 2024-02-10T07:39:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:44 smithi170 ceph-mon[33394]: 6.11 scrub ok 2024-02-10T07:39:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:44 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1846618264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:44 smithi170 ceph-mon[33394]: 8.d scrub starts 2024-02-10T07:39:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:44 smithi170 ceph-mon[33394]: 3.a deep-scrub starts 2024-02-10T07:39:44.412 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[29816]: 4.18 scrub starts 2024-02-10T07:39:44.412 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[29816]: 4.18 scrub ok 2024-02-10T07:39:44.412 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[29816]: 6.5 deep-scrub starts 2024-02-10T07:39:44.412 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[29816]: 6.5 deep-scrub ok 2024-02-10T07:39:44.412 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[29816]: 4.7 scrub starts 2024-02-10T07:39:44.412 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:44.412 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[29816]: 4.7 scrub ok 2024-02-10T07:39:44.413 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:39:44.413 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[29816]: 6.11 scrub starts 2024-02-10T07:39:44.413 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[29816]: 6.11 scrub ok 2024-02-10T07:39:44.413 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1846618264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:44.413 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[29816]: 8.d scrub starts 2024-02-10T07:39:44.413 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[29816]: 3.a deep-scrub starts 2024-02-10T07:39:44.413 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[40743]: 4.18 scrub starts 2024-02-10T07:39:44.413 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[40743]: 4.18 scrub ok 2024-02-10T07:39:44.413 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[40743]: 6.5 deep-scrub starts 2024-02-10T07:39:44.413 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[40743]: 6.5 deep-scrub ok 2024-02-10T07:39:44.413 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[40743]: 4.7 scrub starts 2024-02-10T07:39:44.413 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:44.413 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[40743]: 4.7 scrub ok 2024-02-10T07:39:44.414 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:39:44.414 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[40743]: 6.11 scrub starts 2024-02-10T07:39:44.414 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[40743]: 6.11 scrub ok 2024-02-10T07:39:44.414 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1846618264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:44.414 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[40743]: 8.d scrub starts 2024-02-10T07:39:44.414 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:44 smithi008 ceph-mon[40743]: 3.a deep-scrub starts 2024-02-10T07:39:44.414 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:44.464 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:44.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:44.565 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:44.610 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:44.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:44.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:44.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 78 16 112181 112165 89.8553 54.5 0.00564191 0.0111244 2024-02-10T07:39:44.711 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:44.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:44.753 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:44.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:44.854 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:44.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:44.889 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:44.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:44.990 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:45.021 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:45.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:45.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:45.123 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:45.163 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:45.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:45.265 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:45.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:45.317 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:45.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:45.367 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:45 smithi008 ceph-mon[29816]: pgmap v364: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-10T07:39:45.368 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:45 smithi008 ceph-mon[29816]: 3.a deep-scrub ok 2024-02-10T07:39:45.368 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:45 smithi008 ceph-mon[29816]: 8.a scrub ok 2024-02-10T07:39:45.368 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:45 smithi008 ceph-mon[40743]: pgmap v364: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-10T07:39:45.368 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:45 smithi008 ceph-mon[40743]: 3.a deep-scrub ok 2024-02-10T07:39:45.368 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:45 smithi008 ceph-mon[40743]: 8.a scrub ok 2024-02-10T07:39:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:45 smithi170 ceph-mon[33394]: pgmap v364: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-10T07:39:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:45 smithi170 ceph-mon[33394]: 3.a deep-scrub ok 2024-02-10T07:39:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:45 smithi170 ceph-mon[33394]: 8.a scrub ok 2024-02-10T07:39:45.419 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:45.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:45.467 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:45.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:45.479 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:39:45.480 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-02-10T07:39:45.480 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-02-10T07:39:45.480 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:39:45.480 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.480 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47293 7.2 7:44192736:::benchmark_data_smithi008_82046_object47292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-02-10T07:39:45.480 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.923555+0000", 2024-02-10T07:39:45.480 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.55152911099998, 2024-02-10T07:39:45.480 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0745729319999999, 2024-02-10T07:39:45.481 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.481 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.481 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.481 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.481 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.481 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47293 2024-02-10T07:39:45.481 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.481 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.481 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.482 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.482 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923555+0000", 2024-02-10T07:39:45.482 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.482 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.482 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.482 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.482 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923555+0000", 2024-02-10T07:39:45.482 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.483 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.483 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923557+0000", 2024-02-10T07:39:45.483 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7439999999999999e-06 2024-02-10T07:39:45.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.483 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.483 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923579+0000", 2024-02-10T07:39:45.483 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2155e-05 2024-02-10T07:39:45.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.484 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.484 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923589+0000", 2024-02-10T07:39:45.484 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9410000000000002e-06 2024-02-10T07:39:45.484 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.484 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.484 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.484 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923597+0000", 2024-02-10T07:39:45.484 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7409999999999992e-06 2024-02-10T07:39:45.484 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.484 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.484 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.484 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.927637+0000", 2024-02-10T07:39:45.485 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0040407170000000001 2024-02-10T07:39:45.485 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.485 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.485 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.485 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.927647+0000", 2024-02-10T07:39:45.485 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7249999999999992e-06 2024-02-10T07:39:45.485 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.485 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.485 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.486 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.994086+0000", 2024-02-10T07:39:45.486 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0664391659999999 2024-02-10T07:39:45.486 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.486 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.486 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.486 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.994230+0000", 2024-02-10T07:39:45.486 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014405200000000001 2024-02-10T07:39:45.486 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.487 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.487 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.487 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.994425+0000", 2024-02-10T07:39:45.487 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019504200000000001 2024-02-10T07:39:45.487 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.487 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.487 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.488 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.997950+0000", 2024-02-10T07:39:45.488 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035248499999999999 2024-02-10T07:39:45.488 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.488 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.488 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.488 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998016+0000", 2024-02-10T07:39:45.489 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5520999999999998e-05 2024-02-10T07:39:45.489 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.489 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.489 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.489 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998034+0000", 2024-02-10T07:39:45.490 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8629000000000001e-05 2024-02-10T07:39:45.491 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.491 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.491 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.491 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998103+0000", 2024-02-10T07:39:45.491 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8366000000000002e-05 2024-02-10T07:39:45.491 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.492 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.492 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.492 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998128+0000", 2024-02-10T07:39:45.492 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5282999999999998e-05 2024-02-10T07:39:45.492 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.492 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.492 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.493 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.493 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.493 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47301 7.2 7:4dfdce57:::benchmark_data_smithi008_82046_object47300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.493 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.930507+0000", 2024-02-10T07:39:45.493 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.54457682999998, 2024-02-10T07:39:45.493 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.071251151, 2024-02-10T07:39:45.493 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.494 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.494 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.494 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.494 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.494 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47301 2024-02-10T07:39:45.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.495 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.495 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.495 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.495 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930507+0000", 2024-02-10T07:39:45.495 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.496 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.496 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.496 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930507+0000", 2024-02-10T07:39:45.496 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.496 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.496 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.497 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.497 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930509+0000", 2024-02-10T07:39:45.497 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9690000000000001e-06 2024-02-10T07:39:45.497 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.497 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.497 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.498 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930534+0000", 2024-02-10T07:39:45.498 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4635999999999999e-05 2024-02-10T07:39:45.498 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.498 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.498 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.498 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930545+0000", 2024-02-10T07:39:45.499 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0684e-05 2024-02-10T07:39:45.499 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.499 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.499 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.499 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930553+0000", 2024-02-10T07:39:45.499 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4270000000000008e-06 2024-02-10T07:39:45.500 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.500 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.500 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.500 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930839+0000", 2024-02-10T07:39:45.500 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000285665 2024-02-10T07:39:45.500 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.500 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.500 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.501 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930848+0000", 2024-02-10T07:39:45.501 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8269999999999993e-06 2024-02-10T07:39:45.501 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.501 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.501 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.501 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.996538+0000", 2024-02-10T07:39:45.501 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0656891589999999 2024-02-10T07:39:45.501 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.501 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.501 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.502 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.996677+0000", 2024-02-10T07:39:45.502 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013986200000000001 2024-02-10T07:39:45.502 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.502 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.502 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.502 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.996838+0000", 2024-02-10T07:39:45.502 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016012900000000001 2024-02-10T07:39:45.503 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.503 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.503 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.503 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001568+0000", 2024-02-10T07:39:45.503 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0047307110000000003 2024-02-10T07:39:45.503 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.503 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.503 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.504 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001587+0000", 2024-02-10T07:39:45.504 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8393000000000002e-05 2024-02-10T07:39:45.504 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.504 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.504 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.504 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001655+0000", 2024-02-10T07:39:45.504 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8763999999999994e-05 2024-02-10T07:39:45.504 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.505 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.505 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.505 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001715+0000", 2024-02-10T07:39:45.507 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9156000000000001e-05 2024-02-10T07:39:45.507 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.507 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.507 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.507 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001758+0000", 2024-02-10T07:39:45.507 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3769000000000001e-05 2024-02-10T07:39:45.507 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.508 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.508 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.508 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.508 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.508 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47312 7.2 7:441ce8f4:::benchmark_data_smithi008_82046_object47311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.508 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.938921+0000", 2024-02-10T07:39:45.508 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.53616342999999, 2024-02-10T07:39:45.508 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0674384029999999, 2024-02-10T07:39:45.509 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.509 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.509 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.509 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.509 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.509 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47312 2024-02-10T07:39:45.509 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.509 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.509 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.509 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.510 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938921+0000", 2024-02-10T07:39:45.510 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.510 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.510 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.510 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.510 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938921+0000", 2024-02-10T07:39:45.510 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.510 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.510 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.510 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.511 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938922+0000", 2024-02-10T07:39:45.511 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.539e-06 2024-02-10T07:39:45.511 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.511 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.511 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.511 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938949+0000", 2024-02-10T07:39:45.511 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6395999999999999e-05 2024-02-10T07:39:45.511 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.511 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.511 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.512 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938965+0000", 2024-02-10T07:39:45.512 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6665999999999999e-05 2024-02-10T07:39:45.512 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.512 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.512 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.512 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938974+0000", 2024-02-10T07:39:45.512 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9069999999999996e-06 2024-02-10T07:39:45.512 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.512 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.513 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.513 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939045+0000", 2024-02-10T07:39:45.513 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1165000000000001e-05 2024-02-10T07:39:45.513 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.513 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.513 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.514 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939058+0000", 2024-02-10T07:39:45.514 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3026e-05 2024-02-10T07:39:45.514 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.514 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.514 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.514 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998227+0000", 2024-02-10T07:39:45.514 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.05916901 2024-02-10T07:39:45.514 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.515 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.515 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.515 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998357+0000", 2024-02-10T07:39:45.515 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012962100000000001 2024-02-10T07:39:45.515 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.515 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.515 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.516 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998522+0000", 2024-02-10T07:39:45.516 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000165365 2024-02-10T07:39:45.516 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.516 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.516 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.516 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001821+0000", 2024-02-10T07:39:45.516 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032990379999999998 2024-02-10T07:39:45.516 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.516 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.516 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.517 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001868+0000", 2024-02-10T07:39:45.517 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7039999999999997e-05 2024-02-10T07:39:45.517 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.517 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.517 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.517 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006259+0000", 2024-02-10T07:39:45.517 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0043904510000000001 2024-02-10T07:39:45.518 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.518 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.518 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.518 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006317+0000", 2024-02-10T07:39:45.518 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8346e-05 2024-02-10T07:39:45.518 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.518 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.519 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.519 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006359+0000", 2024-02-10T07:39:45.519 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1832999999999997e-05 2024-02-10T07:39:45.519 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.519 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.519 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.519 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.519 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.519 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47317 7.2 7:477da457:::benchmark_data_smithi008_82046_object47316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.520 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.939750+0000", 2024-02-10T07:39:45.520 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.53533378700001, 2024-02-10T07:39:45.520 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.068997754, 2024-02-10T07:39:45.520 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.520 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.520 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.520 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.520 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.521 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47317 2024-02-10T07:39:45.521 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.521 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.521 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.521 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.521 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939750+0000", 2024-02-10T07:39:45.522 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.522 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.522 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.522 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.522 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939750+0000", 2024-02-10T07:39:45.522 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.522 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.522 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.523 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.523 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939751+0000", 2024-02-10T07:39:45.523 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.034e-06 2024-02-10T07:39:45.523 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.523 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.523 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.524 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939785+0000", 2024-02-10T07:39:45.524 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3855e-05 2024-02-10T07:39:45.524 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.524 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.524 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.524 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939801+0000", 2024-02-10T07:39:45.524 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5654999999999998e-05 2024-02-10T07:39:45.524 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.525 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.525 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.525 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939808+0000", 2024-02-10T07:39:45.525 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6599999999999995e-06 2024-02-10T07:39:45.525 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.525 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.525 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.525 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939884+0000", 2024-02-10T07:39:45.525 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5796000000000004e-05 2024-02-10T07:39:45.525 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.526 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.526 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.526 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939893+0000", 2024-02-10T07:39:45.526 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9460000000000004e-06 2024-02-10T07:39:45.526 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.526 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.526 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.526 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.999976+0000", 2024-02-10T07:39:45.526 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.060082902 2024-02-10T07:39:45.527 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.527 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.527 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.527 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.000113+0000", 2024-02-10T07:39:45.527 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013713899999999999 2024-02-10T07:39:45.527 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.528 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.528 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.528 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.000289+0000", 2024-02-10T07:39:45.528 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017616999999999999 2024-02-10T07:39:45.528 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.528 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.529 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.529 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.003850+0000", 2024-02-10T07:39:45.529 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035609219999999998 2024-02-10T07:39:45.529 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.529 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.529 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.530 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.003902+0000", 2024-02-10T07:39:45.530 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1848000000000002e-05 2024-02-10T07:39:45.530 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.530 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.530 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.530 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008648+0000", 2024-02-10T07:39:45.530 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0047453490000000003 2024-02-10T07:39:45.531 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.531 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.531 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.531 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008704+0000", 2024-02-10T07:39:45.531 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6620999999999999e-05 2024-02-10T07:39:45.531 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.532 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.532 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.532 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008748+0000", 2024-02-10T07:39:45.532 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3856999999999997e-05 2024-02-10T07:39:45.532 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.532 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.533 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.533 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.533 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.533 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47376 7.2 7:4df203ae:::benchmark_data_smithi008_82046_object47375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.533 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.973270+0000", 2024-02-10T07:39:45.533 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.50181428100001, 2024-02-10T07:39:45.534 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.035779212, 2024-02-10T07:39:45.534 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.534 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.534 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.534 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.534 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.534 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47376 2024-02-10T07:39:45.535 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.535 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.535 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.535 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.535 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973270+0000", 2024-02-10T07:39:45.536 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.536 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.536 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.536 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.536 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973270+0000", 2024-02-10T07:39:45.536 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.536 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.537 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.537 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.537 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973272+0000", 2024-02-10T07:39:45.537 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2000000000000001e-06 2024-02-10T07:39:45.537 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.537 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.537 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.538 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973297+0000", 2024-02-10T07:39:45.538 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4769999999999998e-05 2024-02-10T07:39:45.538 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.538 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.538 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.539 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973313+0000", 2024-02-10T07:39:45.539 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5959000000000001e-05 2024-02-10T07:39:45.539 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.539 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.539 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.540 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973321+0000", 2024-02-10T07:39:45.540 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7810000000000005e-06 2024-02-10T07:39:45.540 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.540 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.540 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.540 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973489+0000", 2024-02-10T07:39:45.541 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016799299999999999 2024-02-10T07:39:45.541 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.541 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.541 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.541 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973499+0000", 2024-02-10T07:39:45.542 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0176e-05 2024-02-10T07:39:45.542 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.542 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.542 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.542 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001910+0000", 2024-02-10T07:39:45.542 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0284110420000001 2024-02-10T07:39:45.543 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.543 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.543 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.543 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.002044+0000", 2024-02-10T07:39:45.543 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013467899999999999 2024-02-10T07:39:45.543 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.543 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.543 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.544 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.002181+0000", 2024-02-10T07:39:45.544 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013709900000000001 2024-02-10T07:39:45.544 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.544 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.544 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.544 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008762+0000", 2024-02-10T07:39:45.544 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0065806520000000002 2024-02-10T07:39:45.544 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.544 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.544 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008834+0000", 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2015999999999996e-05 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008896+0000", 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2131000000000005e-05 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009004+0000", 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000107298 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.545 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009049+0000", 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5416e-05 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47408 7.2 7:48404cbe:::benchmark_data_smithi008_82046_object47407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.992841+0000", 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.48224356700001, 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.021168692, 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.546 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47408 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992841+0000", 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992841+0000", 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.547 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992841+0000", 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.13e-07 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992864+0000", 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2410000000000001e-05 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992873+0000", 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9679999999999995e-06 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.548 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992879+0000", 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7769999999999997e-06 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992957+0000", 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7385000000000005e-05 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992967+0000", 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9499999999999996e-06 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.003963+0000", 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.010996497 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.004110+0000", 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000147136 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.004280+0000", 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000170192 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.550 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009080+0000", 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0047996489999999996 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009138+0000", 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7642999999999997e-05 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.013897+0000", 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0047593119999999999 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.551 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.013966+0000", 2024-02-10T07:39:45.552 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9319000000000005e-05 2024-02-10T07:39:45.552 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.552 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.552 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.552 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014009+0000", 2024-02-10T07:39:45.552 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2840999999999999e-05 2024-02-10T07:39:45.552 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.552 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.552 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.552 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.552 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.552 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47416 7.2 7:4733a3ba:::benchmark_data_smithi008_82046_object47415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.552 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.996764+0000", 2024-02-10T07:39:45.552 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.47831979699998, 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0174102730000001, 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47416 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996764+0000", 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.553 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996764+0000", 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996766+0000", 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4500000000000001e-06 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996788+0000", 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1861e-05 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.554 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996799+0000", 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1829e-05 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996807+0000", 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5830000000000001e-06 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996922+0000", 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011469 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.555 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.556 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.556 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996931+0000", 2024-02-10T07:39:45.556 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9910000000000008e-06 2024-02-10T07:39:45.556 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.556 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.556 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.556 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006393+0000", 2024-02-10T07:39:45.556 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.009462222 2024-02-10T07:39:45.556 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.556 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.556 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.556 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006520+0000", 2024-02-10T07:39:45.556 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012698599999999999 2024-02-10T07:39:45.556 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006667+0000", 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014698399999999999 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009091+0000", 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024244480000000001 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009194+0000", 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010273299999999999 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.557 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014084+0000", 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0048898680000000003 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014141+0000", 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6919999999999997e-05 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014175+0000", 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3708000000000003e-05 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.558 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47454 7.2 7:45bf9bfc:::benchmark_data_smithi008_82046_object47453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.019922+0000", 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.45516180599998, 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.99457532599999998, 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47454 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.559 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019922+0000", 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019922+0000", 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019924+0000", 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.979e-06 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.560 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019948+0000", 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3913999999999998e-05 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019965+0000", 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7254000000000001e-05 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019975+0000", 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1290000000000002e-06 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.561 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.020044+0000", 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9432999999999997e-05 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.020055+0000", 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0963e-05 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009296+0000", 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.98924138500000003 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.562 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009424+0000", 2024-02-10T07:39:45.563 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000128059 2024-02-10T07:39:45.563 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.563 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.563 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.563 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009572+0000", 2024-02-10T07:39:45.563 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000147128 2024-02-10T07:39:45.563 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.563 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.563 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.563 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014029+0000", 2024-02-10T07:39:45.563 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0044571560000000003 2024-02-10T07:39:45.563 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.563 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.567 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.568 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014231+0000", 2024-02-10T07:39:45.568 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00020206999999999999 2024-02-10T07:39:45.568 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.568 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.568 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.568 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014335+0000", 2024-02-10T07:39:45.568 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010406799999999999 2024-02-10T07:39:45.568 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.568 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.569 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.569 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014395+0000", 2024-02-10T07:39:45.569 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0081000000000003e-05 2024-02-10T07:39:45.569 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.569 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.569 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.569 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014498+0000", 2024-02-10T07:39:45.569 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000102707 2024-02-10T07:39:45.569 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.569 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.569 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.569 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.569 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.570 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47461 7.2 7:41d2193b:::benchmark_data_smithi008_82046_object47460:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.570 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.026190+0000", 2024-02-10T07:39:45.570 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.448894258, 2024-02-10T07:39:45.570 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.99060737499999996, 2024-02-10T07:39:45.570 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.570 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.570 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.570 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.570 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.570 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47461 2024-02-10T07:39:45.570 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.570 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.570 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.571 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.571 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026190+0000", 2024-02-10T07:39:45.571 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.571 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.571 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.571 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.571 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026190+0000", 2024-02-10T07:39:45.571 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.571 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.571 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.572 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.572 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026191+0000", 2024-02-10T07:39:45.572 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4020000000000001e-06 2024-02-10T07:39:45.572 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.572 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.572 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.572 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026215+0000", 2024-02-10T07:39:45.572 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3498000000000001e-05 2024-02-10T07:39:45.572 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.572 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.572 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.572 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026228+0000", 2024-02-10T07:39:45.572 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.293e-05 2024-02-10T07:39:45.573 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.573 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.573 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.573 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026236+0000", 2024-02-10T07:39:45.573 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1389999999999995e-06 2024-02-10T07:39:45.573 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.573 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.573 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.574 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.027093+0000", 2024-02-10T07:39:45.574 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000857371 2024-02-10T07:39:45.574 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.574 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.574 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.574 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.027104+0000", 2024-02-10T07:39:45.574 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1285e-05 2024-02-10T07:39:45.574 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.574 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.574 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.574 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.011669+0000", 2024-02-10T07:39:45.574 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.98456444099999996 2024-02-10T07:39:45.575 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.575 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.575 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.575 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.011816+0000", 2024-02-10T07:39:45.575 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014664100000000001 2024-02-10T07:39:45.575 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.575 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.575 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.575 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.011969+0000", 2024-02-10T07:39:45.575 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015315700000000001 2024-02-10T07:39:45.575 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.575 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.576 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.576 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014192+0000", 2024-02-10T07:39:45.576 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022234389999999998 2024-02-10T07:39:45.576 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.576 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.576 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.576 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016650+0000", 2024-02-10T07:39:45.576 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002457636 2024-02-10T07:39:45.576 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.576 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.576 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.576 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016693+0000", 2024-02-10T07:39:45.577 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3699999999999998e-05 2024-02-10T07:39:45.577 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.577 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.577 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.577 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016755+0000", 2024-02-10T07:39:45.577 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1278000000000005e-05 2024-02-10T07:39:45.577 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.577 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.577 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.577 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016797+0000", 2024-02-10T07:39:45.578 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2457999999999999e-05 2024-02-10T07:39:45.578 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.578 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.578 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.578 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.578 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.578 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47486 7.2 7:46e23e1a:::benchmark_data_smithi008_82046_object47485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.578 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.041230+0000", 2024-02-10T07:39:45.578 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.433854204, 2024-02-10T07:39:45.578 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.99476295699999995, 2024-02-10T07:39:45.578 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.578 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.579 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.579 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.579 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.579 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47486 2024-02-10T07:39:45.579 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.579 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.579 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.579 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.579 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041230+0000", 2024-02-10T07:39:45.579 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.579 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.579 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.579 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.580 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041230+0000", 2024-02-10T07:39:45.580 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.580 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.580 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.580 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.580 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041231+0000", 2024-02-10T07:39:45.580 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.471e-06 2024-02-10T07:39:45.580 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.580 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.580 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.580 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041254+0000", 2024-02-10T07:39:45.580 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2226e-05 2024-02-10T07:39:45.580 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041270+0000", 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6858e-05 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041279+0000", 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8249999999999994e-06 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041346+0000", 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6929999999999998e-05 2024-02-10T07:39:45.581 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.582 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.582 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.582 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041357+0000", 2024-02-10T07:39:45.582 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0429e-05 2024-02-10T07:39:45.582 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.582 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.582 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.582 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014562+0000", 2024-02-10T07:39:45.582 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97320494599999996 2024-02-10T07:39:45.582 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.582 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.582 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.582 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014756+0000", 2024-02-10T07:39:45.583 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019477900000000001 2024-02-10T07:39:45.583 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.583 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.583 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.583 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014928+0000", 2024-02-10T07:39:45.583 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017193500000000001 2024-02-10T07:39:45.583 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.583 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.583 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.583 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021669+0000", 2024-02-10T07:39:45.583 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0067405290000000003 2024-02-10T07:39:45.583 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.583 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.584 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.584 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021707+0000", 2024-02-10T07:39:45.584 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8451999999999997e-05 2024-02-10T07:39:45.584 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.584 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.584 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.584 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.035895+0000", 2024-02-10T07:39:45.584 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.01418823 2024-02-10T07:39:45.584 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.584 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.584 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.585 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.035953+0000", 2024-02-10T07:39:45.585 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7839e-05 2024-02-10T07:39:45.585 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.585 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.585 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.585 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.035993+0000", 2024-02-10T07:39:45.585 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9508000000000002e-05 2024-02-10T07:39:45.585 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.586 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.586 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.586 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.586 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.586 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47497 7.2 7:492ac9a6:::benchmark_data_smithi008_82046_object47496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.586 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.047446+0000", 2024-02-10T07:39:45.586 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.42763788299999, 2024-02-10T07:39:45.586 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.98990700600000003, 2024-02-10T07:39:45.586 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.586 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.586 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.586 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.586 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.587 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47497 2024-02-10T07:39:45.587 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.587 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.587 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.587 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.587 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047446+0000", 2024-02-10T07:39:45.587 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.587 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.587 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.587 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.587 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047446+0000", 2024-02-10T07:39:45.587 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.588 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.588 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.588 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.588 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047448+0000", 2024-02-10T07:39:45.588 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4330000000000001e-06 2024-02-10T07:39:45.588 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.588 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.588 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.588 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047469+0000", 2024-02-10T07:39:45.589 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1447999999999999e-05 2024-02-10T07:39:45.589 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.589 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.589 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.589 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047481+0000", 2024-02-10T07:39:45.589 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1843e-05 2024-02-10T07:39:45.589 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.589 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.589 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.589 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047540+0000", 2024-02-10T07:39:45.590 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8989000000000001e-05 2024-02-10T07:39:45.590 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.590 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.590 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.590 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047603+0000", 2024-02-10T07:39:45.590 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2974999999999999e-05 2024-02-10T07:39:45.590 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.590 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.590 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.590 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047613+0000", 2024-02-10T07:39:45.591 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8439999999999993e-06 2024-02-10T07:39:45.591 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.591 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.591 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.591 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016852+0000", 2024-02-10T07:39:45.591 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96923946999999999 2024-02-10T07:39:45.591 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.591 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.592 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.592 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016993+0000", 2024-02-10T07:39:45.592 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014067999999999999 2024-02-10T07:39:45.592 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.592 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.592 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.592 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.017152+0000", 2024-02-10T07:39:45.592 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015878999999999999 2024-02-10T07:39:45.592 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.592 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.592 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.592 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.026100+0000", 2024-02-10T07:39:45.592 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0089479090000000004 2024-02-10T07:39:45.592 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.593 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.593 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.593 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032492+0000", 2024-02-10T07:39:45.593 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0063921489999999997 2024-02-10T07:39:45.593 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.593 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.593 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.593 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.036002+0000", 2024-02-10T07:39:45.593 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035099559999999998 2024-02-10T07:39:45.593 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.593 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.593 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.594 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.036042+0000", 2024-02-10T07:39:45.594 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0071999999999999e-05 2024-02-10T07:39:45.594 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.594 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.594 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.594 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037353+0000", 2024-02-10T07:39:45.594 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001311448 2024-02-10T07:39:45.594 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.594 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.594 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.594 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.594 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.595 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47509 7.2 7:46639bc8:::benchmark_data_smithi008_82046_object47508:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.595 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.056564+0000", 2024-02-10T07:39:45.595 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.41851987699999, 2024-02-10T07:39:45.595 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.98093845899999998, 2024-02-10T07:39:45.595 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.595 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.595 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.595 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.595 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.595 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47509 2024-02-10T07:39:45.596 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.596 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.596 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.596 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.596 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056564+0000", 2024-02-10T07:39:45.596 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.596 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.596 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.596 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.596 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056564+0000", 2024-02-10T07:39:45.596 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.597 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.597 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.597 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.597 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056566+0000", 2024-02-10T07:39:45.597 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.516e-06 2024-02-10T07:39:45.597 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.597 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.597 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.597 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056588+0000", 2024-02-10T07:39:45.597 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2405e-05 2024-02-10T07:39:45.597 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.597 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.597 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.597 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056600+0000", 2024-02-10T07:39:45.598 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.236e-05 2024-02-10T07:39:45.598 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.598 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.598 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.598 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056609+0000", 2024-02-10T07:39:45.598 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3380000000000005e-06 2024-02-10T07:39:45.598 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.598 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.598 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.598 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056741+0000", 2024-02-10T07:39:45.598 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013242899999999999 2024-02-10T07:39:45.599 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.599 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.599 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.599 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056751+0000", 2024-02-10T07:39:45.599 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8460000000000003e-06 2024-02-10T07:39:45.599 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.599 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.599 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.599 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.019111+0000", 2024-02-10T07:39:45.599 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96235988699999997 2024-02-10T07:39:45.599 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.599 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.599 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.599 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.019240+0000", 2024-02-10T07:39:45.600 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000128565 2024-02-10T07:39:45.600 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.600 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.600 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.600 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.019393+0000", 2024-02-10T07:39:45.600 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000153294 2024-02-10T07:39:45.600 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.600 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.600 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.600 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032535+0000", 2024-02-10T07:39:45.600 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013142513999999999 2024-02-10T07:39:45.600 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.601 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.601 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.601 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032697+0000", 2024-02-10T07:39:45.601 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016114300000000001 2024-02-10T07:39:45.601 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.601 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.601 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.601 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037374+0000", 2024-02-10T07:39:45.601 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046779819999999998 2024-02-10T07:39:45.601 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.601 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.601 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.601 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037429+0000", 2024-02-10T07:39:45.601 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4669999999999997e-05 2024-02-10T07:39:45.602 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.602 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.602 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.602 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037503+0000", 2024-02-10T07:39:45.602 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3510000000000006e-05 2024-02-10T07:39:45.602 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.602 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.602 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.602 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.602 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.602 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47513 7.2 7:4ccef1d2:::benchmark_data_smithi008_82046_object47512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.602 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.059397+0000", 2024-02-10T07:39:45.603 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.41568664800002, 2024-02-10T07:39:45.603 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97837375900000001, 2024-02-10T07:39:45.603 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.603 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.603 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.603 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.603 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.603 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47513 2024-02-10T07:39:45.603 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.603 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.603 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.603 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.603 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059397+0000", 2024-02-10T07:39:45.603 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.604 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.604 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.604 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.604 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059397+0000", 2024-02-10T07:39:45.604 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.604 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.604 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.604 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.604 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059400+0000", 2024-02-10T07:39:45.604 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1749999999999999e-06 2024-02-10T07:39:45.604 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.605 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.605 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.605 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059425+0000", 2024-02-10T07:39:45.605 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5701e-05 2024-02-10T07:39:45.605 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.605 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.605 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.605 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059443+0000", 2024-02-10T07:39:45.605 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7710000000000002e-05 2024-02-10T07:39:45.605 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.605 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.605 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.606 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059453+0000", 2024-02-10T07:39:45.606 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0029e-05 2024-02-10T07:39:45.606 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.606 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.606 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.606 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059555+0000", 2024-02-10T07:39:45.606 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000102403 2024-02-10T07:39:45.606 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.606 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.606 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.606 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059567+0000", 2024-02-10T07:39:45.606 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1172000000000001e-05 2024-02-10T07:39:45.607 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.607 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.607 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.607 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021735+0000", 2024-02-10T07:39:45.607 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96216885299999999 2024-02-10T07:39:45.607 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.607 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.607 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.607 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021846+0000", 2024-02-10T07:39:45.607 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000110096 2024-02-10T07:39:45.607 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.607 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.608 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.608 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021980+0000", 2024-02-10T07:39:45.608 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013462200000000001 2024-02-10T07:39:45.608 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.608 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.608 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.608 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032576+0000", 2024-02-10T07:39:45.608 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010596202000000001 2024-02-10T07:39:45.608 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.608 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.608 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.608 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032865+0000", 2024-02-10T07:39:45.608 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00028853 2024-02-10T07:39:45.609 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.609 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.609 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.609 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037514+0000", 2024-02-10T07:39:45.609 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046494279999999997 2024-02-10T07:39:45.609 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.609 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.609 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.609 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037580+0000", 2024-02-10T07:39:45.609 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5399000000000002e-05 2024-02-10T07:39:45.609 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.609 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.609 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.609 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037771+0000", 2024-02-10T07:39:45.610 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019143900000000001 2024-02-10T07:39:45.610 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.610 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.610 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.610 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.610 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.610 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47519 7.2 7:474b3507:::benchmark_data_smithi008_82046_object47518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.610 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.063509+0000", 2024-02-10T07:39:45.610 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.411574664, 2024-02-10T07:39:45.610 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97463434199999999, 2024-02-10T07:39:45.610 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.610 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.610 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.610 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.611 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.611 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47519 2024-02-10T07:39:45.611 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.611 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.611 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.611 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.611 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063509+0000", 2024-02-10T07:39:45.611 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.611 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.611 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.611 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.611 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063509+0000", 2024-02-10T07:39:45.611 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.611 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063511+0000", 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3659999999999999e-06 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063535+0000", 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3765e-05 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063546+0000", 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1199000000000001e-05 2024-02-10T07:39:45.612 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063553+0000", 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7920000000000004e-06 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063622+0000", 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9577000000000001e-05 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063633+0000", 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.134e-05 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.613 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.614 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.614 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.023833+0000", 2024-02-10T07:39:45.614 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96019993999999997 2024-02-10T07:39:45.614 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.614 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.614 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.614 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.024002+0000", 2024-02-10T07:39:45.614 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016875700000000001 2024-02-10T07:39:45.614 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.614 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.615 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.615 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.024184+0000", 2024-02-10T07:39:45.615 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000182227 2024-02-10T07:39:45.615 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.615 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.615 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.615 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032615+0000", 2024-02-10T07:39:45.615 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0084308600000000001 2024-02-10T07:39:45.615 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.615 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.615 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.615 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037780+0000", 2024-02-10T07:39:45.615 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0051646779999999998 2024-02-10T07:39:45.616 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.616 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.616 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.616 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.038029+0000", 2024-02-10T07:39:45.616 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00024897100000000001 2024-02-10T07:39:45.616 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.616 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.616 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.616 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.038106+0000", 2024-02-10T07:39:45.616 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6736000000000005e-05 2024-02-10T07:39:45.616 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.616 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.616 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.617 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.038144+0000", 2024-02-10T07:39:45.617 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8133999999999998e-05 2024-02-10T07:39:45.617 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.617 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.617 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.617 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.617 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.617 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47530 7.2 7:44bc3d2a:::benchmark_data_smithi008_82046_object47529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.617 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.076595+0000", 2024-02-10T07:39:45.617 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.39848861399997, 2024-02-10T07:39:45.617 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96853736000000001, 2024-02-10T07:39:45.617 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.618 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.618 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.618 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.618 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.618 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47530 2024-02-10T07:39:45.618 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.618 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.618 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.618 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.618 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076595+0000", 2024-02-10T07:39:45.618 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.618 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.618 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076595+0000", 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076597+0000", 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0899999999999999e-06 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076613+0000", 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.683e-05 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.619 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.620 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.620 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076623+0000", 2024-02-10T07:39:45.620 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9070000000000002e-06 2024-02-10T07:39:45.620 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.620 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.620 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.620 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076629+0000", 2024-02-10T07:39:45.620 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9689999999999999e-06 2024-02-10T07:39:45.620 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.620 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.620 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.620 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076694+0000", 2024-02-10T07:39:45.621 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4873e-05 2024-02-10T07:39:45.621 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.621 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.621 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.621 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076707+0000", 2024-02-10T07:39:45.621 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2424e-05 2024-02-10T07:39:45.621 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.621 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.621 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.621 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.026136+0000", 2024-02-10T07:39:45.621 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.94942972400000003 2024-02-10T07:39:45.622 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.622 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.622 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.622 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.026302+0000", 2024-02-10T07:39:45.622 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000165744 2024-02-10T07:39:45.622 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.622 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.622 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.622 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.026474+0000", 2024-02-10T07:39:45.622 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017199699999999999 2024-02-10T07:39:45.622 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.622 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.623 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.623 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032657+0000", 2024-02-10T07:39:45.623 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0061833010000000004 2024-02-10T07:39:45.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.623 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.623 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.623 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037785+0000", 2024-02-10T07:39:45.623 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0051276610000000004 2024-02-10T07:39:45.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.623 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.623 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.623 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.044963+0000", 2024-02-10T07:39:45.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0071781509999999998 2024-02-10T07:39:45.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.624 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.624 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.624 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045018+0000", 2024-02-10T07:39:45.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4934e-05 2024-02-10T07:39:45.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.624 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.624 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.624 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045133+0000", 2024-02-10T07:39:45.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011475499999999999 2024-02-10T07:39:45.624 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.625 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.625 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.625 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.625 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.625 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47545 7.2 7:41492b52:::benchmark_data_smithi008_82046_object47544:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:39:45.625 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.108305+0000", 2024-02-10T07:39:45.625 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 307.36677883099998, 2024-02-10T07:39:45.625 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.93696623499999998, 2024-02-10T07:39:45.625 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.625 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.625 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.625 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.625 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.625 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47545 2024-02-10T07:39:45.626 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.626 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.626 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.626 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.626 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108305+0000", 2024-02-10T07:39:45.626 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.626 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.626 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.626 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.626 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108305+0000", 2024-02-10T07:39:45.626 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.626 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.627 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.627 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.627 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108306+0000", 2024-02-10T07:39:45.627 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2100000000000001e-06 2024-02-10T07:39:45.627 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.627 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.627 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.627 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108325+0000", 2024-02-10T07:39:45.627 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8094999999999999e-05 2024-02-10T07:39:45.627 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.627 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.627 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.627 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108335+0000", 2024-02-10T07:39:45.627 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0301e-05 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108340+0000", 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6420000000000004e-06 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108392+0000", 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1669999999999998e-05 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108399+0000", 2024-02-10T07:39:45.628 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8279999999999997e-06 2024-02-10T07:39:45.629 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.629 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.629 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.629 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.030442+0000", 2024-02-10T07:39:45.629 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.92204303600000004 2024-02-10T07:39:45.629 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.629 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.629 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.629 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.030575+0000", 2024-02-10T07:39:45.629 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000133089 2024-02-10T07:39:45.629 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.629 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.629 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.030728+0000", 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015333400000000001 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037828+0000", 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.007100014 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037841+0000", 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2738999999999999e-05 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.631 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045181+0000", 2024-02-10T07:39:45.631 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0073395769999999999 2024-02-10T07:39:45.631 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.631 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.631 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.631 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045235+0000", 2024-02-10T07:39:45.631 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4469000000000003e-05 2024-02-10T07:39:45.631 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.631 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.631 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.631 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045271+0000", 2024-02-10T07:39:45.631 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6230999999999999e-05 2024-02-10T07:39:45.631 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.632 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.632 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.632 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.632 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.632 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154801 7.a 7:5577fbe8:::benchmark_data_smithi008_82046_object39663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-02-10T07:39:45.632 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.740728+0000", 2024-02-10T07:39:45.632 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 192.73435558599999, 2024-02-10T07:39:45.632 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0968356680000002, 2024-02-10T07:39:45.632 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.632 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-02-10T07:39:45.632 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.632 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.633 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.633 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154801 2024-02-10T07:39:45.633 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.633 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.633 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.633 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.633 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740728+0000", 2024-02-10T07:39:45.633 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.633 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.633 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.633 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.634 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740728+0000", 2024-02-10T07:39:45.634 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.634 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.634 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.634 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.634 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740730+0000", 2024-02-10T07:39:45.634 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6980000000000001e-06 2024-02-10T07:39:45.634 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.634 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.634 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.634 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740734+0000", 2024-02-10T07:39:45.634 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8720000000000004e-06 2024-02-10T07:39:45.634 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.635 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.635 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.635 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740745+0000", 2024-02-10T07:39:45.635 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0589e-05 2024-02-10T07:39:45.635 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.635 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.635 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.635 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740752+0000", 2024-02-10T07:39:45.635 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2010000000000002e-06 2024-02-10T07:39:45.635 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.635 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.635 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740814+0000", 2024-02-10T07:39:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1679000000000004e-05 2024-02-10T07:39:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740929+0000", 2024-02-10T07:39:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000115658 2024-02-10T07:39:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2", 2024-02-10T07:39:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.741039+0000", 2024-02-10T07:39:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011011 2024-02-10T07:39:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.636 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.637 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.637 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.755960+0000", 2024-02-10T07:39:45.637 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014920487 2024-02-10T07:39:45.637 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.637 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.637 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.637 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.755972+0000", 2024-02-10T07:39:45.637 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1799e-05 2024-02-10T07:39:45.637 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.637 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.637 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.637 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739149+0000", 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.98317701300000004 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739162+0000", 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3864e-05 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.830200+0000", 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.09103803 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for missing object", 2024-02-10T07:39:45.638 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.830364+0000", 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000163325 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.837471+0000", 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0071075069999999999 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.837564+0000", 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2836000000000004e-05 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.639 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154828 7.a 7:58351c3d:::benchmark_data_smithi008_82046_object39690:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.883850+0000", 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 192.59123385000001, 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9570110619999999, 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154828 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.640 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883850+0000", 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883850+0000", 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883852+0000", 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.691e-06 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.641 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883858+0000", 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2029999999999997e-06 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883876+0000", 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7728999999999999e-05 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883886+0000", 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7259999999999997e-06 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.642 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883959+0000", 2024-02-10T07:39:45.643 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3606e-05 2024-02-10T07:39:45.643 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.643 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.643 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.643 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883971+0000", 2024-02-10T07:39:45.643 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:45.644 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1589e-05 2024-02-10T07:39:45.645 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.645 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.645 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.645 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739190+0000", 2024-02-10T07:39:45.645 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.85521957599999998 2024-02-10T07:39:45.645 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.645 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.645 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.645 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739199+0000", 2024-02-10T07:39:45.645 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5290000000000008e-06 2024-02-10T07:39:45.645 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.645 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.646 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.646 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.830636+0000", 2024-02-10T07:39:45.646 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0914373340000001 2024-02-10T07:39:45.646 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.646 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.646 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.646 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.830765+0000", 2024-02-10T07:39:45.646 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000129085 2024-02-10T07:39:45.646 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.646 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.646 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,5", 2024-02-10T07:39:45.647 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.830876+0000", 2024-02-10T07:39:45.647 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011061 2024-02-10T07:39:45.647 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.647 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.647 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.647 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.836879+0000", 2024-02-10T07:39:45.647 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0060032289999999997 2024-02-10T07:39:45.647 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.647 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.647 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.647 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.840426+0000", 2024-02-10T07:39:45.647 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035464889999999999 2024-02-10T07:39:45.648 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.648 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.648 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.648 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.840544+0000", 2024-02-10T07:39:45.648 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011797 2024-02-10T07:39:45.648 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.648 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.648 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.648 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.840618+0000", 2024-02-10T07:39:45.648 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.4357999999999995e-05 2024-02-10T07:39:45.648 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.648 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.649 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.649 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.840861+0000", 2024-02-10T07:39:45.649 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00024333800000000001 2024-02-10T07:39:45.649 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.649 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.649 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.649 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.649 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.649 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154842 7.a 7:5188ae5b:::benchmark_data_smithi008_82046_object39704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:39:45.649 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.905956+0000", 2024-02-10T07:39:45.649 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 192.56912854500001, 2024-02-10T07:39:45.649 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9354436609999999, 2024-02-10T07:39:45.649 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.649 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.650 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.650 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.650 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.650 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154842 2024-02-10T07:39:45.650 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.650 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.650 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.650 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.650 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.905956+0000", 2024-02-10T07:39:45.650 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.650 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.650 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.650 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.650 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.905956+0000", 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.905956+0000", 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7299999999999995e-07 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.905960+0000", 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3840000000000001e-06 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.651 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.905968+0000", 2024-02-10T07:39:45.652 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8319999999999995e-06 2024-02-10T07:39:45.652 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.652 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.652 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.652 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.905975+0000", 2024-02-10T07:39:45.652 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7449999999999998e-06 2024-02-10T07:39:45.652 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.652 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.652 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.652 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.906034+0000", 2024-02-10T07:39:45.652 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9252000000000001e-05 2024-02-10T07:39:45.652 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.652 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.653 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.653 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.906047+0000", 2024-02-10T07:39:45.653 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2366e-05 2024-02-10T07:39:45.653 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.653 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.653 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.653 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739224+0000", 2024-02-10T07:39:45.653 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.83317686999999996 2024-02-10T07:39:45.653 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.653 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.653 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.654 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739232+0000", 2024-02-10T07:39:45.654 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4009999999999997e-06 2024-02-10T07:39:45.654 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.654 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.654 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.654 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.833617+0000", 2024-02-10T07:39:45.654 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.094384676 2024-02-10T07:39:45.654 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.654 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.654 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.654 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.833735+0000", 2024-02-10T07:39:45.654 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000117827 2024-02-10T07:39:45.655 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.655 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.655 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,5", 2024-02-10T07:39:45.655 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.833862+0000", 2024-02-10T07:39:45.655 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012773799999999999 2024-02-10T07:39:45.655 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.655 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.655 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.655 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.840333+0000", 2024-02-10T07:39:45.655 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0064703149999999999 2024-02-10T07:39:45.655 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.655 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.655 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.655 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.840484+0000", 2024-02-10T07:39:45.656 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000151003 2024-02-10T07:39:45.656 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.656 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.656 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.656 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.841207+0000", 2024-02-10T07:39:45.656 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00072376500000000002 2024-02-10T07:39:45.656 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.656 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.656 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.656 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.841266+0000", 2024-02-10T07:39:45.656 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8523000000000002e-05 2024-02-10T07:39:45.656 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.656 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.657 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.657 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.841399+0000", 2024-02-10T07:39:45.657 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013329100000000001 2024-02-10T07:39:45.657 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.657 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.657 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.657 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.657 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.657 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:213588 7.14 7:2f12bf0f:::benchmark_data_smithi008_82046_object98450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-02-10T07:39:45.657 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:39.031951+0000", 2024-02-10T07:39:45.657 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 126.44313333700001, 2024-02-10T07:39:45.657 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0083072099999999, 2024-02-10T07:39:45.658 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:39:45.658 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:39:45.658 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:39:45.658 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:39:45.658 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:39:45.658 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 213588 2024-02-10T07:39:45.658 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.658 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:39:45.658 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.658 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:39:45.658 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.031951+0000", 2024-02-10T07:39:45.658 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.658 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.658 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.659 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:39:45.659 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.031951+0000", 2024-02-10T07:39:45.659 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:39:45.659 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.659 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.659 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:39:45.659 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.031953+0000", 2024-02-10T07:39:45.659 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.057e-06 2024-02-10T07:39:45.659 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.659 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.659 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:39:45.659 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.031960+0000", 2024-02-10T07:39:45.659 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3239999999999997e-06 2024-02-10T07:39:45.659 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.660 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.660 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:39:45.660 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.031977+0000", 2024-02-10T07:39:45.660 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6472000000000001e-05 2024-02-10T07:39:45.660 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.660 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.660 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:39:45.660 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.031987+0000", 2024-02-10T07:39:45.660 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0478e-05 2024-02-10T07:39:45.660 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.660 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.660 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.660 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.032655+0000", 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00066817799999999996 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.032663+0000", 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6650000000000003e-06 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.032047+0000", 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.99938453100000002 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:39:45.661 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.032176+0000", 2024-02-10T07:39:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012841999999999999 2024-02-10T07:39:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,5", 2024-02-10T07:39:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.032551+0000", 2024-02-10T07:39:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00037524200000000002 2024-02-10T07:39:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.037801+0000", 2024-02-10T07:39:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0052499729999999998 2024-02-10T07:39:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.662 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.037860+0000", 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8922000000000003e-05 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.040146+0000", 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022857979999999999 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.040208+0000", 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2510999999999998e-05 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:39:45.663 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:39:45.664 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.040258+0000", 2024-02-10T07:39:45.664 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9638999999999997e-05 2024-02-10T07:39:45.664 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.664 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.664 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.664 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:39:45.664 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:39:45.664 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:39:45.665 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 79 16 113261 113245 89.5723 67.5 0.0160252 0.0111595 2024-02-10T07:39:45.702 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:45.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:45.804 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:45.836 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:45.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:45.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:45.938 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:45.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:45.981 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:45.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:46.082 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:46.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:46.117 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:46.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:46.218 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:46.256 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:46.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:46.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:46.358 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:46 smithi170 ceph-mon[33394]: 4.9 scrub starts 2024-02-10T07:39:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:46 smithi170 ceph-mon[33394]: 4.9 scrub ok 2024-02-10T07:39:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:46 smithi170 ceph-mon[33394]: 5.19 scrub starts 2024-02-10T07:39:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:46 smithi170 ceph-mon[33394]: 5.19 scrub ok 2024-02-10T07:39:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:46 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:46 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:39:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:46 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:39:46.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:46 smithi170 ceph-mon[33394]: 4.17 scrub starts 2024-02-10T07:39:46.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:46 smithi170 ceph-mon[33394]: 4.17 scrub ok 2024-02-10T07:39:46.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:46 smithi170 ceph-mon[33394]: 8.d scrub ok 2024-02-10T07:39:46.397 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:46.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:46.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:46.499 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:46.526 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:46.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:46.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[29816]: 4.9 scrub starts 2024-02-10T07:39:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[29816]: 4.9 scrub ok 2024-02-10T07:39:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[29816]: 5.19 scrub starts 2024-02-10T07:39:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[29816]: 5.19 scrub ok 2024-02-10T07:39:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:39:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:39:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[29816]: 4.17 scrub starts 2024-02-10T07:39:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[29816]: 4.17 scrub ok 2024-02-10T07:39:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[29816]: 8.d scrub ok 2024-02-10T07:39:46.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[40743]: 4.9 scrub starts 2024-02-10T07:39:46.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[40743]: 4.9 scrub ok 2024-02-10T07:39:46.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[40743]: 5.19 scrub starts 2024-02-10T07:39:46.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[40743]: 5.19 scrub ok 2024-02-10T07:39:46.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:39:46.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:39:46.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:39:46.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[40743]: 4.17 scrub starts 2024-02-10T07:39:46.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[40743]: 4.17 scrub ok 2024-02-10T07:39:46.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:46 smithi008 ceph-mon[40743]: 8.d scrub ok 2024-02-10T07:39:46.627 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:46.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:39:46.646920+0000 min lat: 0.00127253 max lat: 0.194119 avg lat: 0.0111866 2024-02-10T07:39:46.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:39:46.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 80 16 114406 114390 89.3472 71.5625 0.0273853 0.0111866 2024-02-10T07:39:46.659 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:46.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:46.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:46.761 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:46.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:46.796 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:46.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:46.897 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:46.928 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:46.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:47.030 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:47.070 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:47.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:47.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:47.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:47.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:47.207 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:47.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:47.308 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:47.342 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:47.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:47.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:47 smithi170 ceph-mon[33394]: 6.14 scrub starts 2024-02-10T07:39:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:47 smithi170 ceph-mon[33394]: 6.14 scrub ok 2024-02-10T07:39:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:47 smithi170 ceph-mon[33394]: pgmap v365: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-02-10T07:39:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:47 smithi170 ceph-mon[33394]: 4.a scrub starts 2024-02-10T07:39:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:47 smithi170 ceph-mon[33394]: 4.a scrub ok 2024-02-10T07:39:47.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:47 smithi170 ceph-mon[33394]: 5.7 scrub starts 2024-02-10T07:39:47.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:47 smithi170 ceph-mon[33394]: 5.7 scrub ok 2024-02-10T07:39:47.443 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:47.481 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:47.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:47 smithi008 ceph-mon[29816]: 6.14 scrub starts 2024-02-10T07:39:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:47 smithi008 ceph-mon[29816]: 6.14 scrub ok 2024-02-10T07:39:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:47 smithi008 ceph-mon[29816]: pgmap v365: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-02-10T07:39:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:47 smithi008 ceph-mon[29816]: 4.a scrub starts 2024-02-10T07:39:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:47 smithi008 ceph-mon[29816]: 4.a scrub ok 2024-02-10T07:39:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:47 smithi008 ceph-mon[29816]: 5.7 scrub starts 2024-02-10T07:39:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:47 smithi008 ceph-mon[29816]: 5.7 scrub ok 2024-02-10T07:39:47.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:47 smithi008 ceph-mon[40743]: 6.14 scrub starts 2024-02-10T07:39:47.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:47 smithi008 ceph-mon[40743]: 6.14 scrub ok 2024-02-10T07:39:47.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:47 smithi008 ceph-mon[40743]: pgmap v365: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-02-10T07:39:47.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:47 smithi008 ceph-mon[40743]: 4.a scrub starts 2024-02-10T07:39:47.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:47 smithi008 ceph-mon[40743]: 4.a scrub ok 2024-02-10T07:39:47.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:47 smithi008 ceph-mon[40743]: 5.7 scrub starts 2024-02-10T07:39:47.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:47 smithi008 ceph-mon[40743]: 5.7 scrub ok 2024-02-10T07:39:47.583 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:47.621 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:47.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:47.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:47.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 81 16 115430 115414 89.0342 64 0.00700833 0.0112271 2024-02-10T07:39:47.708 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:39:47.723 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:47.805 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:47.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:47.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:47.907 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:47.939 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:47.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:48.041 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:48.075 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:48.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:48.176 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:48.211 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:48.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:48.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:48.313 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:48.340 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:48.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:48.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:48 smithi170 ceph-mon[33394]: 4.c scrub starts 2024-02-10T07:39:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:48 smithi170 ceph-mon[33394]: 4.c scrub ok 2024-02-10T07:39:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:48 smithi170 ceph-mon[33394]: 4.b scrub starts 2024-02-10T07:39:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:48 smithi170 ceph-mon[33394]: 4.b scrub ok 2024-02-10T07:39:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:48 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:48.443 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:48.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:48.483 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:48.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:48.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:48 smithi008 ceph-mon[29816]: 4.c scrub starts 2024-02-10T07:39:48.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:48 smithi008 ceph-mon[29816]: 4.c scrub ok 2024-02-10T07:39:48.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:48 smithi008 ceph-mon[29816]: 4.b scrub starts 2024-02-10T07:39:48.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:48 smithi008 ceph-mon[29816]: 4.b scrub ok 2024-02-10T07:39:48.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:48 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:48.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:48 smithi008 ceph-mon[40743]: 4.c scrub starts 2024-02-10T07:39:48.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:48 smithi008 ceph-mon[40743]: 4.c scrub ok 2024-02-10T07:39:48.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:48 smithi008 ceph-mon[40743]: 4.b scrub starts 2024-02-10T07:39:48.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:48 smithi008 ceph-mon[40743]: 4.b scrub ok 2024-02-10T07:39:48.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:48 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:39:48.584 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:48.620 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:48.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:48.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:48.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 82 16 116509 116493 88.7708 67.4375 0.00771315 0.0112602 2024-02-10T07:39:48.722 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:48.751 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:48.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:48.852 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:48.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:48.884 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:48.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:48.985 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:49.027 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:49.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:49.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:39:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:39:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:39:49.129 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:49.156 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:49.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:49.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:49.258 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:49.292 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:49.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:49.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:49 smithi170 ceph-mon[33394]: 3.9 scrub starts 2024-02-10T07:39:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:49 smithi170 ceph-mon[33394]: 3.9 scrub ok 2024-02-10T07:39:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:49 smithi170 ceph-mon[33394]: pgmap v366: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-10T07:39:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:49 smithi170 ceph-mon[33394]: 5.2 scrub starts 2024-02-10T07:39:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:49 smithi170 ceph-mon[33394]: 5.2 scrub ok 2024-02-10T07:39:49.394 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:49.432 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:49.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:49.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:49.533 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:49.557 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:49.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:49.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:49 smithi008 ceph-mon[29816]: 3.9 scrub starts 2024-02-10T07:39:49.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:49 smithi008 ceph-mon[29816]: 3.9 scrub ok 2024-02-10T07:39:49.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:49 smithi008 ceph-mon[29816]: pgmap v366: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-10T07:39:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:49 smithi008 ceph-mon[29816]: 5.2 scrub starts 2024-02-10T07:39:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:49 smithi008 ceph-mon[29816]: 5.2 scrub ok 2024-02-10T07:39:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:49 smithi008 ceph-mon[40743]: 3.9 scrub starts 2024-02-10T07:39:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:49 smithi008 ceph-mon[40743]: 3.9 scrub ok 2024-02-10T07:39:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:49 smithi008 ceph-mon[40743]: pgmap v366: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-10T07:39:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:49 smithi008 ceph-mon[40743]: 5.2 scrub starts 2024-02-10T07:39:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:49 smithi008 ceph-mon[40743]: 5.2 scrub ok 2024-02-10T07:39:49.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 83 16 117413 117397 88.382 56.5 0.0059027 0.0113096 2024-02-10T07:39:49.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:49.690 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:49.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:49.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:49.792 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:49.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:49.839 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:49.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:49.940 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:49.981 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:49.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:49.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:50.083 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:50.120 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:50.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:50.222 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:50.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:50.267 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:50.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:50.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:50 smithi008 ceph-mon[29816]: 5.9 scrub starts 2024-02-10T07:39:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:50 smithi008 ceph-mon[29816]: 5.9 scrub ok 2024-02-10T07:39:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:50 smithi008 ceph-mon[29816]: 5.12 scrub starts 2024-02-10T07:39:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:50 smithi008 ceph-mon[29816]: 5.12 scrub ok 2024-02-10T07:39:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:50 smithi008 ceph-mon[29816]: 6.d deep-scrub starts 2024-02-10T07:39:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:50 smithi008 ceph-mon[29816]: 6.d deep-scrub ok 2024-02-10T07:39:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:50 smithi008 ceph-mon[40743]: 5.9 scrub starts 2024-02-10T07:39:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:50 smithi008 ceph-mon[40743]: 5.9 scrub ok 2024-02-10T07:39:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:50 smithi008 ceph-mon[40743]: 5.12 scrub starts 2024-02-10T07:39:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:50 smithi008 ceph-mon[40743]: 5.12 scrub ok 2024-02-10T07:39:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:50 smithi008 ceph-mon[40743]: 6.d deep-scrub starts 2024-02-10T07:39:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:50 smithi008 ceph-mon[40743]: 6.d deep-scrub ok 2024-02-10T07:39:50.368 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:50 smithi170 ceph-mon[33394]: 5.9 scrub starts 2024-02-10T07:39:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:50 smithi170 ceph-mon[33394]: 5.9 scrub ok 2024-02-10T07:39:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:50 smithi170 ceph-mon[33394]: 5.12 scrub starts 2024-02-10T07:39:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:50 smithi170 ceph-mon[33394]: 5.12 scrub ok 2024-02-10T07:39:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:50 smithi170 ceph-mon[33394]: 6.d deep-scrub starts 2024-02-10T07:39:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:50 smithi170 ceph-mon[33394]: 6.d deep-scrub ok 2024-02-10T07:39:50.408 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:50.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:50.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:50.510 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:50.555 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:50.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:50.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:50.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 84 16 118557 118541 88.181 71.5 0.0096851 0.0113359 2024-02-10T07:39:50.657 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:50.695 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:50.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:50.798 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:50.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:50.837 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:50.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:50.938 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:50.967 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:50.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:50.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:51.069 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:51.115 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:51.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:51.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:51.216 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:51.250 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:51.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:51.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:51.352 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:51.382 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:51.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:51.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:51 smithi170 ceph-mon[33394]: pgmap v367: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 988 op/s 2024-02-10T07:39:51.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:51 smithi170 ceph-mon[33394]: 6.7 scrub starts 2024-02-10T07:39:51.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:51 smithi170 ceph-mon[33394]: 6.7 scrub ok 2024-02-10T07:39:51.484 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:51.520 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:51.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:51.528 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:51 smithi008 ceph-mon[40743]: pgmap v367: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 988 op/s 2024-02-10T07:39:51.528 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:51 smithi008 ceph-mon[40743]: 6.7 scrub starts 2024-02-10T07:39:51.528 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:51 smithi008 ceph-mon[40743]: 6.7 scrub ok 2024-02-10T07:39:51.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:51 smithi008 ceph-mon[29816]: pgmap v367: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 988 op/s 2024-02-10T07:39:51.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:51 smithi008 ceph-mon[29816]: 6.7 scrub starts 2024-02-10T07:39:51.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:51 smithi008 ceph-mon[29816]: 6.7 scrub ok 2024-02-10T07:39:51.588 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-10T07:39:51.601 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:51.621 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:51.649 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 85 16 119656 119640 87.9498 68.6875 0.0186295 0.0113649 2024-02-10T07:39:51.662 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:51.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:51.764 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:51.816 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:51.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:51.917 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:51.955 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:39:51.956 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:51.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:52.057 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:52.095 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:52.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:52.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:52.197 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:52.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:52.229 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:52.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:52.330 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:52.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:52.363 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:52.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:52.464 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:52.504 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:52.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:52.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:52.606 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:52.648 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 86 16 120749 120733 87.7214 68.3125 0.020164 0.0113947 2024-02-10T07:39:52.650 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:52.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:52.752 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:52.790 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:52.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:52.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:52 smithi170 ceph-mon[33394]: 4.e scrub starts 2024-02-10T07:39:52.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:52 smithi170 ceph-mon[33394]: 4.e scrub ok 2024-02-10T07:39:52.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:52 smithi170 ceph-mon[33394]: 6.10 scrub starts 2024-02-10T07:39:52.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:52 smithi170 ceph-mon[33394]: 6.10 scrub ok 2024-02-10T07:39:52.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:52 smithi170 ceph-mon[33394]: pgmap v368: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1018 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-02-10T07:39:52.890 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:52.930 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:52.931 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:52.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:52.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:53.033 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:53.068 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:53.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:53.079 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:52 smithi008 ceph-mon[29816]: 4.e scrub starts 2024-02-10T07:39:53.079 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:52 smithi008 ceph-mon[29816]: 4.e scrub ok 2024-02-10T07:39:53.079 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:52 smithi008 ceph-mon[29816]: 6.10 scrub starts 2024-02-10T07:39:53.079 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:52 smithi008 ceph-mon[29816]: 6.10 scrub ok 2024-02-10T07:39:53.079 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:52 smithi008 ceph-mon[29816]: pgmap v368: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1018 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-02-10T07:39:53.080 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:52 smithi008 ceph-mon[40743]: 4.e scrub starts 2024-02-10T07:39:53.080 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:52 smithi008 ceph-mon[40743]: 4.e scrub ok 2024-02-10T07:39:53.080 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:52 smithi008 ceph-mon[40743]: 6.10 scrub starts 2024-02-10T07:39:53.080 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:52 smithi008 ceph-mon[40743]: 6.10 scrub ok 2024-02-10T07:39:53.080 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:52 smithi008 ceph-mon[40743]: pgmap v368: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1018 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-02-10T07:39:53.169 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:53.200 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:53.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:53.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:53.302 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:53.338 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:53.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:53.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:53.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:39:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:39:53.440 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:53.473 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:53.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:53.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:53.575 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:53.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:53.614 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:53.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:53.648 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 87 16 121875 121859 87.522 70.375 0.0115915 0.0114205 2024-02-10T07:39:53.715 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:53.756 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:53.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:53.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:53.828 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:39:53.860 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:53.896 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:53.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:53.998 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:54.037 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:54.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:54.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:53 smithi008 ceph-mon[29816]: 4.1a scrub starts 2024-02-10T07:39:54.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:53 smithi008 ceph-mon[29816]: 4.1a scrub ok 2024-02-10T07:39:54.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:53 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2354996550' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:39:54.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:53 smithi008 ceph-mon[29816]: 5.1d scrub starts 2024-02-10T07:39:54.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:53 smithi008 ceph-mon[29816]: 5.1d scrub ok 2024-02-10T07:39:54.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:53 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:54.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:53 smithi008 ceph-mon[40743]: 4.1a scrub starts 2024-02-10T07:39:54.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:53 smithi008 ceph-mon[40743]: 4.1a scrub ok 2024-02-10T07:39:54.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:53 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2354996550' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:39:54.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:53 smithi008 ceph-mon[40743]: 5.1d scrub starts 2024-02-10T07:39:54.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:53 smithi008 ceph-mon[40743]: 5.1d scrub ok 2024-02-10T07:39:54.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:53 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:54.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:53 smithi170 ceph-mon[33394]: 4.1a scrub starts 2024-02-10T07:39:54.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:53 smithi170 ceph-mon[33394]: 4.1a scrub ok 2024-02-10T07:39:54.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:53 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2354996550' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:39:54.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:53 smithi170 ceph-mon[33394]: 5.1d scrub starts 2024-02-10T07:39:54.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:53 smithi170 ceph-mon[33394]: 5.1d scrub ok 2024-02-10T07:39:54.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:53 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:39:54.139 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:54.174 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:54.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:54.277 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:54.290 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:39:54.290 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:39:54.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.291 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15087.0:122553 8.b e228/200 8:d4123f8a:::benchmark_data_smithi008_109566_object122552:head v 228'7701, mlcod=228'7697)", 2024-02-10T07:39:54.291 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:39:54.282729+0000", 2024-02-10T07:39:54.291 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.0064028790000000002, 2024-02-10T07:39:54.291 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0064369300000000004, 2024-02-10T07:39:54.291 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:39:54.291 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:39:54.291 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:39:54.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.291 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:39:54.291 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.282729+0000", 2024-02-10T07:39:54.291 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:39:54.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.291 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.282729+0000", 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.282730+0000", 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.034e-06 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.282789+0000", 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8987999999999998e-05 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.282808+0000", 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.825e-05 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.282818+0000", 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0455000000000001e-05 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.283168+0000", 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00034997099999999997 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.283189+0000", 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0771000000000002e-05 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:122562 8.2 8:47534f39:::benchmark_data_smithi008_109566_object122561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:39:54.283881+0000", 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.0052510719999999999, 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.005384537, 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:39:54.294 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 122562 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.283881+0000", 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.283881+0000", 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:39:54.295 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.283883+0000", 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3710000000000001e-06 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.283910+0000", 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7245000000000001e-05 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.283920+0000", 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9890000000000004e-06 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:39:54.296 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.283926+0000", 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5989999999999998e-06 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.285873+0000", 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0019468490000000001 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.286028+0000", 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000154439 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,5", 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:39:54.286197+0000", 2024-02-10T07:39:54.297 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016929799999999999 2024-02-10T07:39:54.298 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:39:54.298 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:39:54.298 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:39:54.298 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:39:54.298 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2024-02-10T07:39:54.298 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 2 2024-02-10T07:39:54.298 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:39:54.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:54.321 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:54.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:54.422 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:39:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:54.460 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:54.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:39:54.561 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:54.598 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:54.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:54.648 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 88 16 122978 122962 87.3108 68.9375 0.0112696 0.0114488 2024-02-10T07:39:54.699 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:54.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:54.750 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:54.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:54.852 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:54.895 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:54.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:54.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:54.921 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[29816]: 5.0 deep-scrub starts 2024-02-10T07:39:54.921 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[29816]: 5.0 deep-scrub ok 2024-02-10T07:39:54.922 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[29816]: 8.2 scrub starts 2024-02-10T07:39:54.922 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[29816]: 4.15 scrub starts 2024-02-10T07:39:54.922 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[29816]: 5.1a scrub starts 2024-02-10T07:39:54.922 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[29816]: 5.1a scrub ok 2024-02-10T07:39:54.922 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[29816]: 4.15 scrub ok 2024-02-10T07:39:54.922 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[29816]: 3.1b scrub starts 2024-02-10T07:39:54.922 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[29816]: 3.1b scrub ok 2024-02-10T07:39:54.922 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:39:54.922 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2354996550' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:39:54.922 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[29816]: osdmap e228: 8 total, 8 up, 8 in 2024-02-10T07:39:54.922 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[29816]: pgmap v370: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-10T07:39:54.922 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[40743]: 5.0 deep-scrub starts 2024-02-10T07:39:54.923 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[40743]: 5.0 deep-scrub ok 2024-02-10T07:39:54.923 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[40743]: 8.2 scrub starts 2024-02-10T07:39:54.923 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[40743]: 4.15 scrub starts 2024-02-10T07:39:54.923 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[40743]: 5.1a scrub starts 2024-02-10T07:39:54.923 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[40743]: 5.1a scrub ok 2024-02-10T07:39:54.923 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[40743]: 4.15 scrub ok 2024-02-10T07:39:54.923 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[40743]: 3.1b scrub starts 2024-02-10T07:39:54.923 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[40743]: 3.1b scrub ok 2024-02-10T07:39:54.923 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:39:54.923 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2354996550' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:39:54.923 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[40743]: osdmap e228: 8 total, 8 up, 8 in 2024-02-10T07:39:54.923 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:54 smithi008 ceph-mon[40743]: pgmap v370: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-10T07:39:54.997 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:55.034 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:55.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:55.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:55.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:54 smithi170 ceph-mon[33394]: 5.0 deep-scrub starts 2024-02-10T07:39:55.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:54 smithi170 ceph-mon[33394]: 5.0 deep-scrub ok 2024-02-10T07:39:55.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:54 smithi170 ceph-mon[33394]: 8.2 scrub starts 2024-02-10T07:39:55.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:54 smithi170 ceph-mon[33394]: 4.15 scrub starts 2024-02-10T07:39:55.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:54 smithi170 ceph-mon[33394]: 5.1a scrub starts 2024-02-10T07:39:55.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:54 smithi170 ceph-mon[33394]: 5.1a scrub ok 2024-02-10T07:39:55.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:54 smithi170 ceph-mon[33394]: 4.15 scrub ok 2024-02-10T07:39:55.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:54 smithi170 ceph-mon[33394]: 3.1b scrub starts 2024-02-10T07:39:55.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:54 smithi170 ceph-mon[33394]: 3.1b scrub ok 2024-02-10T07:39:55.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:54 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:39:55.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:54 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2354996550' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:39:55.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:54 smithi170 ceph-mon[33394]: osdmap e228: 8 total, 8 up, 8 in 2024-02-10T07:39:55.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:54 smithi170 ceph-mon[33394]: pgmap v370: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-10T07:39:55.084 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-10T07:39:55.136 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:55.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:55.179 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:55.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:55.280 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:55.317 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:55.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:55.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:55.419 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:55.466 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:55.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:55.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:55.485 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:39:55.568 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:39:55.607 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:55.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:55.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:39:55.649 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 89 16 123995 123979 87.0439 63.5625 0.0101859 0.0114838 2024-02-10T07:39:55.709 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:55.748 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:55.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:55.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:55.850 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:55 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:39:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:55 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:39:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:55 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:39:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:55 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:39:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:55 smithi170 ceph-mon[33394]: osdmap e229: 8 total, 8 up, 8 in 2024-02-10T07:39:55.889 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:55.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:55.991 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:56.034 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:56.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:56.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:55 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:39:56.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:55 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:39:56.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:55 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:39:56.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:55 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:39:56.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:55 smithi008 ceph-mon[29816]: osdmap e229: 8 total, 8 up, 8 in 2024-02-10T07:39:56.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:55 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:39:56.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:55 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:39:56.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:55 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:39:56.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:55 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:39:56.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:55 smithi008 ceph-mon[40743]: osdmap e229: 8 total, 8 up, 8 in 2024-02-10T07:39:56.135 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:56.171 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:56.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:56.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:56.273 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:56.315 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:56.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:56.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:56.417 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:56.452 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:56.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:56.554 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:56.596 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:56.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:56.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:56.649 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Total time run: 90.0089 2024-02-10T07:39:56.649 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Total writes made: 125084 2024-02-10T07:39:56.649 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Write size: 65536 2024-02-10T07:39:56.650 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Object size: 65536 2024-02-10T07:39:56.650 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Bandwidth (MB/sec): 86.8553 2024-02-10T07:39:56.650 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev Bandwidth: 32.5488 2024-02-10T07:39:56.650 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max bandwidth (MB/sec): 204.938 2024-02-10T07:39:56.650 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min bandwidth (MB/sec): 49.6875 2024-02-10T07:39:56.650 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Average IOPS: 1389 2024-02-10T07:39:56.650 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev IOPS: 520.781 2024-02-10T07:39:56.650 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max IOPS: 3279 2024-02-10T07:39:56.650 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min IOPS: 795 2024-02-10T07:39:56.650 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Average Latency(s): 0.0115092 2024-02-10T07:39:56.650 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev Latency(s): 0.00942506 2024-02-10T07:39:56.650 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max latency(s): 0.194119 2024-02-10T07:39:56.650 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min latency(s): 0.00127253 2024-02-10T07:39:56.652 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Cleaning up (deleting benchmark objects) 2024-02-10T07:39:56.699 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:56.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:56.735 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:56.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:56.836 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:56.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:56.882 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:56.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:56.896 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:56 smithi008 ceph-mon[29816]: pgmap v372: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-10T07:39:56.896 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:56 smithi008 ceph-mon[40743]: pgmap v372: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-10T07:39:56.983 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:57.009 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:56 smithi170 ceph-mon[33394]: pgmap v372: 148 pgs: 148 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-10T07:39:57.023 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:57.023 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:39:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:57.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:57.081 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:39:57.126 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:57.127 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:39:57.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:57.161 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:57.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:57.172 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:39:57.216 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:39:57.262 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:57.264 INFO:teuthology.orchestra.run.smithi008.stderr:no valid command found; 10 closest matches: 2024-02-10T07:39:57.264 INFO:teuthology.orchestra.run.smithi008.stderr:0 2024-02-10T07:39:57.264 INFO:teuthology.orchestra.run.smithi008.stderr:1 2024-02-10T07:39:57.264 INFO:teuthology.orchestra.run.smithi008.stderr:2 2024-02-10T07:39:57.264 INFO:teuthology.orchestra.run.smithi008.stderr:abort 2024-02-10T07:39:57.264 INFO:teuthology.orchestra.run.smithi008.stderr:assert 2024-02-10T07:39:57.264 INFO:teuthology.orchestra.run.smithi008.stderr:bench [] [] [] [] 2024-02-10T07:39:57.264 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs debug_inject_read_zeros 2024-02-10T07:39:57.264 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs files list 2024-02-10T07:39:57.264 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs stats 2024-02-10T07:39:57.264 INFO:teuthology.orchestra.run.smithi008.stderr:bluestore allocator dump block 2024-02-10T07:39:57.265 INFO:teuthology.orchestra.run.smithi008.stderr:Error EINVAL: invalid command 2024-02-10T07:39:57.288 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:39:57.291 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:57.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:57.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:57.334 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:39:57.377 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:39:57.380 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:39:57.381 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:39:57.381 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:39:57.381 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:39:57.381 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:39:57.381 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:39:57.381 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:39:57.381 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:39:57.392 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:57.427 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:57.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:57.508 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:39:57.508 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:39:57.529 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:57.569 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:57.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:57.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:57.671 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:57.702 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:57.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:57.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:57.804 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:57.846 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:57.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:57.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:57.948 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:57.978 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:57.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:58.079 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:58.114 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:58.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:58.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:57 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1739730682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:58.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:57 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:39:58.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:57 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:39:58.217 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:58.232 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-10T07:39:58.255 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:58.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:39:58.315 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:57 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1739730682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:58.315 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:57 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:39:58.315 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:57 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:39:58.316 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:57 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1739730682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:39:58.316 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:57 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:39:58.316 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:57 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:39:58.357 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:58.362 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-10T07:39:58.362 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-10T07:39:58.363 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-02-10T07:39:58.393 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:58.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:58.495 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:58.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:58.541 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:58.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:58.615 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:39:58.642 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:58.649 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:39:58.684 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:58.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:58.786 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:39:58.832 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:58.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:58.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:39:58.934 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:58.981 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:58.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:59.050 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:58 smithi170 ceph-mon[33394]: pgmap v373: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 55 MiB/s wr, 919 op/s 2024-02-10T07:39:59.050 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:58 smithi170 ceph-mon[33394]: osdmap e230: 8 total, 8 up, 8 in 2024-02-10T07:39:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:58 smithi008 ceph-mon[29816]: pgmap v373: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 55 MiB/s wr, 919 op/s 2024-02-10T07:39:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:58 smithi008 ceph-mon[29816]: osdmap e230: 8 total, 8 up, 8 in 2024-02-10T07:39:59.083 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:39:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:39:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:39:59.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:58 smithi008 ceph-mon[40743]: pgmap v373: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 55 MiB/s wr, 919 op/s 2024-02-10T07:39:59.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:58 smithi008 ceph-mon[40743]: osdmap e230: 8 total, 8 up, 8 in 2024-02-10T07:39:59.083 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:59.111 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:59.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:59.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:59.138 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:59.138 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:39:59.213 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:59.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:59.257 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:59.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:59.359 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:39:59.399 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:59.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:59.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:39:59.501 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:39:59.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:59.534 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:39:59.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:39:59.635 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:39:59.666 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:59.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:59.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:39:59.767 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:39:59.799 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:59.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:39:59.901 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:39:59.933 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:39:59.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:39:59.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:00.034 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:00.073 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:00.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:00.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:59 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:40:00.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:39:59 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:40:00.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:59 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:40:00.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:39:59 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:40:00.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:59 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:40:00.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:39:59 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:40:00.174 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:00.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:00.217 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:00.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:00.318 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:00.356 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:00.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:00.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:00.490 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:00.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:00.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:00.592 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:00.631 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:00.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:00.732 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:00.765 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:00.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:00.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:00.866 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:00.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:00.907 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:00.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:00.930 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:40:00.930 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:40:00.947 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:40:00.983 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:00 smithi008 ceph-mon[29816]: pgmap v375: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1018 B/s rd, 41 MiB/s wr, 923 op/s 2024-02-10T07:40:00.983 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:00 smithi008 ceph-mon[29816]: osdmap e231: 8 total, 8 up, 8 in 2024-02-10T07:40:00.984 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:00 smithi008 ceph-mon[29816]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-02-10T07:40:00.984 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:00 smithi008 ceph-mon[29816]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-10T07:40:00.984 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:00 smithi008 ceph-mon[40743]: pgmap v375: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1018 B/s rd, 41 MiB/s wr, 923 op/s 2024-02-10T07:40:00.984 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:00 smithi008 ceph-mon[40743]: osdmap e231: 8 total, 8 up, 8 in 2024-02-10T07:40:00.984 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:00 smithi008 ceph-mon[40743]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-02-10T07:40:00.984 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:00 smithi008 ceph-mon[40743]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-10T07:40:01.008 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:01.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:01.047 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:01.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:01.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:00 smithi170 ceph-mon[33394]: pgmap v375: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1018 B/s rd, 41 MiB/s wr, 923 op/s 2024-02-10T07:40:01.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:00 smithi170 ceph-mon[33394]: osdmap e231: 8 total, 8 up, 8 in 2024-02-10T07:40:01.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:00 smithi170 ceph-mon[33394]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-02-10T07:40:01.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:00 smithi170 ceph-mon[33394]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-10T07:40:01.147 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:01.197 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:01.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:01.298 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:01.348 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:01.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:01.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:01.450 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:01.485 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:01.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:01.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:01.587 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:01.615 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:01.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:01.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:01.717 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:01.734 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-10T07:40:01.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:01.748 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:01.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:01.849 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:01.887 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:01.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:01.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:01.989 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:02.027 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:02.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:02.115 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:40:02.129 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:02.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:02.185 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:02.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:02.286 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:02.338 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:02.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:02.439 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:02.468 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:02.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:02.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:02.570 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:02.604 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:02.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:02.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:02.688 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:02.706 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:02.762 INFO:teuthology.orchestra.run.smithi008.stderr:no valid command found; 10 closest matches: 2024-02-10T07:40:02.763 INFO:teuthology.orchestra.run.smithi008.stderr:0 2024-02-10T07:40:02.763 INFO:teuthology.orchestra.run.smithi008.stderr:1 2024-02-10T07:40:02.763 INFO:teuthology.orchestra.run.smithi008.stderr:2 2024-02-10T07:40:02.763 INFO:teuthology.orchestra.run.smithi008.stderr:abort 2024-02-10T07:40:02.763 INFO:teuthology.orchestra.run.smithi008.stderr:assert 2024-02-10T07:40:02.763 INFO:teuthology.orchestra.run.smithi008.stderr:bench [] [] [] [] 2024-02-10T07:40:02.763 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs debug_inject_read_zeros 2024-02-10T07:40:02.763 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs files list 2024-02-10T07:40:02.763 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs stats 2024-02-10T07:40:02.763 INFO:teuthology.orchestra.run.smithi008.stderr:bluestore allocator dump block 2024-02-10T07:40:02.763 INFO:teuthology.orchestra.run.smithi008.stderr:Error EINVAL: invalid command 2024-02-10T07:40:02.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:02.780 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:02.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:02.864 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:02 smithi008 ceph-mon[29816]: pgmap v377: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 874 B/s rd, 35 MiB/s wr, 948 op/s 2024-02-10T07:40:02.864 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:02 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:40:02.864 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:02 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:02.864 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:02 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/818540482' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:02.865 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:02 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:02.865 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:02 smithi008 ceph-mon[40743]: pgmap v377: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 874 B/s rd, 35 MiB/s wr, 948 op/s 2024-02-10T07:40:02.865 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:02 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:40:02.865 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:02 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:02.865 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:02 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/818540482' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:02.865 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:02 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:02.881 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:02.918 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:02.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:02.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:03.020 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:03.057 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:03.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:03.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:03.113 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:02 smithi170 ceph-mon[33394]: pgmap v377: 148 pgs: 148 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 874 B/s rd, 35 MiB/s wr, 948 op/s 2024-02-10T07:40:03.113 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:02 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:40:03.113 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:02 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:03.113 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:02 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/818540482' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:03.113 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:02 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:03.159 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:03.192 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:03.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:03.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:03.294 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:03.333 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:03.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:03.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:03.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:40:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:40:03.434 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:03.497 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:40:03.523 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:03.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:03.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:03.624 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:03.664 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:03.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:03.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:03.766 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:03.827 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:03.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:03.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:03.978 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:03 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:03.978 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:03 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:40:03.978 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:03 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:40:03.978 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:03 smithi008 ceph-mon[29816]: osdmap e232: 8 total, 8 up, 8 in 2024-02-10T07:40:03.978 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:03 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:03.978 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:03 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:40:03.978 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:03 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:40:03.978 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:03 smithi008 ceph-mon[40743]: osdmap e232: 8 total, 8 up, 8 in 2024-02-10T07:40:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:04.002 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:04.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:04.103 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:04.106 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:40:04.106 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:40:04.106 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:03 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:04.106 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:03 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:40:04.106 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:03 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:40:04.107 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:03 smithi170 ceph-mon[33394]: osdmap e232: 8 total, 8 up, 8 in 2024-02-10T07:40:04.139 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:04.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:04.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:04.240 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:04.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:04.284 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:04.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:04.385 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:04.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:04.422 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:04.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:04.523 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:04.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:04.563 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:04.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:04.665 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:04.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:04.702 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:04.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:04.804 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:04.843 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:04.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:04.856 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-10T07:40:04.944 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:04.978 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:04 smithi008 ceph-mon[29816]: pgmap v379: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 23 MiB/s wr, 1.56k op/s 2024-02-10T07:40:04.979 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:04 smithi008 ceph-mon[40743]: pgmap v379: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 23 MiB/s wr, 1.56k op/s 2024-02-10T07:40:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:04.981 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:04.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:05.082 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:05.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:04 smithi170 ceph-mon[33394]: pgmap v379: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 23 MiB/s wr, 1.56k op/s 2024-02-10T07:40:05.144 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:05.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:05.230 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:40:05.246 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:05.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:05.303 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:05.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:05.406 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:05.418 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-10T07:40:05.419 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-10T07:40:05.419 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-02-10T07:40:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:05.456 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:05.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:05.557 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:05.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:05.604 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:05.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:05.705 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:05.757 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:05.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:05.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:05.859 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:05.901 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:05.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:06.002 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:06.055 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:06.063 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:06.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:06.072 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:40:06.117 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:40:06.156 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:40:06.164 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:06.196 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:40:06.197 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:06.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:06.239 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:40:06.283 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:40:06.298 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:06.329 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:40:06.335 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:06.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:06.377 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:40:06.384 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:40:06.384 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:40:06.384 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:40:06.384 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:40:06.384 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:40:06.384 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:40:06.384 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:40:06.384 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:40:06.437 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:06.437 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:40:06.471 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:06.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:06.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:06.572 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:06.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:06.606 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:06.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:06.707 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:06.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:06.754 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:06.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:06.855 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:06.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:06.888 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:06.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:06.989 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:07.024 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:07.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:07.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:07.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:06 smithi008 ceph-mon[29816]: pgmap v380: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 641 B/s rd, 17 MiB/s wr, 1.18k op/s 2024-02-10T07:40:07.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:06 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1498824686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:07.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:06 smithi008 ceph-mon[40743]: pgmap v380: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 641 B/s rd, 17 MiB/s wr, 1.18k op/s 2024-02-10T07:40:07.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:06 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1498824686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:07.125 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:07.166 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:06 smithi170 ceph-mon[33394]: pgmap v380: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 641 B/s rd, 17 MiB/s wr, 1.18k op/s 2024-02-10T07:40:07.166 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:06 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1498824686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:07.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:07.168 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:07.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:07.269 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:07.312 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:07.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:07.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:07.414 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:07.456 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:07.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:07.486 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:40:07.486 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2024-02-10T07:40:07.487 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2024-02-10T07:40:07.487 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2024-02-10T07:40:07.487 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:40:07.556 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:07.595 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:07.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:07.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:07.697 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:07.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:07.738 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:07.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:07.839 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:07.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:07.880 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:07.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:07.981 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:08.017 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:08.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:08.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:08.119 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:08.154 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:08.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:08.255 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:08.286 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:08.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:08.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:08.388 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:08.430 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:08.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:08.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:08.532 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:08.570 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:08.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:08.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:08.671 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:08.704 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:08.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:08.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:08.767 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:08.799 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-10T07:40:08.806 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:08.861 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:08.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:08.962 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:08.995 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:08.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:08.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:09.066 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:08 smithi008 ceph-mon[29816]: pgmap v381: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-02-10T07:40:09.067 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:40:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:40:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:40:09.067 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:08 smithi008 ceph-mon[40743]: pgmap v381: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-02-10T07:40:09.098 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:08 smithi170 ceph-mon[33394]: pgmap v381: 148 pgs: 148 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-02-10T07:40:09.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:09.169 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:09.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:09.184 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:40:09.270 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:09.311 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:09.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:09.412 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:09.454 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:09.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:09.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:09.555 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:09.589 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:09.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:09.691 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:09.724 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:09.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:09.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:09.826 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:09.863 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:09.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:09.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:09.965 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:10.000 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:10.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:10.102 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:10.138 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:10.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:10.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:10.239 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:10.276 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:10.277 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:40:10.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:10.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:10.379 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:10.412 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:10.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:10.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:10.472 INFO:teuthology.orchestra.run.smithi008.stderr:no valid command found; 10 closest matches: 2024-02-10T07:40:10.472 INFO:teuthology.orchestra.run.smithi008.stderr:0 2024-02-10T07:40:10.472 INFO:teuthology.orchestra.run.smithi008.stderr:1 2024-02-10T07:40:10.472 INFO:teuthology.orchestra.run.smithi008.stderr:2 2024-02-10T07:40:10.472 INFO:teuthology.orchestra.run.smithi008.stderr:abort 2024-02-10T07:40:10.472 INFO:teuthology.orchestra.run.smithi008.stderr:assert 2024-02-10T07:40:10.472 INFO:teuthology.orchestra.run.smithi008.stderr:bench [] [] [] [] 2024-02-10T07:40:10.472 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs debug_inject_read_zeros 2024-02-10T07:40:10.472 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs files list 2024-02-10T07:40:10.472 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs stats 2024-02-10T07:40:10.472 INFO:teuthology.orchestra.run.smithi008.stderr:bluestore allocator dump block 2024-02-10T07:40:10.472 INFO:teuthology.orchestra.run.smithi008.stderr:Error EINVAL: invalid command 2024-02-10T07:40:10.514 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:10.566 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:10.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:10.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:10.668 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:10.701 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:10.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:10.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:10.802 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:10.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:10.847 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:10.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:10.949 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:10.979 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:10.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:11.081 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:11.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:11.131 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:11.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:11.185 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:10 smithi170 ceph-mon[33394]: pgmap v382: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:40:11.186 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:40:11.186 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2024-02-10T07:40:11.186 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2024-02-10T07:40:11.186 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:40:11.186 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.186 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210702 7.12 7:49611789:::benchmark_data_smithi008_82046_object95564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:40:11.186 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.058811+0000", 2024-02-10T07:40:11.186 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 156.12363433100001, 2024-02-10T07:40:11.186 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0023876279999999, 2024-02-10T07:40:11.186 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.186 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.186 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.186 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.186 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.187 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210702 2024-02-10T07:40:11.187 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.187 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.187 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.187 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.187 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058811+0000", 2024-02-10T07:40:11.187 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.187 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.187 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.187 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.187 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058811+0000", 2024-02-10T07:40:11.187 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.187 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.187 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.188 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.188 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058812+0000", 2024-02-10T07:40:11.188 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6799999999999999e-07 2024-02-10T07:40:11.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.188 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.188 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.188 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058815+0000", 2024-02-10T07:40:11.188 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.881e-06 2024-02-10T07:40:11.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.188 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.188 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.188 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058824+0000", 2024-02-10T07:40:11.188 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.2650000000000002e-06 2024-02-10T07:40:11.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.189 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.189 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.189 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058830+0000", 2024-02-10T07:40:11.189 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.242e-06 2024-02-10T07:40:11.189 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.189 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.189 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.189 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058883+0000", 2024-02-10T07:40:11.189 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3631999999999997e-05 2024-02-10T07:40:11.189 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.189 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.189 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.189 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058893+0000", 2024-02-10T07:40:11.189 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0258000000000001e-05 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.042033+0000", 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.98313994000000005 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.042128+0000", 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4141000000000005e-05 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.042439+0000", 2024-02-10T07:40:11.190 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00031155199999999999 2024-02-10T07:40:11.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.191 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.191 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.191 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.045767+0000", 2024-02-10T07:40:11.191 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033280839999999998 2024-02-10T07:40:11.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.191 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.191 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.191 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.048185+0000", 2024-02-10T07:40:11.191 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0024177080000000002 2024-02-10T07:40:11.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.191 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.191 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.191 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061099+0000", 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012914583 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061157+0000", 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7429000000000002e-05 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061199+0000", 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2045000000000001e-05 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.192 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210713 7.12 7:48995235:::benchmark_data_smithi008_82046_object95575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.069890+0000", 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 156.11255583100001, 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.99147127099999999, 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210713 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.193 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.194 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069890+0000", 2024-02-10T07:40:11.194 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.194 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.194 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.194 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.194 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069890+0000", 2024-02-10T07:40:11.194 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.194 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.194 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.194 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069891+0000", 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5880000000000001e-06 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069896+0000", 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9250000000000003e-06 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069909+0000", 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2471e-05 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.195 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069917+0000", 2024-02-10T07:40:11.196 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1850000000000004e-06 2024-02-10T07:40:11.196 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.196 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.196 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.196 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.070014+0000", 2024-02-10T07:40:11.196 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6675999999999994e-05 2024-02-10T07:40:11.196 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.196 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.196 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.196 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.070022+0000", 2024-02-10T07:40:11.196 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7499999999999992e-06 2024-02-10T07:40:11.196 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.196 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.044238+0000", 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.97421551399999995 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.044356+0000", 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000117617 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.044601+0000", 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024531800000000001 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.197 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.198 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049272+0000", 2024-02-10T07:40:11.198 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0046710420000000002 2024-02-10T07:40:11.198 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.198 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.198 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.198 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050384+0000", 2024-02-10T07:40:11.198 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001111721 2024-02-10T07:40:11.198 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.198 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.198 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.198 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061261+0000", 2024-02-10T07:40:11.199 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010877045 2024-02-10T07:40:11.199 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.199 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.199 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.199 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061319+0000", 2024-02-10T07:40:11.199 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8415999999999998e-05 2024-02-10T07:40:11.199 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.199 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.199 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.200 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061361+0000", 2024-02-10T07:40:11.200 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2002999999999997e-05 2024-02-10T07:40:11.200 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.200 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.200 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.200 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.200 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.201 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210719 7.12 7:4a954bf6:::benchmark_data_smithi008_82046_object95581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:40:11.201 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.075516+0000", 2024-02-10T07:40:11.201 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 156.10692959299999, 2024-02-10T07:40:11.201 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.98756840400000001, 2024-02-10T07:40:11.201 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.201 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.201 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.201 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.201 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.201 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210719 2024-02-10T07:40:11.201 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.201 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.201 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.201 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.202 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075516+0000", 2024-02-10T07:40:11.202 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.202 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.202 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.202 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.202 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075516+0000", 2024-02-10T07:40:11.202 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.202 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.202 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.202 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.202 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075517+0000", 2024-02-10T07:40:11.202 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4219999999999999e-06 2024-02-10T07:40:11.202 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.202 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075522+0000", 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7779999999999998e-06 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075534+0000", 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.173e-05 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075544+0000", 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0105000000000001e-05 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.203 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075592+0000", 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8173000000000001e-05 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075601+0000", 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5890000000000003e-06 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.045802+0000", 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.97020084299999998 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.204 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.045871+0000", 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9725000000000001e-05 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.046021+0000", 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014953899999999999 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049278+0000", 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0032569140000000001 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.205 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050430+0000", 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0011520949999999999 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.062999+0000", 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012568823999999999 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063050+0000", 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0788999999999998e-05 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.206 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063084+0000", 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4878e-05 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210730 7.12 7:4fc3a9f7:::benchmark_data_smithi008_82046_object95592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.081362+0000", 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 156.10108331500001, 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.98183658399999996, 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.207 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210730 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081362+0000", 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081362+0000", 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.208 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.209 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.209 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081364+0000", 2024-02-10T07:40:11.209 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4219999999999999e-06 2024-02-10T07:40:11.209 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.209 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.209 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.209 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081368+0000", 2024-02-10T07:40:11.209 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5770000000000004e-06 2024-02-10T07:40:11.209 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.210 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.210 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.210 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081381+0000", 2024-02-10T07:40:11.210 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3118e-05 2024-02-10T07:40:11.210 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.210 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.210 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.210 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081389+0000", 2024-02-10T07:40:11.210 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.379e-06 2024-02-10T07:40:11.210 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.210 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.211 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.211 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081443+0000", 2024-02-10T07:40:11.211 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4475000000000003e-05 2024-02-10T07:40:11.211 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.211 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.211 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.211 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081452+0000", 2024-02-10T07:40:11.211 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.32e-06 2024-02-10T07:40:11.211 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.211 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.211 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.211 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.046907+0000", 2024-02-10T07:40:11.211 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.96545532300000003 2024-02-10T07:40:11.212 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.212 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.212 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.212 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.047023+0000", 2024-02-10T07:40:11.212 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000115832 2024-02-10T07:40:11.212 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.212 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.212 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:40:11.212 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.047228+0000", 2024-02-10T07:40:11.212 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020531600000000001 2024-02-10T07:40:11.212 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.212 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050394+0000", 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0031658239999999998 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055109+0000", 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0047147710000000004 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063136+0000", 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0080273949999999997 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.213 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.214 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.214 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063176+0000", 2024-02-10T07:40:11.214 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0142999999999998e-05 2024-02-10T07:40:11.214 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.214 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.214 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.214 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063199+0000", 2024-02-10T07:40:11.214 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2688999999999999e-05 2024-02-10T07:40:11.214 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.214 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210739 7.12 7:4d4f1063:::benchmark_data_smithi008_82046_object95601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.090931+0000", 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 156.09151441500001, 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.97239903000000005, 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210739 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.215 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090931+0000", 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090931+0000", 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090932+0000", 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2309999999999999e-06 2024-02-10T07:40:11.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090937+0000", 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5650000000000003e-06 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090954+0000", 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6515e-05 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090962+0000", 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0600000000000008e-06 2024-02-10T07:40:11.217 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.091039+0000", 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7632999999999993e-05 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.091050+0000", 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0491e-05 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.048220+0000", 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.95717007499999995 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.218 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.048294+0000", 2024-02-10T07:40:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4536000000000006e-05 2024-02-10T07:40:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:40:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.048444+0000", 2024-02-10T07:40:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014942699999999999 2024-02-10T07:40:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.051558+0000", 2024-02-10T07:40:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0031144889999999998 2024-02-10T07:40:11.219 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.058790+0000", 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0072319430000000002 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063249+0000", 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0044584890000000004 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063296+0000", 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7327999999999998e-05 2024-02-10T07:40:11.220 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063330+0000", 2024-02-10T07:40:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4248000000000001e-05 2024-02-10T07:40:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210750 7.12 7:4d3f77f1:::benchmark_data_smithi008_82046_object95612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:40:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.097423+0000", 2024-02-10T07:40:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 156.085022229, 2024-02-10T07:40:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.96606917000000003, 2024-02-10T07:40:11.221 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210750 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097423+0000", 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097423+0000", 2024-02-10T07:40:11.222 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097425+0000", 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9080000000000002e-06 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097430+0000", 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2390000000000004e-06 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097441+0000", 2024-02-10T07:40:11.223 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1257000000000001e-05 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097449+0000", 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7430000000000002e-06 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097502+0000", 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3361e-05 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097510+0000", 2024-02-10T07:40:11.224 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.4449999999999996e-06 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049307+0000", 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.95179659999999999 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049373+0000", 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5612999999999997e-05 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049521+0000", 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014814799999999999 2024-02-10T07:40:11.225 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.053835+0000", 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0043143499999999998 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061401+0000", 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0075663919999999999 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063378+0000", 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0019766499999999999 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.226 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063453+0000", 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4935e-05 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063493+0000", 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9529e-05 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210807 7.12 7:4f675a45:::benchmark_data_smithi008_82046_object95669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:40:11.227 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.148770+0000", 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 156.033675593, 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.91486412100000003, 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210807 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148770+0000", 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.228 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148770+0000", 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148771+0000", 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1710000000000001e-06 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148775+0000", 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1420000000000003e-06 2024-02-10T07:40:11.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148788+0000", 2024-02-10T07:40:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2897e-05 2024-02-10T07:40:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148797+0000", 2024-02-10T07:40:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2689999999999999e-06 2024-02-10T07:40:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148849+0000", 2024-02-10T07:40:11.230 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2095000000000002e-05 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148857+0000", 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9389999999999997e-06 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050456+0000", 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.90159990199999995 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050527+0000", 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0321000000000007e-05 2024-02-10T07:40:11.231 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050675+0000", 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000147725 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055123+0000", 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0044484379999999999 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061450+0000", 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0063272709999999998 2024-02-10T07:40:11.232 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063551+0000", 2024-02-10T07:40:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0021005830000000001 2024-02-10T07:40:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063603+0000", 2024-02-10T07:40:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2441000000000002e-05 2024-02-10T07:40:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063634+0000", 2024-02-10T07:40:11.233 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0926999999999997e-05 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210838 7.12 7:4d53824f:::benchmark_data_smithi008_82046_object95700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.176589+0000", 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 156.00585688800001, 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.88719636599999996, 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.234 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210838 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176589+0000", 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176589+0000", 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176590+0000", 2024-02-10T07:40:11.235 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.198e-06 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176594+0000", 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8840000000000004e-06 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176607+0000", 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2855e-05 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176616+0000", 2024-02-10T07:40:11.236 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1339999999999993e-06 2024-02-10T07:40:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176667+0000", 2024-02-10T07:40:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0887999999999998e-05 2024-02-10T07:40:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176676+0000", 2024-02-10T07:40:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1009999999999998e-06 2024-02-10T07:40:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.237 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.051582+0000", 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.87490583200000005 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.051647+0000", 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5252999999999993e-05 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.051803+0000", 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000156336 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.238 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055129+0000", 2024-02-10T07:40:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033258559999999999 2024-02-10T07:40:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.062510+0000", 2024-02-10T07:40:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0073805119999999997 2024-02-10T07:40:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063680+0000", 2024-02-10T07:40:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001170508 2024-02-10T07:40:11.239 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.239 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:11.241 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.241 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.241 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063745+0000", 2024-02-10T07:40:11.241 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5273999999999999e-05 2024-02-10T07:40:11.241 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.241 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.241 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.241 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063785+0000", 2024-02-10T07:40:11.241 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9734999999999998e-05 2024-02-10T07:40:11.241 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210862 7.12 7:48875dbe:::benchmark_data_smithi008_82046_object95724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.256071+0000", 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 155.92637422600001, 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.81128573100000001, 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210862 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.242 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256071+0000", 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256071+0000", 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256072+0000", 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5199999999999995e-07 2024-02-10T07:40:11.243 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256076+0000", 2024-02-10T07:40:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7910000000000002e-06 2024-02-10T07:40:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256084+0000", 2024-02-10T07:40:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0860000000000002e-06 2024-02-10T07:40:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256089+0000", 2024-02-10T07:40:11.244 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0549999999999999e-06 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256126+0000", 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6480000000000003e-05 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256131+0000", 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7259999999999999e-06 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.052653+0000", 2024-02-10T07:40:11.245 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.79652150600000005 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.052764+0000", 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000111022 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.052966+0000", 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000202442 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057531+0000", 2024-02-10T07:40:11.246 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0045644800000000001 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.062542+0000", 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0050112000000000004 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.067300+0000", 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0047581560000000004 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.067333+0000", 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2437999999999997e-05 2024-02-10T07:40:11.247 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.067357+0000", 2024-02-10T07:40:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4496999999999999e-05 2024-02-10T07:40:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210891 7.12 7:4e0bd40b:::benchmark_data_smithi008_82046_object95753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:40:11.248 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.296428+0000", 2024-02-10T07:40:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 155.88601776600001, 2024-02-10T07:40:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.77191008299999997, 2024-02-10T07:40:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210891 2024-02-10T07:40:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296428+0000", 2024-02-10T07:40:11.249 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296428+0000", 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296430+0000", 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9659999999999999e-06 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296435+0000", 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8509999999999998e-06 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296452+0000", 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6888000000000002e-05 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296461+0000", 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4429999999999992e-06 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296628+0000", 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000166675 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.251 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296636+0000", 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6770000000000001e-06 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.053881+0000", 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.75724445799999995 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.053969+0000", 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7972000000000004e-05 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.252 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.054161+0000", 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019222400000000001 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057538+0000", 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033767060000000002 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.067399+0000", 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0098609119999999995 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.253 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.068281+0000", 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000881857 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.068312+0000", 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.1659999999999998e-05 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.068338+0000", 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5794000000000001e-05 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210902 7.12 7:4cdd2df1:::benchmark_data_smithi008_82046_object95764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.308067+0000", 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 155.874378506, 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.76995323400000004, 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210902 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308067+0000", 2024-02-10T07:40:11.255 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308067+0000", 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308068+0000", 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.077e-06 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308072+0000", 2024-02-10T07:40:11.256 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8020000000000002e-06 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308083+0000", 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0743e-05 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308091+0000", 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.824e-06 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308147+0000", 2024-02-10T07:40:11.257 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5998000000000002e-05 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308157+0000", 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9250000000000007e-06 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055146+0000", 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.74698950799999997 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055221+0000", 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4597000000000004e-05 2024-02-10T07:40:11.258 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055375+0000", 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015408300000000001 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.058799+0000", 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0034239380000000001 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.068376+0000", 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0095769779999999999 2024-02-10T07:40:11.259 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.077940+0000", 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0095647360000000008 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.077981+0000", 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0688000000000001e-05 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078020+0000", 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9337e-05 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.260 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.261 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.261 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.261 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.261 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210911 7.12 7:4f38fa58:::benchmark_data_smithi008_82046_object95773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:40:11.261 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.318835+0000", 2024-02-10T07:40:11.261 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 155.86361051, 2024-02-10T07:40:11.261 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.75930091600000005, 2024-02-10T07:40:11.261 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.261 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.261 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.261 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.261 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.261 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210911 2024-02-10T07:40:11.261 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318835+0000", 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318835+0000", 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318836+0000", 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1510000000000001e-06 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.262 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318840+0000", 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1309999999999999e-06 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318850+0000", 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6690000000000001e-06 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318855+0000", 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3410000000000002e-06 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318904+0000", 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8712999999999999e-05 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318913+0000", 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.8650000000000007e-06 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.056341+0000", 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.73742814899999998 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.264 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.056452+0000", 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000110511 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.056655+0000", 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020361199999999999 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061057+0000", 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.004401861 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.265 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.076677+0000", 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015619687 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078061+0000", 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0013845089999999999 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078102+0000", 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.108e-05 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.266 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078136+0000", 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3637000000000003e-05 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210950 7.12 7:4bea578c:::benchmark_data_smithi008_82046_object95812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.363086+0000", 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 155.819359539, 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.71519846200000003, 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.267 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210950 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363086+0000", 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363086+0000", 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.268 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363087+0000", 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.232e-06 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363091+0000", 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7780000000000001e-06 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363101+0000", 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9580000000000002e-06 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.269 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363108+0000", 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6239999999999996e-06 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363163+0000", 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4987000000000001e-05 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363173+0000", 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0338e-05 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.270 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057579+0000", 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.69440612899999998 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057654+0000", 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4476999999999999e-05 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057814+0000", 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000160216 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.271 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061207+0000", 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033935850000000002 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.076729+0000", 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015521517 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078191+0000", 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0014615749999999999 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.272 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078243+0000", 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2627000000000003e-05 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078285+0000", 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1418999999999998e-05 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210966 7.12 7:4dc81df4:::benchmark_data_smithi008_82046_object95828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.386841+0000", 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 155.79560448000001, 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.69155063900000002, 2024-02-10T07:40:11.273 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210966 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386841+0000", 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.274 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386841+0000", 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386842+0000", 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3499999999999996e-07 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386844+0000", 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3709999999999998e-06 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.275 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386851+0000", 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2490000000000004e-06 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386856+0000", 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5600000000000004e-06 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386907+0000", 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0946000000000003e-05 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.276 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386916+0000", 2024-02-10T07:40:11.277 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0450000000000006e-06 2024-02-10T07:40:11.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.277 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.277 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.058828+0000", 2024-02-10T07:40:11.277 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.67191242600000001 2024-02-10T07:40:11.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.277 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.277 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.058892+0000", 2024-02-10T07:40:11.277 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3441999999999993e-05 2024-02-10T07:40:11.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.277 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:40:11.278 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.059060+0000", 2024-02-10T07:40:11.278 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000167808 2024-02-10T07:40:11.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.278 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.278 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061212+0000", 2024-02-10T07:40:11.278 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0021523620000000001 2024-02-10T07:40:11.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.278 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.280 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.076780+0000", 2024-02-10T07:40:11.280 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015567713 2024-02-10T07:40:11.280 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.280 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.280 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.280 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078326+0000", 2024-02-10T07:40:11.281 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0015468789999999999 2024-02-10T07:40:11.281 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.281 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.281 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.281 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078357+0000", 2024-02-10T07:40:11.281 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0905000000000003e-05 2024-02-10T07:40:11.281 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.281 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.281 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.281 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078392+0000", 2024-02-10T07:40:11.281 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4397999999999998e-05 2024-02-10T07:40:11.281 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.281 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.281 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211133 7.19 7:9c7b01de:::benchmark_data_smithi008_82046_object95995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.029346+0000", 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 155.15309923699999, 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.75181978400000005, 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211133 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.282 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029346+0000", 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029346+0000", 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029348+0000", 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.347e-06 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.283 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029352+0000", 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4689999999999999e-06 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029364+0000", 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1712999999999999e-05 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029372+0000", 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8869999999999992e-06 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029414+0000", 2024-02-10T07:40:11.284 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2472000000000003e-05 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029421+0000", 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5880000000000003e-06 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.774874+0000", 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.74545264499999997 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.774967+0000", 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3955000000000004e-05 2024-02-10T07:40:11.285 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.775347+0000", 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00037925300000000002 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.780874+0000", 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0055271779999999998 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.780892+0000", 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8397999999999999e-05 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781061+0000", 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016846299999999999 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781118+0000", 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7451000000000003e-05 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781166+0000", 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7964999999999999e-05 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.287 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.288 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.288 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.288 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.288 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211195 7.19 7:98ba502a:::benchmark_data_smithi008_82046_object96057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:40:11.288 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.123282+0000", 2024-02-10T07:40:11.288 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 155.05916360800001, 2024-02-10T07:40:11.288 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.65998720099999997, 2024-02-10T07:40:11.288 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.288 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.288 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.288 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.288 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.288 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211195 2024-02-10T07:40:11.288 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123282+0000", 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123282+0000", 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123283+0000", 2024-02-10T07:40:11.289 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9099999999999991e-07 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123287+0000", 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6839999999999998e-06 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123297+0000", 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0237e-05 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123302+0000", 2024-02-10T07:40:11.290 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4909999999999996e-06 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123356+0000", 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3748000000000001e-05 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123366+0000", 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0139000000000001e-05 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.777248+0000", 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.65388193900000002 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.777361+0000", 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000112723 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.777593+0000", 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00023236400000000001 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781245+0000", 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0036513639999999998 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781258+0000", 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3458999999999999e-05 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783159+0000", 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001900484 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783222+0000", 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3149000000000001e-05 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783269+0000", 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7429000000000003e-05 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211237 7.19 7:9803a4c9:::benchmark_data_smithi008_82046_object96099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.157249+0000", 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 155.025196217, 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.62624425500000003, 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.294 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211237 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157249+0000", 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157249+0000", 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.296 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.296 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157250+0000", 2024-02-10T07:40:11.296 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.47e-07 2024-02-10T07:40:11.296 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.296 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.296 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.296 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157255+0000", 2024-02-10T07:40:11.296 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2889999999999999e-06 2024-02-10T07:40:11.296 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.296 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.296 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.296 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157265+0000", 2024-02-10T07:40:11.296 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0158e-05 2024-02-10T07:40:11.296 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157272+0000", 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3010000000000001e-06 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157324+0000", 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1997999999999999e-05 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157332+0000", 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4839999999999998e-06 2024-02-10T07:40:11.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.298 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.298 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.298 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.778870+0000", 2024-02-10T07:40:11.298 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.62153816500000003 2024-02-10T07:40:11.298 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.298 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.298 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.298 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.778991+0000", 2024-02-10T07:40:11.298 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000121272 2024-02-10T07:40:11.298 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.298 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.298 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:40:11.298 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.779232+0000", 2024-02-10T07:40:11.298 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024073399999999999 2024-02-10T07:40:11.299 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.299 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.299 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.299 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781264+0000", 2024-02-10T07:40:11.299 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020323899999999998 2024-02-10T07:40:11.299 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.299 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.299 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.299 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781322+0000", 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8e-05 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783335+0000", 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020125339999999999 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783454+0000", 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011938300000000001 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783494+0000", 2024-02-10T07:40:11.300 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.96e-05 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211239 7.19 7:9b3bb0be:::benchmark_data_smithi008_82046_object96101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.158720+0000", 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 155.02372574200001, 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.62931068000000001, 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.301 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211239 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158720+0000", 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158720+0000", 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.302 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158721+0000", 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1289999999999999e-06 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158726+0000", 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7010000000000004e-06 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158747+0000", 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1362e-05 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158755+0000", 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7680000000000008e-06 2024-02-10T07:40:11.303 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158807+0000", 2024-02-10T07:40:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2522000000000003e-05 2024-02-10T07:40:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158817+0000", 2024-02-10T07:40:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3870000000000001e-06 2024-02-10T07:40:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781362+0000", 2024-02-10T07:40:11.304 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.62254494699999996 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781463+0000", 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000101592 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781718+0000", 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00025504199999999997 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.785814+0000", 2024-02-10T07:40:11.305 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0040961110000000004 2024-02-10T07:40:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787830+0000", 2024-02-10T07:40:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020153760000000001 2024-02-10T07:40:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787925+0000", 2024-02-10T07:40:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5309000000000005e-05 2024-02-10T07:40:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.306 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787989+0000", 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3578999999999996e-05 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.788031+0000", 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1854999999999998e-05 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211270 7.19 7:9a9ecfcf:::benchmark_data_smithi008_82046_object96132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.185809+0000", 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 154.996636701, 2024-02-10T07:40:11.307 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.60674339600000005, 2024-02-10T07:40:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211270 2024-02-10T07:40:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185809+0000", 2024-02-10T07:40:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.308 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185809+0000", 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185810+0000", 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.215e-06 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185814+0000", 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3259999999999997e-06 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.309 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185827+0000", 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2669e-05 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185835+0000", 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4769999999999997e-06 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.186136+0000", 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000301155 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.310 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.186151+0000", 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5316e-05 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783531+0000", 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.59737997799999998 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783650+0000", 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000118968 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:40:11.311 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783954+0000", 2024-02-10T07:40:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00030440799999999999 2024-02-10T07:40:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787866+0000", 2024-02-10T07:40:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0039120379999999996 2024-02-10T07:40:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787875+0000", 2024-02-10T07:40:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1249999999999993e-06 2024-02-10T07:40:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.312 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792479+0000", 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0046047429999999997 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792525+0000", 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5262000000000001e-05 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792552+0000", 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7716e-05 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.313 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.314 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.314 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.314 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211303 7.19 7:9bbbc725:::benchmark_data_smithi008_82046_object96165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:40:11.314 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.210320+0000", 2024-02-10T07:40:11.314 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 154.972126097, 2024-02-10T07:40:11.314 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.58484913400000005, 2024-02-10T07:40:11.314 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:11.314 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:11.314 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:11.314 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:11.314 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:11.314 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211303 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210320+0000", 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210320+0000", 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210320+0000", 2024-02-10T07:40:11.315 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1200000000000002e-07 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210324+0000", 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2459999999999999e-06 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210344+0000", 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0020000000000001e-05 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210352+0000", 2024-02-10T07:40:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0430000000000008e-06 2024-02-10T07:40:11.317 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.317 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.317 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.317 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210431+0000", 2024-02-10T07:40:11.317 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9482999999999998e-05 2024-02-10T07:40:11.317 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.317 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.317 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:11.317 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210439+0000", 2024-02-10T07:40:11.317 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0909999999999993e-06 2024-02-10T07:40:11.317 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.317 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.317 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:11.317 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.785850+0000", 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.57541028900000002 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.785977+0000", 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012783599999999999 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.786232+0000", 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00025462599999999998 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:11.318 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792371+0000", 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0061389529999999999 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792433+0000", 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2088999999999995e-05 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.795068+0000", 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0026348130000000002 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.795124+0000", 2024-02-10T07:40:11.319 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6365e-05 2024-02-10T07:40:11.320 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:11.320 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:11.320 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:11.320 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.795169+0000", 2024-02-10T07:40:11.320 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4468000000000002e-05 2024-02-10T07:40:11.320 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.320 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.320 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.320 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:11.320 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:11.320 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:40:11.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:11.321 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:11.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:11.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:10 smithi008 ceph-mon[29816]: pgmap v382: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:40:11.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:10 smithi008 ceph-mon[40743]: pgmap v382: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:40:11.422 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:11.458 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:11.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:11.559 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:11.589 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:11.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:11.691 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:11.729 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:11.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:11.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:11.830 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:11.868 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:11.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:11.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:11.970 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:12.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:12.006 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:12.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:12.087 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-10T07:40:12.087 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-10T07:40:12.087 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-02-10T07:40:12.108 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:12.113 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-10T07:40:12.177 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:12.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:12.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:12.225 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:12.279 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:12.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:12.330 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:12.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:12.431 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:12.441 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:40:12.460 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:12.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:12.562 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:12.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:12.601 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:12.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:12.702 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:12.734 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:12.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:12.836 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:12.865 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:12.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:12.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:12 smithi170 ceph-mon[33394]: pgmap v383: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-02-10T07:40:12.965 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:12.997 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:12.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:12.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:13.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:12 smithi008 ceph-mon[29816]: pgmap v383: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-02-10T07:40:13.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:12 smithi008 ceph-mon[40743]: pgmap v383: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-02-10T07:40:13.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:13.131 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:13.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:13.233 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:13.267 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:13.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:13.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:13.369 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:13.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:40:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:40:13.402 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:13.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:13.504 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:13.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:13.528 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:13.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:13.629 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:13.662 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:13.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:13.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:13.764 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:13.792 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:13.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:13.894 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:13.941 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:13.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:13.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:14.042 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:14.070 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:14.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:14.072 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:13 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:14.072 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:13 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:14.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:13 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:14.172 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:14.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:14.217 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:14.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:14.317 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:14.351 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:14.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:14.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:14.453 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:14.480 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:14.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:14.557 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:14.581 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:14.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:14.635 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:14.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:14.736 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:14.770 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:14.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:14.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:14.851 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:40:14.872 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:14.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:14.908 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:14.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:14.971 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:14 smithi008 ceph-mon[29816]: pgmap v384: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 984 B/s rd, 0 B/s wr, 1.33k op/s 2024-02-10T07:40:14.971 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:14 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2978222441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:40:14.971 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:14 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:40:14.972 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:14 smithi008 ceph-mon[40743]: pgmap v384: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 984 B/s rd, 0 B/s wr, 1.33k op/s 2024-02-10T07:40:14.972 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:14 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2978222441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:40:14.972 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:14 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:40:15.009 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:15.050 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:15.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:14 smithi170 ceph-mon[33394]: pgmap v384: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 984 B/s rd, 0 B/s wr, 1.33k op/s 2024-02-10T07:40:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:14 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2978222441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:40:15.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:14 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:40:15.153 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:15.189 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:15.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:15.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:15.291 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:15.324 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:15.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:15.426 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:15.462 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:15.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:15.563 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:15.595 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:15.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:15.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:15.697 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:15.732 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:15.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:15.834 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:15.874 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:15.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:15.975 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:16.014 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:16.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:16.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:16.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:15 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:40:16.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:15 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:40:16.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:15 smithi008 ceph-mon[40743]: osdmap e233: 8 total, 8 up, 8 in 2024-02-10T07:40:16.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:15 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:40:16.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:15 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:40:16.085 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:15 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:40:16.085 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:15 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:40:16.085 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:15 smithi008 ceph-mon[29816]: osdmap e233: 8 total, 8 up, 8 in 2024-02-10T07:40:16.085 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:15 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:40:16.085 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:15 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:40:16.115 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:16.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:15 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:40:16.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:15 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:40:16.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:15 smithi170 ceph-mon[33394]: osdmap e233: 8 total, 8 up, 8 in 2024-02-10T07:40:16.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:15 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:40:16.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:15 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:40:16.151 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:16.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:16.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:16.253 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:16.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:16.306 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:16.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:16.407 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:16.450 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:16.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:16.552 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:16.590 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:16.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:16.692 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:16.719 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:16.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:16.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:16.821 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:16.859 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:16.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:16.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:16.923 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:40:16.923 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:40:16.923 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.923 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15087.0:148416 8.5 e234/200 8:a592a216:::benchmark_data_smithi008_109566_object23329:head v 234'9255, mlcod=234'9253)", 2024-02-10T07:40:16.923 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:16.921046+0000", 2024-02-10T07:40:16.923 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.001488374, 2024-02-10T07:40:16.923 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0015192840000000001, 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.921046+0000", 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.921046+0000", 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.924 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.921047+0000", 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3600000000000002e-07 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.921053+0000", 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0669999999999997e-06 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.921068+0000", 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4579000000000001e-05 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:16.925 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.921075+0000", 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2860000000000002e-06 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.921245+0000", 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017030100000000001 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.921264+0000", 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8873999999999999e-05 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:16.926 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15087.0:148414 8.6 e234/200 8:64112b8c:::benchmark_data_smithi008_109566_object23327:head v 234'9228, mlcod=234'9226)", 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:16.922318+0000", 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.00021704999999999999, 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000311291, 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.922318+0000", 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:16.927 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.922318+0000", 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.922318+0000", 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4799999999999997e-07 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.922325+0000", 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2809999999999997e-06 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:16.928 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.922339+0000", 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4409999999999999e-05 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.922346+0000", 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2169999999999997e-06 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.922408+0000", 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1230999999999997e-05 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:16.929 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:16.922426+0000", 2024-02-10T07:40:16.930 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8034999999999999e-05 2024-02-10T07:40:16.930 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:16.930 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:16.930 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:16.930 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:16.930 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2024-02-10T07:40:16.930 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 2 2024-02-10T07:40:16.930 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:40:16.961 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:16.993 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:16.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:17.095 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:17.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:17.132 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:17.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:17.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:16 smithi170 ceph-mon[33394]: osdmap e234: 8 total, 8 up, 8 in 2024-02-10T07:40:17.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:16 smithi170 ceph-mon[33394]: pgmap v387: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:40:17.233 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:17.270 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:17.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:17.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:16 smithi008 ceph-mon[40743]: osdmap e234: 8 total, 8 up, 8 in 2024-02-10T07:40:17.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:16 smithi008 ceph-mon[40743]: pgmap v387: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:40:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:16 smithi008 ceph-mon[29816]: osdmap e234: 8 total, 8 up, 8 in 2024-02-10T07:40:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:16 smithi008 ceph-mon[29816]: pgmap v387: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:40:17.372 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:17.402 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:17.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:17.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:17.504 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:17.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:17.537 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:17.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:17.638 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:17.676 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:17.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:17.778 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:17.808 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:17.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:17.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:17.823 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-10T07:40:17.909 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:17.943 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:17.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:18.044 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:18.079 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:18.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:18.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:18.135 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:17 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:18.135 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:17 smithi170 ceph-mon[33394]: 6.c scrub starts 2024-02-10T07:40:18.136 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:17 smithi170 ceph-mon[33394]: 6.c scrub ok 2024-02-10T07:40:18.181 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:18.206 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:18.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:18.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:18.249 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:40:18.308 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:18.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:17 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:18.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:17 smithi008 ceph-mon[40743]: 6.c scrub starts 2024-02-10T07:40:18.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:17 smithi008 ceph-mon[40743]: 6.c scrub ok 2024-02-10T07:40:18.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:17 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:18.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:17 smithi008 ceph-mon[29816]: 6.c scrub starts 2024-02-10T07:40:18.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:17 smithi008 ceph-mon[29816]: 6.c scrub ok 2024-02-10T07:40:18.351 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:18.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:18.453 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:18.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:18.481 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:18.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:18.582 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:18.623 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:18.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:18.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:18.725 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:18.764 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:18.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:18.831 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:40:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:40:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:40:18.865 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:18.896 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:18.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:18.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:18.998 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:19.028 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:19.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:19.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:19.130 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:18 smithi170 ceph-mon[33394]: 8.2 scrub starts 2024-02-10T07:40:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:18 smithi170 ceph-mon[33394]: pgmap v388: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 968 op/s 2024-02-10T07:40:19.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:19.165 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:19.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:19.266 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:19.307 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:19.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:19.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:18 smithi008 ceph-mon[29816]: 8.2 scrub starts 2024-02-10T07:40:19.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:18 smithi008 ceph-mon[29816]: pgmap v388: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 968 op/s 2024-02-10T07:40:19.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:18 smithi008 ceph-mon[40743]: 8.2 scrub starts 2024-02-10T07:40:19.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:18 smithi008 ceph-mon[40743]: pgmap v388: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 968 op/s 2024-02-10T07:40:19.408 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:19.449 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:19.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:19.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:19.551 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:19.609 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:19.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:19.710 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:19.741 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:19.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:19.844 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:19.879 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:19.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:19.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:19.982 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:20.017 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:20.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:20.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:20.042 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:19 smithi170 ceph-mon[33394]: 2.0 scrub starts 2024-02-10T07:40:20.042 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:19 smithi170 ceph-mon[33394]: 2.0 scrub ok 2024-02-10T07:40:20.042 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:19 smithi170 ceph-mon[33394]: 4.1d scrub starts 2024-02-10T07:40:20.042 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:19 smithi170 ceph-mon[33394]: 4.1d scrub ok 2024-02-10T07:40:20.119 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:20.156 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:20.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:20.211 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:19 smithi008 ceph-mon[29816]: 2.0 scrub starts 2024-02-10T07:40:20.211 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:19 smithi008 ceph-mon[29816]: 2.0 scrub ok 2024-02-10T07:40:20.211 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:19 smithi008 ceph-mon[29816]: 4.1d scrub starts 2024-02-10T07:40:20.212 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:19 smithi008 ceph-mon[29816]: 4.1d scrub ok 2024-02-10T07:40:20.212 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:19 smithi008 ceph-mon[40743]: 2.0 scrub starts 2024-02-10T07:40:20.212 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:19 smithi008 ceph-mon[40743]: 2.0 scrub ok 2024-02-10T07:40:20.212 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:19 smithi008 ceph-mon[40743]: 4.1d scrub starts 2024-02-10T07:40:20.212 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:19 smithi008 ceph-mon[40743]: 4.1d scrub ok 2024-02-10T07:40:20.230 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:40:20.230 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2024-02-10T07:40:20.230 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2024-02-10T07:40:20.230 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2024-02-10T07:40:20.230 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:40:20.258 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:20.296 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:20.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:20.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:20.357 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:40:20.398 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:20.416 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:40:20.435 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:20.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:20.455 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:40:20.505 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:40:20.538 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:20.546 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:40:20.576 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:20.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:20.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:20.590 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:40:20.631 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:40:20.672 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:40:20.678 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:20.681 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:40:20.681 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:40:20.681 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:40:20.681 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:40:20.681 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:40:20.681 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:40:20.682 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:40:20.682 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:40:20.719 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:20.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:20.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:20.821 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:20.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:20.853 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:20.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:20.954 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:20.994 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:20.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:21.052 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[29816]: 8.2 scrub ok 2024-02-10T07:40:21.052 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[29816]: 4.1c scrub starts 2024-02-10T07:40:21.052 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[29816]: 6.e scrub starts 2024-02-10T07:40:21.053 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[29816]: 4.1c scrub ok 2024-02-10T07:40:21.053 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[29816]: 6.e scrub ok 2024-02-10T07:40:21.053 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[29816]: pgmap v389: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-02-10T07:40:21.053 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[29816]: 4.16 scrub starts 2024-02-10T07:40:21.053 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[29816]: 4.16 scrub ok 2024-02-10T07:40:21.053 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4210724379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:21.053 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:40:21.053 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:40:21.053 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[40743]: 8.2 scrub ok 2024-02-10T07:40:21.053 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[40743]: 4.1c scrub starts 2024-02-10T07:40:21.054 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[40743]: 6.e scrub starts 2024-02-10T07:40:21.054 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[40743]: 4.1c scrub ok 2024-02-10T07:40:21.054 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[40743]: 6.e scrub ok 2024-02-10T07:40:21.054 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[40743]: pgmap v389: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-02-10T07:40:21.054 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[40743]: 4.16 scrub starts 2024-02-10T07:40:21.054 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[40743]: 4.16 scrub ok 2024-02-10T07:40:21.054 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4210724379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:21.054 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:40:21.054 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:20 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:40:21.054 INFO:teuthology.orchestra.run.smithi008.stderr:no valid command found; 10 closest matches: 2024-02-10T07:40:21.054 INFO:teuthology.orchestra.run.smithi008.stderr:0 2024-02-10T07:40:21.054 INFO:teuthology.orchestra.run.smithi008.stderr:1 2024-02-10T07:40:21.055 INFO:teuthology.orchestra.run.smithi008.stderr:2 2024-02-10T07:40:21.055 INFO:teuthology.orchestra.run.smithi008.stderr:abort 2024-02-10T07:40:21.055 INFO:teuthology.orchestra.run.smithi008.stderr:assert 2024-02-10T07:40:21.055 INFO:teuthology.orchestra.run.smithi008.stderr:bench [] [] [] [] 2024-02-10T07:40:21.055 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs debug_inject_read_zeros 2024-02-10T07:40:21.055 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs files list 2024-02-10T07:40:21.055 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs stats 2024-02-10T07:40:21.055 INFO:teuthology.orchestra.run.smithi008.stderr:bluestore allocator dump block 2024-02-10T07:40:21.055 INFO:teuthology.orchestra.run.smithi008.stderr:Error EINVAL: invalid command 2024-02-10T07:40:21.096 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:21.127 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:21.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:20 smithi170 ceph-mon[33394]: 8.2 scrub ok 2024-02-10T07:40:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:20 smithi170 ceph-mon[33394]: 4.1c scrub starts 2024-02-10T07:40:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:20 smithi170 ceph-mon[33394]: 6.e scrub starts 2024-02-10T07:40:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:20 smithi170 ceph-mon[33394]: 4.1c scrub ok 2024-02-10T07:40:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:20 smithi170 ceph-mon[33394]: 6.e scrub ok 2024-02-10T07:40:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:20 smithi170 ceph-mon[33394]: pgmap v389: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-02-10T07:40:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:20 smithi170 ceph-mon[33394]: 4.16 scrub starts 2024-02-10T07:40:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:20 smithi170 ceph-mon[33394]: 4.16 scrub ok 2024-02-10T07:40:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:20 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4210724379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:20 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:40:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:20 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:40:21.229 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:21.261 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:21.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:21.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:21.297 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-10T07:40:21.363 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:21.400 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:21.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:21.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:21.502 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:21.541 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:21.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:21.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:21.643 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:21.680 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:21.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:21.713 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:40:21.782 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:21.821 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:21.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:21.922 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:21.959 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:21.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:22.061 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:22.098 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:22.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:21 smithi170 ceph-mon[33394]: 3.1f scrub starts 2024-02-10T07:40:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:21 smithi170 ceph-mon[33394]: 3.1f scrub ok 2024-02-10T07:40:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:21 smithi170 ceph-mon[33394]: 4.1 scrub starts 2024-02-10T07:40:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:21 smithi170 ceph-mon[33394]: 4.1 scrub ok 2024-02-10T07:40:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:21 smithi170 ceph-mon[33394]: 6.12 scrub starts 2024-02-10T07:40:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:21 smithi170 ceph-mon[33394]: 6.12 scrub ok 2024-02-10T07:40:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:21 smithi170 ceph-mon[33394]: osdmap e235: 8 total, 8 up, 8 in 2024-02-10T07:40:22.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:21 smithi170 ceph-mon[33394]: 6.1 scrub starts 2024-02-10T07:40:22.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:21 smithi170 ceph-mon[33394]: 6.1 scrub ok 2024-02-10T07:40:22.201 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:22.240 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:22.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:22.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[29816]: 3.1f scrub starts 2024-02-10T07:40:22.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[29816]: 3.1f scrub ok 2024-02-10T07:40:22.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[29816]: 4.1 scrub starts 2024-02-10T07:40:22.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[29816]: 4.1 scrub ok 2024-02-10T07:40:22.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[29816]: 6.12 scrub starts 2024-02-10T07:40:22.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[29816]: 6.12 scrub ok 2024-02-10T07:40:22.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[29816]: osdmap e235: 8 total, 8 up, 8 in 2024-02-10T07:40:22.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[29816]: 6.1 scrub starts 2024-02-10T07:40:22.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[29816]: 6.1 scrub ok 2024-02-10T07:40:22.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[40743]: 3.1f scrub starts 2024-02-10T07:40:22.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[40743]: 3.1f scrub ok 2024-02-10T07:40:22.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[40743]: 4.1 scrub starts 2024-02-10T07:40:22.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[40743]: 4.1 scrub ok 2024-02-10T07:40:22.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[40743]: 6.12 scrub starts 2024-02-10T07:40:22.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[40743]: 6.12 scrub ok 2024-02-10T07:40:22.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[40743]: osdmap e235: 8 total, 8 up, 8 in 2024-02-10T07:40:22.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[40743]: 6.1 scrub starts 2024-02-10T07:40:22.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:21 smithi008 ceph-mon[40743]: 6.1 scrub ok 2024-02-10T07:40:22.342 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:22.389 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:22.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-10T07:40:22.397 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-10T07:40:22.397 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-02-10T07:40:22.490 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:22.544 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:22.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:22.646 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:22.671 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:40:22.680 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:22.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:22.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:22.782 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:22.850 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:22.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:22.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:22.902 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:22.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:22.986 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:22.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:22.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:23.042 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[29816]: 6.16 scrub starts 2024-02-10T07:40:23.042 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[29816]: 6.1a scrub starts 2024-02-10T07:40:23.042 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[29816]: 6.1a scrub ok 2024-02-10T07:40:23.042 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[29816]: 6.16 scrub ok 2024-02-10T07:40:23.042 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[29816]: 6.3 scrub starts 2024-02-10T07:40:23.042 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[29816]: 6.3 scrub ok 2024-02-10T07:40:23.042 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[29816]: pgmap v391: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 727 B/s rd, 0 B/s wr, 771 op/s 2024-02-10T07:40:23.042 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[29816]: 4.19 scrub starts 2024-02-10T07:40:23.042 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[29816]: 4.19 scrub ok 2024-02-10T07:40:23.043 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[40743]: 6.16 scrub starts 2024-02-10T07:40:23.043 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[40743]: 6.1a scrub starts 2024-02-10T07:40:23.043 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[40743]: 6.1a scrub ok 2024-02-10T07:40:23.043 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[40743]: 6.16 scrub ok 2024-02-10T07:40:23.043 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[40743]: 6.3 scrub starts 2024-02-10T07:40:23.043 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[40743]: 6.3 scrub ok 2024-02-10T07:40:23.044 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[40743]: pgmap v391: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 727 B/s rd, 0 B/s wr, 771 op/s 2024-02-10T07:40:23.044 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[40743]: 4.19 scrub starts 2024-02-10T07:40:23.044 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:22 smithi008 ceph-mon[40743]: 4.19 scrub ok 2024-02-10T07:40:23.088 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:23.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:23.127 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:23.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:23.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:40:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:40:23.216 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:22 smithi170 ceph-mon[33394]: 6.16 scrub starts 2024-02-10T07:40:23.216 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:22 smithi170 ceph-mon[33394]: 6.1a scrub starts 2024-02-10T07:40:23.216 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:22 smithi170 ceph-mon[33394]: 6.1a scrub ok 2024-02-10T07:40:23.216 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:22 smithi170 ceph-mon[33394]: 6.16 scrub ok 2024-02-10T07:40:23.216 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:22 smithi170 ceph-mon[33394]: 6.3 scrub starts 2024-02-10T07:40:23.216 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:22 smithi170 ceph-mon[33394]: 6.3 scrub ok 2024-02-10T07:40:23.216 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:22 smithi170 ceph-mon[33394]: pgmap v391: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 727 B/s rd, 0 B/s wr, 771 op/s 2024-02-10T07:40:23.216 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:22 smithi170 ceph-mon[33394]: 4.19 scrub starts 2024-02-10T07:40:23.216 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:22 smithi170 ceph-mon[33394]: 4.19 scrub ok 2024-02-10T07:40:23.228 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:23.263 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:23.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:23.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:23.365 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:23.396 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:23.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:23.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:23.498 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:23.536 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:23.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:23.637 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:23.671 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:40:23.676 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:23.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:23.747 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:40:23.747 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:40:23.778 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:23.814 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:23.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:23.915 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:23.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:23.969 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:23.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:24.070 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:24.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:24.113 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:24.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:24.214 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:24.249 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:24.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:24.269 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:23 smithi008 ceph-mon[29816]: 3.5 scrub starts 2024-02-10T07:40:24.269 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:23 smithi008 ceph-mon[29816]: 3.5 scrub ok 2024-02-10T07:40:24.269 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:23 smithi008 ceph-mon[29816]: 6.18 scrub starts 2024-02-10T07:40:24.269 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:23 smithi008 ceph-mon[29816]: 5.10 scrub starts 2024-02-10T07:40:24.269 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:23 smithi008 ceph-mon[29816]: 5.10 scrub ok 2024-02-10T07:40:24.269 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:23 smithi008 ceph-mon[29816]: 6.18 scrub ok 2024-02-10T07:40:24.269 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:23 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:24.269 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:23 smithi008 ceph-mon[40743]: 3.5 scrub starts 2024-02-10T07:40:24.269 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:23 smithi008 ceph-mon[40743]: 3.5 scrub ok 2024-02-10T07:40:24.269 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:23 smithi008 ceph-mon[40743]: 6.18 scrub starts 2024-02-10T07:40:24.270 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:23 smithi008 ceph-mon[40743]: 5.10 scrub starts 2024-02-10T07:40:24.270 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:23 smithi008 ceph-mon[40743]: 5.10 scrub ok 2024-02-10T07:40:24.270 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:23 smithi008 ceph-mon[40743]: 6.18 scrub ok 2024-02-10T07:40:24.270 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:23 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:24.349 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:23 smithi170 ceph-mon[33394]: 3.5 scrub starts 2024-02-10T07:40:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:23 smithi170 ceph-mon[33394]: 3.5 scrub ok 2024-02-10T07:40:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:23 smithi170 ceph-mon[33394]: 6.18 scrub starts 2024-02-10T07:40:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:23 smithi170 ceph-mon[33394]: 5.10 scrub starts 2024-02-10T07:40:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:23 smithi170 ceph-mon[33394]: 5.10 scrub ok 2024-02-10T07:40:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:23 smithi170 ceph-mon[33394]: 6.18 scrub ok 2024-02-10T07:40:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:23 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:24.389 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:24.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:24.489 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T07:40:24.490 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:24.527 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:24.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:24.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:24.629 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:24.671 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:24.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:24.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:24.773 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:24.801 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:24.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:24.837 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:24.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:24.842 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:40:24.938 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:24.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:24.999 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:24.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:25.045 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:25.101 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:25.132 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:25.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:25.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:25.193 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:24 smithi008 ceph-mon[29816]: 3.e scrub starts 2024-02-10T07:40:25.194 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:24 smithi008 ceph-mon[29816]: 3.e scrub ok 2024-02-10T07:40:25.194 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:24 smithi008 ceph-mon[29816]: 4.3 scrub starts 2024-02-10T07:40:25.194 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:24 smithi008 ceph-mon[29816]: 4.3 scrub ok 2024-02-10T07:40:25.194 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:24 smithi008 ceph-mon[29816]: pgmap v392: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-02-10T07:40:25.194 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:24 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:40:25.194 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:24 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:40:25.194 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:24 smithi008 ceph-mon[40743]: 3.e scrub starts 2024-02-10T07:40:25.194 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:24 smithi008 ceph-mon[40743]: 3.e scrub ok 2024-02-10T07:40:25.194 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:24 smithi008 ceph-mon[40743]: 4.3 scrub starts 2024-02-10T07:40:25.194 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:24 smithi008 ceph-mon[40743]: 4.3 scrub ok 2024-02-10T07:40:25.194 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:24 smithi008 ceph-mon[40743]: pgmap v392: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-02-10T07:40:25.194 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:24 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:40:25.195 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:24 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:40:25.234 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:25.270 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:25.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:25.271 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:24 smithi170 ceph-mon[33394]: 3.e scrub starts 2024-02-10T07:40:25.271 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:24 smithi170 ceph-mon[33394]: 3.e scrub ok 2024-02-10T07:40:25.271 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:24 smithi170 ceph-mon[33394]: 4.3 scrub starts 2024-02-10T07:40:25.271 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:24 smithi170 ceph-mon[33394]: 4.3 scrub ok 2024-02-10T07:40:25.271 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:24 smithi170 ceph-mon[33394]: pgmap v392: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-02-10T07:40:25.271 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:24 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:40:25.271 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:24 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:40:25.371 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:25.402 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:25.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:25.504 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:25.540 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:25.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:25.641 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:25.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:25.692 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:25.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:25.793 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:25.822 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:25.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:25.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:25.924 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:25.952 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:25.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:25.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:26.055 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:26.094 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:26.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:26.180 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:25 smithi008 ceph-mon[29816]: 6.0 scrub starts 2024-02-10T07:40:26.180 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:25 smithi008 ceph-mon[29816]: 6.0 scrub ok 2024-02-10T07:40:26.180 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:25 smithi008 ceph-mon[29816]: 5.1e scrub starts 2024-02-10T07:40:26.180 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:25 smithi008 ceph-mon[29816]: 5.1e scrub ok 2024-02-10T07:40:26.180 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:25 smithi008 ceph-mon[29816]: osdmap e236: 8 total, 8 up, 8 in 2024-02-10T07:40:26.180 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:25 smithi008 ceph-mon[40743]: 6.0 scrub starts 2024-02-10T07:40:26.180 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:25 smithi008 ceph-mon[40743]: 6.0 scrub ok 2024-02-10T07:40:26.181 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:25 smithi008 ceph-mon[40743]: 5.1e scrub starts 2024-02-10T07:40:26.181 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:25 smithi008 ceph-mon[40743]: 5.1e scrub ok 2024-02-10T07:40:26.181 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:25 smithi008 ceph-mon[40743]: osdmap e236: 8 total, 8 up, 8 in 2024-02-10T07:40:26.195 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:26.235 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:26.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:26.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:26.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:25 smithi170 ceph-mon[33394]: 6.0 scrub starts 2024-02-10T07:40:26.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:25 smithi170 ceph-mon[33394]: 6.0 scrub ok 2024-02-10T07:40:26.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:25 smithi170 ceph-mon[33394]: 5.1e scrub starts 2024-02-10T07:40:26.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:25 smithi170 ceph-mon[33394]: 5.1e scrub ok 2024-02-10T07:40:26.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:25 smithi170 ceph-mon[33394]: osdmap e236: 8 total, 8 up, 8 in 2024-02-10T07:40:26.337 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:26.364 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:26.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:26.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:26.466 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:26.507 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:26.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:26.609 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:26.638 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:26.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:26.740 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:26.765 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:26.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:26.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:26.867 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:26.910 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:26.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:27.011 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:27.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:27.044 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:27.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:27.092 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:26 smithi008 ceph-mon[29816]: 5.8 scrub starts 2024-02-10T07:40:27.092 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:26 smithi008 ceph-mon[29816]: 5.8 scrub ok 2024-02-10T07:40:27.092 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:26 smithi008 ceph-mon[29816]: 3.1a scrub starts 2024-02-10T07:40:27.092 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:26 smithi008 ceph-mon[29816]: 3.1a scrub ok 2024-02-10T07:40:27.092 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:26 smithi008 ceph-mon[29816]: 3.1 scrub starts 2024-02-10T07:40:27.092 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:26 smithi008 ceph-mon[29816]: 3.1 scrub ok 2024-02-10T07:40:27.092 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:26 smithi008 ceph-mon[29816]: pgmap v394: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.16k op/s 2024-02-10T07:40:27.094 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:26 smithi008 ceph-mon[40743]: 5.8 scrub starts 2024-02-10T07:40:27.094 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:26 smithi008 ceph-mon[40743]: 5.8 scrub ok 2024-02-10T07:40:27.094 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:26 smithi008 ceph-mon[40743]: 3.1a scrub starts 2024-02-10T07:40:27.094 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:26 smithi008 ceph-mon[40743]: 3.1a scrub ok 2024-02-10T07:40:27.094 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:26 smithi008 ceph-mon[40743]: 3.1 scrub starts 2024-02-10T07:40:27.094 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:26 smithi008 ceph-mon[40743]: 3.1 scrub ok 2024-02-10T07:40:27.094 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:26 smithi008 ceph-mon[40743]: pgmap v394: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.16k op/s 2024-02-10T07:40:27.145 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:27.200 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:27.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:27.214 INFO:teuthology.orchestra.run.smithi008.stderr:no valid command found; 10 closest matches: 2024-02-10T07:40:27.214 INFO:teuthology.orchestra.run.smithi008.stderr:0 2024-02-10T07:40:27.214 INFO:teuthology.orchestra.run.smithi008.stderr:1 2024-02-10T07:40:27.215 INFO:teuthology.orchestra.run.smithi008.stderr:2 2024-02-10T07:40:27.215 INFO:teuthology.orchestra.run.smithi008.stderr:abort 2024-02-10T07:40:27.215 INFO:teuthology.orchestra.run.smithi008.stderr:assert 2024-02-10T07:40:27.215 INFO:teuthology.orchestra.run.smithi008.stderr:bench [] [] [] [] 2024-02-10T07:40:27.215 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs debug_inject_read_zeros 2024-02-10T07:40:27.215 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs files list 2024-02-10T07:40:27.215 INFO:teuthology.orchestra.run.smithi008.stderr:bluefs stats 2024-02-10T07:40:27.215 INFO:teuthology.orchestra.run.smithi008.stderr:bluestore allocator dump block 2024-02-10T07:40:27.215 INFO:teuthology.orchestra.run.smithi008.stderr:Error EINVAL: invalid command 2024-02-10T07:40:27.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:26 smithi170 ceph-mon[33394]: 5.8 scrub starts 2024-02-10T07:40:27.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:26 smithi170 ceph-mon[33394]: 5.8 scrub ok 2024-02-10T07:40:27.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:26 smithi170 ceph-mon[33394]: 3.1a scrub starts 2024-02-10T07:40:27.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:26 smithi170 ceph-mon[33394]: 3.1a scrub ok 2024-02-10T07:40:27.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:26 smithi170 ceph-mon[33394]: 3.1 scrub starts 2024-02-10T07:40:27.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:26 smithi170 ceph-mon[33394]: 3.1 scrub ok 2024-02-10T07:40:27.230 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:26 smithi170 ceph-mon[33394]: pgmap v394: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.16k op/s 2024-02-10T07:40:27.302 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:27.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:27.346 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:27.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:27.447 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:27.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:27.486 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:27.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:27.585 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:40:27.585 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:40:27.587 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:27.632 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:27.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:27.733 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:27.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:27.773 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:27.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:27.874 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:27.915 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:27.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:27.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:28.017 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:28.062 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:28.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:28.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:27 smithi008 ceph-mon[29816]: 4.5 scrub starts 2024-02-10T07:40:28.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:27 smithi008 ceph-mon[29816]: 4.5 scrub ok 2024-02-10T07:40:28.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:27 smithi008 ceph-mon[29816]: 6.2 scrub starts 2024-02-10T07:40:28.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:27 smithi008 ceph-mon[29816]: 6.2 scrub ok 2024-02-10T07:40:28.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:27 smithi008 ceph-mon[40743]: 4.5 scrub starts 2024-02-10T07:40:28.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:27 smithi008 ceph-mon[40743]: 4.5 scrub ok 2024-02-10T07:40:28.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:27 smithi008 ceph-mon[40743]: 6.2 scrub starts 2024-02-10T07:40:28.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:27 smithi008 ceph-mon[40743]: 6.2 scrub ok 2024-02-10T07:40:28.164 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:28.202 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:28.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:28.209 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:27 smithi170 ceph-mon[33394]: 4.5 scrub starts 2024-02-10T07:40:28.209 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:27 smithi170 ceph-mon[33394]: 4.5 scrub ok 2024-02-10T07:40:28.209 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:27 smithi170 ceph-mon[33394]: 6.2 scrub starts 2024-02-10T07:40:28.209 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:27 smithi170 ceph-mon[33394]: 6.2 scrub ok 2024-02-10T07:40:28.304 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:28.310 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-10T07:40:28.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:28.343 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:28.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:28.444 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:28.494 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:28.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:28.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:28.596 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:28.628 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:28.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:28.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:28.692 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:40:28.730 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:28.763 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:28.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:28.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:28.866 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:28.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:28.904 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:28.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:29.002 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:40:29.005 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:29.048 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:29.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:29.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[29816]: 5.11 scrub starts 2024-02-10T07:40:29.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[29816]: 5.11 scrub ok 2024-02-10T07:40:29.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[29816]: 5.b scrub starts 2024-02-10T07:40:29.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[29816]: 5.b scrub ok 2024-02-10T07:40:29.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[29816]: pgmap v395: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 991 op/s 2024-02-10T07:40:29.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[29816]: 4.11 scrub starts 2024-02-10T07:40:29.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2734444084' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:29.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:29.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[29816]: 4.11 scrub ok 2024-02-10T07:40:29.084 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:40:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:40:29.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[40743]: 5.11 scrub starts 2024-02-10T07:40:29.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[40743]: 5.11 scrub ok 2024-02-10T07:40:29.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[40743]: 5.b scrub starts 2024-02-10T07:40:29.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[40743]: 5.b scrub ok 2024-02-10T07:40:29.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[40743]: pgmap v395: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 991 op/s 2024-02-10T07:40:29.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[40743]: 4.11 scrub starts 2024-02-10T07:40:29.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2734444084' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:29.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:29.084 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:28 smithi008 ceph-mon[40743]: 4.11 scrub ok 2024-02-10T07:40:29.123 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:28 smithi170 ceph-mon[33394]: 5.11 scrub starts 2024-02-10T07:40:29.123 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:28 smithi170 ceph-mon[33394]: 5.11 scrub ok 2024-02-10T07:40:29.123 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:28 smithi170 ceph-mon[33394]: 5.b scrub starts 2024-02-10T07:40:29.123 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:28 smithi170 ceph-mon[33394]: 5.b scrub ok 2024-02-10T07:40:29.123 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:28 smithi170 ceph-mon[33394]: pgmap v395: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 991 op/s 2024-02-10T07:40:29.123 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:28 smithi170 ceph-mon[33394]: 4.11 scrub starts 2024-02-10T07:40:29.123 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:28 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2734444084' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:29.123 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:28 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:29.123 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:28 smithi170 ceph-mon[33394]: 4.11 scrub ok 2024-02-10T07:40:29.151 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:29.182 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:29.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:29.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:29.283 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:29.318 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:29.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:29.420 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:29.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:29.458 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:29.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:29.497 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-10T07:40:29.498 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-10T07:40:29.559 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:29.602 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:29.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:29.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:29.704 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:29.737 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:29.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:29.838 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:29.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:29.879 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:29.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:29.980 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:30.024 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:30.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:30.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[29816]: 5.1f scrub starts 2024-02-10T07:40:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[29816]: 5.1f scrub ok 2024-02-10T07:40:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[29816]: 6.1d scrub starts 2024-02-10T07:40:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:40:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:40:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:40:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[29816]: osdmap e237: 8 total, 8 up, 8 in 2024-02-10T07:40:30.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[29816]: 6.1d scrub ok 2024-02-10T07:40:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[40743]: 5.1f scrub starts 2024-02-10T07:40:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[40743]: 5.1f scrub ok 2024-02-10T07:40:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[40743]: 6.1d scrub starts 2024-02-10T07:40:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:40:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:40:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:40:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[40743]: osdmap e237: 8 total, 8 up, 8 in 2024-02-10T07:40:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:30 smithi008 ceph-mon[40743]: 6.1d scrub ok 2024-02-10T07:40:30.127 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:30.158 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:30 smithi170 ceph-mon[33394]: 5.1f scrub starts 2024-02-10T07:40:30.158 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:30 smithi170 ceph-mon[33394]: 5.1f scrub ok 2024-02-10T07:40:30.158 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:30 smithi170 ceph-mon[33394]: 6.1d scrub starts 2024-02-10T07:40:30.158 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:30 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:40:30.158 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:30 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:40:30.158 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:30 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:40:30.158 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:30 smithi170 ceph-mon[33394]: osdmap e237: 8 total, 8 up, 8 in 2024-02-10T07:40:30.158 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:30 smithi170 ceph-mon[33394]: 6.1d scrub ok 2024-02-10T07:40:30.164 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:30.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:30.266 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:30.306 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:30.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:30.408 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:30.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:30.445 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:30.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:30.546 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:30.581 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:30.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:30.683 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:30.695 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:40:30.695 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:40:30.714 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:30.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:30.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:30.743 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:40:30.786 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:40:30.816 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:30.832 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:40:30.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:30.853 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:30.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:30.876 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:40:30.918 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:40:30.955 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:30.966 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:40:30.988 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:30.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:30.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:31.007 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:40:31.042 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:40:31.043 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:40:31.043 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:40:31.043 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:40:31.043 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:40:31.043 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:40:31.044 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:40:31.044 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:40:31.044 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:40:31.090 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:31.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:31.119 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:31.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:31.220 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:31.253 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:31.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:31.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[29816]: 3.1d scrub starts 2024-02-10T07:40:31.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[29816]: 3.1d scrub ok 2024-02-10T07:40:31.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[29816]: pgmap v397: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-02-10T07:40:31.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[29816]: 3.6 scrub starts 2024-02-10T07:40:31.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[29816]: 3.6 scrub ok 2024-02-10T07:40:31.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[29816]: 6.6 scrub starts 2024-02-10T07:40:31.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[29816]: 6.6 scrub ok 2024-02-10T07:40:31.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3827111885' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:31.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[40743]: 3.1d scrub starts 2024-02-10T07:40:31.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[40743]: 3.1d scrub ok 2024-02-10T07:40:31.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[40743]: pgmap v397: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-02-10T07:40:31.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[40743]: 3.6 scrub starts 2024-02-10T07:40:31.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[40743]: 3.6 scrub ok 2024-02-10T07:40:31.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[40743]: 6.6 scrub starts 2024-02-10T07:40:31.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[40743]: 6.6 scrub ok 2024-02-10T07:40:31.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:31 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3827111885' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:31.356 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:31.381 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:31.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:31.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:31.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:31 smithi170 ceph-mon[33394]: 3.1d scrub starts 2024-02-10T07:40:31.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:31 smithi170 ceph-mon[33394]: 3.1d scrub ok 2024-02-10T07:40:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:31 smithi170 ceph-mon[33394]: pgmap v397: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-02-10T07:40:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:31 smithi170 ceph-mon[33394]: 3.6 scrub starts 2024-02-10T07:40:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:31 smithi170 ceph-mon[33394]: 3.6 scrub ok 2024-02-10T07:40:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:31 smithi170 ceph-mon[33394]: 6.6 scrub starts 2024-02-10T07:40:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:31 smithi170 ceph-mon[33394]: 6.6 scrub ok 2024-02-10T07:40:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:31 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3827111885' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:31.483 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:31.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:31.514 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:31.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:31.615 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:31.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:31.647 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:31.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:31.747 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:31.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:31.786 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:31.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:31.887 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:31.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:31.927 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:31.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:32.029 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:32.154 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:32.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:32.199 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:32 smithi170 ceph-mon[33394]: 3.3 scrub starts 2024-02-10T07:40:32.199 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:32 smithi170 ceph-mon[33394]: 3.3 scrub ok 2024-02-10T07:40:32.199 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:32 smithi170 ceph-mon[33394]: 4.6 deep-scrub starts 2024-02-10T07:40:32.200 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:32 smithi170 ceph-mon[33394]: 4.6 deep-scrub ok 2024-02-10T07:40:32.242 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-10T07:40:32.255 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:32.324 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:32.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:32.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:32 smithi008 ceph-mon[29816]: 3.3 scrub starts 2024-02-10T07:40:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:32 smithi008 ceph-mon[29816]: 3.3 scrub ok 2024-02-10T07:40:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:32 smithi008 ceph-mon[29816]: 4.6 deep-scrub starts 2024-02-10T07:40:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:32 smithi008 ceph-mon[29816]: 4.6 deep-scrub ok 2024-02-10T07:40:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:32 smithi008 ceph-mon[40743]: 3.3 scrub starts 2024-02-10T07:40:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:32 smithi008 ceph-mon[40743]: 3.3 scrub ok 2024-02-10T07:40:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:32 smithi008 ceph-mon[40743]: 4.6 deep-scrub starts 2024-02-10T07:40:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:32 smithi008 ceph-mon[40743]: 4.6 deep-scrub ok 2024-02-10T07:40:32.425 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:32.456 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:40:32.463 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:32.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:32.566 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:32.605 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:40:32.641 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:32.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:32.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:32.743 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:32.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:32.804 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:32.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:32.884 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:32.905 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:32.937 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:32.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:32.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:33.039 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:33.088 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:33.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:33.189 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:33.234 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:33.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[40743]: 3.1c deep-scrub starts 2024-02-10T07:40:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[40743]: 3.1c deep-scrub ok 2024-02-10T07:40:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[40743]: 5.4 deep-scrub starts 2024-02-10T07:40:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[40743]: 5.4 deep-scrub ok 2024-02-10T07:40:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[40743]: pgmap v398: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 696 op/s 2024-02-10T07:40:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[40743]: 4.1e scrub starts 2024-02-10T07:40:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[40743]: 4.1e scrub ok 2024-02-10T07:40:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[40743]: 3.4 scrub starts 2024-02-10T07:40:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[40743]: 3.4 scrub ok 2024-02-10T07:40:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:40:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:33.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[29816]: 3.1c deep-scrub starts 2024-02-10T07:40:33.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[29816]: 3.1c deep-scrub ok 2024-02-10T07:40:33.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[29816]: 5.4 deep-scrub starts 2024-02-10T07:40:33.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[29816]: 5.4 deep-scrub ok 2024-02-10T07:40:33.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[29816]: pgmap v398: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 696 op/s 2024-02-10T07:40:33.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[29816]: 4.1e scrub starts 2024-02-10T07:40:33.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[29816]: 4.1e scrub ok 2024-02-10T07:40:33.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[29816]: 3.4 scrub starts 2024-02-10T07:40:33.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[29816]: 3.4 scrub ok 2024-02-10T07:40:33.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:40:33.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:33.365 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:33.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:40:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:40:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:33 smithi170 ceph-mon[33394]: 3.1c deep-scrub starts 2024-02-10T07:40:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:33 smithi170 ceph-mon[33394]: 3.1c deep-scrub ok 2024-02-10T07:40:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:33 smithi170 ceph-mon[33394]: 5.4 deep-scrub starts 2024-02-10T07:40:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:33 smithi170 ceph-mon[33394]: 5.4 deep-scrub ok 2024-02-10T07:40:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:33 smithi170 ceph-mon[33394]: pgmap v398: 148 pgs: 148 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 696 op/s 2024-02-10T07:40:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:33 smithi170 ceph-mon[33394]: 4.1e scrub starts 2024-02-10T07:40:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:33 smithi170 ceph-mon[33394]: 4.1e scrub ok 2024-02-10T07:40:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:33 smithi170 ceph-mon[33394]: 3.4 scrub starts 2024-02-10T07:40:33.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:33 smithi170 ceph-mon[33394]: 3.4 scrub ok 2024-02-10T07:40:33.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:40:33.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:33.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:33.405 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:33.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:33.506 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:33.547 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:33.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:33.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:33.649 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:33.729 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:33.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:33.741 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:40:33.832 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:33.867 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:33.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:33.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:33.969 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:34.034 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:34.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:34.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:34 smithi008 ceph-mon[29816]: 5.1b scrub starts 2024-02-10T07:40:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:34 smithi008 ceph-mon[29816]: 5.1b scrub ok 2024-02-10T07:40:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:34 smithi008 ceph-mon[29816]: 4.1b scrub starts 2024-02-10T07:40:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:34 smithi008 ceph-mon[29816]: 4.1b scrub ok 2024-02-10T07:40:34.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:34 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:34.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:34 smithi008 ceph-mon[40743]: 5.1b scrub starts 2024-02-10T07:40:34.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:34 smithi008 ceph-mon[40743]: 5.1b scrub ok 2024-02-10T07:40:34.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:34 smithi008 ceph-mon[40743]: 4.1b scrub starts 2024-02-10T07:40:34.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:34 smithi008 ceph-mon[40743]: 4.1b scrub ok 2024-02-10T07:40:34.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:34 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:34.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:34 smithi170 ceph-mon[33394]: 5.1b scrub starts 2024-02-10T07:40:34.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:34 smithi170 ceph-mon[33394]: 5.1b scrub ok 2024-02-10T07:40:34.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:34 smithi170 ceph-mon[33394]: 4.1b scrub starts 2024-02-10T07:40:34.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:34 smithi170 ceph-mon[33394]: 4.1b scrub ok 2024-02-10T07:40:34.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:34 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:34.137 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:34.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:34.169 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:34.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:34.270 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:34.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:34.294 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:34.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:34.395 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:34.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:34.446 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:34.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:34.499 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-10T07:40:34.499 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:40:34.499 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-02-10T07:40:34.500 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-10T07:40:34.500 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-10T07:40:34.500 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-02-10T07:40:34.547 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:34.565 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:40:34.565 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2024-02-10T07:40:34.565 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2024-02-10T07:40:34.565 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:40:34.565 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.565 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:154826 7.0 7.577e9d00 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:40:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:32.885749+0000", 2024-02-10T07:40:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 241.67478671000001, 2024-02-10T07:40:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.86189586600000001, 2024-02-10T07:40:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:40:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 154826 2024-02-10T07:40:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885749+0000", 2024-02-10T07:40:34.567 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.567 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.567 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.567 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.567 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885749+0000", 2024-02-10T07:40:34.567 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.567 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.567 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.567 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.567 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885751+0000", 2024-02-10T07:40:34.568 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2109999999999999e-06 2024-02-10T07:40:34.568 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.568 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.568 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.568 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885755+0000", 2024-02-10T07:40:34.568 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7960000000000003e-06 2024-02-10T07:40:34.568 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.568 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.568 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.568 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885765+0000", 2024-02-10T07:40:34.568 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0101999999999999e-05 2024-02-10T07:40:34.568 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.568 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.569 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.569 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885775+0000", 2024-02-10T07:40:34.569 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9359999999999994e-06 2024-02-10T07:40:34.569 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.569 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.569 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.569 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885836+0000", 2024-02-10T07:40:34.569 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0615999999999997e-05 2024-02-10T07:40:34.569 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.569 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.569 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:40:34.569 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885856+0000", 2024-02-10T07:40:34.570 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0418999999999999e-05 2024-02-10T07:40:34.570 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.570 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.570 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.570 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:33.747631+0000", 2024-02-10T07:40:34.570 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.86177504800000004 2024-02-10T07:40:34.570 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.570 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.570 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.570 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:33.747645+0000", 2024-02-10T07:40:34.570 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3738e-05 2024-02-10T07:40:34.570 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.570 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.570 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.571 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.571 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:182765 7.0 7:02e686aa:::benchmark_data_smithi008_82046_object67627:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-02-10T07:40:34.571 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:06.052259+0000", 2024-02-10T07:40:34.571 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 208.508277419, 2024-02-10T07:40:34.571 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.111276601, 2024-02-10T07:40:34.571 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.571 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.571 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:34.571 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:34.571 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:34.571 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 182765 2024-02-10T07:40:34.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.571 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.572 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.572 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.572 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052259+0000", 2024-02-10T07:40:34.572 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.572 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.572 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.572 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.572 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052259+0000", 2024-02-10T07:40:34.572 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.572 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.572 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.572 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.572 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052260+0000", 2024-02-10T07:40:34.572 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8080000000000001e-06 2024-02-10T07:40:34.573 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.573 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.573 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.573 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052266+0000", 2024-02-10T07:40:34.573 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4620000000000004e-06 2024-02-10T07:40:34.573 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.573 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.573 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.573 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052283+0000", 2024-02-10T07:40:34.573 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7297000000000001e-05 2024-02-10T07:40:34.573 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.573 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.573 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.573 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052292+0000", 2024-02-10T07:40:34.574 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6379999999999993e-06 2024-02-10T07:40:34.574 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.574 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.574 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052351+0000", 2024-02-10T07:40:34.574 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8943999999999997e-05 2024-02-10T07:40:34.574 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.574 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.574 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052465+0000", 2024-02-10T07:40:34.574 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011377100000000001 2024-02-10T07:40:34.574 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.574 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.052868+0000", 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00040303099999999998 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055300+0000", 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0024328169999999999 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.058232+0000", 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002931978 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.575 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.163436+0000", 2024-02-10T07:40:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.10520370699999999 2024-02-10T07:40:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.163494+0000", 2024-02-10T07:40:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7556999999999998e-05 2024-02-10T07:40:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.163535+0000", 2024-02-10T07:40:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1591000000000002e-05 2024-02-10T07:40:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:182767 7.0 7:063d4601:::benchmark_data_smithi008_82046_object67629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-02-10T07:40:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:06.055797+0000", 2024-02-10T07:40:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 208.50473907099999, 2024-02-10T07:40:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.10783342999999999, 2024-02-10T07:40:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 182767 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055797+0000", 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055797+0000", 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055798+0000", 2024-02-10T07:40:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3340000000000001e-06 2024-02-10T07:40:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055804+0000", 2024-02-10T07:40:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4360000000000001e-06 2024-02-10T07:40:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055817+0000", 2024-02-10T07:40:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3298999999999999e-05 2024-02-10T07:40:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055825+0000", 2024-02-10T07:40:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9950000000000005e-06 2024-02-10T07:40:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.055877+0000", 2024-02-10T07:40:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1536999999999998e-05 2024-02-10T07:40:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.056012+0000", 2024-02-10T07:40:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013553799999999999 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.056365+0000", 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000352686 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.058275+0000", 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001910192 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.058520+0000", 2024-02-10T07:40:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024463000000000002 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.163565+0000", 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.105045084 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.163594+0000", 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8890999999999998e-05 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:06.163630+0000", 2024-02-10T07:40:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6807999999999997e-05 2024-02-10T07:40:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:212177 7.0 7:07f25417:::benchmark_data_smithi008_82046_object97039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-10T07:40:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:37.341966+0000", 2024-02-10T07:40:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 177.218569696, 2024-02-10T07:40:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.10642299299999999, 2024-02-10T07:40:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:40:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:40:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 212177 2024-02-10T07:40:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:37.341966+0000", 2024-02-10T07:40:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:37.341966+0000", 2024-02-10T07:40:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:37.341968+0000", 2024-02-10T07:40:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1570000000000001e-06 2024-02-10T07:40:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:37.341972+0000", 2024-02-10T07:40:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3510000000000004e-06 2024-02-10T07:40:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:37.341983+0000", 2024-02-10T07:40:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1321000000000001e-05 2024-02-10T07:40:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:37.341990+0000", 2024-02-10T07:40:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9369999999999998e-06 2024-02-10T07:40:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:37.342049+0000", 2024-02-10T07:40:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9200000000000002e-05 2024-02-10T07:40:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:37.342186+0000", 2024-02-10T07:40:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013683799999999999 2024-02-10T07:40:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:40:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:37.342561+0000", 2024-02-10T07:40:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00037445099999999998 2024-02-10T07:40:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:37.344174+0000", 2024-02-10T07:40:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001613771 2024-02-10T07:40:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:37.344943+0000", 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00076843099999999998 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:37.448299+0000", 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.10335616 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:37.448351+0000", 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2274999999999997e-05 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:37.448389+0000", 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8101e-05 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:144626 8.4 8:271d8ba5:::benchmark_data_smithi008_109566_object19539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.472241+0000", 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.088295076000001, 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12407452200000001, 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:40:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 144626 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472241+0000", 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472241+0000", 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472242+0000", 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0999999999999997e-07 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472245+0000", 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.394e-06 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472253+0000", 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4460000000000004e-06 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472258+0000", 2024-02-10T07:40:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9049999999999996e-06 2024-02-10T07:40:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472286+0000", 2024-02-10T07:40:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7982000000000001e-05 2024-02-10T07:40:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472361+0000", 2024-02-10T07:40:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5145e-05 2024-02-10T07:40:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:40:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472438+0000", 2024-02-10T07:40:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7408000000000002e-05 2024-02-10T07:40:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.475157+0000", 2024-02-10T07:40:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002719244 2024-02-10T07:40:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.475595+0000", 2024-02-10T07:40:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00043725699999999998 2024-02-10T07:40:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596109+0000", 2024-02-10T07:40:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12051481 2024-02-10T07:40:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596197+0000", 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7805000000000004e-05 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596316+0000", 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000118316 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15087.0:144623 8.0 e232/200 8:07a3f5f8:::benchmark_data_smithi008_109566_object19536:head v 232'8942, mlcod=232'8939)", 2024-02-10T07:40:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.476238+0000", 2024-02-10T07:40:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.084298439000001, 2024-02-10T07:40:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12355805, 2024-02-10T07:40:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476238+0000", 2024-02-10T07:40:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476238+0000", 2024-02-10T07:40:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476238+0000", 2024-02-10T07:40:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2399999999999998e-07 2024-02-10T07:40:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476244+0000", 2024-02-10T07:40:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9610000000000002e-06 2024-02-10T07:40:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476253+0000", 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.4950000000000008e-06 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476258+0000", 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1760000000000001e-06 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476297+0000", 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8695999999999997e-05 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476310+0000", 2024-02-10T07:40:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3533000000000001e-05 2024-02-10T07:40:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.599757+0000", 2024-02-10T07:40:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.123446587 2024-02-10T07:40:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.599796+0000", 2024-02-10T07:40:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8977999999999998e-05 2024-02-10T07:40:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:144631 8.4 8:2f461f31:::benchmark_data_smithi008_109566_object19544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-02-10T07:40:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.476357+0000", 2024-02-10T07:40:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.084179090999999, 2024-02-10T07:40:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12011643399999999, 2024-02-10T07:40:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:40:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:40:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 144631 2024-02-10T07:40:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476357+0000", 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476357+0000", 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476358+0000", 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.16e-07 2024-02-10T07:40:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476360+0000", 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5229999999999998e-06 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476367+0000", 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.703e-06 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476372+0000", 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5199999999999999e-06 2024-02-10T07:40:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476407+0000", 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5549000000000001e-05 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476478+0000", 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0757e-05 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476546+0000", 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.7724000000000004e-05 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596071+0000", 2024-02-10T07:40:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.119525567 2024-02-10T07:40:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596359+0000", 2024-02-10T07:40:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00028778899999999999 2024-02-10T07:40:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596371+0000", 2024-02-10T07:40:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1768e-05 2024-02-10T07:40:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596440+0000", 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8759999999999999e-05 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596473+0000", 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3858e-05 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15087.0:144630 8.2 e232/200 8:4cb0000a:::benchmark_data_smithi008_109566_object19543:head v 232'9097, mlcod=232'9095)", 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.476544+0000", 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.083991700999999, 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11592419800000001, 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476544+0000", 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476544+0000", 2024-02-10T07:40:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476545+0000", 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2600000000000002e-07 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476550+0000", 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9830000000000004e-06 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476558+0000", 2024-02-10T07:40:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7759999999999997e-06 2024-02-10T07:40:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476573+0000", 2024-02-10T07:40:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.502e-05 2024-02-10T07:40:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476699+0000", 2024-02-10T07:40:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012628699999999999 2024-02-10T07:40:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476718+0000", 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9148999999999999e-05 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592411+0000", 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11569304599999999 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592469+0000", 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7411000000000003e-05 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15087.0:144625 8.0 e232/200 8:0c0904a8:::benchmark_data_smithi008_109566_object19538:head v 232'8943, mlcod=232'8939)", 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.477420+0000", 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.083116393000001, 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12644592499999999, 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477420+0000", 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477420+0000", 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477420+0000", 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1999999999999999e-07 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477426+0000", 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6230000000000002e-06 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477435+0000", 2024-02-10T07:40:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.9879999999999993e-06 2024-02-10T07:40:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477441+0000", 2024-02-10T07:40:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1990000000000003e-06 2024-02-10T07:40:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592745+0000", 2024-02-10T07:40:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11530351899999999 2024-02-10T07:40:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592769+0000", 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4640000000000001e-05 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.603827+0000", 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011057255 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.603866+0000", 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9081000000000001e-05 2024-02-10T07:40:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:144633 8.8 8:1f89b4bf:::benchmark_data_smithi008_109566_object19546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-02-10T07:40:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.477783+0000", 2024-02-10T07:40:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.082753497999999, 2024-02-10T07:40:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12616041, 2024-02-10T07:40:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:40:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:40:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 144633 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477783+0000", 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477783+0000", 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477783+0000", 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8999999999999996e-07 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477787+0000", 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.41e-06 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477796+0000", 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0839999999999998e-06 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477805+0000", 2024-02-10T07:40:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5629999999999998e-06 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592422+0000", 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.114616788 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592586+0000", 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016402499999999999 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592728+0000", 2024-02-10T07:40:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014208999999999999 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.602229+0000", 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0095005639999999995 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.602240+0000", 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1272e-05 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.603861+0000", 2024-02-10T07:40:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0016212139999999999 2024-02-10T07:40:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.603915+0000", 2024-02-10T07:40:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3671999999999997e-05 2024-02-10T07:40:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.603943+0000", 2024-02-10T07:40:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8038000000000002e-05 2024-02-10T07:40:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:34.663 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:34.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15087.0:144632 8.a e232/200 8:58d39958:::benchmark_data_smithi008_109566_object19545:head v 232'9186, mlcod=232'9184)", 2024-02-10T07:40:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.478417+0000", 2024-02-10T07:40:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.082119291000001, 2024-02-10T07:40:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11883959199999999, 2024-02-10T07:40:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.478417+0000", 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.478417+0000", 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.478417+0000", 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5999999999999999e-07 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.478423+0000", 2024-02-10T07:40:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2850000000000002e-06 2024-02-10T07:40:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592761+0000", 2024-02-10T07:40:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.114338115 2024-02-10T07:40:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592768+0000", 2024-02-10T07:40:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5410000000000003e-06 2024-02-10T07:40:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592813+0000", 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5006999999999998e-05 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592828+0000", 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4491000000000001e-05 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.597220+0000", 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.004392544 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.597256+0000", 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6149000000000002e-05 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop_reply(client.15087.0:144631 8.4 e232/200 ondisk, result = 0)", 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.478871+0000", 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.081665262000001, 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11720762699999999, 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:40:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.478871+0000", 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.478871+0000", 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.478871+0000", 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0500000000000003e-07 2024-02-10T07:40:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.478875+0000", 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5599999999999998e-06 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592701+0000", 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.113826477 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592715+0000", 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.324e-05 2024-02-10T07:40:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596057+0000", 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033419560000000001 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596065+0000", 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1289999999999996e-06 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596078+0000", 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.366e-05 2024-02-10T07:40:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:144636 8.4 8:2934f2c0:::benchmark_data_smithi008_109566_object19549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-02-10T07:40:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.480560+0000", 2024-02-10T07:40:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 21.079976289000001, 2024-02-10T07:40:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11838391299999999, 2024-02-10T07:40:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:40:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:40:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 144636 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480560+0000", 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480560+0000", 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480560+0000", 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4499999999999997e-07 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480564+0000", 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5039999999999998e-06 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592816+0000", 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11225207399999999 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592831+0000", 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5068999999999999e-05 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592888+0000", 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7164999999999999e-05 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592998+0000", 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000109948 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.593110+0000", 2024-02-10T07:40:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000111422 2024-02-10T07:40:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596481+0000", 2024-02-10T07:40:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033717399999999998 2024-02-10T07:40:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.598790+0000", 2024-02-10T07:40:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002308531 2024-02-10T07:40:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.598824+0000", 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4162999999999998e-05 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.598906+0000", 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1717000000000004e-05 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.598944+0000", 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7935000000000002e-05 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:145042 8.d 8:be6619ce:::benchmark_data_smithi008_109566_object19955:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.981149+0000", 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 20.579386774, 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.110597083, 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 145042 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.981149+0000", 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.981149+0000", 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.981151+0000", 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.319e-06 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.981157+0000", 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9819999999999996e-06 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.981167+0000", 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0869e-05 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.981175+0000", 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7230000000000004e-06 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.983079+0000", 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001903503 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.983178+0000", 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9537000000000006e-05 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.983281+0000", 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010310900000000001 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.986795+0000", 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.003513895 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.988855+0000", 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020596910000000002 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.091616+0000", 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.102760592 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.091681+0000", 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5905e-05 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.091746+0000", 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4957999999999997e-05 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:145056 8.d 8:bf4dde14:::benchmark_data_smithi008_109566_object19969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.988973+0000", 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 20.571562609000001, 2024-02-10T07:40:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.10317021899999999, 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 145056 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.988973+0000", 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.988973+0000", 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.988975+0000", 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.564e-06 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.988980+0000", 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7929999999999997e-06 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.988991+0000", 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0709e-05 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.988998+0000", 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9070000000000007e-06 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.989045+0000", 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6887e-05 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.989172+0000", 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000126561 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.989281+0000", 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010880299999999999 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.992061+0000", 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0027803350000000001 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.992215+0000", 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000153671 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.092030+0000", 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.099815514999999994 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.092106+0000", 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6283000000000001e-05 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.092144+0000", 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7190999999999997e-05 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:145060 8.8 8:12d9ffc4:::benchmark_data_smithi008_109566_object19973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.989576+0000", 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 20.570960230000001, 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.102391923, 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:40:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 145060 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.989576+0000", 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.989576+0000", 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.989577+0000", 2024-02-10T07:40:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2399999999999997e-07 2024-02-10T07:40:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.989580+0000", 2024-02-10T07:40:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.258e-06 2024-02-10T07:40:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.989588+0000", 2024-02-10T07:40:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3899999999999993e-06 2024-02-10T07:40:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.989595+0000", 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3520000000000003e-06 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.989632+0000", 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7518000000000002e-05 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.989730+0000", 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.8243000000000001e-05 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:40:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.989835+0000", 2024-02-10T07:40:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000105078 2024-02-10T07:40:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.992638+0000", 2024-02-10T07:40:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0028025239999999998 2024-02-10T07:40:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.993109+0000", 2024-02-10T07:40:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00047108399999999998 2024-02-10T07:40:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.091880+0000", 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.098770547 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.091928+0000", 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8930000000000001e-05 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.091968+0000", 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9274999999999999e-05 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop_reply(client.15087.0:145042 8.d e232/200 ondisk, result = 0)", 2024-02-10T07:40:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.999751+0000", 2024-02-10T07:40:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 20.560785398, 2024-02-10T07:40:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.092004184000000003, 2024-02-10T07:40:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:40:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.999751+0000", 2024-02-10T07:40:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.999751+0000", 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.999751+0000", 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8299999999999997e-07 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.999755+0000", 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4510000000000001e-06 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.999763+0000", 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1750000000000005e-06 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.999768+0000", 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8450000000000002e-06 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.999797+0000", 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9045000000000001e-05 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.999801+0000", 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4340000000000002e-06 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.091755+0000", 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.091953650999999997 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop_reply(client.15087.0:145056 8.d e232/200 ondisk, result = 0)", 2024-02-10T07:40:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:14.000534+0000", 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 20.560001566, 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.091613778000000007, 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.000534+0000", 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.000534+0000", 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.000535+0000", 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0499999999999999e-07 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.000537+0000", 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1280000000000002e-06 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.091932+0000", 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.091394656000000005 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.091943+0000", 2024-02-10T07:40:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1005000000000001e-05 2024-02-10T07:40:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.092018+0000", 2024-02-10T07:40:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5426000000000006e-05 2024-02-10T07:40:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.092025+0000", 2024-02-10T07:40:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.092e-06 2024-02-10T07:40:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:14.092148+0000", 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012316599999999999 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:164414 8.8 8:17262c4a:::benchmark_data_smithi008_109566_object39327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:31.006540+0000", 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 3.5539959290000001, 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.101550663, 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 164414 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.006540+0000", 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.006540+0000", 2024-02-10T07:40:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.006541+0000", 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.071e-06 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.006546+0000", 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4830000000000001e-06 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.006556+0000", 2024-02-10T07:40:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.876e-06 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.006563+0000", 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0520000000000004e-06 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.006607+0000", 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3958999999999997e-05 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.006713+0000", 2024-02-10T07:40:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010627 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.006823+0000", 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000110142 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.009965+0000", 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0031425160000000001 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.010113+0000", 2024-02-10T07:40:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000147007 2024-02-10T07:40:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.107663+0000", 2024-02-10T07:40:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.097550725000000005 2024-02-10T07:40:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.108042+0000", 2024-02-10T07:40:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00037917400000000003 2024-02-10T07:40:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.108091+0000", 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8387999999999998e-05 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:164768 8.4 8:2e0c2498:::benchmark_data_smithi008_109566_object39681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:31.363731+0000", 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 3.1968048420000001, 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.091115821, 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:40:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 164768 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.363731+0000", 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.363731+0000", 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:40:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.363737+0000", 2024-02-10T07:40:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6579999999999999e-06 2024-02-10T07:40:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:40:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.363743+0000", 2024-02-10T07:40:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3149999999999997e-06 2024-02-10T07:40:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:40:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.363755+0000", 2024-02-10T07:40:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1969e-05 2024-02-10T07:40:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.363762+0000", 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2729999999999997e-06 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.363807+0000", 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4626000000000003e-05 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.363894+0000", 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7267000000000003e-05 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.363967+0000", 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3165999999999997e-05 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.446637+0000", 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.082669346000000005 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.446677+0000", 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0550000000000003e-05 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.454760+0000", 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0080824060000000003 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.454808+0000", 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8436999999999997e-05 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.454847+0000", 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8807999999999998e-05 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:40:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:40:34.714 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:40:34.764 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:34.789 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:34.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:34.891 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:34.934 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:34.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:34.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:35.056 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:35.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:35.141 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:35.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:35.242 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:35.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:35.268 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:35.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:35 smithi008 ceph-mon[40743]: 2.2 scrub starts 2024-02-10T07:40:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:35 smithi008 ceph-mon[40743]: 2.2 scrub ok 2024-02-10T07:40:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:35 smithi008 ceph-mon[40743]: pgmap v399: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-02-10T07:40:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:35 smithi008 ceph-mon[40743]: 5.e scrub starts 2024-02-10T07:40:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:35 smithi008 ceph-mon[40743]: 5.e scrub ok 2024-02-10T07:40:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:35 smithi008 ceph-mon[40743]: 5.c scrub starts 2024-02-10T07:40:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:35 smithi008 ceph-mon[40743]: 5.c scrub ok 2024-02-10T07:40:35.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:35 smithi008 ceph-mon[29816]: 2.2 scrub starts 2024-02-10T07:40:35.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:35 smithi008 ceph-mon[29816]: 2.2 scrub ok 2024-02-10T07:40:35.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:35 smithi008 ceph-mon[29816]: pgmap v399: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-02-10T07:40:35.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:35 smithi008 ceph-mon[29816]: 5.e scrub starts 2024-02-10T07:40:35.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:35 smithi008 ceph-mon[29816]: 5.e scrub ok 2024-02-10T07:40:35.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:35 smithi008 ceph-mon[29816]: 5.c scrub starts 2024-02-10T07:40:35.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:35 smithi008 ceph-mon[29816]: 5.c scrub ok 2024-02-10T07:40:35.369 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:35 smithi170 ceph-mon[33394]: 2.2 scrub starts 2024-02-10T07:40:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:35 smithi170 ceph-mon[33394]: 2.2 scrub ok 2024-02-10T07:40:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:35 smithi170 ceph-mon[33394]: pgmap v399: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-02-10T07:40:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:35 smithi170 ceph-mon[33394]: 5.e scrub starts 2024-02-10T07:40:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:35 smithi170 ceph-mon[33394]: 5.e scrub ok 2024-02-10T07:40:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:35 smithi170 ceph-mon[33394]: 5.c scrub starts 2024-02-10T07:40:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:35 smithi170 ceph-mon[33394]: 5.c scrub ok 2024-02-10T07:40:35.408 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:35.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:35.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:35.510 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:35.549 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:35.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:35.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:35.651 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:35.661 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-10T07:40:35.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:35.684 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:35.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:35.766 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:40:35.785 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:35.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:35.844 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:35.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:35.945 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:35.983 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:35.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:36.085 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:36.128 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:36.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:36.229 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:36.272 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:36.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:36.374 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:36.394 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:36.403 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:36.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:36.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:36.463 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:36.505 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:36.550 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:36.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:36.651 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:36.682 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:36.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:36.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:36.784 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:36.816 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:36.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:36.917 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:36.963 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:36.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:36.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:37.065 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:37.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:37.107 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:37.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:37.207 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:37.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:37.252 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:37.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:37.278 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[29816]: 8.5 scrub starts 2024-02-10T07:40:37.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[29816]: 6.a scrub starts 2024-02-10T07:40:37.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[29816]: 6.a scrub ok 2024-02-10T07:40:37.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[29816]: pgmap v400: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:40:37.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[29816]: 4.f scrub starts 2024-02-10T07:40:37.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[29816]: 5.5 scrub starts 2024-02-10T07:40:37.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[29816]: 5.5 scrub ok 2024-02-10T07:40:37.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[29816]: 4.f scrub ok 2024-02-10T07:40:37.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1849285566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:40:37.279 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:40:37.279 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[40743]: 8.5 scrub starts 2024-02-10T07:40:37.279 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[40743]: 6.a scrub starts 2024-02-10T07:40:37.280 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[40743]: 6.a scrub ok 2024-02-10T07:40:37.280 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[40743]: pgmap v400: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:40:37.280 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[40743]: 4.f scrub starts 2024-02-10T07:40:37.280 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[40743]: 5.5 scrub starts 2024-02-10T07:40:37.280 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[40743]: 5.5 scrub ok 2024-02-10T07:40:37.280 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[40743]: 4.f scrub ok 2024-02-10T07:40:37.280 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1849285566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:40:37.280 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:37 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:40:37.354 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:37 smithi170 ceph-mon[33394]: 8.5 scrub starts 2024-02-10T07:40:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:37 smithi170 ceph-mon[33394]: 6.a scrub starts 2024-02-10T07:40:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:37 smithi170 ceph-mon[33394]: 6.a scrub ok 2024-02-10T07:40:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:37 smithi170 ceph-mon[33394]: pgmap v400: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:40:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:37 smithi170 ceph-mon[33394]: 4.f scrub starts 2024-02-10T07:40:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:37 smithi170 ceph-mon[33394]: 5.5 scrub starts 2024-02-10T07:40:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:37 smithi170 ceph-mon[33394]: 5.5 scrub ok 2024-02-10T07:40:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:37 smithi170 ceph-mon[33394]: 4.f scrub ok 2024-02-10T07:40:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:37 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1849285566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:40:37.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:37 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:40:37.390 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:37.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:37.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:37.492 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:37.523 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:37.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:37.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:37.624 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:37.663 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:37.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:37.764 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:37.791 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:37.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:37.824 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:40:37.892 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:37.929 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:37.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:37.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:38.031 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:38.055 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:38.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:38.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:38.157 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:38.187 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:38.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:38.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:38.289 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:38.294 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:38 smithi170 ceph-mon[33394]: 8.5 scrub ok 2024-02-10T07:40:38.294 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:38 smithi170 ceph-mon[33394]: 6.8 deep-scrub starts 2024-02-10T07:40:38.294 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:38 smithi170 ceph-mon[33394]: 6.8 deep-scrub ok 2024-02-10T07:40:38.294 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:38 smithi170 ceph-mon[33394]: 6.17 scrub starts 2024-02-10T07:40:38.294 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:38 smithi170 ceph-mon[33394]: 6.17 scrub ok 2024-02-10T07:40:38.294 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:38 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:40:38.294 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:38 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:40:38.294 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:38 smithi170 ceph-mon[33394]: osdmap e238: 8 total, 8 up, 8 in 2024-02-10T07:40:38.320 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:38.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:38.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:38.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[29816]: 8.5 scrub ok 2024-02-10T07:40:38.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[29816]: 6.8 deep-scrub starts 2024-02-10T07:40:38.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[29816]: 6.8 deep-scrub ok 2024-02-10T07:40:38.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[29816]: 6.17 scrub starts 2024-02-10T07:40:38.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[29816]: 6.17 scrub ok 2024-02-10T07:40:38.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:40:38.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:40:38.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[29816]: osdmap e238: 8 total, 8 up, 8 in 2024-02-10T07:40:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[40743]: 8.5 scrub ok 2024-02-10T07:40:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[40743]: 6.8 deep-scrub starts 2024-02-10T07:40:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[40743]: 6.8 deep-scrub ok 2024-02-10T07:40:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[40743]: 6.17 scrub starts 2024-02-10T07:40:38.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[40743]: 6.17 scrub ok 2024-02-10T07:40:38.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:40:38.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:40:38.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-mon[40743]: osdmap e238: 8 total, 8 up, 8 in 2024-02-10T07:40:38.422 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:38.460 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:38.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:38.562 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:38.588 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:38.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:38.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:38.690 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:38.724 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:38.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:38.826 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:38.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:38.851 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:38.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:38.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:38.985 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:38.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:38.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:39.042 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:40:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:40:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:40:39.087 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:39.124 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:39.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:39.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:39.226 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:39.267 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:39.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:39.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:39.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:39 smithi008 ceph-mon[40743]: 5.6 scrub starts 2024-02-10T07:40:39.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:39 smithi008 ceph-mon[40743]: pgmap v402: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 869 op/s 2024-02-10T07:40:39.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:39 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:40:39.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:39 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:40:39.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:39 smithi008 ceph-mon[40743]: osdmap e239: 8 total, 8 up, 8 in 2024-02-10T07:40:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:39 smithi008 ceph-mon[29816]: 5.6 scrub starts 2024-02-10T07:40:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:39 smithi008 ceph-mon[29816]: pgmap v402: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 869 op/s 2024-02-10T07:40:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:39 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:40:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:39 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:40:39.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:39 smithi008 ceph-mon[29816]: osdmap e239: 8 total, 8 up, 8 in 2024-02-10T07:40:39.369 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:39 smithi170 ceph-mon[33394]: 5.6 scrub starts 2024-02-10T07:40:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:39 smithi170 ceph-mon[33394]: pgmap v402: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 869 op/s 2024-02-10T07:40:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:39 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:40:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:39 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:40:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:39 smithi170 ceph-mon[33394]: osdmap e239: 8 total, 8 up, 8 in 2024-02-10T07:40:39.400 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:39.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:39.502 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:39.539 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:39.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:39.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:39.641 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:39.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:39.681 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:39.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:39.782 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:39.818 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:39.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:39.919 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:39.991 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:39.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:40.092 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:40.167 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:40.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:40.213 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:40:40.213 INFO:teuthology.orchestra.run.smithi170.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-10T07:40:40.213 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:40:40.268 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:40.300 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:40.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:40.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:40.402 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:40.441 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:40.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:40.543 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:40.566 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:40.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:40.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:40.667 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:40.702 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:40.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:40.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:40.804 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:40.842 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:40.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:40.944 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:40.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:40.983 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:40.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:41.084 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:41.093 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:40:41.121 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:41.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:41.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:41.223 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:41.287 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:41 smithi170 ceph-mon[33394]: pgmap v404: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-02-10T07:40:41.288 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:41.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:41.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:41.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:41 smithi008 ceph-mon[29816]: pgmap v404: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-02-10T07:40:41.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:41 smithi008 ceph-mon[40743]: pgmap v404: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-02-10T07:40:41.390 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:41.421 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:41.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:41.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:41.523 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:41.555 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:41.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:41.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:41.656 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:41.699 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:41.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:41.795 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:40:41.795 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:40:41.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.795 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15087.0:175820 8.c 8:3c112b4a:::benchmark_data_smithi008_109566_object50733:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-02-10T07:40:41.796 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:40:41.781829+0000", 2024-02-10T07:40:41.796 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.01228663, 2024-02-10T07:40:41.796 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.01232104, 2024-02-10T07:40:41.796 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:40:41.796 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:40:41.796 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:40:41.796 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15087", 2024-02-10T07:40:41.796 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:40:41.796 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 175820 2024-02-10T07:40:41.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.796 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:40:41.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:40:41.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.781829+0000", 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.781829+0000", 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.781831+0000", 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9769999999999999e-06 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.781837+0000", 2024-02-10T07:40:41.797 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5620000000000003e-06 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.781855+0000", 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8218000000000001e-05 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.781865+0000", 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0424e-05 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.782153+0000", 2024-02-10T07:40:41.798 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000288127 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.782284+0000", 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013040600000000001 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.782408+0000", 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012412100000000001 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.785919+0000", 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035114460000000001 2024-02-10T07:40:41.799 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15087.0:175824 8.c 8:3912c0b0:::benchmark_data_smithi008_109566_object50737:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:40:41.784501+0000", 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0096146640000000002, 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0097597670000000008, 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15087", 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 175824 2024-02-10T07:40:41.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.801 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:40:41.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:40:41.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.784501+0000", 2024-02-10T07:40:41.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:40:41.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:40:41.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.784501+0000", 2024-02-10T07:40:41.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:40:41.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:40:41.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.784502+0000", 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4100000000000001e-06 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.784508+0000", 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6400000000000002e-06 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.784525+0000", 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6898999999999999e-05 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.784533+0000", 2024-02-10T07:40:41.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3469999999999999e-06 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.786085+0000", 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015515990000000001 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.786230+0000", 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014547599999999999 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.786408+0000", 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000177985 2024-02-10T07:40:41.803 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15087.0:175826 8.c 8:3a43f764:::benchmark_data_smithi008_109566_object50739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:40:41.786278+0000", 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0078380610000000003, 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0080742109999999995, 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15087", 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 175826 2024-02-10T07:40:41.804 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.786278+0000", 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.786278+0000", 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.786279+0000", 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3099999999999996e-07 2024-02-10T07:40:41.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.786283+0000", 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5469999999999998e-06 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.786293+0000", 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6660000000000003e-06 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.786300+0000", 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1600000000000001e-06 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.790075+0000", 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0037746400000000001 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.790179+0000", 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000104706 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.790283+0000", 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010424700000000001 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.807 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:40:41.808 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.808 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.808 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.808 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop_reply(client.15087.0:175820 8.c e239/200)", 2024-02-10T07:40:41.808 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:41.809 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:40:41.787171+0000", 2024-02-10T07:40:41.809 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0069444909999999997, 2024-02-10T07:40:41.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0072818470000000001, 2024-02-10T07:40:41.809 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:40:41.809 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:40:41.809 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:40:41.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.809 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:40:41.809 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.787171+0000", 2024-02-10T07:40:41.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:40:41.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.809 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:40:41.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.787171+0000", 2024-02-10T07:40:41.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:40:41.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.810 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:40:41.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.787172+0000", 2024-02-10T07:40:41.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0100000000000004e-07 2024-02-10T07:40:41.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.810 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:40:41.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.787180+0000", 2024-02-10T07:40:41.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5799999999999992e-06 2024-02-10T07:40:41.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.787192+0000", 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1804000000000001e-05 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.787200+0000", 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.357e-06 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15087.0:175828 8.6 8:613c19fa:::benchmark_data_smithi008_109566_object50741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:40:41.789010+0000", 2024-02-10T07:40:41.811 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0051053619999999996, 2024-02-10T07:40:41.812 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0054988220000000004, 2024-02-10T07:40:41.812 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:40:41.812 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:40:41.812 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:40:41.812 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15087", 2024-02-10T07:40:41.812 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:40:41.812 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 175828 2024-02-10T07:40:41.812 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.812 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:40:41.812 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.812 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:40:41.812 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789010+0000", 2024-02-10T07:40:41.812 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:40:41.812 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789010+0000", 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789011+0000", 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1820000000000001e-06 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789016+0000", 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1579999999999998e-06 2024-02-10T07:40:41.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789029+0000", 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3064999999999999e-05 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789038+0000", 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3749999999999992e-06 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789215+0000", 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017656099999999999 2024-02-10T07:40:41.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789351+0000", 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000136098 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 6,7", 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789484+0000", 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000132946 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.792807+0000", 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033230019999999998 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.794199+0000", 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0013918660000000001 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15087.0:175829 8.e 8:7c965ac9:::benchmark_data_smithi008_109566_object50742:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:40:41.789060+0000", 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0050555870000000003, 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0055387630000000004, 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:40:41.816 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15087", 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 175829 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789060+0000", 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789060+0000", 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789061+0000", 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3000000000000001e-07 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789064+0000", 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1760000000000002e-06 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789077+0000", 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.346e-05 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.789084+0000", 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7970000000000003e-06 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.791570+0000", 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002485648 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.791695+0000", 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000124948 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.820 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:40:41.820 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.791828+0000", 2024-02-10T07:40:41.820 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000133411 2024-02-10T07:40:41.820 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.820 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:40:41.820 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.820 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.820 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.820 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop_reply(client.15087.0:175824 8.c e239/200)", 2024-02-10T07:40:41.820 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:40:41.791324+0000", 2024-02-10T07:40:41.820 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0027918130000000002, 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033467990000000001, 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.791324+0000", 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.791324+0000", 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.791325+0000", 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1900000000000001e-07 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.791333+0000", 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1629999999999996e-06 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.791345+0000", 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2007999999999999e-05 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:40:41.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:40:41.823 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:40:41.791353+0000", 2024-02-10T07:40:41.823 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0909999999999993e-06 2024-02-10T07:40:41.823 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.823 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:40:41.823 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.823 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:40:41.823 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:40:41.823 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 7 2024-02-10T07:40:41.823 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:40:41.852 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:41.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:41.954 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:41.995 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:41.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:41.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:42.097 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:42.128 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:42.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:42.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:42.230 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:42.237 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:40:42.238 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:40:42.238 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:40:42.247 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:40:42.265 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:42.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:42.287 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:40:42.323 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:40:42.362 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:40:42.367 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:42.391 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:42.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:42.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:42.411 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:40:42.419 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:40:42.419 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:40:42.419 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:40:42.419 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:40:42.419 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:40:42.419 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:40:42.419 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:40:42.419 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:40:42.493 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:42.524 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:42.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:42.625 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:42.653 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:42.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:42.754 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:42.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:42.802 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:42.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:42.849 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-10T07:40:42.903 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:42.929 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:42.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:42.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:43.030 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:43.084 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:42 smithi008 ceph-mon[29816]: pgmap v405: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 646 op/s 2024-02-10T07:40:43.084 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:42 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2474454216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:43.086 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:42 smithi008 ceph-mon[40743]: pgmap v405: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 646 op/s 2024-02-10T07:40:43.086 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:42 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2474454216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:43.105 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:43.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:43.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:43.160 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:42 smithi170 ceph-mon[33394]: pgmap v405: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 646 op/s 2024-02-10T07:40:43.160 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:42 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2474454216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:43.207 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:43.245 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:43.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:43.346 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:43.379 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:43.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:43.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:43.480 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:43.540 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:43.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:43.598 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:43.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:40:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:40:43.633 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:43.642 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:43.671 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:43.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:43.773 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:43.805 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:43.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:43.906 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:43.933 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:43.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:43.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:44.010 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:43 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:44.011 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:43 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:44.034 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:44.065 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:44.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:44.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:44.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:44.202 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:44.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:44.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:44.304 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:44.334 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:44.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:44.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:43 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:44.435 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:44.467 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:44.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:44.569 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:44.599 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:44.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:44.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:44.701 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:44.732 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:44.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:44.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:44.834 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:44.858 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:44.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:44.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:44.960 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:45.000 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:45.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:45.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:45.102 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:45.142 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:45.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:45.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:45.244 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:45.277 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:45.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:45.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:45.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:44 smithi008 ceph-mon[29816]: pgmap v406: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-02-10T07:40:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:44 smithi008 ceph-mon[40743]: pgmap v406: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-02-10T07:40:45.378 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:45.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:44 smithi170 ceph-mon[33394]: pgmap v406: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-02-10T07:40:45.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:45.406 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:45.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:45.507 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:45.543 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:40:45.550 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:45.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:45.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:45.656 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:45.680 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:45.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:45.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:45.782 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:45.824 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:45.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:45.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:45.925 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:45.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:45.962 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:45.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:46.063 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:46.096 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:46.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:46.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:46.198 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:46.227 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:46.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:46.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:46.329 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:46.330 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-10T07:40:46.330 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:40:46.331 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-02-10T07:40:46.331 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.0 reset_purged_snaps_last 2024-02-10T07:40:46.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:45 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:40:46.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:45 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:40:46.360 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:46.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:46.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:45 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:40:46.462 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:46.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:46.503 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:46.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:46.604 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:46.633 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:46.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:46.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:46.735 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:46.781 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:46.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:46.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:46.883 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:46.934 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:46.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:46.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:47.036 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:47.063 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:47.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:47.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:46 smithi008 ceph-mon[29816]: pgmap v407: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 634 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-10T07:40:47.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:46 smithi008 ceph-mon[40743]: pgmap v407: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 634 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-10T07:40:47.165 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:47.201 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:47.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:47.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:47.303 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:47.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:47.354 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:47.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:47.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:46 smithi170 ceph-mon[33394]: pgmap v407: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 634 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-10T07:40:47.454 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:47.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:47.490 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:47.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:47.591 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:47.631 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:47.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:47.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:47.733 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:47.783 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:47.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:47.885 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:47.934 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:47.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:47.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:48.041 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:48.082 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:48.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:48.147 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:47 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:48.147 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:47 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:40:48.148 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:47 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:40:48.148 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:47 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:40:48.148 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:47 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:48.148 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:47 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:48.148 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:47 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:40:48.148 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:47 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:40:48.148 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:47 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:40:48.148 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:47 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:48.192 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:48.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:48.227 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:48.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:48.328 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:48.375 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:48.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:48.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:47 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:40:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:47 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:40:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:47 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:40:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:47 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:40:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:47 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:40:48.477 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:48.509 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:48.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:48.610 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:48.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:48.650 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:48.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:48.719 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:40:48.719 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:40:48.751 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:48.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:48.792 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:48.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:48.893 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:48.924 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:48.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:49.026 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:49.053 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:48 smithi008 ceph-mon[29816]: pgmap v408: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-02-10T07:40:49.053 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:40:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:40:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:40:49.054 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:48 smithi008 ceph-mon[40743]: pgmap v408: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-02-10T07:40:49.059 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:49.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:49.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:49.161 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:49.193 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:49.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:49.294 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:49.327 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:49.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:48 smithi170 ceph-mon[33394]: pgmap v408: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-02-10T07:40:49.430 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:49.435 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:49.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:49.474 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:49.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:49.479 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-10T07:40:49.575 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:49.606 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:49.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:49.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:49.708 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:49.762 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:49.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:49.770 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:49.864 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:49.905 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:49.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:49.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:49.943 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:40:50.007 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:50.011 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:49 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/219209566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:50.011 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:49 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:50.011 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:49 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/219209566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:50.011 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:49 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:50.047 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:50.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:50.149 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:50.186 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:50.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:50.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:50.288 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:50.320 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:50.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:49 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/219209566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:49 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:40:50.422 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:50.453 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:50.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:50.555 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:50.595 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:50.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:50.697 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:50.730 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:50.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:50.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:50.832 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:50.871 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:50.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:50.972 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:51.014 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:51.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:51.115 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:51.136 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:50 smithi008 ceph-mon[29816]: pgmap v409: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 925 B/s rd, 0 B/s wr, 958 op/s 2024-02-10T07:40:51.136 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:50 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:40:51.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:50 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:40:51.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:50 smithi008 ceph-mon[29816]: osdmap e240: 8 total, 8 up, 8 in 2024-02-10T07:40:51.137 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:50 smithi008 ceph-mon[40743]: pgmap v409: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 925 B/s rd, 0 B/s wr, 958 op/s 2024-02-10T07:40:51.137 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:50 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:40:51.137 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:50 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:40:51.137 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:50 smithi008 ceph-mon[40743]: osdmap e240: 8 total, 8 up, 8 in 2024-02-10T07:40:51.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:51.159 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:51.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:51.260 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:51.292 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:51.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:51.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:50 smithi170 ceph-mon[33394]: pgmap v409: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 925 B/s rd, 0 B/s wr, 958 op/s 2024-02-10T07:40:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:50 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:40:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:50 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:40:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:50 smithi170 ceph-mon[33394]: osdmap e240: 8 total, 8 up, 8 in 2024-02-10T07:40:51.394 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:51.426 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:51.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:51.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:51.528 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:51.565 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:51.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:51.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:51.667 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:51.702 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:51.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:51.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:51.804 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:51.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:51.844 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:51.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:51.945 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:52.002 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:52.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:52.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:52.104 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:52.158 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:52.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:52.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:52.260 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:52.296 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:52.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:52.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:52.397 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:52.435 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:52.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:52.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:52.537 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:52.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:52.570 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:52.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:52.671 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:52.701 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:52.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:52.801 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:52.837 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:52.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:52.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:52.907 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:40:52.939 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:52.967 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:52.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:52.983 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:53.069 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:53.122 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:52 smithi008 ceph-mon[29816]: pgmap v411: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:40:53.122 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:52 smithi008 ceph-mon[40743]: pgmap v411: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:40:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:53.130 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:53.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:53.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:52 smithi170 ceph-mon[33394]: pgmap v411: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:40:53.231 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:53.277 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:53.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:53.377 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:53.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:53.416 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:53.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:53.517 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:53.564 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:53.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:53.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:40:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:40:53.666 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:53.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:53.718 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:53.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:53.819 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:53.858 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:53.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:53.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:53.960 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:53.992 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:53.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:53.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:54.094 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:54.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:53 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:54.134 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:54.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:54.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:54.236 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:54.274 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:54.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:54.327 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:53 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:54.327 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:53 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:40:54.372 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.1 reset_purged_snaps_last 2024-02-10T07:40:54.376 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:54.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:54.461 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:54.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:54.562 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:54.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:54.609 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:54.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:54.710 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:54.759 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:54.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:54.861 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:54.896 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:54.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:54.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:54.998 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:55.039 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:55.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:55.080 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:55.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:54 smithi008 ceph-mon[29816]: pgmap v412: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:40:55.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:54 smithi008 ceph-mon[40743]: pgmap v412: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:40:55.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:54 smithi170 ceph-mon[33394]: pgmap v412: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:40:55.141 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:55.172 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:55.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:55.273 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:55.322 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:55.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:55.423 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:55.470 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:55.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:55.571 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:55.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:55.603 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:55.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:55.704 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:55.742 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:55.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:55.844 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:55.897 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:55.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:55.998 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:56.003 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:55 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1848526644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:56.004 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:55 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1848526644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:56.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:56.038 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:56.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:56.139 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:56.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:56.173 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:56.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:56.267 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:40:56.267 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:40:56.267 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:40:56.267 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:40:56.268 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:40:56.268 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:40:56.268 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:40:56.274 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:56.282 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:40:56.295 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:40:56.301 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:56.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:56.307 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:40:56.307 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:40:56.307 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:40:56.307 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:40:56.307 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:40:56.307 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:40:56.307 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:40:56.307 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:40:56.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:55 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1848526644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:40:56.402 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:56.449 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:56.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:56.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:56.550 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:56.586 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:56.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:56.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:56.688 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:40:56.736 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:56.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:40:56.838 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:56.875 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:56.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:56.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:56.977 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:57.020 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:57.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:57.121 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:57.156 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:57.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:57.258 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:57.287 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:57.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:57.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:56 smithi008 ceph-mon[29816]: pgmap v413: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:40:57.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:56 smithi008 ceph-mon[40743]: pgmap v413: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:40:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:56 smithi170 ceph-mon[33394]: pgmap v413: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:40:57.391 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:57.431 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:57.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:57.532 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:57.571 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:57.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:57.673 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:57.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:57.718 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:57.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:57.819 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:57.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:57.856 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:57.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:57.957 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:57.980 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:57.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:58.082 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:58.119 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:58.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:58.221 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:58.257 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:58.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:58.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:58.359 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:58.391 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:58.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:58.493 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:58.526 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:58.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:58.628 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:58.657 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:58.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:58.759 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:40:58.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:58.794 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:58.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:40:58.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:40:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:58.929 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:58.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:40:59.029 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:59.063 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:59.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:59.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:40:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:40:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:40:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:40:58 smithi008 ceph-mon[29816]: pgmap v414: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:40:59.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:40:58 smithi008 ceph-mon[40743]: pgmap v414: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:40:59.165 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:59.191 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:59.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:59.293 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:40:59.328 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:59.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:40:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:40:58 smithi170 ceph-mon[33394]: pgmap v414: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:40:59.430 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:40:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:59.467 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:59.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:40:59.568 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:40:59.603 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:59.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:59.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:40:59.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:40:59.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:59.741 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:40:59.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:40:59.842 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:40:59.881 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:40:59.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:40:59.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:40:59.983 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:00.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:00.025 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:00.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:00.126 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:00.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:00.169 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:00.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:00.270 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:00.303 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:00.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:00.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:00.405 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:00.441 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:00.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:00.542 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:00.579 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:00.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:00.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:00.681 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:00.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:00.721 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:00.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:00.822 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:00.855 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:00.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:00.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:00.957 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:00.999 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:00.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:01.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:01.100 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:01.134 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:01.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:01.149 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:00 smithi008 ceph-mon[29816]: pgmap v415: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:41:01.149 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:00 smithi008 ceph-mon[40743]: pgmap v415: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:41:01.167 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:41:01.167 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-02-10T07:41:01.167 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-02-10T07:41:01.168 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:41:01.168 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.168 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127184 7.9 7:930e04c1:::benchmark_data_smithi008_82046_object12046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-10T07:41:01.168 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:55.989856+0000", 2024-02-10T07:41:01.168 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 305.17265811999999, 2024-02-10T07:41:01.168 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.25485934199999999, 2024-02-10T07:41:01.168 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.168 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.168 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:41:01.168 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:41:01.168 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:41:01.168 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127184 2024-02-10T07:41:01.169 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.169 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.169 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.169 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.169 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989856+0000", 2024-02-10T07:41:01.169 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.169 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.169 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.169 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.169 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989856+0000", 2024-02-10T07:41:01.169 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.169 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.169 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.170 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.170 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989857+0000", 2024-02-10T07:41:01.170 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3799999999999996e-07 2024-02-10T07:41:01.170 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.170 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.170 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.170 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989861+0000", 2024-02-10T07:41:01.170 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.703e-06 2024-02-10T07:41:01.170 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.170 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.170 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.170 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989870+0000", 2024-02-10T07:41:01.170 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0869999999999996e-06 2024-02-10T07:41:01.170 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989877+0000", 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.153e-06 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.015176+0000", 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.025298960999999998 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.015291+0000", 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000115123 2024-02-10T07:41:01.171 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.172 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.172 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:41:01.172 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.015667+0000", 2024-02-10T07:41:01.172 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00037586599999999998 2024-02-10T07:41:01.172 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.172 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.172 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:41:01.172 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.020630+0000", 2024-02-10T07:41:01.172 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0049631440000000001 2024-02-10T07:41:01.172 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.172 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.172 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.172 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.020900+0000", 2024-02-10T07:41:01.172 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026987000000000002 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244581+0000", 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.22368058900000001 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244640+0000", 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9453999999999997e-05 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244716+0000", 2024-02-10T07:41:01.173 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5654000000000004e-05 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127211 7.9 7:97623005:::benchmark_data_smithi008_82046_object12073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.027194+0000", 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 305.135320025, 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.217660456, 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:41:01.174 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127211 2024-02-10T07:41:01.175 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.175 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.175 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.175 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.175 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027194+0000", 2024-02-10T07:41:01.175 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.175 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.175 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.175 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.175 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027194+0000", 2024-02-10T07:41:01.175 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.175 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.175 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.175 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.176 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027195+0000", 2024-02-10T07:41:01.176 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7499999999999998e-07 2024-02-10T07:41:01.176 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.176 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.176 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.176 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027200+0000", 2024-02-10T07:41:01.176 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6829999999999999e-06 2024-02-10T07:41:01.176 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.176 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.176 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.176 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027212+0000", 2024-02-10T07:41:01.176 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1522000000000001e-05 2024-02-10T07:41:01.176 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027219+0000", 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6979999999999998e-06 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.031348+0000", 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0041287870000000001 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.031490+0000", 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014215199999999999 2024-02-10T07:41:01.177 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.031908+0000", 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00041737399999999998 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.038097+0000", 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0061895500000000003 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.038176+0000", 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8963999999999998e-05 2024-02-10T07:41:01.178 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244761+0000", 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.206584868 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244806+0000", 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5368000000000003e-05 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244855+0000", 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8514999999999999e-05 2024-02-10T07:41:01.179 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127243 7.14 7:2d50c885:::benchmark_data_smithi008_82046_object12105:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.076957+0000", 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 305.08555691499998, 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.074139102, 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127243 2024-02-10T07:41:01.180 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076957+0000", 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076957+0000", 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076958+0000", 2024-02-10T07:41:01.181 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9300000000000006e-07 2024-02-10T07:41:01.182 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.182 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.182 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.182 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076963+0000", 2024-02-10T07:41:01.182 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2520000000000001e-06 2024-02-10T07:41:01.182 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.182 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.182 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.182 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076974+0000", 2024-02-10T07:41:01.182 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0808999999999999e-05 2024-02-10T07:41:01.182 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.182 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.183 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.183 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076981+0000", 2024-02-10T07:41:01.183 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.391e-06 2024-02-10T07:41:01.183 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.183 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.183 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.183 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.077036+0000", 2024-02-10T07:41:01.183 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5529999999999999e-05 2024-02-10T07:41:01.183 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.183 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.183 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.183 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.077172+0000", 2024-02-10T07:41:01.183 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013559100000000001 2024-02-10T07:41:01.184 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.184 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.184 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,3,4,5", 2024-02-10T07:41:01.184 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.077478+0000", 2024-02-10T07:41:01.184 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00030572100000000001 2024-02-10T07:41:01.184 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.184 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.184 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.184 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.080684+0000", 2024-02-10T07:41:01.184 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032062390000000001 2024-02-10T07:41:01.184 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.184 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.184 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:41:01.184 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.081086+0000", 2024-02-10T07:41:01.185 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00040183900000000003 2024-02-10T07:41:01.185 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.185 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.185 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.185 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.081152+0000", 2024-02-10T07:41:01.185 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6185000000000004e-05 2024-02-10T07:41:01.185 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.185 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.185 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.185 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.083905+0000", 2024-02-10T07:41:01.185 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027525420000000002 2024-02-10T07:41:01.185 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.186 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.186 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.186 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.090838+0000", 2024-02-10T07:41:01.186 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0069333470000000003 2024-02-10T07:41:01.186 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.186 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.186 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.186 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.233892+0000", 2024-02-10T07:41:01.186 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.143053929 2024-02-10T07:41:01.186 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.186 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.186 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:41:01.186 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.233903+0000", 2024-02-10T07:41:01.187 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1412e-05 2024-02-10T07:41:01.187 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.187 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.187 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.187 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.151012+0000", 2024-02-10T07:41:01.187 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.91710869900000003 2024-02-10T07:41:01.187 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.187 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.187 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.187 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.151097+0000", 2024-02-10T07:41:01.187 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4623000000000006e-05 2024-02-10T07:41:01.187 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.187 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.188 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.188 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.188 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.188 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127271 7.17 7:ee10418c:::benchmark_data_smithi008_82046_object12133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:41:01.188 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.283167+0000", 2024-02-10T07:41:01.188 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 304.87934737799998, 2024-02-10T07:41:01.188 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.87381412300000005, 2024-02-10T07:41:01.188 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.188 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.188 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:41:01.188 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:41:01.188 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:41:01.189 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127271 2024-02-10T07:41:01.189 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.189 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.189 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.189 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.189 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283167+0000", 2024-02-10T07:41:01.189 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.189 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.189 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.189 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.189 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283167+0000", 2024-02-10T07:41:01.189 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.189 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.190 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.190 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.190 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283168+0000", 2024-02-10T07:41:01.190 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6199999999999996e-07 2024-02-10T07:41:01.190 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.190 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.190 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.190 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283173+0000", 2024-02-10T07:41:01.190 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6589999999999998e-06 2024-02-10T07:41:01.190 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.190 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.190 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.191 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283184+0000", 2024-02-10T07:41:01.191 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1175999999999999e-05 2024-02-10T07:41:01.191 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.191 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.191 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.191 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283192+0000", 2024-02-10T07:41:01.191 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1049999999999995e-06 2024-02-10T07:41:01.191 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.191 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.191 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.191 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283249+0000", 2024-02-10T07:41:01.191 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7491000000000002e-05 2024-02-10T07:41:01.191 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.192 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.192 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:41:01.192 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283259+0000", 2024-02-10T07:41:01.192 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2790000000000004e-06 2024-02-10T07:41:01.192 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.192 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.192 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.192 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.145677+0000", 2024-02-10T07:41:01.192 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.86241815099999997 2024-02-10T07:41:01.192 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.192 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.192 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.193 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.145793+0000", 2024-02-10T07:41:01.193 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011595 2024-02-10T07:41:01.193 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.193 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.193 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,3,6,7", 2024-02-10T07:41:01.193 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.146056+0000", 2024-02-10T07:41:01.193 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026355900000000002 2024-02-10T07:41:01.193 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.193 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.193 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.193 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.147787+0000", 2024-02-10T07:41:01.193 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0017307329999999999 2024-02-10T07:41:01.193 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.194 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.194 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.194 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.155806+0000", 2024-02-10T07:41:01.194 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0080188640000000005 2024-02-10T07:41:01.194 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.194 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.194 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.194 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.155945+0000", 2024-02-10T07:41:01.194 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013933700000000001 2024-02-10T07:41:01.194 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.194 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.194 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:41:01.195 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156537+0000", 2024-02-10T07:41:01.195 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00059228199999999996 2024-02-10T07:41:01.195 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.195 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.195 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.195 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156801+0000", 2024-02-10T07:41:01.195 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026387999999999997 2024-02-10T07:41:01.195 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.195 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.195 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.195 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156860+0000", 2024-02-10T07:41:01.196 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8900999999999997e-05 2024-02-10T07:41:01.196 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.196 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.196 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.197 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156981+0000", 2024-02-10T07:41:01.197 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000120894 2024-02-10T07:41:01.197 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.197 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.197 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.197 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.197 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.197 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127281 7.17 7:ec58e9f8:::benchmark_data_smithi008_82046_object12143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:41:01.197 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.333207+0000", 2024-02-10T07:41:01.197 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 304.82930707399998, 2024-02-10T07:41:01.197 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.82508648100000004, 2024-02-10T07:41:01.197 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.197 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.198 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:41:01.198 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:41:01.198 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:41:01.198 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127281 2024-02-10T07:41:01.198 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.198 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.198 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.198 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.198 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333207+0000", 2024-02-10T07:41:01.198 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.198 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.198 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.198 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.199 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333207+0000", 2024-02-10T07:41:01.199 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.199 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.199 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.199 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.199 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333209+0000", 2024-02-10T07:41:01.199 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.415e-06 2024-02-10T07:41:01.199 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.199 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.199 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.199 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333213+0000", 2024-02-10T07:41:01.199 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2230000000000001e-06 2024-02-10T07:41:01.199 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.200 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.200 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.200 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333224+0000", 2024-02-10T07:41:01.200 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1483e-05 2024-02-10T07:41:01.200 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.200 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.200 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.200 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333232+0000", 2024-02-10T07:41:01.200 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.4970000000000004e-06 2024-02-10T07:41:01.200 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.200 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.200 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333278+0000", 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6362000000000001e-05 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333287+0000", 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6230000000000003e-06 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.147857+0000", 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.81456973200000005 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.201 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.147984+0000", 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012703200000000001 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,3,6,7", 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.148252+0000", 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000268278 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.155857+0000", 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0076047550000000004 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.202 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.155901+0000", 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4283000000000003e-05 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156520+0000", 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00061886199999999997 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156552+0000", 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1786999999999999e-05 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.158034+0000", 2024-02-10T07:41:01.203 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001482141 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.158140+0000", 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010587 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.158294+0000", 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015413800000000001 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.204 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24356.0:1373 5.3 5:c609908c:::notify.5:head [watch ping cookie 93827140438272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.411579+0000", 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 304.75093528299999, 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.71068318600000002, 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24356", 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1933562162", 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 1373 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411579+0000", 2024-02-10T07:41:01.205 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411579+0000", 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411581+0000", 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4839999999999999e-06 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411585+0000", 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8799999999999999e-06 2024-02-10T07:41:01.206 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411598+0000", 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2388999999999999e-05 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411606+0000", 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4659999999999999e-06 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411704+0000", 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8095000000000002e-05 2024-02-10T07:41:01.207 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.208 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.208 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:41:01.208 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411714+0000", 2024-02-10T07:41:01.208 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0044000000000001e-05 2024-02-10T07:41:01.208 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.208 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.208 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.208 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.115532+0000", 2024-02-10T07:41:01.208 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.70381718199999999 2024-02-10T07:41:01.208 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.208 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.208 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.208 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.115689+0000", 2024-02-10T07:41:01.208 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015699900000000001 2024-02-10T07:41:01.209 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.209 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.209 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.209 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.122262+0000", 2024-02-10T07:41:01.209 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0065736470000000002 2024-02-10T07:41:01.209 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.209 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.209 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.209 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.209 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.209 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24356.0:1383 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 93827140438272 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:41:01.209 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.412062+0000", 2024-02-10T07:41:01.209 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 304.75045225000002, 2024-02-10T07:41:01.210 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.725469484, 2024-02-10T07:41:01.210 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.210 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:41:01.210 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:41:01.210 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24356", 2024-02-10T07:41:01.210 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1933562162", 2024-02-10T07:41:01.210 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 1383 2024-02-10T07:41:01.210 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.210 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.210 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.210 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.210 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412062+0000", 2024-02-10T07:41:01.210 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.211 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.211 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.211 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.211 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412062+0000", 2024-02-10T07:41:01.211 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.211 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.211 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.211 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.211 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412063+0000", 2024-02-10T07:41:01.211 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7700000000000004e-07 2024-02-10T07:41:01.211 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.211 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.211 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.212 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412067+0000", 2024-02-10T07:41:01.212 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9330000000000003e-06 2024-02-10T07:41:01.212 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.212 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.212 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.212 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412081+0000", 2024-02-10T07:41:01.212 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3991e-05 2024-02-10T07:41:01.212 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.212 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.212 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.212 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412088+0000", 2024-02-10T07:41:01.212 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6459999999999993e-06 2024-02-10T07:41:01.213 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.213 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.213 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.213 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.498465+0000", 2024-02-10T07:41:01.213 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.086376877000000005 2024-02-10T07:41:01.213 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.213 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.213 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:41:01.213 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.498476+0000", 2024-02-10T07:41:01.213 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0975000000000001e-05 2024-02-10T07:41:01.213 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.213 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.214 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.214 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.122295+0000", 2024-02-10T07:41:01.214 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.62381883100000002 2024-02-10T07:41:01.214 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.214 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.214 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.214 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.122389+0000", 2024-02-10T07:41:01.214 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3941000000000001e-05 2024-02-10T07:41:01.214 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.214 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.214 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.214 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.137532+0000", 2024-02-10T07:41:01.215 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015142513 2024-02-10T07:41:01.215 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.215 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.215 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.215 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.215 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.215 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:131746 7.9 7:97040099:::benchmark_data_smithi008_82046_object16608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:41:01.215 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:05.798493+0000", 2024-02-10T07:41:01.215 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 295.364020977, 2024-02-10T07:41:01.215 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.26161331199999999, 2024-02-10T07:41:01.215 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.215 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.215 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:41:01.216 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:41:01.216 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:41:01.216 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 131746 2024-02-10T07:41:01.216 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.216 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.216 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.216 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.216 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798493+0000", 2024-02-10T07:41:01.216 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.216 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.216 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.216 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.217 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798493+0000", 2024-02-10T07:41:01.217 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.217 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.217 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.217 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.217 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798495+0000", 2024-02-10T07:41:01.217 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2240000000000001e-06 2024-02-10T07:41:01.217 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.217 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.217 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.217 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798499+0000", 2024-02-10T07:41:01.217 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2359999999999998e-06 2024-02-10T07:41:01.217 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.218 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.218 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.218 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798518+0000", 2024-02-10T07:41:01.218 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9519000000000001e-05 2024-02-10T07:41:01.218 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.218 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.218 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.218 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798530+0000", 2024-02-10T07:41:01.218 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.184e-05 2024-02-10T07:41:01.218 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.218 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.218 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.219 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.830695+0000", 2024-02-10T07:41:01.219 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.032164618999999998 2024-02-10T07:41:01.219 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.219 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.219 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.219 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.830907+0000", 2024-02-10T07:41:01.219 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00021249600000000001 2024-02-10T07:41:01.219 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.219 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.219 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:41:01.219 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.831383+0000", 2024-02-10T07:41:01.219 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000476016 2024-02-10T07:41:01.220 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.220 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.220 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:41:01.220 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.834488+0000", 2024-02-10T07:41:01.220 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0031048759999999999 2024-02-10T07:41:01.220 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.220 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.220 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.220 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.834673+0000", 2024-02-10T07:41:01.220 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00018433600000000001 2024-02-10T07:41:01.220 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.220 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.221 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.221 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.059995+0000", 2024-02-10T07:41:01.221 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.22532233199999999 2024-02-10T07:41:01.221 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.221 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.221 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.221 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060046+0000", 2024-02-10T07:41:01.221 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1076999999999999e-05 2024-02-10T07:41:01.221 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.221 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.221 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.221 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060107+0000", 2024-02-10T07:41:01.222 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0741e-05 2024-02-10T07:41:01.222 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.222 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.222 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.222 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.222 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.222 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:131775 7.19 7:9fe66b4c:::benchmark_data_smithi008_82046_object16637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:41:01.222 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:05.844733+0000", 2024-02-10T07:41:01.222 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 295.31778125, 2024-02-10T07:41:01.222 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.216159669, 2024-02-10T07:41:01.222 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.222 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.222 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:41:01.222 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:41:01.223 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:41:01.223 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 131775 2024-02-10T07:41:01.223 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.223 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.223 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.223 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.223 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844733+0000", 2024-02-10T07:41:01.223 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.223 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.223 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.223 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.223 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844733+0000", 2024-02-10T07:41:01.224 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.224 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.224 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.224 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.224 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844734+0000", 2024-02-10T07:41:01.224 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3340000000000001e-06 2024-02-10T07:41:01.224 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.224 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.224 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.225 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844739+0000", 2024-02-10T07:41:01.225 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.065e-06 2024-02-10T07:41:01.225 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.225 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.225 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.225 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844748+0000", 2024-02-10T07:41:01.225 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0960000000000007e-06 2024-02-10T07:41:01.225 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.225 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.225 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.225 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844754+0000", 2024-02-10T07:41:01.225 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6900000000000003e-06 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844936+0000", 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00018217400000000001 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.845059+0000", 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000122815 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.845348+0000", 2024-02-10T07:41:01.226 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00028826999999999998 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.856837+0000", 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011489256999999999 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.856879+0000", 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2268000000000003e-05 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.856919+0000", 2024-02-10T07:41:01.227 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9681000000000002e-05 2024-02-10T07:41:01.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.228 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.856958+0000", 2024-02-10T07:41:01.228 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9623999999999999e-05 2024-02-10T07:41:01.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:41:01.228 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.860314+0000", 2024-02-10T07:41:01.228 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033553839999999999 2024-02-10T07:41:01.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.228 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060424+0000", 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.20010988099999999 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060484+0000", 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0136000000000002e-05 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060893+0000", 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00040899399999999998 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:131874 7.9 7:90ce5fa2:::benchmark_data_smithi008_82046_object16736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:06.060485+0000", 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 295.10202952600002, 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.22169967700000001, 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 131874 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.230 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060485+0000", 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060485+0000", 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060486+0000", 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7999999999999993e-07 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.231 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060491+0000", 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.814e-06 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060503+0000", 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1915e-05 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060511+0000", 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6479999999999992e-06 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.082347+0000", 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.021835391999999999 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.082588+0000", 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00024096600000000001 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.083045+0000", 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00045718000000000001 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.233 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.091368+0000", 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0083230149999999996 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.091406+0000", 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8444e-05 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.282075+0000", 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.19066877500000001 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.282128+0000", 2024-02-10T07:41:01.234 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3090000000000002e-05 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.282185+0000", 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6458000000000001e-05 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:132742 7.19 7:991f4e26:::benchmark_data_smithi008_82046_object17604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:07.446202+0000", 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 293.71631266200001, 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.25343312400000001, 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.235 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.236 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:41:01.236 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:41:01.236 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:41:01.236 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 132742 2024-02-10T07:41:01.236 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.236 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.236 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.236 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446202+0000", 2024-02-10T07:41:01.236 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.236 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.236 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.236 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446202+0000", 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446203+0000", 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.159e-06 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446207+0000", 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.738e-06 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.237 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446217+0000", 2024-02-10T07:41:01.238 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0518e-05 2024-02-10T07:41:01.238 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.238 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.238 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.238 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446225+0000", 2024-02-10T07:41:01.238 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3760000000000002e-06 2024-02-10T07:41:01.238 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:01.240 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.240 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.240 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446280+0000", 2024-02-10T07:41:01.240 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5883e-05 2024-02-10T07:41:01.240 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.240 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.240 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446423+0000", 2024-02-10T07:41:01.240 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014267300000000001 2024-02-10T07:41:01.241 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.241 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.241 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-02-10T07:41:01.241 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446714+0000", 2024-02-10T07:41:01.241 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00029046200000000001 2024-02-10T07:41:01.241 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.241 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.241 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.241 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450612+0000", 2024-02-10T07:41:01.241 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0038986860000000002 2024-02-10T07:41:01.241 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.241 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:41:01.242 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450626+0000", 2024-02-10T07:41:01.242 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4151999999999999e-05 2024-02-10T07:41:01.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.242 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450672+0000", 2024-02-10T07:41:01.242 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6119999999999999e-05 2024-02-10T07:41:01.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.242 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450720+0000", 2024-02-10T07:41:01.243 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7803999999999998e-05 2024-02-10T07:41:01.243 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.243 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.243 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450765+0000", 2024-02-10T07:41:01.243 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4594e-05 2024-02-10T07:41:01.243 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.243 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.243 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699383+0000", 2024-02-10T07:41:01.243 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24861809200000001 2024-02-10T07:41:01.243 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.244 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.244 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699507+0000", 2024-02-10T07:41:01.244 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000124156 2024-02-10T07:41:01.244 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.244 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.244 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.244 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699635+0000", 2024-02-10T07:41:01.244 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000127711 2024-02-10T07:41:01.244 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.244 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.244 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.244 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.245 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.245 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:132743 7.19 7:9cfb864f:::benchmark_data_smithi008_82046_object17605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:41:01.245 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:07.446450+0000", 2024-02-10T07:41:01.245 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 293.71606435000001, 2024-02-10T07:41:01.245 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.25339128999999999, 2024-02-10T07:41:01.245 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.245 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.245 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:41:01.245 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:41:01.245 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:41:01.245 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 132743 2024-02-10T07:41:01.245 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.246 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.246 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.246 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446450+0000", 2024-02-10T07:41:01.246 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.246 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.246 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.246 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446450+0000", 2024-02-10T07:41:01.246 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.246 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.246 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.247 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446451+0000", 2024-02-10T07:41:01.247 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8599999999999997e-07 2024-02-10T07:41:01.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.247 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.247 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446455+0000", 2024-02-10T07:41:01.247 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.5839999999999999e-06 2024-02-10T07:41:01.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.247 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.247 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446466+0000", 2024-02-10T07:41:01.247 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1242e-05 2024-02-10T07:41:01.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.248 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446474+0000", 2024-02-10T07:41:01.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2309999999999996e-06 2024-02-10T07:41:01.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.248 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.448986+0000", 2024-02-10T07:41:01.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0025125059999999999 2024-02-10T07:41:01.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.248 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.249 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.449131+0000", 2024-02-10T07:41:01.249 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014447200000000001 2024-02-10T07:41:01.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.249 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-02-10T07:41:01.249 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.449392+0000", 2024-02-10T07:41:01.249 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026119499999999998 2024-02-10T07:41:01.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.249 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.249 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450809+0000", 2024-02-10T07:41:01.249 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0014168469999999999 2024-02-10T07:41:01.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:41:01.250 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450905+0000", 2024-02-10T07:41:01.250 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6237999999999996e-05 2024-02-10T07:41:01.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.250 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450997+0000", 2024-02-10T07:41:01.250 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1774999999999995e-05 2024-02-10T07:41:01.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.251 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.452168+0000", 2024-02-10T07:41:01.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00117117 2024-02-10T07:41:01.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.251 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.452212+0000", 2024-02-10T07:41:01.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3548999999999999e-05 2024-02-10T07:41:01.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.251 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699673+0000", 2024-02-10T07:41:01.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24746167399999999 2024-02-10T07:41:01.252 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.252 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.252 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.252 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699723+0000", 2024-02-10T07:41:01.252 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9421e-05 2024-02-10T07:41:01.252 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.252 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.252 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.252 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699841+0000", 2024-02-10T07:41:01.252 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011849999999999999 2024-02-10T07:41:01.252 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.252 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.253 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.253 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154815 7.14 7:28f3695e:::benchmark_data_smithi008_82046_object39677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:41:01.253 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.856040+0000", 2024-02-10T07:41:01.253 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 268.30647473499999, 2024-02-10T07:41:01.253 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96816402099999999, 2024-02-10T07:41:01.253 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.253 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.253 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:41:01.253 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:41:01.253 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:41:01.253 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154815 2024-02-10T07:41:01.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.254 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856040+0000", 2024-02-10T07:41:01.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856040+0000", 2024-02-10T07:41:01.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.255 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.255 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856042+0000", 2024-02-10T07:41:01.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8700000000000001e-06 2024-02-10T07:41:01.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.255 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.255 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.255 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856046+0000", 2024-02-10T07:41:01.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9649999999999999e-06 2024-02-10T07:41:01.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.255 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.255 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.256 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856063+0000", 2024-02-10T07:41:01.256 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6838999999999999e-05 2024-02-10T07:41:01.256 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.256 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.256 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.256 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856083+0000", 2024-02-10T07:41:01.256 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9191e-05 2024-02-10T07:41:01.256 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.256 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.256 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.256 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856152+0000", 2024-02-10T07:41:01.256 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9622999999999994e-05 2024-02-10T07:41:01.256 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.256 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856165+0000", 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2604e-05 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.820015+0000", 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.963850031 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.820144+0000", 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012971799999999999 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.257 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,3,4,5", 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.820489+0000", 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00034403500000000002 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.822408+0000", 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001919624 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.822852+0000", 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00044381899999999997 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.258 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.822915+0000", 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.279e-05 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.822955+0000", 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0460999999999998e-05 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.823010+0000", 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4537999999999999e-05 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.259 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.823994+0000", 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00098379999999999995 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.824060+0000", 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6548e-05 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.824204+0000", 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014356500000000001 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.260 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:212345 7.f 7:f80fa090:::benchmark_data_smithi008_82046_object97207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.578926+0000", 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 203.58358802199999, 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24282195200000001, 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 212345 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.261 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578926+0000", 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578926+0000", 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578928+0000", 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.486e-06 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.262 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578932+0000", 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2699999999999998e-06 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578945+0000", 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2703e-05 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578954+0000", 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8710000000000003e-06 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.579336+0000", 2024-02-10T07:41:01.263 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00038255500000000003 2024-02-10T07:41:01.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.264 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.264 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.264 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.579461+0000", 2024-02-10T07:41:01.264 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000124621 2024-02-10T07:41:01.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.264 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.264 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:41:01.264 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.579564+0000", 2024-02-10T07:41:01.264 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000103317 2024-02-10T07:41:01.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.264 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.264 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:41:01.264 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.582702+0000", 2024-02-10T07:41:01.265 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0031373400000000002 2024-02-10T07:41:01.265 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.265 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.265 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.265 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583326+0000", 2024-02-10T07:41:01.265 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00062478000000000002 2024-02-10T07:41:01.265 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.265 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.265 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:41:01.265 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821642+0000", 2024-02-10T07:41:01.265 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.238315844 2024-02-10T07:41:01.266 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.266 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.266 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821699+0000", 2024-02-10T07:41:01.266 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6635000000000002e-05 2024-02-10T07:41:01.266 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.266 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.266 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821748+0000", 2024-02-10T07:41:01.266 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9530000000000002e-05 2024-02-10T07:41:01.266 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.266 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.266 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.266 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.267 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.267 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212347 7.12 e187/185 7:4b6dc533:::benchmark_data_smithi008_82046_object97209:head v 187'10763, mlcod=187'10761)", 2024-02-10T07:41:01.267 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.583166+0000", 2024-02-10T07:41:01.267 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 203.579348318, 2024-02-10T07:41:01.267 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24110515900000001, 2024-02-10T07:41:01.267 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.267 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.267 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.267 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.267 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.267 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583166+0000", 2024-02-10T07:41:01.267 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.267 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.268 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.268 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.268 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583166+0000", 2024-02-10T07:41:01.268 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.268 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.268 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.268 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.268 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583167+0000", 2024-02-10T07:41:01.268 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1299999999999999e-07 2024-02-10T07:41:01.268 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.268 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.268 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.269 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583172+0000", 2024-02-10T07:41:01.269 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6740000000000002e-06 2024-02-10T07:41:01.269 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.269 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.269 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.269 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583182+0000", 2024-02-10T07:41:01.269 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8130000000000007e-06 2024-02-10T07:41:01.269 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.269 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.269 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.269 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583189+0000", 2024-02-10T07:41:01.269 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9140000000000002e-06 2024-02-10T07:41:01.269 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.270 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.270 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583624+0000", 2024-02-10T07:41:01.270 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00043515399999999999 2024-02-10T07:41:01.270 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.270 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.270 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583646+0000", 2024-02-10T07:41:01.270 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1474000000000001e-05 2024-02-10T07:41:01.270 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.270 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.271 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.824236+0000", 2024-02-10T07:41:01.271 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24059001699999999 2024-02-10T07:41:01.271 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.271 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.271 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.824271+0000", 2024-02-10T07:41:01.271 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.54e-05 2024-02-10T07:41:01.271 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.271 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.271 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.271 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.271 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212349 7.e e187/166 7:7df8d3ee:::benchmark_data_smithi008_82046_object97211:head v 187'13215, mlcod=187'13213)", 2024-02-10T07:41:01.272 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.583663+0000", 2024-02-10T07:41:01.272 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 203.578851047, 2024-02-10T07:41:01.272 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23797284099999999, 2024-02-10T07:41:01.272 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.272 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.272 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.272 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.272 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.272 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583663+0000", 2024-02-10T07:41:01.272 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.272 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.272 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.272 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.273 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583663+0000", 2024-02-10T07:41:01.273 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.273 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.273 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.273 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.273 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583664+0000", 2024-02-10T07:41:01.273 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3900000000000004e-07 2024-02-10T07:41:01.273 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.273 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.273 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.273 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583670+0000", 2024-02-10T07:41:01.273 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7790000000000001e-06 2024-02-10T07:41:01.274 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.274 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.274 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583680+0000", 2024-02-10T07:41:01.274 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7419999999999992e-06 2024-02-10T07:41:01.274 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.274 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.274 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583686+0000", 2024-02-10T07:41:01.274 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5740000000000001e-06 2024-02-10T07:41:01.274 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.274 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.275 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.584048+0000", 2024-02-10T07:41:01.275 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00036165999999999998 2024-02-10T07:41:01.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.275 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.275 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.275 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.584076+0000", 2024-02-10T07:41:01.275 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8121000000000001e-05 2024-02-10T07:41:01.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.275 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.275 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.275 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821584+0000", 2024-02-10T07:41:01.275 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.237508318 2024-02-10T07:41:01.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.276 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.276 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.276 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821636+0000", 2024-02-10T07:41:01.276 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1907999999999998e-05 2024-02-10T07:41:01.276 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.276 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.276 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.276 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.277 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.277 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop_reply(client.24740.0:212345 7.f e187/98 ondisk, result = 0)", 2024-02-10T07:41:01.277 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.585629+0000", 2024-02-10T07:41:01.277 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 203.57688571099999, 2024-02-10T07:41:01.277 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23612559999999999, 2024-02-10T07:41:01.277 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.277 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:41:01.277 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.277 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.277 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.277 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585629+0000", 2024-02-10T07:41:01.277 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.278 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.278 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.278 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.278 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585629+0000", 2024-02-10T07:41:01.278 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.278 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.278 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.278 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.278 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585630+0000", 2024-02-10T07:41:01.278 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0999999999999997e-07 2024-02-10T07:41:01.278 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.278 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.278 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.279 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585635+0000", 2024-02-10T07:41:01.279 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1050000000000003e-06 2024-02-10T07:41:01.279 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.279 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.279 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.279 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585645+0000", 2024-02-10T07:41:01.279 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0484e-05 2024-02-10T07:41:01.279 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.279 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.279 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.279 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585652+0000", 2024-02-10T07:41:01.279 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2570000000000001e-06 2024-02-10T07:41:01.279 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.280 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.280 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.280 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821625+0000", 2024-02-10T07:41:01.280 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23597299399999999 2024-02-10T07:41:01.280 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.280 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.280 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.280 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821635+0000", 2024-02-10T07:41:01.280 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9240000000000002e-06 2024-02-10T07:41:01.280 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.280 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.280 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.281 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821754+0000", 2024-02-10T07:41:01.281 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011892600000000001 2024-02-10T07:41:01.281 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.281 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.281 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.281 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.281 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.281 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDPGPushReply(7.14 187/156 [PushReplyOp(7:2f8c8aed:::benchmark_data_smithi008_82046_object99852:head)])", 2024-02-10T07:41:01.281 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:41:01.281 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 203.57432426899999, 2024-02-10T07:41:01.281 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23338334399999999, 2024-02-10T07:41:01.281 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.281 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:41:01.282 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2499999999999999e-07 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588193+0000", 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.881e-06 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588200+0000", 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.703e-06 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588206+0000", 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4869999999999999e-06 2024-02-10T07:41:01.283 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.284 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.284 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.284 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588865+0000", 2024-02-10T07:41:01.285 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00065820199999999998 2024-02-10T07:41:01.285 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.285 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.285 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.285 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821573+0000", 2024-02-10T07:41:01.285 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.232708846 2024-02-10T07:41:01.285 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.285 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.285 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.285 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.285 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212353 7.12 e187/185 7:4af281cd:::benchmark_data_smithi008_82046_object97215:head v 187'10764, mlcod=187'10761)", 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.589498+0000", 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 203.57301653299999, 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23719131800000001, 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589498+0000", 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.286 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589498+0000", 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589499+0000", 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8500000000000001e-07 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589505+0000", 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4060000000000002e-06 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821581+0000", 2024-02-10T07:41:01.287 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23207629699999999 2024-02-10T07:41:01.288 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.288 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.288 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.288 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821593+0000", 2024-02-10T07:41:01.288 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1387e-05 2024-02-10T07:41:01.288 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.288 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.288 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.288 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821991+0000", 2024-02-10T07:41:01.288 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00039868400000000001 2024-02-10T07:41:01.288 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.288 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.288 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.288 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.822013+0000", 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1191000000000001e-05 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.826650+0000", 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046371650000000004 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.826689+0000", 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9502999999999998e-05 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.289 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.290 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212363 7.5 e187/135 7:a6287109:::benchmark_data_smithi008_82046_object97225:head v 187'10252, mlcod=187'10250)", 2024-02-10T07:41:01.290 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.594086+0000", 2024-02-10T07:41:01.290 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 203.56842876100001, 2024-02-10T07:41:01.290 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23397765000000001, 2024-02-10T07:41:01.290 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:41:01.290 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:41:01.290 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:41:01.290 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.290 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:41:01.290 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594086+0000", 2024-02-10T07:41:01.290 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.290 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.290 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.290 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594086+0000", 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594087+0000", 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.595e-06 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594097+0000", 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4560000000000006e-06 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.291 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:41:01.292 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821570+0000", 2024-02-10T07:41:01.292 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.22747376999999999 2024-02-10T07:41:01.292 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.292 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.292 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:41:01.292 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821582+0000", 2024-02-10T07:41:01.292 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1093e-05 2024-02-10T07:41:01.292 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.292 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.292 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:41:01.292 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821684+0000", 2024-02-10T07:41:01.292 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010225699999999999 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821708+0000", 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4641e-05 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.828021+0000", 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0063127080000000002 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.828063+0000", 2024-02-10T07:41:01.293 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2129999999999998e-05 2024-02-10T07:41:01.294 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.294 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.294 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.294 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:41:01.294 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:41:01.294 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:41:01.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:01.295 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:01.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:01.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:00 smithi170 ceph-mon[33394]: pgmap v415: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:41:01.396 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:01.431 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:01.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:01.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:01.533 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:01.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:01.569 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:01.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:01.670 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:01.708 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:01.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:01.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:01.810 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:01.849 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:01.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:01.951 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:01.992 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:01.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:02.094 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:02.128 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:02.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:02.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:02.230 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:02.267 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:02.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:02.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:02.274 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:41:02.369 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:02.402 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:02.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:02.504 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:02.555 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:02.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:02.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:02.658 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:02.690 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:02.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:02.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:02.792 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:02.833 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:02.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:02.934 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:03.027 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:03.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:03.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:02 smithi008 ceph-mon[29816]: pgmap v416: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 855 B/s rd, 0 B/s wr, 955 op/s 2024-02-10T07:41:03.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:02 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:03.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:02 smithi008 ceph-mon[40743]: pgmap v416: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 855 B/s rd, 0 B/s wr, 955 op/s 2024-02-10T07:41:03.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:02 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:03.128 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:02 smithi170 ceph-mon[33394]: pgmap v416: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 855 B/s rd, 0 B/s wr, 955 op/s 2024-02-10T07:41:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:02 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:03.150 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-10T07:41:03.158 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:03.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:03.260 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:03.323 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:03.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:03.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:03.424 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:03.473 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:03.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:03.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:03.575 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:03.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:03.619 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:03.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:03.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:41:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:41:03.720 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:03.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:03.771 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:03.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:03.872 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:03.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:03.909 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:03.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:03.955 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:03 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:03.955 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:03 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:04.010 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:04.075 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:04.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:04.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:04.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:03 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:04.177 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:04.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:04.213 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:04.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:04.314 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:04.351 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:04.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:04.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:04.453 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:04.494 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:04.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:04.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:04.596 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:04.637 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:04.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:04.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:04.738 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:04.778 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:04.835 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:04.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:04.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:04.937 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:04.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:04.993 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:04.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:05.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:04 smithi008 ceph-mon[29816]: pgmap v417: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-02-10T07:41:05.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:04 smithi008 ceph-mon[40743]: pgmap v417: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-02-10T07:41:05.095 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:05.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:05.133 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:05.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:05.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:04 smithi170 ceph-mon[33394]: pgmap v417: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-02-10T07:41:05.234 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:05.265 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:05.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:05.368 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:05.403 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:05.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:05.504 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:05.546 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:05.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:05.648 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:05.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:05.759 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:05.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:05.860 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:05.864 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:41:05.908 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:05.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:05.965 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.2 reset_purged_snaps_last 2024-02-10T07:41:06.010 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:06.127 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:06.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:05 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2970289436' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:41:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:05 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:41:06.229 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:06.256 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:05 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2970289436' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:41:06.256 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:05 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:41:06.257 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:05 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2970289436' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:41:06.257 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:05 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:41:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:06.282 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:06.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:06.384 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:06.422 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:06.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:06.526 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:06.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:06.562 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:06.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:06.664 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:06.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:06.703 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:06.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:06.804 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:06.844 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:06.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:06.945 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:06.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:06.981 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:06.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:07.083 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:07.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:07.117 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:07.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:07.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:06 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:41:07.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:06 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:41:07.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:06 smithi170 ceph-mon[33394]: osdmap e241: 8 total, 8 up, 8 in 2024-02-10T07:41:07.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:06 smithi170 ceph-mon[33394]: pgmap v419: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:41:07.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:06 smithi170 ceph-mon[33394]: 4.4 scrub starts 2024-02-10T07:41:07.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:06 smithi170 ceph-mon[33394]: 4.4 scrub ok 2024-02-10T07:41:07.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:06 smithi170 ceph-mon[33394]: 5.6 scrub starts 2024-02-10T07:41:07.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:06 smithi170 ceph-mon[33394]: 5.6 scrub ok 2024-02-10T07:41:07.209 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:41:07.210 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:41:07.210 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[29816]: osdmap e241: 8 total, 8 up, 8 in 2024-02-10T07:41:07.210 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[29816]: pgmap v419: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:41:07.210 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[29816]: 4.4 scrub starts 2024-02-10T07:41:07.210 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[29816]: 4.4 scrub ok 2024-02-10T07:41:07.210 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[29816]: 5.6 scrub starts 2024-02-10T07:41:07.211 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[29816]: 5.6 scrub ok 2024-02-10T07:41:07.211 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:41:07.211 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:41:07.211 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[40743]: osdmap e241: 8 total, 8 up, 8 in 2024-02-10T07:41:07.212 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[40743]: pgmap v419: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:41:07.212 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[40743]: 4.4 scrub starts 2024-02-10T07:41:07.212 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[40743]: 4.4 scrub ok 2024-02-10T07:41:07.212 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[40743]: 5.6 scrub starts 2024-02-10T07:41:07.212 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:06 smithi008 ceph-mon[40743]: 5.6 scrub ok 2024-02-10T07:41:07.219 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:07.267 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:07.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:07.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:07.368 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:07.413 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:07.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:07.514 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:07.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:07.549 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:07.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:07.650 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:07.675 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:07.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:07.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:07.778 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:07.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:07.821 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:07.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:07.923 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:07.967 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:07.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:08.069 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:08.103 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:08.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:08.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:08.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:07 smithi170 ceph-mon[33394]: 4.1f scrub starts 2024-02-10T07:41:08.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:07 smithi170 ceph-mon[33394]: 4.1f scrub ok 2024-02-10T07:41:08.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:07 smithi170 ceph-mon[33394]: 5.a scrub starts 2024-02-10T07:41:08.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:07 smithi170 ceph-mon[33394]: 5.a scrub ok 2024-02-10T07:41:08.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:07 smithi008 ceph-mon[29816]: 4.1f scrub starts 2024-02-10T07:41:08.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:07 smithi008 ceph-mon[29816]: 4.1f scrub ok 2024-02-10T07:41:08.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:07 smithi008 ceph-mon[29816]: 5.a scrub starts 2024-02-10T07:41:08.137 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:07 smithi008 ceph-mon[29816]: 5.a scrub ok 2024-02-10T07:41:08.137 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:07 smithi008 ceph-mon[40743]: 4.1f scrub starts 2024-02-10T07:41:08.137 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:07 smithi008 ceph-mon[40743]: 4.1f scrub ok 2024-02-10T07:41:08.137 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:07 smithi008 ceph-mon[40743]: 5.a scrub starts 2024-02-10T07:41:08.137 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:07 smithi008 ceph-mon[40743]: 5.a scrub ok 2024-02-10T07:41:08.206 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:08.245 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:08.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:08.347 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:08.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:08.373 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:08.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:08.474 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:08.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:08.512 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:08.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:08.613 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:08.635 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:41:08.648 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:08.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:08.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:08.750 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:08.800 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:08.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:08.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:08.902 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:08.936 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:08.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:08.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:09.036 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:41:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:41:09.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[29816]: 8.9 scrub starts 2024-02-10T07:41:09.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[29816]: 3.11 scrub starts 2024-02-10T07:41:09.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[29816]: 3.11 scrub ok 2024-02-10T07:41:09.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[29816]: 5.1 scrub starts 2024-02-10T07:41:09.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[29816]: 5.1 scrub ok 2024-02-10T07:41:09.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[29816]: 6.4 scrub starts 2024-02-10T07:41:09.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[29816]: 6.4 scrub ok 2024-02-10T07:41:09.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[29816]: pgmap v420: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:41:09.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[29816]: 4.2 scrub starts 2024-02-10T07:41:09.038 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[29816]: 4.2 scrub ok 2024-02-10T07:41:09.038 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[40743]: 8.9 scrub starts 2024-02-10T07:41:09.039 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[40743]: 3.11 scrub starts 2024-02-10T07:41:09.039 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[40743]: 3.11 scrub ok 2024-02-10T07:41:09.039 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[40743]: 5.1 scrub starts 2024-02-10T07:41:09.039 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[40743]: 5.1 scrub ok 2024-02-10T07:41:09.039 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[40743]: 6.4 scrub starts 2024-02-10T07:41:09.039 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[40743]: 6.4 scrub ok 2024-02-10T07:41:09.039 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[40743]: pgmap v420: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:41:09.039 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[40743]: 4.2 scrub starts 2024-02-10T07:41:09.039 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:08 smithi008 ceph-mon[40743]: 4.2 scrub ok 2024-02-10T07:41:09.039 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:09.071 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:09.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:09.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:09.173 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:09.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:09.213 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:09.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:09.314 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:09.352 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:09.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:08 smithi170 ceph-mon[33394]: 8.9 scrub starts 2024-02-10T07:41:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:08 smithi170 ceph-mon[33394]: 3.11 scrub starts 2024-02-10T07:41:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:08 smithi170 ceph-mon[33394]: 3.11 scrub ok 2024-02-10T07:41:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:08 smithi170 ceph-mon[33394]: 5.1 scrub starts 2024-02-10T07:41:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:08 smithi170 ceph-mon[33394]: 5.1 scrub ok 2024-02-10T07:41:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:08 smithi170 ceph-mon[33394]: 6.4 scrub starts 2024-02-10T07:41:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:08 smithi170 ceph-mon[33394]: 6.4 scrub ok 2024-02-10T07:41:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:08 smithi170 ceph-mon[33394]: pgmap v420: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:41:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:08 smithi170 ceph-mon[33394]: 4.2 scrub starts 2024-02-10T07:41:09.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:08 smithi170 ceph-mon[33394]: 4.2 scrub ok 2024-02-10T07:41:09.455 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:09.475 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:09.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:09.551 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:09.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:09.561 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:09.569 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:09.652 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:09.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:09.705 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:09.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:09.806 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:09.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:09.847 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:09.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:09.948 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:09.957 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:09 smithi008 ceph-mon[29816]: 6.13 scrub starts 2024-02-10T07:41:09.957 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:09 smithi008 ceph-mon[29816]: 6.13 scrub ok 2024-02-10T07:41:09.957 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:09 smithi008 ceph-mon[29816]: 8.9 scrub ok 2024-02-10T07:41:09.958 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:09 smithi008 ceph-mon[29816]: 3.10 scrub starts 2024-02-10T07:41:09.958 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:09 smithi008 ceph-mon[29816]: 3.10 scrub ok 2024-02-10T07:41:09.958 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:09 smithi008 ceph-mon[29816]: 6.f scrub starts 2024-02-10T07:41:09.958 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:09 smithi008 ceph-mon[40743]: 6.13 scrub starts 2024-02-10T07:41:09.958 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:09 smithi008 ceph-mon[40743]: 6.13 scrub ok 2024-02-10T07:41:09.958 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:09 smithi008 ceph-mon[40743]: 8.9 scrub ok 2024-02-10T07:41:09.958 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:09 smithi008 ceph-mon[40743]: 3.10 scrub starts 2024-02-10T07:41:09.958 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:09 smithi008 ceph-mon[40743]: 3.10 scrub ok 2024-02-10T07:41:09.958 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:09 smithi008 ceph-mon[40743]: 6.f scrub starts 2024-02-10T07:41:09.980 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:09.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:09.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:10.082 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:10.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:10.117 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:10.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:10.200 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:09 smithi170 ceph-mon[33394]: 6.13 scrub starts 2024-02-10T07:41:10.200 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:09 smithi170 ceph-mon[33394]: 6.13 scrub ok 2024-02-10T07:41:10.201 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:09 smithi170 ceph-mon[33394]: 8.9 scrub ok 2024-02-10T07:41:10.201 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:09 smithi170 ceph-mon[33394]: 3.10 scrub starts 2024-02-10T07:41:10.201 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:09 smithi170 ceph-mon[33394]: 3.10 scrub ok 2024-02-10T07:41:10.201 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:09 smithi170 ceph-mon[33394]: 6.f scrub starts 2024-02-10T07:41:10.218 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:10.251 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:10.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:10.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:10.353 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:10.394 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:10.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:10.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:10.496 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:10.530 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:10.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:10.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:10.632 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:10.669 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:10.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:10.705 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:41:10.752 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:41:10.770 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:10.794 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:41:10.811 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:10.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:10.842 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:41:10.880 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:41:10.913 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:10.920 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:41:10.954 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:10.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:10.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:10.957 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:41:11.007 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:41:11.010 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:41:11.011 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:41:11.011 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:41:11.011 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:41:11.011 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:41:11.011 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:41:11.011 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:41:11.011 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:41:11.057 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:11.085 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:11.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:11.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:11.187 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:11.223 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:11.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:11.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:10 smithi008 ceph-mon[29816]: 6.f scrub ok 2024-02-10T07:41:11.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:10 smithi008 ceph-mon[29816]: pgmap v421: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:41:11.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:10 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2294731924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:41:11.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:10 smithi008 ceph-mon[40743]: 6.f scrub ok 2024-02-10T07:41:11.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:10 smithi008 ceph-mon[40743]: pgmap v421: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:41:11.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:10 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2294731924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:41:11.324 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:11.364 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:11.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:11.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:11.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:10 smithi170 ceph-mon[33394]: 6.f scrub ok 2024-02-10T07:41:11.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:10 smithi170 ceph-mon[33394]: pgmap v421: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:41:11.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:10 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2294731924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:41:11.465 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:11.503 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:11.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:11.605 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:11.635 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:11.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:11.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:11.737 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:11.770 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:11.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:11.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:11.871 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:11.904 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:11.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:11.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:12.006 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:12.049 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:12.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:12.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:11 smithi008 ceph-mon[29816]: 3.a scrub starts 2024-02-10T07:41:12.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:11 smithi008 ceph-mon[29816]: 3.a scrub ok 2024-02-10T07:41:12.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:11 smithi008 ceph-mon[29816]: 3.0 scrub starts 2024-02-10T07:41:12.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:11 smithi008 ceph-mon[29816]: 3.0 scrub ok 2024-02-10T07:41:12.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:11 smithi008 ceph-mon[29816]: 6.1f scrub starts 2024-02-10T07:41:12.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:11 smithi008 ceph-mon[29816]: 6.1f scrub ok 2024-02-10T07:41:12.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:11 smithi008 ceph-mon[40743]: 3.a scrub starts 2024-02-10T07:41:12.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:11 smithi008 ceph-mon[40743]: 3.a scrub ok 2024-02-10T07:41:12.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:11 smithi008 ceph-mon[40743]: 3.0 scrub starts 2024-02-10T07:41:12.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:11 smithi008 ceph-mon[40743]: 3.0 scrub ok 2024-02-10T07:41:12.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:11 smithi008 ceph-mon[40743]: 6.1f scrub starts 2024-02-10T07:41:12.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:11 smithi008 ceph-mon[40743]: 6.1f scrub ok 2024-02-10T07:41:12.151 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:12.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:12.188 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:12.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:12.289 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:12.333 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:12.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:12.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:12.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:11 smithi170 ceph-mon[33394]: 3.a scrub starts 2024-02-10T07:41:12.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:11 smithi170 ceph-mon[33394]: 3.a scrub ok 2024-02-10T07:41:12.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:11 smithi170 ceph-mon[33394]: 3.0 scrub starts 2024-02-10T07:41:12.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:11 smithi170 ceph-mon[33394]: 3.0 scrub ok 2024-02-10T07:41:12.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:11 smithi170 ceph-mon[33394]: 6.1f scrub starts 2024-02-10T07:41:12.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:11 smithi170 ceph-mon[33394]: 6.1f scrub ok 2024-02-10T07:41:12.434 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:12.473 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:12.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:12.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:12.574 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:12.638 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:12.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:12.739 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:12.785 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:12.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:12.886 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:12.919 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:12.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:13.021 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:13.065 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:13.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:13.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:13.154 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[29816]: 3.13 scrub starts 2024-02-10T07:41:13.154 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[29816]: 3.13 scrub ok 2024-02-10T07:41:13.154 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[29816]: 4.13 scrub starts 2024-02-10T07:41:13.154 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[29816]: pgmap v422: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:41:13.154 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[29816]: 4.13 scrub ok 2024-02-10T07:41:13.154 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[29816]: 8.1 scrub starts 2024-02-10T07:41:13.154 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[29816]: 5.d scrub starts 2024-02-10T07:41:13.155 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[29816]: 5.d scrub ok 2024-02-10T07:41:13.155 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[40743]: 3.13 scrub starts 2024-02-10T07:41:13.155 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[40743]: 3.13 scrub ok 2024-02-10T07:41:13.155 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[40743]: 4.13 scrub starts 2024-02-10T07:41:13.155 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[40743]: pgmap v422: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:41:13.155 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[40743]: 4.13 scrub ok 2024-02-10T07:41:13.155 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[40743]: 8.1 scrub starts 2024-02-10T07:41:13.155 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[40743]: 5.d scrub starts 2024-02-10T07:41:13.155 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:12 smithi008 ceph-mon[40743]: 5.d scrub ok 2024-02-10T07:41:13.168 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:13.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:12 smithi170 ceph-mon[33394]: 3.13 scrub starts 2024-02-10T07:41:13.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:12 smithi170 ceph-mon[33394]: 3.13 scrub ok 2024-02-10T07:41:13.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:12 smithi170 ceph-mon[33394]: 4.13 scrub starts 2024-02-10T07:41:13.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:12 smithi170 ceph-mon[33394]: pgmap v422: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:41:13.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:12 smithi170 ceph-mon[33394]: 4.13 scrub ok 2024-02-10T07:41:13.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:12 smithi170 ceph-mon[33394]: 8.1 scrub starts 2024-02-10T07:41:13.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:12 smithi170 ceph-mon[33394]: 5.d scrub starts 2024-02-10T07:41:13.193 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:12 smithi170 ceph-mon[33394]: 5.d scrub ok 2024-02-10T07:41:13.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:13.199 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:13.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:13.300 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:13.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:13.342 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:13.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:13.443 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:13.473 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:13.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:13.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:13.575 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:13.604 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:13.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:13.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:13.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:41:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:41:13.706 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:13.744 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:13.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:13.811 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:41:13.847 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:13.873 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:13.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:13.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:13.975 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:13.976 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[29816]: 4.a scrub starts 2024-02-10T07:41:13.976 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[29816]: 4.a scrub ok 2024-02-10T07:41:13.977 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[29816]: 5.f scrub starts 2024-02-10T07:41:13.977 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[29816]: 5.f scrub ok 2024-02-10T07:41:13.977 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[29816]: 3.17 scrub starts 2024-02-10T07:41:13.977 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[29816]: 4.12 scrub starts 2024-02-10T07:41:13.977 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[29816]: 3.17 scrub ok 2024-02-10T07:41:13.977 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[29816]: 4.12 scrub ok 2024-02-10T07:41:13.977 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[29816]: 8.1 scrub ok 2024-02-10T07:41:13.977 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:13.977 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[40743]: 4.a scrub starts 2024-02-10T07:41:13.977 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[40743]: 4.a scrub ok 2024-02-10T07:41:13.977 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[40743]: 5.f scrub starts 2024-02-10T07:41:13.978 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[40743]: 5.f scrub ok 2024-02-10T07:41:13.978 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[40743]: 3.17 scrub starts 2024-02-10T07:41:13.978 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[40743]: 4.12 scrub starts 2024-02-10T07:41:13.978 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[40743]: 3.17 scrub ok 2024-02-10T07:41:13.978 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[40743]: 4.12 scrub ok 2024-02-10T07:41:13.978 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[40743]: 8.1 scrub ok 2024-02-10T07:41:13.978 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:13 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:14.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:14.124 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:14.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:14.225 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:14.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:14.273 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:14.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:14.332 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:41:14.332 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:41:14.374 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:13 smithi170 ceph-mon[33394]: 4.a scrub starts 2024-02-10T07:41:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:13 smithi170 ceph-mon[33394]: 4.a scrub ok 2024-02-10T07:41:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:13 smithi170 ceph-mon[33394]: 5.f scrub starts 2024-02-10T07:41:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:13 smithi170 ceph-mon[33394]: 5.f scrub ok 2024-02-10T07:41:14.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:13 smithi170 ceph-mon[33394]: 3.17 scrub starts 2024-02-10T07:41:14.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:13 smithi170 ceph-mon[33394]: 4.12 scrub starts 2024-02-10T07:41:14.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:13 smithi170 ceph-mon[33394]: 3.17 scrub ok 2024-02-10T07:41:14.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:13 smithi170 ceph-mon[33394]: 4.12 scrub ok 2024-02-10T07:41:14.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:13 smithi170 ceph-mon[33394]: 8.1 scrub ok 2024-02-10T07:41:14.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:13 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:14.404 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:14.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:14.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:14.506 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:14.552 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:14.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:14.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:14.654 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:14.686 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:14.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:14.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:14.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:14.820 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:14.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:14.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:14.921 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:14.960 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:14.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:14.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:15.033 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:14 smithi008 ceph-mon[29816]: 6.1c scrub starts 2024-02-10T07:41:15.033 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:14 smithi008 ceph-mon[29816]: 6.1c scrub ok 2024-02-10T07:41:15.034 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:14 smithi008 ceph-mon[29816]: 5.18 scrub starts 2024-02-10T07:41:15.034 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:14 smithi008 ceph-mon[29816]: 5.18 scrub ok 2024-02-10T07:41:15.034 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:14 smithi008 ceph-mon[29816]: pgmap v423: 148 pgs: 148 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:41:15.034 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:14 smithi008 ceph-mon[29816]: 8.3 scrub starts 2024-02-10T07:41:15.034 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:14 smithi008 ceph-mon[29816]: 8.3 scrub ok 2024-02-10T07:41:15.034 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:14 smithi008 ceph-mon[40743]: 6.1c scrub starts 2024-02-10T07:41:15.034 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:14 smithi008 ceph-mon[40743]: 6.1c scrub ok 2024-02-10T07:41:15.034 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:14 smithi008 ceph-mon[40743]: 5.18 scrub starts 2024-02-10T07:41:15.034 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:14 smithi008 ceph-mon[40743]: 5.18 scrub ok 2024-02-10T07:41:15.034 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:14 smithi008 ceph-mon[40743]: pgmap v423: 148 pgs: 148 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:41:15.034 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:14 smithi008 ceph-mon[40743]: 8.3 scrub starts 2024-02-10T07:41:15.034 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:14 smithi008 ceph-mon[40743]: 8.3 scrub ok 2024-02-10T07:41:15.063 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:15.094 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:15.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:15.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:15.196 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:15.219 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:15.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:15.321 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:15.355 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:15.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:15.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:15.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:14 smithi170 ceph-mon[33394]: 6.1c scrub starts 2024-02-10T07:41:15.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:14 smithi170 ceph-mon[33394]: 6.1c scrub ok 2024-02-10T07:41:15.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:14 smithi170 ceph-mon[33394]: 5.18 scrub starts 2024-02-10T07:41:15.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:14 smithi170 ceph-mon[33394]: 5.18 scrub ok 2024-02-10T07:41:15.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:14 smithi170 ceph-mon[33394]: pgmap v423: 148 pgs: 148 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-10T07:41:15.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:14 smithi170 ceph-mon[33394]: 8.3 scrub starts 2024-02-10T07:41:15.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:14 smithi170 ceph-mon[33394]: 8.3 scrub ok 2024-02-10T07:41:15.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:15.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:15.491 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:15.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:15.592 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:15.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:15.627 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:15.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:15.728 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:15.763 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:15.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:15.865 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:15.901 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:15.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:15.988 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-10T07:41:16.003 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:16.079 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:16.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:16.181 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:16.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:16.219 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:16.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:16.320 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:16.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:15 smithi008 ceph-mon[29816]: 6.14 scrub starts 2024-02-10T07:41:16.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:15 smithi008 ceph-mon[40743]: 6.14 scrub starts 2024-02-10T07:41:16.355 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:16.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:16.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:15 smithi170 ceph-mon[33394]: 6.14 scrub starts 2024-02-10T07:41:16.456 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:16.493 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:16.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:16.595 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:16.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:16.637 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:16.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:16.738 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:16.777 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:16.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:16.878 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:16.910 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:16.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:17.012 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:17.061 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:17.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:17.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:17.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:16 smithi008 ceph-mon[29816]: 3.d scrub starts 2024-02-10T07:41:17.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:16 smithi008 ceph-mon[29816]: 3.d scrub ok 2024-02-10T07:41:17.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:16 smithi008 ceph-mon[29816]: 6.14 scrub ok 2024-02-10T07:41:17.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:16 smithi008 ceph-mon[29816]: pgmap v424: 148 pgs: 148 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1020 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:41:17.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:16 smithi008 ceph-mon[40743]: 3.d scrub starts 2024-02-10T07:41:17.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:16 smithi008 ceph-mon[40743]: 3.d scrub ok 2024-02-10T07:41:17.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:16 smithi008 ceph-mon[40743]: 6.14 scrub ok 2024-02-10T07:41:17.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:16 smithi008 ceph-mon[40743]: pgmap v424: 148 pgs: 148 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1020 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:41:17.163 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:17.190 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:17.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:17.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:17.292 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:17.336 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:17.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:17.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:16 smithi170 ceph-mon[33394]: 3.d scrub starts 2024-02-10T07:41:17.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:16 smithi170 ceph-mon[33394]: 3.d scrub ok 2024-02-10T07:41:17.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:16 smithi170 ceph-mon[33394]: 6.14 scrub ok 2024-02-10T07:41:17.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:16 smithi170 ceph-mon[33394]: pgmap v424: 148 pgs: 148 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1020 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-10T07:41:17.433 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:17.438 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:17.468 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:17.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:17.482 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:17.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:17.583 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:17.629 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:17.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:17.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:17.731 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:17.768 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:17.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:17.870 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:17.902 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:17.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:17.980 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:41:18.003 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:18.056 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:18.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:18.158 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:18.202 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:18.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:18.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:18.232 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:41:18.232 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:41:18.232 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: 6.15 scrub starts 2024-02-10T07:41:18.232 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: 6.15 scrub ok 2024-02-10T07:41:18.232 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: 2.1 scrub starts 2024-02-10T07:41:18.232 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: 2.1 scrub ok 2024-02-10T07:41:18.232 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: 3.14 scrub starts 2024-02-10T07:41:18.232 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: 3.14 scrub ok 2024-02-10T07:41:18.232 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: osdmap e242: 8 total, 8 up, 8 in 2024-02-10T07:41:18.232 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: 6.9 scrub starts 2024-02-10T07:41:18.232 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: 8.b scrub starts 2024-02-10T07:41:18.232 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: 6.9 scrub ok 2024-02-10T07:41:18.233 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3447386552' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:41:18.233 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:41:18.233 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:18.233 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:41:18.233 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:41:18.233 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: 6.15 scrub starts 2024-02-10T07:41:18.233 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: 6.15 scrub ok 2024-02-10T07:41:18.233 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: 2.1 scrub starts 2024-02-10T07:41:18.233 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: 2.1 scrub ok 2024-02-10T07:41:18.233 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: 3.14 scrub starts 2024-02-10T07:41:18.234 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: 3.14 scrub ok 2024-02-10T07:41:18.234 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: osdmap e242: 8 total, 8 up, 8 in 2024-02-10T07:41:18.234 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: 6.9 scrub starts 2024-02-10T07:41:18.234 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: 8.b scrub starts 2024-02-10T07:41:18.234 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: 6.9 scrub ok 2024-02-10T07:41:18.234 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3447386552' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:41:18.234 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:41:18.234 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:17 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:18.303 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:18.340 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:18.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:18.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:18.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:41:18.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:41:18.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: 6.15 scrub starts 2024-02-10T07:41:18.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: 6.15 scrub ok 2024-02-10T07:41:18.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: 2.1 scrub starts 2024-02-10T07:41:18.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: 2.1 scrub ok 2024-02-10T07:41:18.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: 3.14 scrub starts 2024-02-10T07:41:18.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: 3.14 scrub ok 2024-02-10T07:41:18.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: osdmap e242: 8 total, 8 up, 8 in 2024-02-10T07:41:18.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: 6.9 scrub starts 2024-02-10T07:41:18.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: 8.b scrub starts 2024-02-10T07:41:18.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: 6.9 scrub ok 2024-02-10T07:41:18.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3447386552' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:41:18.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:41:18.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:17 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:18.442 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:18.475 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:18.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:18.576 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:18.613 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:18.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:18.715 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:18.751 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:18.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:18.852 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:18.889 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:18.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:18.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:18.952 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:41:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:41:18.952 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[40743]: 3.15 scrub starts 2024-02-10T07:41:18.952 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[40743]: 3.15 scrub ok 2024-02-10T07:41:18.991 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:19.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:19.028 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:19.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:19.129 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:19.167 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:19.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:19.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:19.269 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:19.310 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:19.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:19.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:19.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[29816]: 3.15 scrub starts 2024-02-10T07:41:19.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[29816]: 3.15 scrub ok 2024-02-10T07:41:19.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[29816]: 4.18 scrub starts 2024-02-10T07:41:19.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[29816]: pgmap v426: 148 pgs: 148 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:41:19.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[29816]: 4.18 scrub ok 2024-02-10T07:41:19.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:41:19.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:41:19.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:41:19.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:41:19.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:41:19.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[29816]: osdmap e243: 8 total, 8 up, 8 in 2024-02-10T07:41:19.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[29816]: 8.b scrub ok 2024-02-10T07:41:19.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:41:19.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:41:19.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[40743]: 4.18 scrub starts 2024-02-10T07:41:19.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[40743]: pgmap v426: 148 pgs: 148 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:41:19.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[40743]: 4.18 scrub ok 2024-02-10T07:41:19.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:41:19.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:41:19.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:41:19.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:41:19.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:41:19.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[40743]: osdmap e243: 8 total, 8 up, 8 in 2024-02-10T07:41:19.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[40743]: 8.b scrub ok 2024-02-10T07:41:19.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:41:19.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:18 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:41:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:18 smithi170 ceph-mon[33394]: 3.15 scrub starts 2024-02-10T07:41:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:18 smithi170 ceph-mon[33394]: 3.15 scrub ok 2024-02-10T07:41:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:18 smithi170 ceph-mon[33394]: 4.18 scrub starts 2024-02-10T07:41:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:18 smithi170 ceph-mon[33394]: pgmap v426: 148 pgs: 148 active+clean; 2.3 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:41:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:18 smithi170 ceph-mon[33394]: 4.18 scrub ok 2024-02-10T07:41:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:18 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:41:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:18 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:41:19.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:18 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:41:19.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:18 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:41:19.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:18 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:41:19.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:18 smithi170 ceph-mon[33394]: osdmap e243: 8 total, 8 up, 8 in 2024-02-10T07:41:19.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:18 smithi170 ceph-mon[33394]: 8.b scrub ok 2024-02-10T07:41:19.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:18 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:41:19.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:18 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:41:19.412 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:19.459 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:19.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:19.562 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:19.608 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:19.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:19.710 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:19.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:19.745 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:19.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:19.846 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:19.883 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:19.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:19.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:19.988 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:20.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:20.029 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:20.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:20.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:19 smithi008 ceph-mon[29816]: 5.2 scrub starts 2024-02-10T07:41:20.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:19 smithi008 ceph-mon[29816]: 5.2 scrub ok 2024-02-10T07:41:20.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:19 smithi008 ceph-mon[29816]: 6.19 scrub starts 2024-02-10T07:41:20.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:19 smithi008 ceph-mon[29816]: 6.19 scrub ok 2024-02-10T07:41:20.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:19 smithi008 ceph-mon[29816]: osdmap e244: 8 total, 8 up, 8 in 2024-02-10T07:41:20.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:19 smithi008 ceph-mon[40743]: 5.2 scrub starts 2024-02-10T07:41:20.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:19 smithi008 ceph-mon[40743]: 5.2 scrub ok 2024-02-10T07:41:20.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:19 smithi008 ceph-mon[40743]: 6.19 scrub starts 2024-02-10T07:41:20.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:19 smithi008 ceph-mon[40743]: 6.19 scrub ok 2024-02-10T07:41:20.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:19 smithi008 ceph-mon[40743]: osdmap e244: 8 total, 8 up, 8 in 2024-02-10T07:41:20.130 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:20.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:20.170 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:20.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:20.271 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:20.311 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:20.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:20.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:19 smithi170 ceph-mon[33394]: 5.2 scrub starts 2024-02-10T07:41:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:19 smithi170 ceph-mon[33394]: 5.2 scrub ok 2024-02-10T07:41:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:19 smithi170 ceph-mon[33394]: 6.19 scrub starts 2024-02-10T07:41:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:19 smithi170 ceph-mon[33394]: 6.19 scrub ok 2024-02-10T07:41:20.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:19 smithi170 ceph-mon[33394]: osdmap e244: 8 total, 8 up, 8 in 2024-02-10T07:41:20.413 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:20.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:20.453 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:20.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:20.554 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:20.586 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:20.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:20.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:20.688 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:20.726 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:20.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:20.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:20.828 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:20.850 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:20.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:20.952 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:20.994 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:20.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:20.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:21.096 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:21.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:21.132 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:21.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:21.233 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:21.272 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:21.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:21.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:21.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:41:21.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:41:21.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[29816]: 5.9 scrub starts 2024-02-10T07:41:21.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[29816]: 5.9 scrub ok 2024-02-10T07:41:21.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[29816]: 3.9 scrub starts 2024-02-10T07:41:21.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[29816]: 3.9 scrub ok 2024-02-10T07:41:21.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[29816]: 3.19 scrub starts 2024-02-10T07:41:21.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[29816]: 3.19 scrub ok 2024-02-10T07:41:21.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[29816]: pgmap v429: 148 pgs: 148 active+clean; 2.1 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 904 op/s 2024-02-10T07:41:21.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[29816]: osdmap e245: 8 total, 8 up, 8 in 2024-02-10T07:41:21.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:41:21.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:41:21.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[40743]: 5.9 scrub starts 2024-02-10T07:41:21.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[40743]: 5.9 scrub ok 2024-02-10T07:41:21.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[40743]: 3.9 scrub starts 2024-02-10T07:41:21.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[40743]: 3.9 scrub ok 2024-02-10T07:41:21.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[40743]: 3.19 scrub starts 2024-02-10T07:41:21.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[40743]: 3.19 scrub ok 2024-02-10T07:41:21.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[40743]: pgmap v429: 148 pgs: 148 active+clean; 2.1 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 904 op/s 2024-02-10T07:41:21.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:20 smithi008 ceph-mon[40743]: osdmap e245: 8 total, 8 up, 8 in 2024-02-10T07:41:21.374 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:21.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:20 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:41:21.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:20 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:41:21.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:20 smithi170 ceph-mon[33394]: 5.9 scrub starts 2024-02-10T07:41:21.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:20 smithi170 ceph-mon[33394]: 5.9 scrub ok 2024-02-10T07:41:21.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:20 smithi170 ceph-mon[33394]: 3.9 scrub starts 2024-02-10T07:41:21.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:20 smithi170 ceph-mon[33394]: 3.9 scrub ok 2024-02-10T07:41:21.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:20 smithi170 ceph-mon[33394]: 3.19 scrub starts 2024-02-10T07:41:21.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:20 smithi170 ceph-mon[33394]: 3.19 scrub ok 2024-02-10T07:41:21.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:20 smithi170 ceph-mon[33394]: pgmap v429: 148 pgs: 148 active+clean; 2.1 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 904 op/s 2024-02-10T07:41:21.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:20 smithi170 ceph-mon[33394]: osdmap e245: 8 total, 8 up, 8 in 2024-02-10T07:41:21.410 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:21.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:21.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:21.429 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:41:21.513 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:21.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:21.537 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:21.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:21.638 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:21.697 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:21.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:21.798 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:21.831 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:21.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:21.935 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:21.968 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:21.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:21.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:22.070 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:22.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:22.096 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:22.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:22.197 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:22.229 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:22.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:22.330 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:22.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:21 smithi008 ceph-mon[40743]: 6.1b scrub starts 2024-02-10T07:41:22.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:21 smithi008 ceph-mon[40743]: 4.7 scrub starts 2024-02-10T07:41:22.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:21 smithi008 ceph-mon[40743]: 4.7 scrub ok 2024-02-10T07:41:22.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:21 smithi008 ceph-mon[40743]: 6.1b scrub ok 2024-02-10T07:41:22.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:21 smithi008 ceph-mon[40743]: 8.e scrub starts 2024-02-10T07:41:22.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:21 smithi008 ceph-mon[29816]: 6.1b scrub starts 2024-02-10T07:41:22.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:21 smithi008 ceph-mon[29816]: 4.7 scrub starts 2024-02-10T07:41:22.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:21 smithi008 ceph-mon[29816]: 4.7 scrub ok 2024-02-10T07:41:22.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:21 smithi008 ceph-mon[29816]: 6.1b scrub ok 2024-02-10T07:41:22.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:21 smithi008 ceph-mon[29816]: 8.e scrub starts 2024-02-10T07:41:22.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:22.369 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:22.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:21 smithi170 ceph-mon[33394]: 6.1b scrub starts 2024-02-10T07:41:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:21 smithi170 ceph-mon[33394]: 4.7 scrub starts 2024-02-10T07:41:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:21 smithi170 ceph-mon[33394]: 4.7 scrub ok 2024-02-10T07:41:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:21 smithi170 ceph-mon[33394]: 6.1b scrub ok 2024-02-10T07:41:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:21 smithi170 ceph-mon[33394]: 8.e scrub starts 2024-02-10T07:41:22.471 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:22.507 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:22.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:22.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:22.609 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:22.638 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:22.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:22.740 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:22.771 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:22.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:22.872 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:22.906 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:22.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:22.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:23.008 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:23.043 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:23.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:23.143 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:23.171 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:23.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:23.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:23.273 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:23.310 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:23.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:23.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:22 smithi008 ceph-mon[29816]: 4.9 deep-scrub starts 2024-02-10T07:41:23.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:22 smithi008 ceph-mon[29816]: 4.9 deep-scrub ok 2024-02-10T07:41:23.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:22 smithi008 ceph-mon[29816]: pgmap v431: 148 pgs: 148 active+clean; 2.1 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 885 op/s 2024-02-10T07:41:23.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:22 smithi008 ceph-mon[29816]: 8.e scrub ok 2024-02-10T07:41:23.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:22 smithi008 ceph-mon[40743]: 4.9 deep-scrub starts 2024-02-10T07:41:23.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:22 smithi008 ceph-mon[40743]: 4.9 deep-scrub ok 2024-02-10T07:41:23.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:22 smithi008 ceph-mon[40743]: pgmap v431: 148 pgs: 148 active+clean; 2.1 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 885 op/s 2024-02-10T07:41:23.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:22 smithi008 ceph-mon[40743]: 8.e scrub ok 2024-02-10T07:41:23.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:41:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:41:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:22 smithi170 ceph-mon[33394]: 4.9 deep-scrub starts 2024-02-10T07:41:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:22 smithi170 ceph-mon[33394]: 4.9 deep-scrub ok 2024-02-10T07:41:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:22 smithi170 ceph-mon[33394]: pgmap v431: 148 pgs: 148 active+clean; 2.1 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 885 op/s 2024-02-10T07:41:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:22 smithi170 ceph-mon[33394]: 8.e scrub ok 2024-02-10T07:41:23.413 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:23.446 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:23.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:23.548 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:23.582 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:23.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:23.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:23.685 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:23.710 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:23.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:23.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:23.812 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:23.853 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:23.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:23.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:23.955 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:23.985 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:23.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:24.087 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:24.117 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:24.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:24.219 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:24.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:24.255 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:24.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:24.291 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:23 smithi008 ceph-mon[29816]: 3.c scrub starts 2024-02-10T07:41:24.291 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:23 smithi008 ceph-mon[29816]: 3.c scrub ok 2024-02-10T07:41:24.291 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:23 smithi008 ceph-mon[29816]: 8.f scrub starts 2024-02-10T07:41:24.291 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:23 smithi008 ceph-mon[29816]: 5.1d scrub starts 2024-02-10T07:41:24.291 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:23 smithi008 ceph-mon[29816]: 5.1d scrub ok 2024-02-10T07:41:24.291 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:23 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:24.292 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:23 smithi008 ceph-mon[40743]: 3.c scrub starts 2024-02-10T07:41:24.292 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:23 smithi008 ceph-mon[40743]: 3.c scrub ok 2024-02-10T07:41:24.292 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:23 smithi008 ceph-mon[40743]: 8.f scrub starts 2024-02-10T07:41:24.292 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:23 smithi008 ceph-mon[40743]: 5.1d scrub starts 2024-02-10T07:41:24.292 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:23 smithi008 ceph-mon[40743]: 5.1d scrub ok 2024-02-10T07:41:24.292 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:23 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:24.355 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:23 smithi170 ceph-mon[33394]: 3.c scrub starts 2024-02-10T07:41:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:23 smithi170 ceph-mon[33394]: 3.c scrub ok 2024-02-10T07:41:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:23 smithi170 ceph-mon[33394]: 8.f scrub starts 2024-02-10T07:41:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:23 smithi170 ceph-mon[33394]: 5.1d scrub starts 2024-02-10T07:41:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:23 smithi170 ceph-mon[33394]: 5.1d scrub ok 2024-02-10T07:41:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:23 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:24.400 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:24.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:24.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:24.503 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:24.545 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:24.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:24.647 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:24.685 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:24.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:24.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:24.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:24.821 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:24.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:24.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:24.923 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:24.962 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:24.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:24.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:25.064 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:25.105 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:25.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:25.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:25.207 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:25.216 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.3 reset_purged_snaps_last 2024-02-10T07:41:25.239 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:25.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:25.256 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[29816]: 3.f deep-scrub starts 2024-02-10T07:41:25.256 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[29816]: 3.f deep-scrub ok 2024-02-10T07:41:25.256 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[29816]: pgmap v432: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.58k op/s 2024-02-10T07:41:25.256 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[29816]: 6.1e scrub starts 2024-02-10T07:41:25.256 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[29816]: 6.1e scrub ok 2024-02-10T07:41:25.256 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[29816]: 8.f scrub ok 2024-02-10T07:41:25.256 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[29816]: 3.12 scrub starts 2024-02-10T07:41:25.257 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[29816]: 3.12 scrub ok 2024-02-10T07:41:25.257 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[40743]: 3.f deep-scrub starts 2024-02-10T07:41:25.257 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[40743]: 3.f deep-scrub ok 2024-02-10T07:41:25.257 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[40743]: pgmap v432: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.58k op/s 2024-02-10T07:41:25.257 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[40743]: 6.1e scrub starts 2024-02-10T07:41:25.257 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[40743]: 6.1e scrub ok 2024-02-10T07:41:25.257 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[40743]: 8.f scrub ok 2024-02-10T07:41:25.257 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[40743]: 3.12 scrub starts 2024-02-10T07:41:25.257 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:24 smithi008 ceph-mon[40743]: 3.12 scrub ok 2024-02-10T07:41:25.341 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:25.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:25.370 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:25.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:24 smithi170 ceph-mon[33394]: 3.f deep-scrub starts 2024-02-10T07:41:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:24 smithi170 ceph-mon[33394]: 3.f deep-scrub ok 2024-02-10T07:41:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:24 smithi170 ceph-mon[33394]: pgmap v432: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.58k op/s 2024-02-10T07:41:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:24 smithi170 ceph-mon[33394]: 6.1e scrub starts 2024-02-10T07:41:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:24 smithi170 ceph-mon[33394]: 6.1e scrub ok 2024-02-10T07:41:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:24 smithi170 ceph-mon[33394]: 8.f scrub ok 2024-02-10T07:41:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:24 smithi170 ceph-mon[33394]: 3.12 scrub starts 2024-02-10T07:41:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:24 smithi170 ceph-mon[33394]: 3.12 scrub ok 2024-02-10T07:41:25.471 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:25.501 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:25.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:25.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:25.602 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:25.635 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:25.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:25.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:25.737 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:25.762 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:25.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:25.795 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:25.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:25.890 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:41:25.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:25.923 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:25.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:25.931 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:41:25.977 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:41:26.025 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:26.033 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:41:26.049 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:26.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:26.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:26.075 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:41:26.115 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:41:26.151 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:26.158 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:41:26.158 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:26 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:41:26.158 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:26 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:41:26.158 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:26 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1283935178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:41:26.159 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:26 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:41:26.159 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:26 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:41:26.159 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:26 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1283935178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:41:26.197 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:41:26.204 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:26.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:26.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:26.227 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:41:26.228 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:41:26.228 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:41:26.228 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:41:26.228 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:41:26.228 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:41:26.228 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:41:26.228 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:41:26.307 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:26.349 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:26.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:26 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:41:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:26 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:41:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:26 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1283935178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:41:26.450 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:26.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:26.494 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:26.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:26.595 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:26.639 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:26.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:26.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:26.741 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:26.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:26.780 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:26.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:26.882 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:26.921 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:26.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:26.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:27.022 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:27.051 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:27.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:27.153 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:27.183 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:27.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:27.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:27.227 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:27 smithi008 ceph-mon[29816]: 4.14 scrub starts 2024-02-10T07:41:27.227 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:27 smithi008 ceph-mon[29816]: 4.14 scrub ok 2024-02-10T07:41:27.228 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:27 smithi008 ceph-mon[29816]: 4.d deep-scrub starts 2024-02-10T07:41:27.228 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:27 smithi008 ceph-mon[29816]: 4.d deep-scrub ok 2024-02-10T07:41:27.228 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:27 smithi008 ceph-mon[29816]: pgmap v433: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 644 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-10T07:41:27.228 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:27 smithi008 ceph-mon[29816]: osdmap e246: 8 total, 8 up, 8 in 2024-02-10T07:41:27.228 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:27 smithi008 ceph-mon[29816]: 8.7 scrub starts 2024-02-10T07:41:27.228 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:27 smithi008 ceph-mon[40743]: 4.14 scrub starts 2024-02-10T07:41:27.228 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:27 smithi008 ceph-mon[40743]: 4.14 scrub ok 2024-02-10T07:41:27.228 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:27 smithi008 ceph-mon[40743]: 4.d deep-scrub starts 2024-02-10T07:41:27.228 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:27 smithi008 ceph-mon[40743]: 4.d deep-scrub ok 2024-02-10T07:41:27.228 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:27 smithi008 ceph-mon[40743]: pgmap v433: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 644 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-10T07:41:27.229 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:27 smithi008 ceph-mon[40743]: osdmap e246: 8 total, 8 up, 8 in 2024-02-10T07:41:27.229 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:27 smithi008 ceph-mon[40743]: 8.7 scrub starts 2024-02-10T07:41:27.285 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:27.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:27.333 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:27.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:27 smithi170 ceph-mon[33394]: 4.14 scrub starts 2024-02-10T07:41:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:27 smithi170 ceph-mon[33394]: 4.14 scrub ok 2024-02-10T07:41:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:27 smithi170 ceph-mon[33394]: 4.d deep-scrub starts 2024-02-10T07:41:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:27 smithi170 ceph-mon[33394]: 4.d deep-scrub ok 2024-02-10T07:41:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:27 smithi170 ceph-mon[33394]: pgmap v433: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.7 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 644 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-10T07:41:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:27 smithi170 ceph-mon[33394]: osdmap e246: 8 total, 8 up, 8 in 2024-02-10T07:41:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:27 smithi170 ceph-mon[33394]: 8.7 scrub starts 2024-02-10T07:41:27.434 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:27.477 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:27.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:27.579 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:27.609 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:27.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:27.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:27.710 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:27.745 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:27.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:27.846 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:27.884 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:27.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:27.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:27.986 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:28.067 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:28.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:28.169 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:28.229 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:28.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:28.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-mon[29816]: 1.0 scrub starts 2024-02-10T07:41:28.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-mon[29816]: 1.0 scrub ok 2024-02-10T07:41:28.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-mon[29816]: 4.8 deep-scrub starts 2024-02-10T07:41:28.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-mon[29816]: 4.8 deep-scrub ok 2024-02-10T07:41:28.274 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-mon[29816]: 8.6 scrub starts 2024-02-10T07:41:28.274 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-mon[29816]: 8.7 scrub ok 2024-02-10T07:41:28.274 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-mon[29816]: 8.6 scrub ok 2024-02-10T07:41:28.274 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-mon[40743]: 1.0 scrub starts 2024-02-10T07:41:28.274 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-mon[40743]: 1.0 scrub ok 2024-02-10T07:41:28.275 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-mon[40743]: 4.8 deep-scrub starts 2024-02-10T07:41:28.275 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-mon[40743]: 4.8 deep-scrub ok 2024-02-10T07:41:28.275 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-mon[40743]: 8.6 scrub starts 2024-02-10T07:41:28.275 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-mon[40743]: 8.7 scrub ok 2024-02-10T07:41:28.275 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-mon[40743]: 8.6 scrub ok 2024-02-10T07:41:28.294 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:41:28.294 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:41:28.294 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-02-10T07:41:28.294 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-02-10T07:41:28.294 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:41:28.332 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:28.360 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:28.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:28.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:28 smithi170 ceph-mon[33394]: 1.0 scrub starts 2024-02-10T07:41:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:28 smithi170 ceph-mon[33394]: 1.0 scrub ok 2024-02-10T07:41:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:28 smithi170 ceph-mon[33394]: 4.8 deep-scrub starts 2024-02-10T07:41:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:28 smithi170 ceph-mon[33394]: 4.8 deep-scrub ok 2024-02-10T07:41:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:28 smithi170 ceph-mon[33394]: 8.6 scrub starts 2024-02-10T07:41:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:28 smithi170 ceph-mon[33394]: 8.7 scrub ok 2024-02-10T07:41:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:28 smithi170 ceph-mon[33394]: 8.6 scrub ok 2024-02-10T07:41:28.462 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:28.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:28.494 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:28.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:28.595 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:28.633 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:28.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:28.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:28.735 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:28.770 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:28.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:28.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:28.872 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:28.903 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:28.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:28.962 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:29.005 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:29.025 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:41:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:41:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:41:29.036 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:29.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:29.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:29.050 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:41:29.138 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:29.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:29.177 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:29.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:29.277 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:29.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:29 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:41:29.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:29 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:41:29.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:29 smithi170 ceph-mon[33394]: pgmap v435: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 841 op/s 2024-02-10T07:41:29.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:29 smithi170 ceph-mon[33394]: 8.0 deep-scrub starts 2024-02-10T07:41:29.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:29 smithi170 ceph-mon[33394]: osdmap e247: 8 total, 8 up, 8 in 2024-02-10T07:41:29.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:29 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2185680205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:41:29.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:29 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:41:29.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:29 smithi170 ceph-mon[33394]: 5.12 scrub starts 2024-02-10T07:41:29.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:29 smithi170 ceph-mon[33394]: 5.12 scrub ok 2024-02-10T07:41:29.311 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:29.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:29.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:41:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:41:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[29816]: pgmap v435: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 841 op/s 2024-02-10T07:41:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[29816]: 8.0 deep-scrub starts 2024-02-10T07:41:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[29816]: osdmap e247: 8 total, 8 up, 8 in 2024-02-10T07:41:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2185680205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:41:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:41:29.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[29816]: 5.12 scrub starts 2024-02-10T07:41:29.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[29816]: 5.12 scrub ok 2024-02-10T07:41:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:41:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:41:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[40743]: pgmap v435: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 841 op/s 2024-02-10T07:41:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[40743]: 8.0 deep-scrub starts 2024-02-10T07:41:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[40743]: osdmap e247: 8 total, 8 up, 8 in 2024-02-10T07:41:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2185680205' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:41:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:41:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[40743]: 5.12 scrub starts 2024-02-10T07:41:29.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:29 smithi008 ceph-mon[40743]: 5.12 scrub ok 2024-02-10T07:41:29.413 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:29.453 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:29.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:29.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:29.549 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-10T07:41:29.555 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:29.589 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:29.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:29.691 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:29.730 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:29.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:29.745 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:41:29.831 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:29.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:29.897 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:29.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:29.999 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:30.027 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:30.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[29816]: 5.1a scrub starts 2024-02-10T07:41:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[29816]: 5.1a scrub ok 2024-02-10T07:41:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[29816]: 5.1c deep-scrub starts 2024-02-10T07:41:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[29816]: 5.1c deep-scrub ok 2024-02-10T07:41:30.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[29816]: 5.13 scrub starts 2024-02-10T07:41:30.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[29816]: 5.13 scrub ok 2024-02-10T07:41:30.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:41:30.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:41:30.083 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[29816]: osdmap e248: 8 total, 8 up, 8 in 2024-02-10T07:41:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[40743]: 5.1a scrub starts 2024-02-10T07:41:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[40743]: 5.1a scrub ok 2024-02-10T07:41:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[40743]: 5.1c deep-scrub starts 2024-02-10T07:41:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[40743]: 5.1c deep-scrub ok 2024-02-10T07:41:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[40743]: 5.13 scrub starts 2024-02-10T07:41:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[40743]: 5.13 scrub ok 2024-02-10T07:41:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:41:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:41:30.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:30 smithi008 ceph-mon[40743]: osdmap e248: 8 total, 8 up, 8 in 2024-02-10T07:41:30.129 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:30.167 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:30.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:30.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:30.269 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:30.301 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:30.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:30.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:30 smithi170 ceph-mon[33394]: 5.1a scrub starts 2024-02-10T07:41:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:30 smithi170 ceph-mon[33394]: 5.1a scrub ok 2024-02-10T07:41:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:30 smithi170 ceph-mon[33394]: 5.1c deep-scrub starts 2024-02-10T07:41:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:30 smithi170 ceph-mon[33394]: 5.1c deep-scrub ok 2024-02-10T07:41:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:30 smithi170 ceph-mon[33394]: 5.13 scrub starts 2024-02-10T07:41:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:30 smithi170 ceph-mon[33394]: 5.13 scrub ok 2024-02-10T07:41:30.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:30 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:41:30.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:30 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:41:30.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:30 smithi170 ceph-mon[33394]: osdmap e248: 8 total, 8 up, 8 in 2024-02-10T07:41:30.403 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:30.443 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:30.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:30.544 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:30.594 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:30.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:30.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:30.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:30.731 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:30.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:30.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:30.833 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:30.864 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:30.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:30.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:30.966 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:31.000 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:31.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:31.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:31.102 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:31.136 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:31.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:31.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:31.238 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:31.275 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:31.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:31.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:31.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:31 smithi008 ceph-mon[29816]: pgmap v438: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 857 op/s 2024-02-10T07:41:31.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:31 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:41:31.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:31 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:41:31.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:31 smithi008 ceph-mon[40743]: pgmap v438: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 857 op/s 2024-02-10T07:41:31.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:31 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:41:31.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:31 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:41:31.378 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:31.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:31 smithi170 ceph-mon[33394]: pgmap v438: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 857 op/s 2024-02-10T07:41:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:31 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:41:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:31 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:41:31.408 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:31.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:31.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:31.510 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:31.540 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:31.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:31.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:31.641 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:31.677 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:31.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:31.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:31.779 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:31.823 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:31.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:31.926 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:31.964 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:31.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:32.065 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:32.093 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:32.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:32.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:32.195 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:32.236 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:32.236 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:32.262 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:32.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:32.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:32 smithi008 ceph-mon[29816]: osdmap e249: 8 total, 8 up, 8 in 2024-02-10T07:41:32.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:32 smithi008 ceph-mon[40743]: osdmap e249: 8 total, 8 up, 8 in 2024-02-10T07:41:32.363 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:32 smithi170 ceph-mon[33394]: osdmap e249: 8 total, 8 up, 8 in 2024-02-10T07:41:32.394 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:32.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:32.496 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:32.522 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:41:32.531 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:32.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:32.634 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:32.680 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:32.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:32.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:32.782 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:32.823 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:32.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:32.924 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:32.959 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:32.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:33.061 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:33.095 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:33.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:33.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:33.197 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:33.227 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:33.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:33.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:33.329 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:33 smithi008 ceph-mon[29816]: pgmap v440: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 870 B/s rd, 0 B/s wr, 875 op/s 2024-02-10T07:41:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:33 smithi008 ceph-mon[40743]: pgmap v440: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 870 B/s rd, 0 B/s wr, 875 op/s 2024-02-10T07:41:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:33.358 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:33.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:33.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:33 smithi170 ceph-mon[33394]: pgmap v440: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.4 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 870 B/s rd, 0 B/s wr, 875 op/s 2024-02-10T07:41:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:33.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:41:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:41:33.460 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:33.493 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:33.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:33.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:33.595 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:33.635 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:33.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:33.737 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:33.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:33.776 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:33.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:33.877 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:33.915 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:33.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:33.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:34.017 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:34.054 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:34.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:34.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:34.156 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:34.186 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:34.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:34.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:34.287 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:34.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:34 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:34.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:34 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:34.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:34.359 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:34.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:34 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:34.460 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:34.488 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:34.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:34.589 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:34.630 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:34.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:34.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:34.732 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:34.768 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:34.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:34.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:34.870 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:34.908 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:34.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:34.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:35.009 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:35.046 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:35.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:35.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:35.131 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:35 smithi008 ceph-mon[29816]: pgmap v441: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.28k op/s 2024-02-10T07:41:35.131 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:35 smithi008 ceph-mon[40743]: pgmap v441: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.28k op/s 2024-02-10T07:41:35.148 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:35.189 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:35.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:35.291 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:35.330 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:35.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:35.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:35 smithi170 ceph-mon[33394]: pgmap v441: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.28k op/s 2024-02-10T07:41:35.431 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:35.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:35.464 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:35.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:35.568 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:35.600 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:35.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:35.702 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:35.733 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:35.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:35.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:35.835 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:35.864 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:35.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:35.966 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:36.005 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:36.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:36.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:36.107 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:36.139 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.4 reset_purged_snaps_last 2024-02-10T07:41:36.142 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:36.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:36.244 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:36.293 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:36.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:36.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:36 smithi008 ceph-mon[29816]: 8.0 deep-scrub ok 2024-02-10T07:41:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:36 smithi008 ceph-mon[40743]: 8.0 deep-scrub ok 2024-02-10T07:41:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:36 smithi170 ceph-mon[33394]: 8.0 deep-scrub ok 2024-02-10T07:41:36.394 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:36.425 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:36.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:36.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:36.469 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:41:36.510 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:41:36.528 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:36.562 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:41:36.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:36.584 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:36.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:36.598 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:36.607 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:41:36.652 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:41:36.685 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:36.706 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:41:36.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:36.713 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:36.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:36.754 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:41:36.803 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:41:36.807 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:41:36.808 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:41:36.808 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:41:36.808 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:41:36.808 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:41:36.808 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:41:36.808 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:41:36.808 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:41:36.815 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:36.843 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:36.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:36.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:36.945 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:36.978 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:36.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:36.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:37.080 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:37.123 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:37.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:37.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:37.224 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:37.269 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:37.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:37.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:37 smithi008 ceph-mon[29816]: pgmap v442: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 650 B/s rd, 0 B/s wr, 1.01k op/s 2024-02-10T07:41:37.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:37 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3457939407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:41:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:37 smithi008 ceph-mon[40743]: pgmap v442: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 650 B/s rd, 0 B/s wr, 1.01k op/s 2024-02-10T07:41:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:37 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3457939407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:41:37.372 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:37 smithi170 ceph-mon[33394]: pgmap v442: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 650 B/s rd, 0 B/s wr, 1.01k op/s 2024-02-10T07:41:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:37 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3457939407' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:41:37.407 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:37.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:37.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:37.509 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:37.549 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:37.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:37.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:37.651 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:37.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:37.681 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:37.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:37.782 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:37.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:37.866 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:37.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:37.967 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:38.005 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:38.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:38.106 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:38.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:38.146 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:38.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:38.248 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:38.288 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:38.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:38.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:38.390 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:38.424 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:38.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:38.526 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:38.572 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:38.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:38.660 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:38.671 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:41:38.671 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:41:38.673 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:38.705 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:38.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:38.806 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:38.843 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:38.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:38.945 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:38.992 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:38.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:39.066 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:41:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:41:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:41:39.094 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:39.144 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:39.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:39.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:39.249 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:39.297 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:39.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:39.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:39.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:39 smithi008 ceph-mon[29816]: pgmap v443: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 668 op/s 2024-02-10T07:41:39.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:39 smithi008 ceph-mon[40743]: pgmap v443: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 668 op/s 2024-02-10T07:41:39.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:39 smithi170 ceph-mon[33394]: pgmap v443: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 668 op/s 2024-02-10T07:41:39.399 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:39.436 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:39.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:39.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:39.537 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:39.563 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:39.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:39.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:39.665 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:39.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:39.697 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:39.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:39.798 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:39.834 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:39.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:39.935 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:39.978 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:39.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:39.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:40.080 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:40.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:40.118 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:40.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:40.215 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:41:40.219 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:40.304 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:40.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:40.406 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:40.438 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:40.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:40.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:40.540 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:40.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:40.580 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:40.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:40.681 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:40.747 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-10T07:41:40.761 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:40.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:40.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:40.862 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:40.911 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:40.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:41.013 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:41.061 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:41.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:41.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:41.163 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:41.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:41.228 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:41.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:41.237 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:41 smithi008 ceph-mon[29816]: pgmap v444: 148 pgs: 148 active+clean; 824 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 997 op/s 2024-02-10T07:41:41.237 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:41 smithi008 ceph-mon[40743]: pgmap v444: 148 pgs: 148 active+clean; 824 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 997 op/s 2024-02-10T07:41:41.329 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:41.356 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:41.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:41.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:41.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:41 smithi170 ceph-mon[33394]: pgmap v444: 148 pgs: 148 active+clean; 824 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 997 op/s 2024-02-10T07:41:41.458 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:41.485 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:41.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:41.587 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:41.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:41.636 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:41.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:41.737 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:41.762 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:41.768 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:41.805 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:41.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:41.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:41.907 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:41.950 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:41.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:42.052 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:42.082 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:42.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:42.118 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:41:42.184 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:42.217 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:42.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:42.319 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:42.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:42 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1348975367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:41:42.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:42 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:41:42.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:42 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1348975367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:41:42.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:42 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:41:42.351 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:42.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:42.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:42 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1348975367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:41:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:42 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:41:42.452 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:42.492 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:42.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:42.594 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:42.626 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:42.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:42.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:42.728 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:42.766 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:42.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:42.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:42.867 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:42.899 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:42.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:43.000 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:43.046 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:43.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:43.146 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:43.189 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:43.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:43.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:43.259 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:43 smithi008 ceph-mon[29816]: pgmap v445: 148 pgs: 148 active+clean; 824 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 943 B/s rd, 0 B/s wr, 919 op/s 2024-02-10T07:41:43.259 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:43 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:41:43.260 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:43 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:41:43.260 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:43 smithi008 ceph-mon[29816]: osdmap e250: 8 total, 8 up, 8 in 2024-02-10T07:41:43.260 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:43 smithi008 ceph-mon[40743]: pgmap v445: 148 pgs: 148 active+clean; 824 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 943 B/s rd, 0 B/s wr, 919 op/s 2024-02-10T07:41:43.260 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:43 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:41:43.260 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:43 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:41:43.260 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:43 smithi008 ceph-mon[40743]: osdmap e250: 8 total, 8 up, 8 in 2024-02-10T07:41:43.290 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:43.319 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:43.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:43.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:41:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:41:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:43 smithi170 ceph-mon[33394]: pgmap v445: 148 pgs: 148 active+clean; 824 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 943 B/s rd, 0 B/s wr, 919 op/s 2024-02-10T07:41:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:43 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:41:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:43 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:41:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:43 smithi170 ceph-mon[33394]: osdmap e250: 8 total, 8 up, 8 in 2024-02-10T07:41:43.421 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:43.467 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:43.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:43.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:43.568 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:43.606 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:43.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:43.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:43.708 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:43.746 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:43.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:43.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:43.847 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:43.870 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:43.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:43.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:43.972 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:44.005 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:44.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:44.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:44.107 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:44.139 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:44.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:44.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:44.240 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:44.272 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:44.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:44.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:44.374 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:44 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:44.408 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:44.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:44.424 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:44 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:44.425 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:44 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:44.510 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:44.546 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:44.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:44.648 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:44.683 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:44.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:44.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:44.785 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:44.825 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:44.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:44.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:44.926 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:44.954 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:44.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:44.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:45.056 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:45.058 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:41:45.102 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:45.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:45.204 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:45.235 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:45.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:45.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:45.337 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:45.368 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:45.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:45.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:45 smithi170 ceph-mon[33394]: pgmap v447: 148 pgs: 148 active+clean; 548 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:41:45.470 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:45.514 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:45.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:45.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:45.520 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:45 smithi008 ceph-mon[29816]: pgmap v447: 148 pgs: 148 active+clean; 548 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:41:45.520 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:45 smithi008 ceph-mon[40743]: pgmap v447: 148 pgs: 148 active+clean; 548 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:41:45.616 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:45.646 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:45.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:45.748 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:45.780 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:45.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:45.882 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:45.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:45.915 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:45.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:46.016 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:46.040 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:46.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:46.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:46.141 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:46.179 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:46.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:46.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:46.281 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:46.318 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:46.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:46.419 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:46.454 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:46.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:46.556 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:46.588 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:46.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:46.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:46.689 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:46.712 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:46.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:46.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:46.814 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:46.848 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:46.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:46.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:46.950 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:46.976 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:46.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:46.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:47.078 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:47.113 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:47.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:47.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:47.214 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:47.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:47.238 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:47.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:47.339 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:47.372 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:47.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:47 smithi170 ceph-mon[33394]: pgmap v448: 148 pgs: 148 active+clean; 483 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-02-10T07:41:47.473 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:47.519 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:47.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:47.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:47 smithi008 ceph-mon[29816]: pgmap v448: 148 pgs: 148 active+clean; 483 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-02-10T07:41:47.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:47 smithi008 ceph-mon[40743]: pgmap v448: 148 pgs: 148 active+clean; 483 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.07k op/s 2024-02-10T07:41:47.622 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:47.661 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:47.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:47.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:47.762 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:47.800 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:47.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:47.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:47.901 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:47.930 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:47.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:47.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:48.031 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:48.068 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:48.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:48.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:48.170 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:48.193 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:48.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:48.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:48.295 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:48.331 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:48.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:48.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:48 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:41:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:48 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:48.425 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:48 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:41:48.425 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:48 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:48.426 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:48 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:41:48.426 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:48 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:41:48.433 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:48.469 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:48.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:48.571 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:48.602 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:48.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:48.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:48.704 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:48.743 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:48.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:48.845 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:48.890 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:48.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:48.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:48.992 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:49.021 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:49.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:49.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:49.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:41:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:41:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:41:49.123 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:49.149 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:49.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:49.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:49.251 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:49.281 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:49.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:49.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:49 smithi170 ceph-mon[33394]: pgmap v449: 148 pgs: 148 active+clean; 404 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s 2024-02-10T07:41:49.383 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:49.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:49.415 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:49.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:49.516 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:49.540 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:49 smithi008 ceph-mon[29816]: pgmap v449: 148 pgs: 148 active+clean; 404 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s 2024-02-10T07:41:49.540 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:49 smithi008 ceph-mon[40743]: pgmap v449: 148 pgs: 148 active+clean; 404 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s 2024-02-10T07:41:49.553 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:49.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:49.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:49.655 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:49.678 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:49.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:49.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:49.780 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:49.811 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:49.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:49.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:49.913 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:49.952 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:49.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:50.053 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:50.088 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:50.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:50.189 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:50.219 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:50.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:50.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:50.294 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.5 reset_purged_snaps_last 2024-02-10T07:41:50.321 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:50.353 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:50.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:50.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:50.455 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:50.498 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:50.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:50.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:50.499 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Removed 125084 objects 2024-02-10T07:41:50.499 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Clean up completed and total clean up time :113.846 2024-02-10T07:41:50.600 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:50.623 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:50.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:50.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:50.725 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:50.751 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:50.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:50.853 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:50.876 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:50.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:50.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:50.978 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:51.005 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:51.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:51.107 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:51.132 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:51.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:51.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:51.218 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:51.235 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:51.260 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:41:51.262 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:51.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:51.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:51.280 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:41:51.280 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:51 smithi008 ceph-mon[29816]: pgmap v450: 148 pgs: 148 active+clean; 161 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:41:51.280 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:51 smithi008 ceph-mon[40743]: pgmap v450: 148 pgs: 148 active+clean; 161 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:41:51.299 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:41:51.319 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:41:51.340 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:41:51.360 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:41:51.364 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:51.382 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:41:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:51 smithi170 ceph-mon[33394]: pgmap v450: 148 pgs: 148 active+clean; 161 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:41:51.389 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:51.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:51.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:51.404 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:41:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:41:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:41:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:41:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:41:51.405 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:41:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:41:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:41:51.406 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:41:51.490 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:51.517 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:51.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:51.618 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:51.645 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:51.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:51.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:51.747 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:51.770 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:51.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:51.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:51.872 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:51.899 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:51.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:51.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:52.001 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:52.024 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:52.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:52.126 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:52.153 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:52.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:52.255 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:52.278 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:52.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:52.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:52.380 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:52 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2177759917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:41:52.402 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:52 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2177759917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:41:52.402 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:52 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2177759917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:41:52.405 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:52.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:52.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:52.507 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:52.533 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:52.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:52.634 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:52.659 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:52.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:52.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:52.761 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:52.784 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:52.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:52.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:52.886 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:52.909 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:52.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:52.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:53.011 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:53.034 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:53.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:53.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:53.136 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:53.164 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:53.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:53.265 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:53.289 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:53.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:53.377 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:53 smithi008 ceph-mon[29816]: pgmap v451: 148 pgs: 148 active+clean; 161 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:41:53.377 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:53 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:53.377 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:53 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:41:53.377 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:53 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:41:53.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:41:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:41:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:53 smithi170 ceph-mon[33394]: pgmap v451: 148 pgs: 148 active+clean; 161 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:41:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:53 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:53 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:41:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:53 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:41:53.392 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:53.396 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:53 smithi008 ceph-mon[40743]: pgmap v451: 148 pgs: 148 active+clean; 161 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:41:53.396 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:53 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:41:53.396 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:53 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:41:53.396 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:53 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:41:53.420 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:53.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:53.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:53.522 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:53.546 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:53.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:53.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:53.648 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:53.671 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:53.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:53.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:53.773 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:53.801 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:53.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:53.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:53.802 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:41:53.802 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:41:53.802 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-02-10T07:41:53.802 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:41:53.903 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:53.926 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:53.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:54.028 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:54.031 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-02-10T07:41:54.031 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-02-10T07:41:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:54.052 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:54.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:54.153 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:54.176 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:54.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:54.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:54.278 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:54.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:54.308 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:54.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:54.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:54 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:54.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:54 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:54.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:54 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:41:54.409 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:54.433 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:54.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:54.434 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:54.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:54.535 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:54.563 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:54.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:54.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:54.630 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:41:54.664 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:54.691 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:54.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:54.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:54.793 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:54.820 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:54.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:54.862 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.6 reset_purged_snaps_last 2024-02-10T07:41:54.922 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:54.949 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:54.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:55.051 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:55.078 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:55.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:55.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:55.180 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:55.208 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:55.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:55.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:55.309 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:55.333 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:55.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:55.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:55 smithi170 ceph-mon[33394]: pgmap v452: 148 pgs: 148 active+clean; 21 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1k op/s 2024-02-10T07:41:55.407 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:55 smithi008 ceph-mon[29816]: pgmap v452: 148 pgs: 148 active+clean; 21 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1k op/s 2024-02-10T07:41:55.407 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:55 smithi008 ceph-mon[40743]: pgmap v452: 148 pgs: 148 active+clean; 21 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1k op/s 2024-02-10T07:41:55.417 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:55.418 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:55.435 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:55.448 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-10T07:41:55.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:55.464 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:55.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:55.565 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:55.588 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:55.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:55.690 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:55.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:55.714 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:55.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:55.815 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:55.843 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:55.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:55.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:55.944 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:55.972 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:55.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:55.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:56.074 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:56.099 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:56.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:56.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:56.143 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:41:56.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:56.229 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:56.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:56.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:56.330 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:56.354 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:56.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:56.456 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:56.479 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:56.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:56.581 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:56.604 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:56.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:56.706 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:56.734 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:56.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:56.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:56.836 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:56.863 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:56.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:56.964 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:56.989 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:56.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:56.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:57.091 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:57.116 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:57.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:57.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:57.218 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:57.242 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:57.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:57.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:57.343 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:57.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:57.369 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:57.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:57.469 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:57.497 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:57.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:57.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:57.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:57 smithi008 ceph-mon[29816]: pgmap v453: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 730 op/s 2024-02-10T07:41:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:57 smithi008 ceph-mon[40743]: pgmap v453: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 730 op/s 2024-02-10T07:41:57.599 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:41:57.623 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:57.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:57.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:41:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:57 smithi170 ceph-mon[33394]: pgmap v453: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 730 op/s 2024-02-10T07:41:57.724 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:57.748 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:57.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:57.849 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:57.877 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:57.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:57.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:57.979 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:58.007 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:58.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:58.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:58.109 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:58.137 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:58.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:58.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:58.239 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:58.261 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:58.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:58.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:58.363 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:58.391 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:58.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:58.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:58.493 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:58.517 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:58.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:58.618 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:58.647 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:58.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:58.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:58.749 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:41:58.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:58.779 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:58.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:41:58.880 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:41:58.909 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:58.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:58.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:41:59.011 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:59.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:59.035 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:59.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:59.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:41:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:41:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:41:59.136 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:59.159 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:59.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:59.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:59.261 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:59.285 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:59.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:59.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:41:59.362 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:41:59 smithi008 ceph-mon[29816]: pgmap v454: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 643 op/s 2024-02-10T07:41:59.363 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:41:59 smithi008 ceph-mon[40743]: pgmap v454: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 643 op/s 2024-02-10T07:41:59.387 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:41:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:59.412 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:59.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:41:59.434 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:41:59 smithi170 ceph-mon[33394]: pgmap v454: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 643 op/s 2024-02-10T07:41:59.513 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:41:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:59.537 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:59.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:41:59.638 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:41:59.661 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:59.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:59.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:41:59.762 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:41:59.788 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:41:59.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:59.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:41:59.890 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:41:59.913 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:41:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:41:59.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:00.015 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:00.038 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:00.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:00.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:00.140 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:00.163 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:00.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:00.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:00.265 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:00.291 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:00.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:00.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:00.393 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:00.418 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:00.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:00.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:00.520 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:00.544 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:00.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:00.642 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:00.645 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:00.669 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:00.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:00.771 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:00.795 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:00.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:00.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:00.896 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:00.920 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:00.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:00.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:01.021 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:01.044 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:01.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:01.146 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:01.161 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'unique_pool_1' removed 2024-02-10T07:42:01.174 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:01.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:01.179 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:42:01.276 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:01.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:01.303 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:01.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:01.404 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:01.416 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:01 smithi008 ceph-mon[29816]: pgmap v455: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 0 B/s wr, 549 op/s 2024-02-10T07:42:01.416 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:01 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2258569346' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:42:01.416 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:01 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2548684640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:42:01.416 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:01 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:42:01.417 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:01 smithi008 ceph-mon[40743]: pgmap v455: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 0 B/s wr, 549 op/s 2024-02-10T07:42:01.417 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:01 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2258569346' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:42:01.417 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:01 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2548684640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:42:01.417 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:01 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:42:01.429 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:01.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:01.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:01.531 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:01.569 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:01.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:01 smithi170 ceph-mon[33394]: pgmap v455: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 0 B/s wr, 549 op/s 2024-02-10T07:42:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:01 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2258569346' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:42:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:01 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2548684640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:42:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:01 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:42:01.671 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:01.695 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:01.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:01.796 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:01.821 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:01.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:01.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:01.923 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:01.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:01.947 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:01.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:02.048 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:02.073 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:02.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:02.175 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:02.199 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:02.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:02.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:02.301 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:02.315 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:02 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:42:02.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:02 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2258569346' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:42:02.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:02 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:42:02.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:02 smithi008 ceph-mon[29816]: osdmap e251: 8 total, 8 up, 8 in 2024-02-10T07:42:02.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:02 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:42:02.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:02 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:42:02.316 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:02 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:42:02.316 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:02 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2258569346' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:42:02.316 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:02 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:42:02.316 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:02 smithi008 ceph-mon[40743]: osdmap e251: 8 total, 8 up, 8 in 2024-02-10T07:42:02.316 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:02 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:42:02.317 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:02 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:42:02.325 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:02.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:02.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:02.427 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:02.450 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:02.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:02.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:02.552 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:02.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:02.576 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:02.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:02 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:42:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:02 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2258569346' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:42:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:02 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:42:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:02 smithi170 ceph-mon[33394]: osdmap e251: 8 total, 8 up, 8 in 2024-02-10T07:42:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:02 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:42:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:02 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:42:02.677 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:02.701 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:02.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:02.802 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:02.827 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:02.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:02.878 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-10T07:42:02.879 INFO:tasks.radosbench:Beginning radosbench... 2024-02-10T07:42:02.879 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-02-10T07:42:02.879 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-02-10T07:42:02.928 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:02.952 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:02.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:02.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:03.054 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:03.078 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:03.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:03.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:03.134 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:42:03.157 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:42:03.176 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:42:03.179 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:03.196 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:42:03.202 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:03.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:03.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:03.216 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:42:03.236 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:42:03.257 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:42:03.257 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[29816]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 9 op/s 2024-02-10T07:42:03.257 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[29816]: 6.5 scrub starts 2024-02-10T07:42:03.257 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[29816]: 6.5 scrub ok 2024-02-10T07:42:03.257 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[29816]: osdmap e252: 8 total, 8 up, 8 in 2024-02-10T07:42:03.257 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:03.258 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[29816]: 4.1c scrub starts 2024-02-10T07:42:03.258 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[29816]: 4.1c scrub ok 2024-02-10T07:42:03.258 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2925028998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:42:03.258 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[40743]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 9 op/s 2024-02-10T07:42:03.258 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[40743]: 6.5 scrub starts 2024-02-10T07:42:03.258 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[40743]: 6.5 scrub ok 2024-02-10T07:42:03.258 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[40743]: osdmap e252: 8 total, 8 up, 8 in 2024-02-10T07:42:03.258 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:03.258 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[40743]: 4.1c scrub starts 2024-02-10T07:42:03.258 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[40743]: 4.1c scrub ok 2024-02-10T07:42:03.258 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:03 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2925028998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:42:03.277 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:42:03.278 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:42:03.279 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:42:03.279 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:42:03.279 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:42:03.279 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:42:03.279 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:42:03.279 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:42:03.279 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:42:03.304 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:03.328 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:03.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:03.429 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:03.455 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:03.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:03.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:03.557 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:03.582 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:03.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:03.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:03.625 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:42:03.625 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:42:03.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:42:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:42:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:03 smithi170 ceph-mon[33394]: pgmap v457: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 9 op/s 2024-02-10T07:42:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:03 smithi170 ceph-mon[33394]: 6.5 scrub starts 2024-02-10T07:42:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:03 smithi170 ceph-mon[33394]: 6.5 scrub ok 2024-02-10T07:42:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:03 smithi170 ceph-mon[33394]: osdmap e252: 8 total, 8 up, 8 in 2024-02-10T07:42:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:03 smithi170 ceph-mon[33394]: 4.1c scrub starts 2024-02-10T07:42:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:03 smithi170 ceph-mon[33394]: 4.1c scrub ok 2024-02-10T07:42:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:03 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2925028998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:42:03.685 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:03.708 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:03.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:03.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:03.810 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:03.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:03.837 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:03.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:03.938 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:03.961 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:03.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:03.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:04.039 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:42:04.063 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:04.104 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:04.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:04.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:04.206 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:04.232 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:04.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:04.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:04 smithi008 ceph-mon[40743]: 3.3 scrub starts 2024-02-10T07:42:04.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:04 smithi008 ceph-mon[40743]: 3.3 scrub ok 2024-02-10T07:42:04.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:04 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:04.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:04 smithi008 ceph-mon[40743]: 5.8 scrub starts 2024-02-10T07:42:04.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:04 smithi008 ceph-mon[40743]: 5.8 scrub ok 2024-02-10T07:42:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:04 smithi008 ceph-mon[29816]: 3.3 scrub starts 2024-02-10T07:42:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:04 smithi008 ceph-mon[29816]: 3.3 scrub ok 2024-02-10T07:42:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:04 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:04 smithi008 ceph-mon[29816]: 5.8 scrub starts 2024-02-10T07:42:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:04 smithi008 ceph-mon[29816]: 5.8 scrub ok 2024-02-10T07:42:04.334 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:04.357 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:04.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:04.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:04.459 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:04.470 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:04.485 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:04.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:04.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:04.587 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:04.610 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:04.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:04.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:04 smithi170 ceph-mon[33394]: 3.3 scrub starts 2024-02-10T07:42:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:04 smithi170 ceph-mon[33394]: 3.3 scrub ok 2024-02-10T07:42:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:04 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:04 smithi170 ceph-mon[33394]: 5.8 scrub starts 2024-02-10T07:42:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:04 smithi170 ceph-mon[33394]: 5.8 scrub ok 2024-02-10T07:42:04.711 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:04.743 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:04.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:04.845 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:04.873 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:04.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:04.974 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:04.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:04.999 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:04.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:05.100 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:05.128 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:05.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:05.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:05.230 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:05 smithi008 ceph-mon[29816]: pgmap v459: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 54 op/s 2024-02-10T07:42:05.230 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:05 smithi008 ceph-mon[29816]: 6.3 scrub starts 2024-02-10T07:42:05.230 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:05 smithi008 ceph-mon[29816]: 6.3 scrub ok 2024-02-10T07:42:05.230 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:05 smithi008 ceph-mon[29816]: 3.7 scrub starts 2024-02-10T07:42:05.230 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:05 smithi008 ceph-mon[29816]: 3.7 scrub ok 2024-02-10T07:42:05.230 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:05 smithi008 ceph-mon[29816]: 5.1f scrub starts 2024-02-10T07:42:05.230 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:05 smithi008 ceph-mon[29816]: 5.1f scrub ok 2024-02-10T07:42:05.230 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:05 smithi008 ceph-mon[40743]: pgmap v459: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 54 op/s 2024-02-10T07:42:05.230 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:05 smithi008 ceph-mon[40743]: 6.3 scrub starts 2024-02-10T07:42:05.230 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:05 smithi008 ceph-mon[40743]: 6.3 scrub ok 2024-02-10T07:42:05.230 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:05 smithi008 ceph-mon[40743]: 3.7 scrub starts 2024-02-10T07:42:05.231 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:05 smithi008 ceph-mon[40743]: 3.7 scrub ok 2024-02-10T07:42:05.231 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:05 smithi008 ceph-mon[40743]: 5.1f scrub starts 2024-02-10T07:42:05.231 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:05 smithi008 ceph-mon[40743]: 5.1f scrub ok 2024-02-10T07:42:05.231 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:05.257 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:05.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:05.358 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:05.382 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:05.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:05.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:05.484 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:05.508 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:05.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:05.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:05.609 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:05 smithi170 ceph-mon[33394]: pgmap v459: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 54 op/s 2024-02-10T07:42:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:05 smithi170 ceph-mon[33394]: 6.3 scrub starts 2024-02-10T07:42:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:05 smithi170 ceph-mon[33394]: 6.3 scrub ok 2024-02-10T07:42:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:05 smithi170 ceph-mon[33394]: 3.7 scrub starts 2024-02-10T07:42:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:05 smithi170 ceph-mon[33394]: 3.7 scrub ok 2024-02-10T07:42:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:05 smithi170 ceph-mon[33394]: 5.1f scrub starts 2024-02-10T07:42:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:05 smithi170 ceph-mon[33394]: 5.1f scrub ok 2024-02-10T07:42:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:05.634 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:05.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:05.735 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:05.759 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:05.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:05.861 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:05.889 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:05.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:05.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:05.990 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:06.015 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:06.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:06.116 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:06.144 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:06.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:06.246 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:06.273 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:06.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:06.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:06.315 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:06 smithi008 ceph-mon[40743]: 2.0 scrub starts 2024-02-10T07:42:06.315 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:06 smithi008 ceph-mon[40743]: 2.0 scrub ok 2024-02-10T07:42:06.315 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:06 smithi008 ceph-mon[40743]: 5.19 scrub starts 2024-02-10T07:42:06.315 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:06 smithi008 ceph-mon[40743]: 5.19 scrub ok 2024-02-10T07:42:06.315 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:06 smithi008 ceph-mon[40743]: 4.b scrub starts 2024-02-10T07:42:06.316 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:06 smithi008 ceph-mon[40743]: 4.b scrub ok 2024-02-10T07:42:06.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:06 smithi008 ceph-mon[29816]: 2.0 scrub starts 2024-02-10T07:42:06.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:06 smithi008 ceph-mon[29816]: 2.0 scrub ok 2024-02-10T07:42:06.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:06 smithi008 ceph-mon[29816]: 5.19 scrub starts 2024-02-10T07:42:06.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:06 smithi008 ceph-mon[29816]: 5.19 scrub ok 2024-02-10T07:42:06.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:06 smithi008 ceph-mon[29816]: 4.b scrub starts 2024-02-10T07:42:06.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:06 smithi008 ceph-mon[29816]: 4.b scrub ok 2024-02-10T07:42:06.376 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:06.400 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:06.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:06.502 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:06.530 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:06.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:06.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:06.632 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:06 smithi170 ceph-mon[33394]: 2.0 scrub starts 2024-02-10T07:42:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:06 smithi170 ceph-mon[33394]: 2.0 scrub ok 2024-02-10T07:42:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:06 smithi170 ceph-mon[33394]: 5.19 scrub starts 2024-02-10T07:42:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:06 smithi170 ceph-mon[33394]: 5.19 scrub ok 2024-02-10T07:42:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:06 smithi170 ceph-mon[33394]: 4.b scrub starts 2024-02-10T07:42:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:06 smithi170 ceph-mon[33394]: 4.b scrub ok 2024-02-10T07:42:06.656 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:06.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:06.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:06.741 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-10T07:42:06.759 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:06.786 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:06.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:06.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:06.889 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:06.917 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:06.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:06.947 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:42:07.019 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:07.046 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:07.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:07.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:07.148 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:07.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:07.177 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:07.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:07.243 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:07 smithi008 ceph-mon[29816]: pgmap v460: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 54 KiB/s rd, 54 op/s 2024-02-10T07:42:07.243 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:07 smithi008 ceph-mon[29816]: 4.1a scrub starts 2024-02-10T07:42:07.244 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:07 smithi008 ceph-mon[29816]: 4.1a scrub ok 2024-02-10T07:42:07.244 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:07 smithi008 ceph-mon[29816]: 6.1 scrub starts 2024-02-10T07:42:07.244 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:07 smithi008 ceph-mon[29816]: 6.1 scrub ok 2024-02-10T07:42:07.244 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:07 smithi008 ceph-mon[29816]: 5.b scrub starts 2024-02-10T07:42:07.244 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:07 smithi008 ceph-mon[29816]: 5.b scrub ok 2024-02-10T07:42:07.244 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:07 smithi008 ceph-mon[40743]: pgmap v460: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 54 KiB/s rd, 54 op/s 2024-02-10T07:42:07.244 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:07 smithi008 ceph-mon[40743]: 4.1a scrub starts 2024-02-10T07:42:07.244 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:07 smithi008 ceph-mon[40743]: 4.1a scrub ok 2024-02-10T07:42:07.244 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:07 smithi008 ceph-mon[40743]: 6.1 scrub starts 2024-02-10T07:42:07.244 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:07 smithi008 ceph-mon[40743]: 6.1 scrub ok 2024-02-10T07:42:07.244 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:07 smithi008 ceph-mon[40743]: 5.b scrub starts 2024-02-10T07:42:07.244 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:07 smithi008 ceph-mon[40743]: 5.b scrub ok 2024-02-10T07:42:07.278 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:07.301 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:07.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:07.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:07.345 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:07.403 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:07.431 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:07.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:07.532 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:07.560 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:07.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:07 smithi170 ceph-mon[33394]: pgmap v460: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 54 KiB/s rd, 54 op/s 2024-02-10T07:42:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:07 smithi170 ceph-mon[33394]: 4.1a scrub starts 2024-02-10T07:42:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:07 smithi170 ceph-mon[33394]: 4.1a scrub ok 2024-02-10T07:42:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:07 smithi170 ceph-mon[33394]: 6.1 scrub starts 2024-02-10T07:42:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:07 smithi170 ceph-mon[33394]: 6.1 scrub ok 2024-02-10T07:42:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:07 smithi170 ceph-mon[33394]: 5.b scrub starts 2024-02-10T07:42:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:07 smithi170 ceph-mon[33394]: 5.b scrub ok 2024-02-10T07:42:07.662 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:07.686 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:07.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:07.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:07.814 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:07.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:07.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:07.916 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:07.939 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:07.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:07.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:08.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:08.069 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:08.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:08.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:08.171 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:08.172 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'unique_pool_2' created 2024-02-10T07:42:08.196 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:08.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:08.298 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:08.326 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:08.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:08.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:08.428 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:08.452 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:08.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:08.553 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:08.578 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:08.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:08.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[40743]: 5.16 scrub starts 2024-02-10T07:42:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[40743]: 5.16 scrub ok 2024-02-10T07:42:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[40743]: 6.1d scrub starts 2024-02-10T07:42:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[40743]: 6.1d scrub ok 2024-02-10T07:42:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[40743]: 3.1b scrub starts 2024-02-10T07:42:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[40743]: 3.1b scrub ok 2024-02-10T07:42:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1061414764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-10T07:42:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-10T07:42:08.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[29816]: 5.16 scrub starts 2024-02-10T07:42:08.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[29816]: 5.16 scrub ok 2024-02-10T07:42:08.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[29816]: 6.1d scrub starts 2024-02-10T07:42:08.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[29816]: 6.1d scrub ok 2024-02-10T07:42:08.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[29816]: 3.1b scrub starts 2024-02-10T07:42:08.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[29816]: 3.1b scrub ok 2024-02-10T07:42:08.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1061414764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-10T07:42:08.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-10T07:42:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:08 smithi170 ceph-mon[33394]: 5.16 scrub starts 2024-02-10T07:42:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:08 smithi170 ceph-mon[33394]: 5.16 scrub ok 2024-02-10T07:42:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:08 smithi170 ceph-mon[33394]: 6.1d scrub starts 2024-02-10T07:42:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:08 smithi170 ceph-mon[33394]: 6.1d scrub ok 2024-02-10T07:42:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:08 smithi170 ceph-mon[33394]: 3.1b scrub starts 2024-02-10T07:42:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:08 smithi170 ceph-mon[33394]: 3.1b scrub ok 2024-02-10T07:42:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:08 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1061414764' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-10T07:42:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:08 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-10T07:42:08.680 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:08.708 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:08.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:08.886 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:08.911 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:08.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:08.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:09.013 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:09.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:09.037 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:09.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:09.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:42:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:42:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:42:09.139 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:09.169 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:09.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:09.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:09.270 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:09.294 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:09.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:09.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:09.396 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:09.423 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:09.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:09.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:09.525 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:09.550 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:09.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:09.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:09.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:09 smithi008 ceph-mon[40743]: pgmap v461: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 44 op/s 2024-02-10T07:42:09.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:09 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-10T07:42:09.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:09 smithi008 ceph-mon[40743]: osdmap e253: 8 total, 8 up, 8 in 2024-02-10T07:42:09.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:09 smithi008 ceph-mon[40743]: 5.11 scrub starts 2024-02-10T07:42:09.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:09 smithi008 ceph-mon[40743]: 5.11 scrub ok 2024-02-10T07:42:09.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:09 smithi008 ceph-mon[29816]: pgmap v461: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 44 op/s 2024-02-10T07:42:09.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:09 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-10T07:42:09.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:09 smithi008 ceph-mon[29816]: osdmap e253: 8 total, 8 up, 8 in 2024-02-10T07:42:09.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:09 smithi008 ceph-mon[29816]: 5.11 scrub starts 2024-02-10T07:42:09.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:09 smithi008 ceph-mon[29816]: 5.11 scrub ok 2024-02-10T07:42:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:09 smithi170 ceph-mon[33394]: pgmap v461: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 44 op/s 2024-02-10T07:42:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:09 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-10T07:42:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:09 smithi170 ceph-mon[33394]: osdmap e253: 8 total, 8 up, 8 in 2024-02-10T07:42:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:09 smithi170 ceph-mon[33394]: 5.11 scrub starts 2024-02-10T07:42:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:09 smithi170 ceph-mon[33394]: 5.11 scrub ok 2024-02-10T07:42:09.651 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:09.679 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:09.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:09.781 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:09.808 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:09.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:09.910 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:09.936 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:09.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:09.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:10.038 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:10.064 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:10.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:10.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:10.193 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:10.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:10.294 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:10.321 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:10.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:10.332 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.7 reset_purged_snaps_last 2024-02-10T07:42:10.423 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:10.449 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:10.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:10.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:10.551 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:10.575 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:10.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:10.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:10 smithi008 ceph-mon[29816]: osdmap e254: 8 total, 8 up, 8 in 2024-02-10T07:42:10.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:10 smithi008 ceph-mon[29816]: 3.18 scrub starts 2024-02-10T07:42:10.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:10 smithi008 ceph-mon[29816]: 3.18 scrub ok 2024-02-10T07:42:10.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:10 smithi008 ceph-mon[40743]: osdmap e254: 8 total, 8 up, 8 in 2024-02-10T07:42:10.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:10 smithi008 ceph-mon[40743]: 3.18 scrub starts 2024-02-10T07:42:10.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:10 smithi008 ceph-mon[40743]: 3.18 scrub ok 2024-02-10T07:42:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:10 smithi170 ceph-mon[33394]: osdmap e254: 8 total, 8 up, 8 in 2024-02-10T07:42:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:10 smithi170 ceph-mon[33394]: 3.18 scrub starts 2024-02-10T07:42:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:10 smithi170 ceph-mon[33394]: 3.18 scrub ok 2024-02-10T07:42:10.676 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:10.703 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:10.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:10.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:10.805 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:10.828 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:10.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:10.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:10.865 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-02-10T07:42:10.930 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:10.957 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:10.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:11.059 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:11.084 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:11.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:11.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:11.112 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:11.112 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:11.186 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:11.211 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:11.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:11.312 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:11.340 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:11.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:11.427 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[40743]: pgmap v464: 148 pgs: 9 creating+peering, 7 unknown, 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-02-10T07:42:11.427 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[40743]: 3.e scrub starts 2024-02-10T07:42:11.428 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[40743]: 3.e scrub ok 2024-02-10T07:42:11.428 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[40743]: 4.c scrub starts 2024-02-10T07:42:11.428 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[40743]: 4.c scrub ok 2024-02-10T07:42:11.428 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[40743]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:42:11.428 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[40743]: osdmap e255: 8 total, 8 up, 8 in 2024-02-10T07:42:11.428 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[40743]: 6.0 scrub starts 2024-02-10T07:42:11.428 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[40743]: 6.0 scrub ok 2024-02-10T07:42:11.428 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[40743]: 5.3 scrub starts 2024-02-10T07:42:11.428 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[40743]: 5.3 scrub ok 2024-02-10T07:42:11.428 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[29816]: pgmap v464: 148 pgs: 9 creating+peering, 7 unknown, 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-02-10T07:42:11.428 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[29816]: 3.e scrub starts 2024-02-10T07:42:11.429 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[29816]: 3.e scrub ok 2024-02-10T07:42:11.429 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[29816]: 4.c scrub starts 2024-02-10T07:42:11.429 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[29816]: 4.c scrub ok 2024-02-10T07:42:11.429 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[29816]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:42:11.429 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[29816]: osdmap e255: 8 total, 8 up, 8 in 2024-02-10T07:42:11.429 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[29816]: 6.0 scrub starts 2024-02-10T07:42:11.429 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[29816]: 6.0 scrub ok 2024-02-10T07:42:11.429 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[29816]: 5.3 scrub starts 2024-02-10T07:42:11.429 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:11 smithi008 ceph-mon[29816]: 5.3 scrub ok 2024-02-10T07:42:11.441 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:11.468 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:11.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:11.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:11.569 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:11.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:11.597 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:11.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:11 smithi170 ceph-mon[33394]: pgmap v464: 148 pgs: 9 creating+peering, 7 unknown, 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 66 op/s 2024-02-10T07:42:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:11 smithi170 ceph-mon[33394]: 3.e scrub starts 2024-02-10T07:42:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:11 smithi170 ceph-mon[33394]: 3.e scrub ok 2024-02-10T07:42:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:11 smithi170 ceph-mon[33394]: 4.c scrub starts 2024-02-10T07:42:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:11 smithi170 ceph-mon[33394]: 4.c scrub ok 2024-02-10T07:42:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:11 smithi170 ceph-mon[33394]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:42:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:11 smithi170 ceph-mon[33394]: osdmap e255: 8 total, 8 up, 8 in 2024-02-10T07:42:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:11 smithi170 ceph-mon[33394]: 6.0 scrub starts 2024-02-10T07:42:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:11 smithi170 ceph-mon[33394]: 6.0 scrub ok 2024-02-10T07:42:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:11 smithi170 ceph-mon[33394]: 5.3 scrub starts 2024-02-10T07:42:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:11 smithi170 ceph-mon[33394]: 5.3 scrub ok 2024-02-10T07:42:11.698 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:11.723 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:11.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:11.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:11.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:11.826 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:11.856 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:11.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:11.957 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:11.982 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:11.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:12.084 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:12.109 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:12.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:12.184 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:42:12.211 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:12.237 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:12.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:12.313 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:12 smithi008 ceph-mon[29816]: 4.e scrub starts 2024-02-10T07:42:12.313 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:12 smithi008 ceph-mon[29816]: 4.e scrub ok 2024-02-10T07:42:12.313 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:12 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4042287923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:42:12.313 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:12 smithi008 ceph-mon[29816]: 3.4 scrub starts 2024-02-10T07:42:12.313 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:12 smithi008 ceph-mon[29816]: 3.4 scrub ok 2024-02-10T07:42:12.313 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:12 smithi008 ceph-mon[40743]: 4.e scrub starts 2024-02-10T07:42:12.313 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:12 smithi008 ceph-mon[40743]: 4.e scrub ok 2024-02-10T07:42:12.313 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:12 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4042287923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:42:12.313 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:12 smithi008 ceph-mon[40743]: 3.4 scrub starts 2024-02-10T07:42:12.313 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:12 smithi008 ceph-mon[40743]: 3.4 scrub ok 2024-02-10T07:42:12.339 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:12.361 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:12.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:12.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:12.463 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:12.489 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:12.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:12.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:12.590 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:12.614 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:12.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:12.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:12 smithi170 ceph-mon[33394]: 4.e scrub starts 2024-02-10T07:42:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:12 smithi170 ceph-mon[33394]: 4.e scrub ok 2024-02-10T07:42:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:12 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4042287923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:42:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:12 smithi170 ceph-mon[33394]: 3.4 scrub starts 2024-02-10T07:42:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:12 smithi170 ceph-mon[33394]: 3.4 scrub ok 2024-02-10T07:42:12.716 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:12.739 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:12.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:12.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:12.841 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:12.869 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:12.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:12.971 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:12.996 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:12.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:13.098 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:13.122 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:13.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:13.223 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:13.247 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:13.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:13.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:13.326 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[29816]: pgmap v466: 148 pgs: 9 creating+peering, 7 unknown, 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:13.326 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:42:13.326 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4042287923' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:42:13.326 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[29816]: osdmap e256: 8 total, 8 up, 8 in 2024-02-10T07:42:13.326 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[29816]: 5.c scrub starts 2024-02-10T07:42:13.326 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[29816]: 5.c scrub ok 2024-02-10T07:42:13.326 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[29816]: 6.10 scrub starts 2024-02-10T07:42:13.326 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[29816]: 6.10 scrub ok 2024-02-10T07:42:13.326 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[40743]: pgmap v466: 148 pgs: 9 creating+peering, 7 unknown, 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:13.326 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:42:13.326 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4042287923' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:42:13.327 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[40743]: osdmap e256: 8 total, 8 up, 8 in 2024-02-10T07:42:13.327 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[40743]: 5.c scrub starts 2024-02-10T07:42:13.327 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[40743]: 5.c scrub ok 2024-02-10T07:42:13.327 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[40743]: 6.10 scrub starts 2024-02-10T07:42:13.327 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:13 smithi008 ceph-mon[40743]: 6.10 scrub ok 2024-02-10T07:42:13.349 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:13.373 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:13.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:13.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:13.474 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:13.502 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:13.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:13.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:13.604 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:13.629 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:13.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:13.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:42:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:42:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:13 smithi170 ceph-mon[33394]: pgmap v466: 148 pgs: 9 creating+peering, 7 unknown, 132 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:42:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:13 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:42:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:13 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4042287923' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:42:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:13 smithi170 ceph-mon[33394]: osdmap e256: 8 total, 8 up, 8 in 2024-02-10T07:42:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:13 smithi170 ceph-mon[33394]: 5.c scrub starts 2024-02-10T07:42:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:13 smithi170 ceph-mon[33394]: 5.c scrub ok 2024-02-10T07:42:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:13 smithi170 ceph-mon[33394]: 6.10 scrub starts 2024-02-10T07:42:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:13 smithi170 ceph-mon[33394]: 6.10 scrub ok 2024-02-10T07:42:13.730 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:13.758 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:13.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:13.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:13.860 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:13.885 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:13.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:13.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:13.987 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:14.015 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:14.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:14.116 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:14.140 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:14.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:14.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:14.242 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:14.270 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:14.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:14.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:14.372 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:14.396 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:14.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:14.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:14.498 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:14.521 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:14.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:14.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:14.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:14 smithi008 ceph-mon[29816]: 5.4 scrub starts 2024-02-10T07:42:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:14 smithi008 ceph-mon[29816]: 5.4 scrub ok 2024-02-10T07:42:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:14 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:14.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:14 smithi008 ceph-mon[40743]: 5.4 scrub starts 2024-02-10T07:42:14.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:14 smithi008 ceph-mon[40743]: 5.4 scrub ok 2024-02-10T07:42:14.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:14 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:14.624 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:14.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:14 smithi170 ceph-mon[33394]: 5.4 scrub starts 2024-02-10T07:42:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:14 smithi170 ceph-mon[33394]: 5.4 scrub ok 2024-02-10T07:42:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:14 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:14.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:14.652 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:14.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:14.753 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:14.776 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:14.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:14.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:14.880 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:14.908 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:14.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:14.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:15.009 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:15.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:15.034 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:15.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:15.135 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:15.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:15.161 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:15.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:15.262 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:15.290 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:15.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:15 smithi008 ceph-mon[29816]: pgmap v468: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-02-10T07:42:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:15 smithi008 ceph-mon[29816]: 4.5 scrub starts 2024-02-10T07:42:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:15 smithi008 ceph-mon[29816]: 4.5 scrub ok 2024-02-10T07:42:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:15 smithi008 ceph-mon[29816]: 6.16 scrub starts 2024-02-10T07:42:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:15 smithi008 ceph-mon[29816]: 6.16 scrub ok 2024-02-10T07:42:15.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:15 smithi008 ceph-mon[40743]: pgmap v468: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-02-10T07:42:15.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:15 smithi008 ceph-mon[40743]: 4.5 scrub starts 2024-02-10T07:42:15.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:15 smithi008 ceph-mon[40743]: 4.5 scrub ok 2024-02-10T07:42:15.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:15 smithi008 ceph-mon[40743]: 6.16 scrub starts 2024-02-10T07:42:15.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:15 smithi008 ceph-mon[40743]: 6.16 scrub ok 2024-02-10T07:42:15.392 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:15.416 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:15.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:15.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:15.517 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:15.541 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:15.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:15.596 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:15 smithi170 ceph-mon[33394]: pgmap v468: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-02-10T07:42:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:15 smithi170 ceph-mon[33394]: 4.5 scrub starts 2024-02-10T07:42:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:15 smithi170 ceph-mon[33394]: 4.5 scrub ok 2024-02-10T07:42:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:15 smithi170 ceph-mon[33394]: 6.16 scrub starts 2024-02-10T07:42:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:15 smithi170 ceph-mon[33394]: 6.16 scrub ok 2024-02-10T07:42:15.643 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:15.672 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:15.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:15.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:15.773 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:15.800 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:15.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:15.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:15.902 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:15.925 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:15.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:15.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:16.027 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:16.050 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:16.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:16.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:16.115 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:42:16.152 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:16.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:16.182 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:16.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:16.283 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:16.307 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:16.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:16.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:16.400 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:16 smithi008 ceph-mon[29816]: 5.15 scrub starts 2024-02-10T07:42:16.400 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:16 smithi008 ceph-mon[29816]: 5.15 scrub ok 2024-02-10T07:42:16.400 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:16 smithi008 ceph-mon[29816]: 3.10 scrub starts 2024-02-10T07:42:16.400 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:16 smithi008 ceph-mon[29816]: 3.10 scrub ok 2024-02-10T07:42:16.400 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:16 smithi008 ceph-mon[29816]: 5.7 scrub starts 2024-02-10T07:42:16.401 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:16 smithi008 ceph-mon[40743]: 5.15 scrub starts 2024-02-10T07:42:16.401 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:16 smithi008 ceph-mon[40743]: 5.15 scrub ok 2024-02-10T07:42:16.401 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:16 smithi008 ceph-mon[40743]: 3.10 scrub starts 2024-02-10T07:42:16.401 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:16 smithi008 ceph-mon[40743]: 3.10 scrub ok 2024-02-10T07:42:16.401 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:16 smithi008 ceph-mon[40743]: 5.7 scrub starts 2024-02-10T07:42:16.409 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:16.433 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:16.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:16.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:16.535 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:16.563 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:16.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:16.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:16 smithi170 ceph-mon[33394]: 5.15 scrub starts 2024-02-10T07:42:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:16 smithi170 ceph-mon[33394]: 5.15 scrub ok 2024-02-10T07:42:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:16 smithi170 ceph-mon[33394]: 3.10 scrub starts 2024-02-10T07:42:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:16 smithi170 ceph-mon[33394]: 3.10 scrub ok 2024-02-10T07:42:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:16 smithi170 ceph-mon[33394]: 5.7 scrub starts 2024-02-10T07:42:16.665 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:16.688 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:16.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:16.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:16.790 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:16.818 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:16.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:16.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:16.920 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:16.943 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:16.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:16.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:17.045 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:17.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:17.072 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:17.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:17.173 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:17.198 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:17.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:17.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:17.202 INFO:teuthology.orchestra.run.smithi008.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-02-10T07:42:17.301 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:17.325 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:17.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:17.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:17.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[29816]: pgmap v469: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 759 B/s rd, 0 op/s 2024-02-10T07:42:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[29816]: 6.12 scrub starts 2024-02-10T07:42:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[29816]: 6.12 scrub ok 2024-02-10T07:42:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[29816]: 3.1e scrub starts 2024-02-10T07:42:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[29816]: 3.1e scrub ok 2024-02-10T07:42:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[29816]: 5.7 scrub ok 2024-02-10T07:42:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/284022566' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:42:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[29816]: 6.18 scrub starts 2024-02-10T07:42:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[29816]: 6.18 scrub ok 2024-02-10T07:42:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[29816]: 6.7 deep-scrub starts 2024-02-10T07:42:17.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[40743]: pgmap v469: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 759 B/s rd, 0 op/s 2024-02-10T07:42:17.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[40743]: 6.12 scrub starts 2024-02-10T07:42:17.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[40743]: 6.12 scrub ok 2024-02-10T07:42:17.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[40743]: 3.1e scrub starts 2024-02-10T07:42:17.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[40743]: 3.1e scrub ok 2024-02-10T07:42:17.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[40743]: 5.7 scrub ok 2024-02-10T07:42:17.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/284022566' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:42:17.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[40743]: 6.18 scrub starts 2024-02-10T07:42:17.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[40743]: 6.18 scrub ok 2024-02-10T07:42:17.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:17 smithi008 ceph-mon[40743]: 6.7 deep-scrub starts 2024-02-10T07:42:17.426 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:17.451 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:17.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:17.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:17.553 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:17.577 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:17.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:17.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:17 smithi170 ceph-mon[33394]: pgmap v469: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 759 B/s rd, 0 op/s 2024-02-10T07:42:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:17 smithi170 ceph-mon[33394]: 6.12 scrub starts 2024-02-10T07:42:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:17 smithi170 ceph-mon[33394]: 6.12 scrub ok 2024-02-10T07:42:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:17 smithi170 ceph-mon[33394]: 3.1e scrub starts 2024-02-10T07:42:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:17 smithi170 ceph-mon[33394]: 3.1e scrub ok 2024-02-10T07:42:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:17 smithi170 ceph-mon[33394]: 5.7 scrub ok 2024-02-10T07:42:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:17 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/284022566' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:42:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:17 smithi170 ceph-mon[33394]: 6.18 scrub starts 2024-02-10T07:42:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:17 smithi170 ceph-mon[33394]: 6.18 scrub ok 2024-02-10T07:42:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:17 smithi170 ceph-mon[33394]: 6.7 deep-scrub starts 2024-02-10T07:42:17.679 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:17.704 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:17.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:17.805 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:17.830 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:17.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:17.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:17.922 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:42:17.922 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:42:17.932 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:17.957 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:17.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:17.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:18.058 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:18.085 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:18.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:18.187 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:18.212 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:18.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:18.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:18.313 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:18.339 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:18.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:18.441 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:18.467 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:18.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:18.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:18.569 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: 6.f scrub starts 2024-02-10T07:42:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: 6.f scrub ok 2024-02-10T07:42:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: 6.8 scrub starts 2024-02-10T07:42:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: 6.8 scrub ok 2024-02-10T07:42:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: 5.e scrub starts 2024-02-10T07:42:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: 5.e scrub ok 2024-02-10T07:42:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: 5.14 scrub starts 2024-02-10T07:42:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: 5.14 scrub ok 2024-02-10T07:42:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: 6.7 deep-scrub ok 2024-02-10T07:42:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/284022566' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:42:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: osdmap e257: 8 total, 8 up, 8 in 2024-02-10T07:42:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: 4.f scrub starts 2024-02-10T07:42:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: 4.f scrub ok 2024-02-10T07:42:18.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:42:18.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:18.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: 4.1e deep-scrub starts 2024-02-10T07:42:18.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[29816]: 4.1e deep-scrub ok 2024-02-10T07:42:18.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: 6.f scrub starts 2024-02-10T07:42:18.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: 6.f scrub ok 2024-02-10T07:42:18.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: 6.8 scrub starts 2024-02-10T07:42:18.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: 6.8 scrub ok 2024-02-10T07:42:18.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: 5.e scrub starts 2024-02-10T07:42:18.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: 5.e scrub ok 2024-02-10T07:42:18.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: 5.14 scrub starts 2024-02-10T07:42:18.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: 5.14 scrub ok 2024-02-10T07:42:18.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: 6.7 deep-scrub ok 2024-02-10T07:42:18.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/284022566' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:42:18.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: osdmap e257: 8 total, 8 up, 8 in 2024-02-10T07:42:18.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: 4.f scrub starts 2024-02-10T07:42:18.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: 4.f scrub ok 2024-02-10T07:42:18.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:42:18.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:18.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: 4.1e deep-scrub starts 2024-02-10T07:42:18.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-mon[40743]: 4.1e deep-scrub ok 2024-02-10T07:42:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:18.596 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:18.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: 6.f scrub starts 2024-02-10T07:42:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: 6.f scrub ok 2024-02-10T07:42:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: 6.8 scrub starts 2024-02-10T07:42:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: 6.8 scrub ok 2024-02-10T07:42:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: 5.e scrub starts 2024-02-10T07:42:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: 5.e scrub ok 2024-02-10T07:42:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: 5.14 scrub starts 2024-02-10T07:42:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: 5.14 scrub ok 2024-02-10T07:42:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: 6.7 deep-scrub ok 2024-02-10T07:42:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/284022566' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:42:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: osdmap e257: 8 total, 8 up, 8 in 2024-02-10T07:42:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: 4.f scrub starts 2024-02-10T07:42:18.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: 4.f scrub ok 2024-02-10T07:42:18.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:42:18.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:18.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: 4.1e deep-scrub starts 2024-02-10T07:42:18.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:18 smithi170 ceph-mon[33394]: 4.1e deep-scrub ok 2024-02-10T07:42:18.697 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:18.724 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:18.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:18.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:18.763 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:42:18.783 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:42:18.802 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:42:18.823 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:42:18.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:18.843 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:42:18.852 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:18.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:18.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:18.864 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:42:18.884 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:42:18.884 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:42:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:42:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:42:18.904 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:42:18.905 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:42:18.905 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:42:18.905 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:42:18.905 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:42:18.905 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:42:18.905 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:42:18.905 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:42:18.905 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:42:18.954 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:18.980 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:18.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:19.082 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:19.105 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:19.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:19.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:19.207 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:19.230 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:19.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:19.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:19.332 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:19.355 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:19.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:19.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:19.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:19.485 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:19.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:19.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[29816]: pgmap v471: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:42:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[29816]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:42:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:42:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[29816]: 4.4 scrub starts 2024-02-10T07:42:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[29816]: 4.4 scrub ok 2024-02-10T07:42:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3589057333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:42:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[29816]: 4.16 scrub starts 2024-02-10T07:42:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[29816]: 4.16 scrub ok 2024-02-10T07:42:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:42:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:42:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[40743]: pgmap v471: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:42:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[40743]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:42:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:42:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[40743]: 4.4 scrub starts 2024-02-10T07:42:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[40743]: 4.4 scrub ok 2024-02-10T07:42:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3589057333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:42:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[40743]: 4.16 scrub starts 2024-02-10T07:42:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[40743]: 4.16 scrub ok 2024-02-10T07:42:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:42:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:19 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:42:19.586 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:19.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:19.612 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:19.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:19 smithi170 ceph-mon[33394]: pgmap v471: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:42:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:19 smithi170 ceph-mon[33394]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:42:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:19 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:42:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:19 smithi170 ceph-mon[33394]: 4.4 scrub starts 2024-02-10T07:42:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:19 smithi170 ceph-mon[33394]: 4.4 scrub ok 2024-02-10T07:42:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:19 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3589057333' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:42:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:19 smithi170 ceph-mon[33394]: 4.16 scrub starts 2024-02-10T07:42:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:19 smithi170 ceph-mon[33394]: 4.16 scrub ok 2024-02-10T07:42:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:19 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:42:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:19 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:42:19.712 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:19.740 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:19.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:19.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:19.841 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:19.869 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:19.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:19.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:19.970 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:19.993 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:19.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:20.095 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:20.123 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:20.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:20.225 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:20.248 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:20.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:20.350 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:20.378 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:20.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:20.435 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-10T07:42:20.479 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:20.506 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:20.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:20.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:20.608 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:20.631 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:20.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:20.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:20.714 DEBUG:teuthology.orchestra.run.smithi008:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-02-10T07:42:20.716 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-10T07:42:20.716 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-10T07:42:20.732 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:20.760 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:20.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:20.766 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:hints = 1 2024-02-10T07:42:20.766 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-10T07:42:20.766 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Object prefix: benchmark_data_smithi008_134920 2024-02-10T07:42:20.766 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:42:20.766 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 0 0 0 0 0 0 - 0 2024-02-10T07:42:20.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:20 smithi008 ceph-mon[29816]: 4.12 scrub starts 2024-02-10T07:42:20.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:20 smithi008 ceph-mon[29816]: 4.12 scrub ok 2024-02-10T07:42:20.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:20 smithi008 ceph-mon[29816]: osdmap e258: 8 total, 8 up, 8 in 2024-02-10T07:42:20.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:20 smithi008 ceph-mon[29816]: pgmap v473: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:42:20.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:20 smithi008 ceph-mon[40743]: 4.12 scrub starts 2024-02-10T07:42:20.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:20 smithi008 ceph-mon[40743]: 4.12 scrub ok 2024-02-10T07:42:20.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:20 smithi008 ceph-mon[40743]: osdmap e258: 8 total, 8 up, 8 in 2024-02-10T07:42:20.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:20 smithi008 ceph-mon[40743]: pgmap v473: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:42:20.862 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:20.907 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:20.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:20.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:21.009 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:21.045 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:21.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:21.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:20 smithi170 ceph-mon[33394]: 4.12 scrub starts 2024-02-10T07:42:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:20 smithi170 ceph-mon[33394]: 4.12 scrub ok 2024-02-10T07:42:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:20 smithi170 ceph-mon[33394]: osdmap e258: 8 total, 8 up, 8 in 2024-02-10T07:42:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:20 smithi170 ceph-mon[33394]: pgmap v473: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:42:21.147 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:21.181 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:21.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:21.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:21.282 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:21.319 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:21.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:21.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:21.421 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:21.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:21.463 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:21.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:21.480 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:21.545 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:21.564 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:21.596 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:21.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:21.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:21.697 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:21.734 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:21.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:21.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:21.766 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 1 15 3136 3121 195.051 195.062 0.00400299 0.00510961 2024-02-10T07:42:21.788 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[29816]: 4.2 scrub starts 2024-02-10T07:42:21.789 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[29816]: 4.2 scrub ok 2024-02-10T07:42:21.789 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[29816]: 4.6 scrub starts 2024-02-10T07:42:21.789 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[29816]: 4.6 scrub ok 2024-02-10T07:42:21.789 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[29816]: 3.16 scrub starts 2024-02-10T07:42:21.789 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[29816]: 3.16 scrub ok 2024-02-10T07:42:21.789 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[29816]: 5.10 scrub starts 2024-02-10T07:42:21.789 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[29816]: 5.10 scrub ok 2024-02-10T07:42:21.789 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[40743]: 4.2 scrub starts 2024-02-10T07:42:21.789 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[40743]: 4.2 scrub ok 2024-02-10T07:42:21.789 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[40743]: 4.6 scrub starts 2024-02-10T07:42:21.789 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[40743]: 4.6 scrub ok 2024-02-10T07:42:21.789 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[40743]: 3.16 scrub starts 2024-02-10T07:42:21.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[40743]: 3.16 scrub ok 2024-02-10T07:42:21.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[40743]: 5.10 scrub starts 2024-02-10T07:42:21.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:21 smithi008 ceph-mon[40743]: 5.10 scrub ok 2024-02-10T07:42:21.835 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:21.869 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:21.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:21.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:21.971 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:22.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:22.007 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:22.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:22.108 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:21 smithi170 ceph-mon[33394]: 4.2 scrub starts 2024-02-10T07:42:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:21 smithi170 ceph-mon[33394]: 4.2 scrub ok 2024-02-10T07:42:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:21 smithi170 ceph-mon[33394]: 4.6 scrub starts 2024-02-10T07:42:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:21 smithi170 ceph-mon[33394]: 4.6 scrub ok 2024-02-10T07:42:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:21 smithi170 ceph-mon[33394]: 3.16 scrub starts 2024-02-10T07:42:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:21 smithi170 ceph-mon[33394]: 3.16 scrub ok 2024-02-10T07:42:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:21 smithi170 ceph-mon[33394]: 5.10 scrub starts 2024-02-10T07:42:22.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:21 smithi170 ceph-mon[33394]: 5.10 scrub ok 2024-02-10T07:42:22.149 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:22.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:22.251 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:22.290 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:22.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:22.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:22.392 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:22.427 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:22.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:22.529 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:22.566 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:22.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:22.668 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:22.708 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:22.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:22.766 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 2 16 6328 6312 197.236 199.438 0.00412638 0.00493923 2024-02-10T07:42:22.810 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:22.845 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:22.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:22.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:22.947 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:22.980 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:22.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:22.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:23.082 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:23.121 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:23.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:23.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:22 smithi170 ceph-mon[33394]: 3.a scrub starts 2024-02-10T07:42:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:22 smithi170 ceph-mon[33394]: 3.a scrub ok 2024-02-10T07:42:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:22 smithi170 ceph-mon[33394]: 3.1f scrub starts 2024-02-10T07:42:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:22 smithi170 ceph-mon[33394]: 3.1f scrub ok 2024-02-10T07:42:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:22 smithi170 ceph-mon[33394]: pgmap v474: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:42:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:22 smithi170 ceph-mon[33394]: 5.5 deep-scrub starts 2024-02-10T07:42:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:22 smithi170 ceph-mon[33394]: 5.5 deep-scrub ok 2024-02-10T07:42:23.223 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:23.245 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:42:23.258 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:23.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:23.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:23.283 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:22 smithi008 ceph-mon[29816]: 3.a scrub starts 2024-02-10T07:42:23.283 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:22 smithi008 ceph-mon[29816]: 3.a scrub ok 2024-02-10T07:42:23.284 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:22 smithi008 ceph-mon[29816]: 3.1f scrub starts 2024-02-10T07:42:23.284 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:22 smithi008 ceph-mon[29816]: 3.1f scrub ok 2024-02-10T07:42:23.284 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:22 smithi008 ceph-mon[29816]: pgmap v474: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:42:23.284 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:22 smithi008 ceph-mon[29816]: 5.5 deep-scrub starts 2024-02-10T07:42:23.284 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:22 smithi008 ceph-mon[29816]: 5.5 deep-scrub ok 2024-02-10T07:42:23.284 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:22 smithi008 ceph-mon[40743]: 3.a scrub starts 2024-02-10T07:42:23.284 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:22 smithi008 ceph-mon[40743]: 3.a scrub ok 2024-02-10T07:42:23.284 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:22 smithi008 ceph-mon[40743]: 3.1f scrub starts 2024-02-10T07:42:23.284 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:22 smithi008 ceph-mon[40743]: 3.1f scrub ok 2024-02-10T07:42:23.284 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:22 smithi008 ceph-mon[40743]: pgmap v474: 148 pgs: 148 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:42:23.284 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:22 smithi008 ceph-mon[40743]: 5.5 deep-scrub starts 2024-02-10T07:42:23.284 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:22 smithi008 ceph-mon[40743]: 5.5 deep-scrub ok 2024-02-10T07:42:23.360 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:23.394 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:23.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:23.495 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:23.538 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:23.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:23.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:42:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:42:23.639 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:23.680 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:23.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:23.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:23.766 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 3 16 9428 9412 196.069 193.75 0.00274023 0.00509043 2024-02-10T07:42:23.782 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:23.817 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:23.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:23.919 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:23.973 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:23.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:23.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:24.075 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:24.110 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:24.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:24.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:24.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:23 smithi170 ceph-mon[33394]: 4.0 scrub starts 2024-02-10T07:42:24.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:23 smithi170 ceph-mon[33394]: 4.0 scrub ok 2024-02-10T07:42:24.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:23 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:24.212 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:24.241 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:23 smithi008 ceph-mon[29816]: 4.0 scrub starts 2024-02-10T07:42:24.241 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:23 smithi008 ceph-mon[29816]: 4.0 scrub ok 2024-02-10T07:42:24.241 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:23 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:24.241 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:23 smithi008 ceph-mon[40743]: 4.0 scrub starts 2024-02-10T07:42:24.241 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:23 smithi008 ceph-mon[40743]: 4.0 scrub ok 2024-02-10T07:42:24.241 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:23 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:24.244 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:24.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:24.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:24.345 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:24.381 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:24.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:24.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:24.482 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:24.524 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:24.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:24.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:24.626 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:24.660 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:24.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:24.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:24.762 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:24.767 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 4 15 12336 12321 192.501 181.812 0.00654288 0.00518809 2024-02-10T07:42:24.797 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:24.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:24.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:24.899 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:24.943 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:24.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:24.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:25.045 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:25.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:24 smithi008 ceph-mon[29816]: 3.8 scrub starts 2024-02-10T07:42:25.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:24 smithi008 ceph-mon[29816]: 3.8 scrub ok 2024-02-10T07:42:25.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:24 smithi008 ceph-mon[29816]: 5.a scrub starts 2024-02-10T07:42:25.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:24 smithi008 ceph-mon[29816]: 5.a scrub ok 2024-02-10T07:42:25.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:24 smithi008 ceph-mon[29816]: pgmap v475: 148 pgs: 148 active+clean; 309 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 39 MiB/s wr, 618 op/s 2024-02-10T07:42:25.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:24 smithi008 ceph-mon[40743]: 3.8 scrub starts 2024-02-10T07:42:25.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:24 smithi008 ceph-mon[40743]: 3.8 scrub ok 2024-02-10T07:42:25.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:24 smithi008 ceph-mon[40743]: 5.a scrub starts 2024-02-10T07:42:25.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:24 smithi008 ceph-mon[40743]: 5.a scrub ok 2024-02-10T07:42:25.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:24 smithi008 ceph-mon[40743]: pgmap v475: 148 pgs: 148 active+clean; 309 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 39 MiB/s wr, 618 op/s 2024-02-10T07:42:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:25.086 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:25.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:24 smithi170 ceph-mon[33394]: 3.8 scrub starts 2024-02-10T07:42:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:24 smithi170 ceph-mon[33394]: 3.8 scrub ok 2024-02-10T07:42:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:24 smithi170 ceph-mon[33394]: 5.a scrub starts 2024-02-10T07:42:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:24 smithi170 ceph-mon[33394]: 5.a scrub ok 2024-02-10T07:42:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:24 smithi170 ceph-mon[33394]: pgmap v475: 148 pgs: 148 active+clean; 309 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 39 MiB/s wr, 618 op/s 2024-02-10T07:42:25.187 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:25.236 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:25.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:25.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:25.339 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:25.374 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:25.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:25.475 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:25.510 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:25.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:25.612 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:25.647 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:25.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:25.706 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:25.749 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:25.767 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 5 16 15008 14992 187.377 166.938 0.0102589 0.0053294 2024-02-10T07:42:25.780 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:25.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:25.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:25.882 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:25.901 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:42:25.931 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:25.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:25.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:26.033 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:26.068 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:26.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:26.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:26.169 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:26.204 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:26.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:26.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:26.306 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:26.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:25 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2564617670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:42:26.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:25 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:42:26.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:25 smithi008 ceph-mon[29816]: 3.1 scrub starts 2024-02-10T07:42:26.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:25 smithi008 ceph-mon[29816]: 3.1 scrub ok 2024-02-10T07:42:26.309 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:25 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2564617670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:42:26.309 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:25 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:42:26.310 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:25 smithi008 ceph-mon[40743]: 3.1 scrub starts 2024-02-10T07:42:26.310 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:25 smithi008 ceph-mon[40743]: 3.1 scrub ok 2024-02-10T07:42:26.339 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:26.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:25 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2564617670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:42:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:25 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:42:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:25 smithi170 ceph-mon[33394]: 3.1 scrub starts 2024-02-10T07:42:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:25 smithi170 ceph-mon[33394]: 3.1 scrub ok 2024-02-10T07:42:26.444 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:26.480 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:26.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:26.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:26.581 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:26.623 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:26.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:26.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:26.724 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:26.749 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:26.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:26.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:26.767 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 6 16 17636 17620 183.52 164.25 0.00441125 0.00543832 2024-02-10T07:42:26.850 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:26.874 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:26.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:26.977 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:27.016 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:27.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:27.118 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:27.156 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:27.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:27.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:27.258 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:27.291 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:27.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:27.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:27.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:26 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:42:27.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:26 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:42:27.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:26 smithi008 ceph-mon[29816]: osdmap e259: 8 total, 8 up, 8 in 2024-02-10T07:42:27.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:26 smithi008 ceph-mon[29816]: pgmap v477: 148 pgs: 148 active+clean; 468 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 934 op/s 2024-02-10T07:42:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:26 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:42:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:26 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:42:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:26 smithi008 ceph-mon[40743]: osdmap e259: 8 total, 8 up, 8 in 2024-02-10T07:42:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:26 smithi008 ceph-mon[40743]: pgmap v477: 148 pgs: 148 active+clean; 468 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 934 op/s 2024-02-10T07:42:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:26 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:42:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:26 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:42:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:26 smithi170 ceph-mon[33394]: osdmap e259: 8 total, 8 up, 8 in 2024-02-10T07:42:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:26 smithi170 ceph-mon[33394]: pgmap v477: 148 pgs: 148 active+clean; 468 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 58 MiB/s wr, 934 op/s 2024-02-10T07:42:27.393 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:27.426 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:27.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:27.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:27.528 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:27.581 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:27.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:27.682 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:27.723 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:27.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:27.767 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 7 16 20068 20052 178.999 152 0.0104735 0.00557864 2024-02-10T07:42:27.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:27.859 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:27.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:27.961 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:28.000 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:28.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:28.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:28.102 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:28.136 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:28.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:28.238 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:28.278 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:28.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:28.379 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:28.412 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:28.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:28.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:28.514 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:28.548 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:28.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:28.650 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:28.688 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:28.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:28.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:28.768 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 8 16 22464 22448 175.342 149.75 0.00382005 0.00569673 2024-02-10T07:42:28.789 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:28.825 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:28.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:28.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:28.927 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:28.928 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-10T07:42:28.928 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:42:28.928 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-02-10T07:42:28.929 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-10T07:42:28.929 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-10T07:42:28.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-02-10T07:42:28.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:28.971 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:28.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:29.072 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:29.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:42:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:42:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:42:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:28 smithi008 ceph-mon[40743]: pgmap v478: 148 pgs: 148 active+clean; 843 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-02-10T07:42:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:28 smithi008 ceph-mon[29816]: pgmap v478: 148 pgs: 148 active+clean; 843 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-02-10T07:42:29.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:29.116 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:29.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:29.217 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:29.248 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:29.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:29.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:29.350 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:29.374 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:29.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:28 smithi170 ceph-mon[33394]: pgmap v478: 148 pgs: 148 active+clean; 843 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-02-10T07:42:29.476 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:29.512 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:29.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:29.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:29.613 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:29.636 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:29.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:29.738 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:29.768 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 9 16 24404 24388 169.331 121.25 0.00711169 0.00590018 2024-02-10T07:42:29.775 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:29.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:29.877 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:29.911 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:29.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:29.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:30.012 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:30.048 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:30.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:30.150 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:30.183 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:30.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:30.250 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:42:30.285 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:30.339 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:30.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:30.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:30.379 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:42:30.380 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:42:30.380 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.380 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25283.0:23704 9.1 9:87b3a5b3:::benchmark_data_smithi008_134920_object23703:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-02-10T07:42:30.380 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:42:29.344886+0000", 2024-02-10T07:42:30.380 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.017965277000000002, 2024-02-10T07:42:30.380 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.017996631999999999, 2024-02-10T07:42:30.380 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:30.380 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:42:30.380 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:30.380 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25283", 2024-02-10T07:42:30.380 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:42:30.380 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 23704 2024-02-10T07:42:30.380 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.380 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.344886+0000", 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.344886+0000", 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.344888+0000", 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8020000000000001e-06 2024-02-10T07:42:30.381 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.344912+0000", 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4116000000000001e-05 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.344925+0000", 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2575000000000001e-05 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.344933+0000", 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0949999999999996e-06 2024-02-10T07:42:30.382 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.344995+0000", 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2617e-05 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.345145+0000", 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000149915 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 6,7", 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.345297+0000", 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015218299999999999 2024-02-10T07:42:30.383 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348626+0000", 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003328134 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348672+0000", 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6613000000000002e-05 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25283.0:23706 9.1 9:893007de:::benchmark_data_smithi008_134920_object23705:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-02-10T07:42:30.384 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:42:29.344949+0000", 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.017902530999999999, 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.017996970000000001, 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25283", 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 23706 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.344949+0000", 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.385 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.344949+0000", 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.344949+0000", 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.82e-07 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.344969+0000", 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9621000000000001e-05 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.386 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.344977+0000", 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4300000000000006e-06 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.344984+0000", 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2090000000000002e-06 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.346950+0000", 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0019662030000000001 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.387 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.347091+0000", 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000141602 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 6,7", 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.347245+0000", 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015320000000000001 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348632+0000", 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0013868369999999999 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.350975+0000", 2024-02-10T07:42:30.388 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0023433569999999999 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25283.0:23715 9.c 9:373e3d73:::benchmark_data_smithi008_134920_object23714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:42:29.348388+0000", 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.01446357, 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014612454, 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25283", 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:42:30.389 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 23715 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348388+0000", 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348388+0000", 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348389+0000", 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4440000000000001e-06 2024-02-10T07:42:30.390 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348412+0000", 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2787999999999999e-05 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348423+0000", 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0733999999999999e-05 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348431+0000", 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.875e-06 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.391 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348827+0000", 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00039652600000000002 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348990+0000", 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016286299999999999 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 4,7", 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349129+0000", 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000138727 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:30.392 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25283.0:23716 9.1 9:8ce0ebc5:::benchmark_data_smithi008_134920_object23715:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:42:29.348700+0000", 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.014150879, 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014346691999999999, 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25283", 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 23716 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.393 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348700+0000", 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348700+0000", 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348701+0000", 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3200000000000004e-07 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348725+0000", 2024-02-10T07:42:30.394 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3394e-05 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348738+0000", 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2884000000000001e-05 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348746+0000", 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0050000000000004e-06 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.348860+0000", 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011413600000000001 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.395 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349218+0000", 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000358263 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 6,7", 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349375+0000", 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015741 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25283.0:23717 9.8 9:11c7b0e6:::benchmark_data_smithi008_134920_object23716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-02-10T07:42:30.396 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:42:29.349493+0000", 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.013358586, 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013605598, 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25283", 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 23717 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349493+0000", 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.397 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349493+0000", 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349494+0000", 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4e-07 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349513+0000", 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9086e-05 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:30.398 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349523+0000", 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0101999999999999e-05 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349530+0000", 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8469999999999998e-06 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349587+0000", 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7575999999999999e-05 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349705+0000", 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000117736 2024-02-10T07:42:30.399 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,5", 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349825+0000", 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000119951 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25283.0:23718 9.1 9:85ce9aa3:::benchmark_data_smithi008_134920_object23717:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:42:29.349547+0000", 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.013303907, 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013595612999999999, 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:30.400 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25283", 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 23718 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349547+0000", 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349547+0000", 2024-02-10T07:42:30.401 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349548+0000", 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0900000000000002e-07 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349563+0000", 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.518e-05 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349571+0000", 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3930000000000002e-06 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.402 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.349576+0000", 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.485e-06 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351011+0000", 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001434493 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351137+0000", 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000126179 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.403 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 6,7", 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351282+0000", 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014557200000000001 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25283.0:23722 9.c 9:3d48f80f:::benchmark_data_smithi008_134920_object23721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:42:29.350423+0000", 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.012428326, 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012766395, 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:42:30.404 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25283", 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 23722 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.350423+0000", 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.350423+0000", 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.405 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.350424+0000", 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2580000000000001e-06 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.350446+0000", 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.145e-05 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.350458+0000", 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1823e-05 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.406 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.350465+0000", 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5460000000000003e-06 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.350522+0000", 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7225999999999997e-05 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.350648+0000", 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000126055 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.407 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:30.408 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.408 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.408 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.408 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop_reply(client.25283.0:23704 9.1 e259/253)", 2024-02-10T07:42:30.408 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:42:29.351093+0000", 2024-02-10T07:42:30.408 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.011758407, 2024-02-10T07:42:30.408 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012133718999999999, 2024-02-10T07:42:30.408 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:30.408 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:42:30.408 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:30.408 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.408 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:30.408 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351093+0000", 2024-02-10T07:42:30.408 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.409 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.409 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.409 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:30.409 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351093+0000", 2024-02-10T07:42:30.409 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.409 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.409 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.409 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:30.409 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351094+0000", 2024-02-10T07:42:30.409 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.263e-06 2024-02-10T07:42:30.409 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.409 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.409 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:30.409 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351099+0000", 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3150000000000002e-06 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351108+0000", 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4110000000000002e-06 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351115+0000", 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7950000000000002e-06 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.410 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25283.0:23719 9.3 e259/253)", 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:42:29.351132+0000", 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.011718994999999999, 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012121596, 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351132+0000", 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351132+0000", 2024-02-10T07:42:30.411 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351133+0000", 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9099999999999999e-07 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351164+0000", 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1503e-05 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351172+0000", 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2309999999999996e-06 2024-02-10T07:42:30.412 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351178+0000", 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9089999999999997e-06 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:42:29.351288+0000", 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000109311 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 9 2024-02-10T07:42:30.413 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:42:30.443 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:30.473 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:30.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:30.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:30.574 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:30.610 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:30.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:30.712 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:30.751 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:30.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:30.768 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 10 16 26415 26399 164.966 125.688 0.00436368 0.00605611 2024-02-10T07:42:30.853 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:30.887 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:30.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:30.989 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:31.024 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:31.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:31.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:31.126 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:31.164 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:31.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:31.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:31.266 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:31.300 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:31.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:31.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:30 smithi008 ceph-mon[40743]: pgmap v479: 148 pgs: 148 active+clean; 1.3 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.10k op/s 2024-02-10T07:42:31.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:30 smithi008 ceph-mon[29816]: pgmap v479: 148 pgs: 148 active+clean; 1.3 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.10k op/s 2024-02-10T07:42:31.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:30 smithi170 ceph-mon[33394]: pgmap v479: 148 pgs: 148 active+clean; 1.3 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 131 MiB/s wr, 2.10k op/s 2024-02-10T07:42:31.402 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:31.448 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:31.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:31.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:31.550 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:31.585 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:31.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:31.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:31.687 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:31.720 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:31.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:31.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:31.768 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 11 16 28641 28625 162.616 139.125 0.00633701 0.00614363 2024-02-10T07:42:31.821 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:31.828 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:31.880 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-10T07:42:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:31.883 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:31.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:31.983 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:32.033 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:32.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:32.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:32.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:32 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:42:32.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:32 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:42:32.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:32 smithi008 ceph-mon[40743]: osdmap e260: 8 total, 8 up, 8 in 2024-02-10T07:42:32.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:32 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:42:32.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:32 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:42:32.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:32 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:42:32.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:32 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:42:32.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:32 smithi008 ceph-mon[29816]: osdmap e260: 8 total, 8 up, 8 in 2024-02-10T07:42:32.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:32 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:42:32.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:32 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:42:32.135 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:32.164 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:32.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:32.266 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:32.312 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:32.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:32 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:42:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:32 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:42:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:32 smithi170 ceph-mon[33394]: osdmap e260: 8 total, 8 up, 8 in 2024-02-10T07:42:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:32 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:42:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:32 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:42:32.414 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:32.419 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:32.445 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:32.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:32.531 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:42:32.547 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:32.569 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:42:32.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:32.595 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:32.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:32.612 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:42:32.657 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:42:32.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:32.697 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:42:32.725 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:32.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:32.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:32.739 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:42:32.768 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 12 16 30434 30418 158.402 112.062 0.0135155 0.00630598 2024-02-10T07:42:32.785 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:42:32.826 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:32.833 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:42:32.846 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:42:32.846 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:42:32.846 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:42:32.846 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:42:32.847 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:42:32.847 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:42:32.847 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:42:32.847 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:42:32.855 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:32.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:32.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:32.957 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:32.991 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:32.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:32.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:33.092 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:33.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:33.128 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:33.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:33.228 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:33.268 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:33.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:33.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:33.272 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:33 smithi170 ceph-mon[33394]: pgmap v481: 148 pgs: 148 active+clean; 1.3 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-02-10T07:42:33.272 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:33 smithi170 ceph-mon[33394]: osdmap e261: 8 total, 8 up, 8 in 2024-02-10T07:42:33.272 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:33 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4243143710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:42:33.272 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:42:33.272 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:33.274 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:33 smithi008 ceph-mon[29816]: pgmap v481: 148 pgs: 148 active+clean; 1.3 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-02-10T07:42:33.274 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:33 smithi008 ceph-mon[29816]: osdmap e261: 8 total, 8 up, 8 in 2024-02-10T07:42:33.274 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:33 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4243143710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:42:33.274 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:42:33.274 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:33.274 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:33 smithi008 ceph-mon[40743]: pgmap v481: 148 pgs: 148 active+clean; 1.3 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 125 MiB/s wr, 2.00k op/s 2024-02-10T07:42:33.274 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:33 smithi008 ceph-mon[40743]: osdmap e261: 8 total, 8 up, 8 in 2024-02-10T07:42:33.274 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:33 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4243143710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:42:33.274 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:42:33.274 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:33.370 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:33.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:33.410 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:33.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:33.510 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:33.542 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:33.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:33.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:33.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:42:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:42:33.645 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:33.677 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:33.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:33.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:33.768 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 13 16 32302 32286 155.198 116.75 0.00297831 0.00643729 2024-02-10T07:42:33.779 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:33.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:33.822 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:33.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:33.923 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:33.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:33.965 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:33.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:34.066 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:34.108 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:34.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:34.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:34.210 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:34.250 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:34.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:34.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:34.314 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:34 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:34.314 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:34 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:34.352 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:34 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:34.386 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:34.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:34.487 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:34.526 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:34.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:34.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:34.628 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:34.667 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:34.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:34.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:34.768 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 14 16 34144 34128 152.335 115.125 0.006677 0.00655873 2024-02-10T07:42:34.770 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:34.811 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:34.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:34.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:34.913 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:34.943 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:34.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:34.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:35.044 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:35.080 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:35.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:35.182 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:35.220 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:35.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:35.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:35.321 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:35.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:35 smithi008 ceph-mon[29816]: pgmap v483: 148 pgs: 148 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 166 MiB/s wr, 2.65k op/s 2024-02-10T07:42:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:35 smithi008 ceph-mon[40743]: pgmap v483: 148 pgs: 148 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 166 MiB/s wr, 2.65k op/s 2024-02-10T07:42:35.360 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:35.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:35 smithi170 ceph-mon[33394]: pgmap v483: 148 pgs: 148 active+clean; 1.8 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 166 MiB/s wr, 2.65k op/s 2024-02-10T07:42:35.463 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:35.486 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:35.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:35.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:35.587 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:35.628 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:35.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:35.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:35.730 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:35.762 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:35.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:35.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:35.768 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 15 16 36128 36112 150.445 124 0.0052519 0.00664073 2024-02-10T07:42:35.865 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:35.900 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:35.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:36.002 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:36.036 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:36.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:36.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:36.137 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:36.175 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:36.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:36.276 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:36.299 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:36.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:36.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:36.400 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:36.439 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:36.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:36.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:36.543 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:36.577 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:36.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:36.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:36.678 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:36.712 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:36.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:36.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:36.768 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 16 15 37911 37896 148.011 111.5 0.0117469 0.00675104 2024-02-10T07:42:36.814 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:36.838 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:36.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:36.939 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:36.974 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:36.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:36.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:37.076 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:37.115 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:37.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:37.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:37.217 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:37.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:37.254 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:37.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:37.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:37 smithi008 ceph-mon[29816]: pgmap v484: 148 pgs: 148 active+clean; 1.9 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 138 MiB/s wr, 2.20k op/s 2024-02-10T07:42:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:37 smithi008 ceph-mon[40743]: pgmap v484: 148 pgs: 148 active+clean; 1.9 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 138 MiB/s wr, 2.20k op/s 2024-02-10T07:42:37.355 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:37 smithi170 ceph-mon[33394]: pgmap v484: 148 pgs: 148 active+clean; 1.9 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 138 MiB/s wr, 2.20k op/s 2024-02-10T07:42:37.389 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:37.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:37.491 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:37.514 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:37.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:37.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:37.616 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:37.655 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:37.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:37.757 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:37.768 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 17 16 39300 39284 144.407 86.75 0.00524231 0.00691947 2024-02-10T07:42:37.791 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:37.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:37.892 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:37.917 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:37.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:38.018 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:38.042 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:38.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:38.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:38.112 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:42:38.112 INFO:teuthology.orchestra.run.smithi008.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-10T07:42:38.112 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:42:38.144 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:38.182 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:38.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:38.284 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:38.320 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:38.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:38.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:38.422 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:38.446 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:38.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:38.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:38.548 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:38.583 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:38.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:38.685 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:38.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:38.721 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:38.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:38.769 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 18 16 40719 40703 141.311 88.6875 0.0145257 0.0070714 2024-02-10T07:42:38.822 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:38.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:38.863 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:38.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:38.964 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:39.001 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:39.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:39.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:39.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:42:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:42:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:42:39.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:39 smithi008 ceph-mon[29816]: pgmap v485: 148 pgs: 148 active+clean; 2.1 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-02-10T07:42:39.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:39 smithi008 ceph-mon[40743]: pgmap v485: 148 pgs: 148 active+clean; 2.1 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-02-10T07:42:39.102 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:39.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:39.142 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:39.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:39.243 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:39.283 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:39.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:39.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:39 smithi170 ceph-mon[33394]: pgmap v485: 148 pgs: 148 active+clean; 2.1 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.76k op/s 2024-02-10T07:42:39.384 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:39.410 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:39.416 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:39.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:39.518 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:39.556 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:39.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:39.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:39.657 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:39.691 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:39.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:39.769 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 19 16 42551 42535 139.9 114.5 0.00759714 0.00714282 2024-02-10T07:42:39.793 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:39.828 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:39.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:39.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:39.930 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:39.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:39.966 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:39.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:40.067 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:40.100 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:40.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:40.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:40.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:40.236 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:40.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:40.338 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:40.368 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:40.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:40.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:40.470 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:40.504 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:40.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:40.606 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:40.638 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:40.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:40.739 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:40.769 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:42:40.769776+0000 min lat: 0.001562 max lat: 0.194217 avg lat: 0.00721337 2024-02-10T07:42:40.769 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:42:40.769 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 20 16 44350 44334 138.526 112.438 0.0103495 0.00721337 2024-02-10T07:42:40.772 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:40.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:40.874 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:40.896 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:40.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:40.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:40.997 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:41.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:41.038 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:41.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:41.138 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:41.180 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:41.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:41.281 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:41.310 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:41.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:41.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:41 smithi008 ceph-mon[40743]: pgmap v486: 148 pgs: 148 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-02-10T07:42:41.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:41 smithi008 ceph-mon[29816]: pgmap v486: 148 pgs: 148 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-02-10T07:42:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:41 smithi170 ceph-mon[33394]: pgmap v486: 148 pgs: 148 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-02-10T07:42:41.412 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:41.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:41.452 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:41.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:41.553 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:41.589 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:41.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:41.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:41.690 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:41.723 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:41.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:41.769 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 21 16 46047 46031 136.98 106.062 0.0052209 0.0072954 2024-02-10T07:42:41.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:41.860 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:41.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:41.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:41.962 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:41.984 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:42:42.007 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:42.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:42.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:42.109 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:42.152 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:42.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:42.255 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:42.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:42.309 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:42.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:42.410 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:42.433 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:42.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:42.534 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:42.574 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:42.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:42.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:42.675 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:42.712 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:42.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:42.769 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 22 16 47525 47509 134.952 92.375 0.0116651 0.0074042 2024-02-10T07:42:42.814 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:42.844 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:42.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:42.877 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:42:42.945 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:42.969 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:42 smithi008 ceph-mon[29816]: pgmap v487: 148 pgs: 148 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-02-10T07:42:42.969 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:42 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3813183470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:42:42.969 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:42 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:42:42.969 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:42 smithi008 ceph-mon[40743]: pgmap v487: 148 pgs: 148 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-02-10T07:42:42.970 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:42 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3813183470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:42:42.970 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:42 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:42:42.985 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:42.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:42.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:43.088 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:43.120 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:43.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:43.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:43.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:42 smithi170 ceph-mon[33394]: pgmap v487: 148 pgs: 148 active+clean; 2.4 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-02-10T07:42:43.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:42 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3813183470' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:42:43.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:42 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:42:43.221 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:43.252 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:43.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:43.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:43.353 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:43.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:43.395 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:43.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:43.496 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:43.531 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:43.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:43.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:43.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:42:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:42:43.633 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:43.669 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:43.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:43.769 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 23 16 49216 49200 133.679 105.688 0.00922964 0.00747558 2024-02-10T07:42:43.771 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:43.810 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:43.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:43.911 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:43.948 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:43.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:44.049 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:44.079 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:44.094 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:44.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:44.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:44.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:43 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:42:44.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:43 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:42:44.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:43 smithi170 ceph-mon[33394]: osdmap e262: 8 total, 8 up, 8 in 2024-02-10T07:42:44.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:43 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:44.175 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:43 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:42:44.175 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:43 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:42:44.175 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:43 smithi008 ceph-mon[29816]: osdmap e262: 8 total, 8 up, 8 in 2024-02-10T07:42:44.175 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:43 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:44.176 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:42:44.176 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:42:44.176 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:43 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:42:44.176 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:43 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:42:44.176 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:43 smithi008 ceph-mon[40743]: osdmap e262: 8 total, 8 up, 8 in 2024-02-10T07:42:44.177 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:43 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:44.196 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:44.230 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:44.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:44.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:44.331 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:44.367 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:44.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:44.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:44.469 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:44.503 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:44.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:44.605 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:44.639 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:44.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:44.741 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:44.769 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 24 16 50864 50848 132.4 103 0.00794093 0.00754765 2024-02-10T07:42:44.781 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:44.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:44.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:44.883 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:44.921 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:44.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:44.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:45.022 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:45.064 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:45.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:45.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:45.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:44 smithi008 ceph-mon[29816]: pgmap v489: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-02-10T07:42:45.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:44 smithi008 ceph-mon[40743]: pgmap v489: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-02-10T07:42:45.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:44 smithi170 ceph-mon[33394]: pgmap v489: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-02-10T07:42:45.167 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:45.200 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:45.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:45.302 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:45.333 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:45.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:45.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:45.434 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:45.470 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:45.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:45.571 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:45.605 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:45.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:45.706 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:45.742 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:45.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:45.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:45.761 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-10T07:42:45.769 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 25 16 52567 52551 131.362 106.438 0.0147986 0.00760669 2024-02-10T07:42:45.844 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:45.882 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:45.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:45.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:45.917 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:42:45.964 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:45 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:42:45.964 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:45 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:42:45.964 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:45 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:42:45.965 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:45 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:42:45.984 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:46.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:46.019 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:46.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:46.120 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:46.152 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:46.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:46.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:46.254 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:46.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:46.304 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:46.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:46.371 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:46.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:45 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:42:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:45 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:42:46.405 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:46.448 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:46.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:46.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:46.550 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:46.582 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:46.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:46.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:46.684 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:46.722 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:46.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:46.769 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 26 16 54010 53994 129.778 90.1875 0.0142944 0.00770068 2024-02-10T07:42:46.824 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:46.858 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:46.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:46.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:46.960 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:46.977 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-10T07:42:46.977 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 0 1 2024-02-10T07:42:46.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:46.998 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:46.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:47.042 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:46 smithi008 ceph-mon[29816]: osdmap e263: 8 total, 8 up, 8 in 2024-02-10T07:42:47.043 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:46 smithi008 ceph-mon[29816]: pgmap v491: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-02-10T07:42:47.043 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:46 smithi008 ceph-mon[40743]: osdmap e263: 8 total, 8 up, 8 in 2024-02-10T07:42:47.043 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:46 smithi008 ceph-mon[40743]: pgmap v491: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-02-10T07:42:47.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:47.131 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:47.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:47.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:47.233 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:47.265 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:47.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:47.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:47.367 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:47.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:46 smithi170 ceph-mon[33394]: osdmap e263: 8 total, 8 up, 8 in 2024-02-10T07:42:47.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:46 smithi170 ceph-mon[33394]: pgmap v491: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-02-10T07:42:47.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:47.391 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:47.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:47.492 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:47.532 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:47.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:47.634 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:47.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:47.662 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:47.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:47.762 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:47.769 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 27 16 55472 55456 128.355 91.375 0.00776877 0.00778532 2024-02-10T07:42:47.799 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:47.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:47.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:47.901 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:47.936 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:47.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:48.038 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:48.076 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:48.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:48.178 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:48.201 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:48.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:48.302 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:48.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:47 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:42:48.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:47 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:42:48.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:47 smithi008 ceph-mon[29816]: osdmap e264: 8 total, 8 up, 8 in 2024-02-10T07:42:48.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:47 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:42:48.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:47 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:42:48.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:47 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:48.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:47 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:42:48.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:47 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:42:48.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:47 smithi008 ceph-mon[40743]: osdmap e264: 8 total, 8 up, 8 in 2024-02-10T07:42:48.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:47 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:42:48.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:47 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:42:48.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:47 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:48.335 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:48.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:48.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:47 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:42:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:47 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:42:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:47 smithi170 ceph-mon[33394]: osdmap e264: 8 total, 8 up, 8 in 2024-02-10T07:42:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:47 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:42:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:47 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:42:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:47 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:42:48.438 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:48.475 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:48.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:48.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:48.577 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:48.617 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:48.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:48.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:48.719 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:48.756 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:48.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:48.769 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 28 16 56955 56939 127.081 92.6875 0.0172343 0.00786335 2024-02-10T07:42:48.857 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:48.896 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:48.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:48.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:48.998 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:49.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:49.036 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:49.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:48 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:42:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:48 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:42:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:48 smithi008 ceph-mon[29816]: pgmap v493: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-02-10T07:42:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:48 smithi008 ceph-mon[29816]: osdmap e265: 8 total, 8 up, 8 in 2024-02-10T07:42:49.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:42:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:42:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:42:49.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:48 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:42:49.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:48 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:42:49.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:48 smithi008 ceph-mon[40743]: pgmap v493: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-02-10T07:42:49.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:48 smithi008 ceph-mon[40743]: osdmap e265: 8 total, 8 up, 8 in 2024-02-10T07:42:49.137 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:49.175 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:49.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:49.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:49.276 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:49.311 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:49.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:48 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:42:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:48 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:42:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:48 smithi170 ceph-mon[33394]: pgmap v493: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-02-10T07:42:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:48 smithi170 ceph-mon[33394]: osdmap e265: 8 total, 8 up, 8 in 2024-02-10T07:42:49.414 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:49.448 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:49.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:49.550 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:49.584 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:49.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:49.686 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:49.722 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:49.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:49.769 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 29 15 58554 58539 126.147 100 0.0126643 0.0079219 2024-02-10T07:42:49.824 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:49.859 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:49.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:49.961 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:49.994 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:49.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:49.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:50.096 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:50.137 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:50.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:50.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:50.239 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:50.274 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:50.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:50.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:50.376 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:50.414 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:50.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:50.516 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:50.562 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:50.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:50.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:50.665 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:50.673 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:42:50.715 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:42:50.725 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:50.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:50.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:50.731 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:50.753 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:42:50.770 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 30 16 59984 59968 124.919 89.3125 0.0212932 0.00799899 2024-02-10T07:42:50.798 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:42:50.828 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:50.840 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:42:50.876 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:50.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:50.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:50.885 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:42:50.928 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:42:50.973 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:50 smithi008 ceph-mon[40743]: pgmap v495: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-02-10T07:42:50.973 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:50 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3524719158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:42:50.977 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:50 smithi008 ceph-mon[29816]: pgmap v495: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-02-10T07:42:50.977 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:50 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3524719158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:42:50.977 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:42:50.978 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:50.987 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:42:50.987 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:42:50.987 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:42:50.987 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:42:50.987 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:42:50.988 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:42:50.988 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:42:50.988 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:42:51.002 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:51.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:51.104 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:51.141 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:51.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:51.243 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:51.277 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:51.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:51.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:51.379 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:50 smithi170 ceph-mon[33394]: pgmap v495: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-02-10T07:42:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:50 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3524719158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:42:51.414 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:51.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:51.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:51.516 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:51.551 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:51.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:51.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:51.652 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:51.701 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:51.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:51.770 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 31 16 61399 61383 123.742 88.4375 0.0123261 0.00807588 2024-02-10T07:42:51.790 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:42:51.790 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-02-10T07:42:51.790 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-02-10T07:42:51.790 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:42:51.790 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.790 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47293 7.2 7:44192736:::benchmark_data_smithi008_82046_object47292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-02-10T07:42:51.790 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.923555+0000", 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.85964090800002, 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0745729319999999, 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47293 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923555+0000", 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.792 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.792 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923555+0000", 2024-02-10T07:42:51.792 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.792 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.792 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.792 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923557+0000", 2024-02-10T07:42:51.792 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7439999999999999e-06 2024-02-10T07:42:51.792 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.792 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.792 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923579+0000", 2024-02-10T07:42:51.792 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2155e-05 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923589+0000", 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9410000000000002e-06 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.923597+0000", 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7409999999999992e-06 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.927637+0000", 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0040407170000000001 2024-02-10T07:42:51.793 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.927647+0000", 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7249999999999992e-06 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.994086+0000", 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0664391659999999 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.994230+0000", 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014405200000000001 2024-02-10T07:42:51.794 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.994425+0000", 2024-02-10T07:42:51.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019504200000000001 2024-02-10T07:42:51.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.997950+0000", 2024-02-10T07:42:51.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035248499999999999 2024-02-10T07:42:51.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998016+0000", 2024-02-10T07:42:51.796 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5520999999999998e-05 2024-02-10T07:42:51.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998034+0000", 2024-02-10T07:42:51.796 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8629000000000001e-05 2024-02-10T07:42:51.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.797 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998103+0000", 2024-02-10T07:42:51.797 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8366000000000002e-05 2024-02-10T07:42:51.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.797 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.797 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998128+0000", 2024-02-10T07:42:51.797 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5282999999999998e-05 2024-02-10T07:42:51.797 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.797 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.797 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.797 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47301 7.2 7:4dfdce57:::benchmark_data_smithi008_82046_object47300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.798 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.930507+0000", 2024-02-10T07:42:51.798 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.85268862700002, 2024-02-10T07:42:51.798 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.071251151, 2024-02-10T07:42:51.798 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.798 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.798 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.798 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.798 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.798 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47301 2024-02-10T07:42:51.798 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.798 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.798 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.798 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.798 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930507+0000", 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930507+0000", 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930509+0000", 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9690000000000001e-06 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930534+0000", 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4635999999999999e-05 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930545+0000", 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0684e-05 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930553+0000", 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4270000000000008e-06 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.800 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930839+0000", 2024-02-10T07:42:51.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000285665 2024-02-10T07:42:51.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.930848+0000", 2024-02-10T07:42:51.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8269999999999993e-06 2024-02-10T07:42:51.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.996538+0000", 2024-02-10T07:42:51.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0656891589999999 2024-02-10T07:42:51.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.996677+0000", 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013986200000000001 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.996838+0000", 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016012900000000001 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001568+0000", 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0047307110000000003 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001587+0000", 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8393000000000002e-05 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001655+0000", 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8763999999999994e-05 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001715+0000", 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9156000000000001e-05 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.803 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001758+0000", 2024-02-10T07:42:51.804 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3769000000000001e-05 2024-02-10T07:42:51.804 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.804 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.804 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.804 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.804 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.804 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47312 7.2 7:441ce8f4:::benchmark_data_smithi008_82046_object47311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.804 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.938921+0000", 2024-02-10T07:42:51.804 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.84427522700003, 2024-02-10T07:42:51.804 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0674384029999999, 2024-02-10T07:42:51.804 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.804 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.804 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.804 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47312 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938921+0000", 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938921+0000", 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.805 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938922+0000", 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.539e-06 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938949+0000", 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6395999999999999e-05 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938965+0000", 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6665999999999999e-05 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.806 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.938974+0000", 2024-02-10T07:42:51.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9069999999999996e-06 2024-02-10T07:42:51.807 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.807 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939045+0000", 2024-02-10T07:42:51.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1165000000000001e-05 2024-02-10T07:42:51.807 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.807 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.807 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939058+0000", 2024-02-10T07:42:51.807 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3026e-05 2024-02-10T07:42:51.807 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.807 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.808 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.808 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998227+0000", 2024-02-10T07:42:51.808 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.05916901 2024-02-10T07:42:51.808 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.808 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.808 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.808 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998357+0000", 2024-02-10T07:42:51.808 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012962100000000001 2024-02-10T07:42:51.808 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.808 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.808 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.808 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.998522+0000", 2024-02-10T07:42:51.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000165365 2024-02-10T07:42:51.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.809 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.809 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001821+0000", 2024-02-10T07:42:51.809 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032990379999999998 2024-02-10T07:42:51.809 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.809 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.809 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.809 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001868+0000", 2024-02-10T07:42:51.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7039999999999997e-05 2024-02-10T07:42:51.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.810 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.810 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.810 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006259+0000", 2024-02-10T07:42:51.810 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0043904510000000001 2024-02-10T07:42:51.810 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.811 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.811 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.811 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006317+0000", 2024-02-10T07:42:51.811 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8346e-05 2024-02-10T07:42:51.811 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.811 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.812 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.812 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006359+0000", 2024-02-10T07:42:51.812 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1832999999999997e-05 2024-02-10T07:42:51.812 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.812 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.812 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47317 7.2 7:477da457:::benchmark_data_smithi008_82046_object47316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.939750+0000", 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.84344558399999, 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.068997754, 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47317 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.813 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939750+0000", 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939750+0000", 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939751+0000", 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.034e-06 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.814 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939785+0000", 2024-02-10T07:42:51.815 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3855e-05 2024-02-10T07:42:51.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939801+0000", 2024-02-10T07:42:51.815 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5654999999999998e-05 2024-02-10T07:42:51.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.815 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.815 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939808+0000", 2024-02-10T07:42:51.815 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6599999999999995e-06 2024-02-10T07:42:51.815 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.815 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.816 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.816 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939884+0000", 2024-02-10T07:42:51.816 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5796000000000004e-05 2024-02-10T07:42:51.816 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.816 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.816 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.816 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.939893+0000", 2024-02-10T07:42:51.817 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9460000000000004e-06 2024-02-10T07:42:51.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.817 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.817 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.999976+0000", 2024-02-10T07:42:51.817 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.060082902 2024-02-10T07:42:51.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.817 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.817 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.000113+0000", 2024-02-10T07:42:51.817 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013713899999999999 2024-02-10T07:42:51.817 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.817 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.000289+0000", 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017616999999999999 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.003850+0000", 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035609219999999998 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.003902+0000", 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1848000000000002e-05 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.818 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008648+0000", 2024-02-10T07:42:51.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0047453490000000003 2024-02-10T07:42:51.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008704+0000", 2024-02-10T07:42:51.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6620999999999999e-05 2024-02-10T07:42:51.819 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.819 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.819 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.819 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008748+0000", 2024-02-10T07:42:51.819 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3856999999999997e-05 2024-02-10T07:42:51.819 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47376 7.2 7:4df203ae:::benchmark_data_smithi008_82046_object47375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.973270+0000", 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.80992607799999, 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.035779212, 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47376 2024-02-10T07:42:51.820 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973270+0000", 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973270+0000", 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973272+0000", 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2000000000000001e-06 2024-02-10T07:42:51.821 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973297+0000", 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4769999999999998e-05 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973313+0000", 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5959000000000001e-05 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973321+0000", 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7810000000000005e-06 2024-02-10T07:42:51.822 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973489+0000", 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016799299999999999 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.973499+0000", 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0176e-05 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.001910+0000", 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0284110420000001 2024-02-10T07:42:51.823 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.824 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.824 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.824 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.002044+0000", 2024-02-10T07:42:51.824 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013467899999999999 2024-02-10T07:42:51.824 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.824 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.824 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.824 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.002181+0000", 2024-02-10T07:42:51.824 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013709900000000001 2024-02-10T07:42:51.824 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.824 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.824 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008762+0000", 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0065806520000000002 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008834+0000", 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2015999999999996e-05 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.008896+0000", 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2131000000000005e-05 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.825 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009004+0000", 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000107298 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009049+0000", 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5416e-05 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47408 7.2 7:48404cbe:::benchmark_data_smithi008_82046_object47407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.992841+0000", 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.79035536399999, 2024-02-10T07:42:51.826 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.021168692, 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47408 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992841+0000", 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.827 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992841+0000", 2024-02-10T07:42:51.828 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.828 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.828 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.828 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.828 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992841+0000", 2024-02-10T07:42:51.828 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.13e-07 2024-02-10T07:42:51.828 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.828 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.828 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.828 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992864+0000", 2024-02-10T07:42:51.828 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2410000000000001e-05 2024-02-10T07:42:51.828 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.828 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.829 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.829 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992873+0000", 2024-02-10T07:42:51.829 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9679999999999995e-06 2024-02-10T07:42:51.829 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.829 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.829 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.829 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992879+0000", 2024-02-10T07:42:51.829 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7769999999999997e-06 2024-02-10T07:42:51.829 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.829 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.829 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.829 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992957+0000", 2024-02-10T07:42:51.829 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7385000000000005e-05 2024-02-10T07:42:51.829 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.992967+0000", 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9499999999999996e-06 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.003963+0000", 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.010996497 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.004110+0000", 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000147136 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.830 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.831 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.831 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.004280+0000", 2024-02-10T07:42:51.831 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000170192 2024-02-10T07:42:51.831 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.831 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.831 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.831 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009080+0000", 2024-02-10T07:42:51.831 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0047996489999999996 2024-02-10T07:42:51.831 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.831 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.831 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.831 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009138+0000", 2024-02-10T07:42:51.831 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7642999999999997e-05 2024-02-10T07:42:51.832 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.832 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.832 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:51.834 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.834 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.013897+0000", 2024-02-10T07:42:51.834 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0047593119999999999 2024-02-10T07:42:51.834 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.834 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.835 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.835 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.013966+0000", 2024-02-10T07:42:51.835 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9319000000000005e-05 2024-02-10T07:42:51.835 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.835 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.835 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.835 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014009+0000", 2024-02-10T07:42:51.835 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2840999999999999e-05 2024-02-10T07:42:51.835 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.835 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.835 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47416 7.2 7:4733a3ba:::benchmark_data_smithi008_82046_object47415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:37.996764+0000", 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.78643159400002, 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0174102730000001, 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47416 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.836 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996764+0000", 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996764+0000", 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996766+0000", 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4500000000000001e-06 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.837 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996788+0000", 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1861e-05 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996799+0000", 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1829e-05 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996807+0000", 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5830000000000001e-06 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.838 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996922+0000", 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011469 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:37.996931+0000", 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9910000000000008e-06 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006393+0000", 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.009462222 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.839 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.840 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.840 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006520+0000", 2024-02-10T07:42:51.840 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012698599999999999 2024-02-10T07:42:51.840 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.840 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.840 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.840 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.006667+0000", 2024-02-10T07:42:51.840 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014698399999999999 2024-02-10T07:42:51.840 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.840 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.840 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.840 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009091+0000", 2024-02-10T07:42:51.840 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0024244480000000001 2024-02-10T07:42:51.840 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009194+0000", 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010273299999999999 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014084+0000", 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0048898680000000003 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014141+0000", 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6919999999999997e-05 2024-02-10T07:42:51.841 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014175+0000", 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3708000000000003e-05 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47454 7.2 7:45bf9bfc:::benchmark_data_smithi008_82046_object47453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.019922+0000", 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.76327360300002, 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.99457532599999998, 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.842 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47454 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019922+0000", 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019922+0000", 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.843 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.844 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.844 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.844 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019924+0000", 2024-02-10T07:42:51.844 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.979e-06 2024-02-10T07:42:51.844 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.844 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.844 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.847 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019948+0000", 2024-02-10T07:42:51.847 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3913999999999998e-05 2024-02-10T07:42:51.847 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.847 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.847 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.847 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019965+0000", 2024-02-10T07:42:51.847 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7254000000000001e-05 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.019975+0000", 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1290000000000002e-06 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.020044+0000", 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9432999999999997e-05 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.020055+0000", 2024-02-10T07:42:51.848 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0963e-05 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009296+0000", 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.98924138500000003 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009424+0000", 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000128059 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.009572+0000", 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000147128 2024-02-10T07:42:51.849 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014029+0000", 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0044571560000000003 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014231+0000", 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00020206999999999999 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014335+0000", 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010406799999999999 2024-02-10T07:42:51.850 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014395+0000", 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0081000000000003e-05 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014498+0000", 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000102707 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.851 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47461 7.2 7:41d2193b:::benchmark_data_smithi008_82046_object47460:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.852 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.026190+0000", 2024-02-10T07:42:51.852 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.75700605499998, 2024-02-10T07:42:51.852 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.99060737499999996, 2024-02-10T07:42:51.852 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.852 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.852 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.852 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.852 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.852 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47461 2024-02-10T07:42:51.852 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.852 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.852 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.852 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.852 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026190+0000", 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026190+0000", 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026191+0000", 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4020000000000001e-06 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026215+0000", 2024-02-10T07:42:51.853 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3498000000000001e-05 2024-02-10T07:42:51.854 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.854 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.854 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.854 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026228+0000", 2024-02-10T07:42:51.854 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.293e-05 2024-02-10T07:42:51.854 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.854 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.854 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.854 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.026236+0000", 2024-02-10T07:42:51.854 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1389999999999995e-06 2024-02-10T07:42:51.854 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.854 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.854 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.854 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.027093+0000", 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000857371 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.027104+0000", 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1285e-05 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.011669+0000", 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.98456444099999996 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.855 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.011816+0000", 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014664100000000001 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.011969+0000", 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015315700000000001 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014192+0000", 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022234389999999998 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.856 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016650+0000", 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002457636 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016693+0000", 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3699999999999998e-05 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016755+0000", 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1278000000000005e-05 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.857 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016797+0000", 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2457999999999999e-05 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47486 7.2 7:46e23e1a:::benchmark_data_smithi008_82046_object47485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.041230+0000", 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.74196600099998, 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.99476295699999995, 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.858 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47486 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041230+0000", 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041230+0000", 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.859 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041231+0000", 2024-02-10T07:42:51.860 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.471e-06 2024-02-10T07:42:51.860 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.860 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.860 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.860 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041254+0000", 2024-02-10T07:42:51.860 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2226e-05 2024-02-10T07:42:51.860 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.860 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.860 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.860 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041270+0000", 2024-02-10T07:42:51.860 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6858e-05 2024-02-10T07:42:51.860 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.860 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.860 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.861 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041279+0000", 2024-02-10T07:42:51.861 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8249999999999994e-06 2024-02-10T07:42:51.861 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.861 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.861 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.861 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041346+0000", 2024-02-10T07:42:51.861 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6929999999999998e-05 2024-02-10T07:42:51.861 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.861 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.861 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.861 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.041357+0000", 2024-02-10T07:42:51.861 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0429e-05 2024-02-10T07:42:51.861 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.861 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014562+0000", 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97320494599999996 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014756+0000", 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019477900000000001 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.014928+0000", 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017193500000000001 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.862 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.863 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021669+0000", 2024-02-10T07:42:51.863 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0067405290000000003 2024-02-10T07:42:51.863 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.863 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.863 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.863 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021707+0000", 2024-02-10T07:42:51.863 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8451999999999997e-05 2024-02-10T07:42:51.863 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.863 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.863 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.863 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.035895+0000", 2024-02-10T07:42:51.863 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.01418823 2024-02-10T07:42:51.863 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.863 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.035953+0000", 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7839e-05 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.035993+0000", 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9508000000000002e-05 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47497 7.2 7:492ac9a6:::benchmark_data_smithi008_82046_object47496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.864 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.047446+0000", 2024-02-10T07:42:51.865 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.73574968000003, 2024-02-10T07:42:51.865 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.98990700600000003, 2024-02-10T07:42:51.865 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.865 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.865 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.865 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.865 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.865 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47497 2024-02-10T07:42:51.865 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.865 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.865 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.865 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.865 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047446+0000", 2024-02-10T07:42:51.865 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047446+0000", 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047448+0000", 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4330000000000001e-06 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047469+0000", 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1447999999999999e-05 2024-02-10T07:42:51.866 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047481+0000", 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1843e-05 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047540+0000", 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8989000000000001e-05 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047603+0000", 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2974999999999999e-05 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.867 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.868 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.868 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.047613+0000", 2024-02-10T07:42:51.868 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8439999999999993e-06 2024-02-10T07:42:51.868 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.868 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.868 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.868 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016852+0000", 2024-02-10T07:42:51.868 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96923946999999999 2024-02-10T07:42:51.868 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.868 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.868 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.868 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.016993+0000", 2024-02-10T07:42:51.868 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014067999999999999 2024-02-10T07:42:51.868 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.017152+0000", 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015878999999999999 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.026100+0000", 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0089479090000000004 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032492+0000", 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0063921489999999997 2024-02-10T07:42:51.869 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.036002+0000", 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035099559999999998 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.036042+0000", 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0071999999999999e-05 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037353+0000", 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001311448 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.870 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47509 7.2 7:46639bc8:::benchmark_data_smithi008_82046_object47508:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.056564+0000", 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.72663167399998, 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.98093845899999998, 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47509 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.871 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056564+0000", 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056564+0000", 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056566+0000", 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.516e-06 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.872 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056588+0000", 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2405e-05 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056600+0000", 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.236e-05 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056609+0000", 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3380000000000005e-06 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.873 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056741+0000", 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013242899999999999 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.056751+0000", 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8460000000000003e-06 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.019111+0000", 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96235988699999997 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.874 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.875 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.019240+0000", 2024-02-10T07:42:51.875 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000128565 2024-02-10T07:42:51.875 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.875 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.875 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.876 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.019393+0000", 2024-02-10T07:42:51.876 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000153294 2024-02-10T07:42:51.876 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.876 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.876 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.876 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032535+0000", 2024-02-10T07:42:51.876 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013142513999999999 2024-02-10T07:42:51.876 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032697+0000", 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016114300000000001 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037374+0000", 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046779819999999998 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037429+0000", 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4669999999999997e-05 2024-02-10T07:42:51.877 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037503+0000", 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3510000000000006e-05 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47513 7.2 7:4ccef1d2:::benchmark_data_smithi008_82046_object47512:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.059397+0000", 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.723798445, 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97837375900000001, 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.878 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47513 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059397+0000", 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059397+0000", 2024-02-10T07:42:51.879 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.880 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.880 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.880 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.880 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059400+0000", 2024-02-10T07:42:51.880 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1749999999999999e-06 2024-02-10T07:42:51.880 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.880 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.880 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.880 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059425+0000", 2024-02-10T07:42:51.880 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.5701e-05 2024-02-10T07:42:51.880 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.880 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.880 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.880 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059443+0000", 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7710000000000002e-05 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059453+0000", 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0029e-05 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059555+0000", 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000102403 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.059567+0000", 2024-02-10T07:42:51.881 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1172000000000001e-05 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021735+0000", 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96216885299999999 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021846+0000", 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000110096 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.021980+0000", 2024-02-10T07:42:51.882 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013462200000000001 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032576+0000", 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010596202000000001 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032865+0000", 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00028853 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037514+0000", 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046494279999999997 2024-02-10T07:42:51.883 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.884 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.884 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.884 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037580+0000", 2024-02-10T07:42:51.884 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5399000000000002e-05 2024-02-10T07:42:51.884 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.884 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.884 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.884 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037771+0000", 2024-02-10T07:42:51.884 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019143900000000001 2024-02-10T07:42:51.884 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.884 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.884 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.884 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.884 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47519 7.2 7:474b3507:::benchmark_data_smithi008_82046_object47518:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.063509+0000", 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.71968646099998, 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97463434199999999, 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47519 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.885 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063509+0000", 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063509+0000", 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063511+0000", 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3659999999999999e-06 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.886 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063535+0000", 2024-02-10T07:42:51.887 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3765e-05 2024-02-10T07:42:51.887 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.887 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.887 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.887 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063546+0000", 2024-02-10T07:42:51.887 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1199000000000001e-05 2024-02-10T07:42:51.887 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.887 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.887 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.887 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063553+0000", 2024-02-10T07:42:51.887 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7920000000000004e-06 2024-02-10T07:42:51.887 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.887 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.887 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063622+0000", 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9577000000000001e-05 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.063633+0000", 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.134e-05 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.023833+0000", 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96019993999999997 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.888 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.024002+0000", 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016875700000000001 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.024184+0000", 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000182227 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032615+0000", 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0084308600000000001 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.889 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037780+0000", 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0051646779999999998 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.038029+0000", 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00024897100000000001 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.038106+0000", 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6736000000000005e-05 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.890 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.038144+0000", 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8133999999999998e-05 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47530 7.2 7:44bc3d2a:::benchmark_data_smithi008_82046_object47529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.076595+0000", 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.70660041100001, 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96853736000000001, 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.891 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47530 2024-02-10T07:42:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076595+0000", 2024-02-10T07:42:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076595+0000", 2024-02-10T07:42:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.892 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076597+0000", 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0899999999999999e-06 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076613+0000", 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.683e-05 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076623+0000", 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9070000000000002e-06 2024-02-10T07:42:51.893 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.894 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.894 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.894 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076629+0000", 2024-02-10T07:42:51.894 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9689999999999999e-06 2024-02-10T07:42:51.894 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.894 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.894 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.894 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076694+0000", 2024-02-10T07:42:51.894 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4873e-05 2024-02-10T07:42:51.894 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.894 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.894 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.894 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.076707+0000", 2024-02-10T07:42:51.894 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2424e-05 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.026136+0000", 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.94942972400000003 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.026302+0000", 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000165744 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.026474+0000", 2024-02-10T07:42:51.895 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017199699999999999 2024-02-10T07:42:51.896 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.896 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.896 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.896 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.032657+0000", 2024-02-10T07:42:51.896 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0061833010000000004 2024-02-10T07:42:51.896 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.896 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.896 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.896 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037785+0000", 2024-02-10T07:42:51.896 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0051276610000000004 2024-02-10T07:42:51.896 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.896 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.896 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.896 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.044963+0000", 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0071781509999999998 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045018+0000", 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4934e-05 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045133+0000", 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011475499999999999 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.897 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.898 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.898 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:47545 7.2 7:41492b52:::benchmark_data_smithi008_82046_object47544:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-10T07:42:51.898 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:34:38.108305+0000", 2024-02-10T07:42:51.898 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.67489062800001, 2024-02-10T07:42:51.898 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.93696623499999998, 2024-02-10T07:42:51.898 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.898 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.898 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.898 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.898 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.898 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 47545 2024-02-10T07:42:51.898 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.898 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.898 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108305+0000", 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108305+0000", 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108306+0000", 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2100000000000001e-06 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.899 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108325+0000", 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8094999999999999e-05 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108335+0000", 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0301e-05 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108340+0000", 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6420000000000004e-06 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.900 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108392+0000", 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1669999999999998e-05 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:38.108399+0000", 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8279999999999997e-06 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.030442+0000", 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.92204303600000004 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.901 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.030575+0000", 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000133089 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.030728+0000", 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015333400000000001 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037828+0000", 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.007100014 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.902 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.037841+0000", 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2738999999999999e-05 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045181+0000", 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0073395769999999999 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045235+0000", 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4469000000000003e-05 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.903 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.904 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.904 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:34:39.045271+0000", 2024-02-10T07:42:51.904 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6230999999999999e-05 2024-02-10T07:42:51.904 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.904 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.904 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.904 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.904 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.904 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154801 7.a 7:5577fbe8:::benchmark_data_smithi008_82046_object39663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-02-10T07:42:51.904 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.740728+0000", 2024-02-10T07:42:51.904 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 379.04246738299997, 2024-02-10T07:42:51.904 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0968356680000002, 2024-02-10T07:42:51.904 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.904 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154801 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740728+0000", 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740728+0000", 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.905 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740730+0000", 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6980000000000001e-06 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740734+0000", 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8720000000000004e-06 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740745+0000", 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0589e-05 2024-02-10T07:42:51.906 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740752+0000", 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2010000000000002e-06 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740814+0000", 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1679000000000004e-05 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.740929+0000", 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000115658 2024-02-10T07:42:51.907 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2", 2024-02-10T07:42:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.741039+0000", 2024-02-10T07:42:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011011 2024-02-10T07:42:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.755960+0000", 2024-02-10T07:42:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014920487 2024-02-10T07:42:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.755972+0000", 2024-02-10T07:42:51.908 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1799e-05 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739149+0000", 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.98317701300000004 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739162+0000", 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3864e-05 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.830200+0000", 2024-02-10T07:42:51.909 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.09103803 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for missing object", 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.830364+0000", 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000163325 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.837471+0000", 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0071075069999999999 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.837564+0000", 2024-02-10T07:42:51.910 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2836000000000004e-05 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154828 7.a 7:58351c3d:::benchmark_data_smithi008_82046_object39690:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.883850+0000", 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 378.89934564700002, 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9570110619999999, 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154828 2024-02-10T07:42:51.911 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883850+0000", 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883850+0000", 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883852+0000", 2024-02-10T07:42:51.912 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.691e-06 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883858+0000", 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2029999999999997e-06 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883876+0000", 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7728999999999999e-05 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883886+0000", 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7259999999999997e-06 2024-02-10T07:42:51.913 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.914 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.914 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.914 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883959+0000", 2024-02-10T07:42:51.914 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3606e-05 2024-02-10T07:42:51.914 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.914 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.914 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.914 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.883971+0000", 2024-02-10T07:42:51.915 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1589e-05 2024-02-10T07:42:51.915 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.915 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.915 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.915 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739190+0000", 2024-02-10T07:42:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.85521957599999998 2024-02-10T07:42:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739199+0000", 2024-02-10T07:42:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5290000000000008e-06 2024-02-10T07:42:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.830636+0000", 2024-02-10T07:42:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0914373340000001 2024-02-10T07:42:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.916 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.830765+0000", 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000129085 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,5", 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.830876+0000", 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011061 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.836879+0000", 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0060032289999999997 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.917 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.840426+0000", 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035464889999999999 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.840544+0000", 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011797 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.840618+0000", 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.4357999999999995e-05 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.918 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.840861+0000", 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00024333800000000001 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154842 7.a 7:5188ae5b:::benchmark_data_smithi008_82046_object39704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.905956+0000", 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 378.87724034199999, 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9354436609999999, 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.919 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154842 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.905956+0000", 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.905956+0000", 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.920 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.905956+0000", 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7299999999999995e-07 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.905960+0000", 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3840000000000001e-06 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.905968+0000", 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8319999999999995e-06 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.921 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.905975+0000", 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7449999999999998e-06 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.906034+0000", 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9252000000000001e-05 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.906047+0000", 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2366e-05 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.922 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739224+0000", 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.83317686999999996 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.739232+0000", 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4009999999999997e-06 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.833617+0000", 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.094384676 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.923 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.833735+0000", 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000117827 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,5", 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.833862+0000", 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012773799999999999 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.840333+0000", 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0064703149999999999 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.924 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.840484+0000", 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000151003 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.841207+0000", 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00072376500000000002 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.841266+0000", 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8523000000000002e-05 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.925 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:34.841399+0000", 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013329100000000001 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:213588 7.14 7:2f12bf0f:::benchmark_data_smithi008_82046_object98450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:39.031951+0000", 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 312.75124513399999, 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0083072099999999, 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:42:51.926 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 213588 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.031951+0000", 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.031951+0000", 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:42:51.927 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.031953+0000", 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.057e-06 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.031960+0000", 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3239999999999997e-06 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.031977+0000", 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6472000000000001e-05 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:42:51.928 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.031987+0000", 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0478e-05 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.032655+0000", 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00066817799999999996 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:39.032663+0000", 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6650000000000003e-06 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.032047+0000", 2024-02-10T07:42:51.929 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.99938453100000002 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.032176+0000", 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012841999999999999 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,5", 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.032551+0000", 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00037524200000000002 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.037801+0000", 2024-02-10T07:42:51.930 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0052499729999999998 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.037860+0000", 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8922000000000003e-05 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.040146+0000", 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022857979999999999 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.040208+0000", 2024-02-10T07:42:51.931 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2510999999999998e-05 2024-02-10T07:42:51.932 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:42:51.932 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:42:51.932 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:42:51.932 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:40.040258+0000", 2024-02-10T07:42:51.932 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9638999999999997e-05 2024-02-10T07:42:51.932 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.932 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.932 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.932 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:42:51.932 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:42:51.932 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:42:51.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:51.933 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:51.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:52.034 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:52.085 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:52.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:52.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:52.187 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:52.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:52.227 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:52.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:52.328 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:52.369 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:52.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:52.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:52.471 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:52.504 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:52.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:52.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:52.606 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:52.648 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:52.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:52.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:52.750 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:52.770 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 32 16 62763 62747 122.539 85.25 0.0155242 0.0081558 2024-02-10T07:42:52.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:52.786 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:52.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:52.887 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:52.921 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:52.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:53.022 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:53.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:53.065 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:53.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:53.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:52 smithi170 ceph-mon[33394]: osdmap e266: 8 total, 8 up, 8 in 2024-02-10T07:42:53.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:52 smithi170 ceph-mon[33394]: pgmap v497: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-02-10T07:42:53.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:52 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:42:53.165 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:53.209 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:53.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:53.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:53.263 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:53.311 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:53.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:52 smithi008 ceph-mon[40743]: osdmap e266: 8 total, 8 up, 8 in 2024-02-10T07:42:53.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:52 smithi008 ceph-mon[40743]: pgmap v497: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-02-10T07:42:53.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:52 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:42:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:52 smithi008 ceph-mon[29816]: osdmap e266: 8 total, 8 up, 8 in 2024-02-10T07:42:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:52 smithi008 ceph-mon[29816]: pgmap v497: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-02-10T07:42:53.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:52 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:42:53.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:53.362 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:53.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:53.463 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:53.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:53.501 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:53.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:53.602 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:53.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:42:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:42:53.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:53.642 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:53.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:53.743 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:53.770 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 33 16 64100 64084 121.357 83.5625 0.0165993 0.00823509 2024-02-10T07:42:53.778 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:53.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:53.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:53.880 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:53.922 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:53.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:53.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:54.024 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:54.060 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:54.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:54.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:53 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:54.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:53 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:42:54.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:53 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:42:54.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:53 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:42:54.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:53 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:42:54.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:53 smithi008 ceph-mon[29816]: Adjusting osd_memory_target on smithi170 to 3248M 2024-02-10T07:42:54.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:53 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:42:54.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:53 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:54.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:53 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:42:54.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:53 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:42:54.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:53 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:42:54.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:53 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:42:54.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:53 smithi008 ceph-mon[40743]: Adjusting osd_memory_target on smithi170 to 3248M 2024-02-10T07:42:54.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:53 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:42:54.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:53 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:42:54.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:53 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:42:54.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:53 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:42:54.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:53 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:42:54.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:53 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:42:54.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:53 smithi170 ceph-mon[33394]: Adjusting osd_memory_target on smithi170 to 3248M 2024-02-10T07:42:54.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:53 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:42:54.161 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:54.195 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:54.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:54.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:54.297 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:54.335 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:54.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:54.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:54.437 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:54.472 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:54.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:54.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:54.574 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:54.607 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:54.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:54.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:54.709 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:54.745 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:54.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:54.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:54.770 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 34 16 65495 65479 120.352 87.1875 0.00503567 0.00830329 2024-02-10T07:42:54.846 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:54.879 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:54.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:54.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:54.981 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:55.015 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:55.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:55.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:55.117 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:55.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:54 smithi170 ceph-mon[33394]: pgmap v498: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-02-10T07:42:55.151 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:54 smithi008 ceph-mon[29816]: pgmap v498: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-02-10T07:42:55.152 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-10T07:42:55.153 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:54 smithi008 ceph-mon[40743]: pgmap v498: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-02-10T07:42:55.170 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:55.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:55.272 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:55.308 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:55.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:55.411 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:55.443 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:55.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:55.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:55.528 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:42:55.545 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:55.585 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:55.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:55.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:55.686 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:55.728 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:55.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:55.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:55.770 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 35 16 66839 66823 119.314 84 0.00544951 0.00837607 2024-02-10T07:42:55.829 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:55.865 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:55.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:55.966 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:55.986 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:55 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:42:55.986 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:55 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:42:55.986 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:55 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:42:56.004 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:56.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:56.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:56.106 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:56.159 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:56.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:56.192 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:55 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:42:56.192 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:55 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:42:56.192 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:55 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:42:56.192 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:55 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:42:56.192 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:55 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:42:56.192 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:55 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:42:56.261 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:56.299 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:56.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:56.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:56.401 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:56.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:56.435 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:56.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:56.536 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:42:56.577 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:56.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:42:56.679 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:56.713 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:56.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:56.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:56.770 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 36 16 68203 68187 118.367 85.25 0.0110963 0.00844376 2024-02-10T07:42:56.815 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:56.843 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:56.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:56.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:56.945 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:56.985 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:56.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:56.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:57.053 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:56 smithi008 ceph-mon[29816]: pgmap v499: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-02-10T07:42:57.053 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:56 smithi008 ceph-mon[29816]: osdmap e267: 8 total, 8 up, 8 in 2024-02-10T07:42:57.053 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:56 smithi008 ceph-mon[40743]: pgmap v499: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-02-10T07:42:57.053 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:56 smithi008 ceph-mon[40743]: osdmap e267: 8 total, 8 up, 8 in 2024-02-10T07:42:57.086 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:57.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:56 smithi170 ceph-mon[33394]: pgmap v499: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-02-10T07:42:57.097 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:56 smithi170 ceph-mon[33394]: osdmap e267: 8 total, 8 up, 8 in 2024-02-10T07:42:57.129 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:57.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:57.231 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:57.273 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:57.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:57.375 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:42:57.409 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:57.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:57.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:42:57.511 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:57.551 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:57.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:57.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:57.653 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:57.692 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:57.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:57.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:57.740 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:42:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:42:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.25283.0:69419 9.4 e267/253 9:2620e63e:::benchmark_data_smithi008_134920_object69418:head v 267'4352, mlcod=267'4348)", 2024-02-10T07:42:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:42:57.737443+0000", 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.0024731190000000002, 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0025040140000000002, 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.737443+0000", 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.737443+0000", 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.737444+0000", 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2699999999999999e-07 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.737499+0000", 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4484000000000002e-05 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.737511+0000", 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2469999999999999e-05 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:42:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.737518+0000", 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2679999999999997e-06 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.737585+0000", 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6717999999999994e-05 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.737610+0000", 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4532000000000002e-05 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.25283.0:69425 9.e e267/253 9:7d9b5428:::benchmark_data_smithi008_134920_object69424:head v 267'4323, mlcod=267'4321)", 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:42:57.739002+0000", 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.00091400699999999999, 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0010118309999999999, 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.739002+0000", 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.739002+0000", 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:42:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.739003+0000", 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5000000000000002e-07 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.739028+0000", 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5083999999999999e-05 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.739045+0000", 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6951000000000001e-05 2024-02-10T07:42:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.739052+0000", 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2150000000000004e-06 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.739129+0000", 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6341000000000007e-05 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:42:57.739148+0000", 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9032e-05 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:42:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:42:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:42:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:42:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2024-02-10T07:42:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 2 2024-02-10T07:42:57.747 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:42:57.770 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 37 16 69468 69452 117.305 79.0625 0.0138128 0.00852007 2024-02-10T07:42:57.795 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:57.818 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:57.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:57.902 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:42:57.920 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:57.962 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:57.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:57.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:58.064 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:58.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:57 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1534879218' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:42:58.091 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:42:58.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:58.103 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:58.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:58.204 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:58.254 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:58.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:58.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:57 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1534879218' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:42:58.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:57 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1534879218' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:42:58.354 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:58.404 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:58.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:58.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:58.506 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:42:58.541 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:58.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:58.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:42:58.642 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:42:58.684 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:58.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:42:58.770 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 38 15 70577 70562 116.043 69.375 0.00890962 0.00861219 2024-02-10T07:42:58.786 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:58.852 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:58.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:58.953 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:58.992 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:58.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:59.058 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-10T07:42:59.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:42:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:42:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:42:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:58 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:42:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:58 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1534879218' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:42:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:58 smithi008 ceph-mon[29816]: osdmap e268: 8 total, 8 up, 8 in 2024-02-10T07:42:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:58 smithi008 ceph-mon[29816]: pgmap v502: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.6 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-02-10T07:42:59.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:58 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:42:59.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:58 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1534879218' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:42:59.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:58 smithi008 ceph-mon[40743]: osdmap e268: 8 total, 8 up, 8 in 2024-02-10T07:42:59.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:58 smithi008 ceph-mon[40743]: pgmap v502: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.6 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-02-10T07:42:59.095 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:59.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:59.133 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:59.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:59.234 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:59.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:59.279 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:59.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:59.380 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:42:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:58 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:42:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:58 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1534879218' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:42:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:58 smithi170 ceph-mon[33394]: osdmap e268: 8 total, 8 up, 8 in 2024-02-10T07:42:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:58 smithi170 ceph-mon[33394]: pgmap v502: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.6 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-02-10T07:42:59.412 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:59.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:59.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:42:59.485 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:42:59.514 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:42:59.564 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:42:59.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:59.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:42:59.592 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:42:59.666 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:59.701 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:59.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:59.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:42:59.770 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 39 16 71734 71718 114.92 72.25 0.0193543 0.00869687 2024-02-10T07:42:59.803 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:42:59.837 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:59.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:59.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:42:59.932 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.0 to 1 (10000) 2024-02-10T07:42:59.939 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:42:59.951 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:59 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/986502134' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:42:59.951 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:59 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:42:59.951 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:59 smithi170 ceph-mon[33394]: 3.11 scrub starts 2024-02-10T07:42:59.951 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:42:59 smithi170 ceph-mon[33394]: 3.11 scrub ok 2024-02-10T07:42:59.987 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:42:59.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:42:59.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:00.018 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:59 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/986502134' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:43:00.018 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:59 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:43:00.018 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:59 smithi008 ceph-mon[29816]: 3.11 scrub starts 2024-02-10T07:43:00.018 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:42:59 smithi008 ceph-mon[29816]: 3.11 scrub ok 2024-02-10T07:43:00.019 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:59 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/986502134' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:43:00.019 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:59 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:43:00.019 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:59 smithi008 ceph-mon[40743]: 3.11 scrub starts 2024-02-10T07:43:00.019 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:42:59 smithi008 ceph-mon[40743]: 3.11 scrub ok 2024-02-10T07:43:00.089 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:00.126 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:00.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:00.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:00.228 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:00.259 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:00.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:00.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:00.360 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:00.395 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:00.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:00.497 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:00.540 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:00.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:00.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:00.642 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:00.679 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:00.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:00.771 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:43:00.771518+0000 min lat: 0.001562 max lat: 0.194217 avg lat: 0.00875755 2024-02-10T07:43:00.771 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:43:00.771 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 40 15 73060 73045 114.121 82.9375 0.0157067 0.00875755 2024-02-10T07:43:00.781 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:00.819 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:00.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:00.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:00.920 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:00.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:00.972 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:00.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:01.023 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:00 smithi170 ceph-mon[33394]: 6.14 scrub starts 2024-02-10T07:43:01.023 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:00 smithi170 ceph-mon[33394]: 6.14 scrub ok 2024-02-10T07:43:01.023 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:00 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:43:01.023 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:00 smithi170 ceph-mon[33394]: pgmap v504: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-10T07:43:01.023 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:00 smithi170 ceph-mon[33394]: osdmap e269: 8 total, 8 up, 8 in 2024-02-10T07:43:01.073 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:01.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:00 smithi008 ceph-mon[29816]: 6.14 scrub starts 2024-02-10T07:43:01.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:00 smithi008 ceph-mon[29816]: 6.14 scrub ok 2024-02-10T07:43:01.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:00 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:43:01.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:00 smithi008 ceph-mon[29816]: pgmap v504: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-10T07:43:01.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:00 smithi008 ceph-mon[29816]: osdmap e269: 8 total, 8 up, 8 in 2024-02-10T07:43:01.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:00 smithi008 ceph-mon[40743]: 6.14 scrub starts 2024-02-10T07:43:01.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:00 smithi008 ceph-mon[40743]: 6.14 scrub ok 2024-02-10T07:43:01.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:00 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:43:01.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:00 smithi008 ceph-mon[40743]: pgmap v504: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-10T07:43:01.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:00 smithi008 ceph-mon[40743]: osdmap e269: 8 total, 8 up, 8 in 2024-02-10T07:43:01.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:01.108 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:01.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:01.208 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:01.253 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:01.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:01.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:01.356 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:01.401 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:01.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:01.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:01.503 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:01.545 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:01.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:01.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:01.579 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:43:01.593 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:43:01.593 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2024-02-10T07:43:01.594 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2024-02-10T07:43:01.594 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2024-02-10T07:43:01.594 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:43:01.647 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:01.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:01.702 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:01.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:01.772 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 41 16 74308 74292 113.233 77.9375 0.0292495 0.00882507 2024-02-10T07:43:01.789 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 1 1 2024-02-10T07:43:01.802 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:01.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:01.889 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:01.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:01.990 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:02.022 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:02.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:02.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:02.124 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:02.157 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:02.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:02.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:02.197 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:01 smithi008 ceph-mon[29816]: 3.2 scrub starts 2024-02-10T07:43:02.197 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:01 smithi008 ceph-mon[29816]: 3.2 scrub ok 2024-02-10T07:43:02.197 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:01 smithi008 ceph-mon[29816]: 5.1d scrub starts 2024-02-10T07:43:02.197 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:01 smithi008 ceph-mon[29816]: 5.1d scrub ok 2024-02-10T07:43:02.197 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:01 smithi008 ceph-mon[29816]: 6.1c scrub starts 2024-02-10T07:43:02.197 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:01 smithi008 ceph-mon[29816]: 6.1c scrub ok 2024-02-10T07:43:02.197 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:01 smithi008 ceph-mon[29816]: osdmap e270: 8 total, 8 up, 8 in 2024-02-10T07:43:02.197 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:01 smithi008 ceph-mon[40743]: 3.2 scrub starts 2024-02-10T07:43:02.197 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:01 smithi008 ceph-mon[40743]: 3.2 scrub ok 2024-02-10T07:43:02.197 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:01 smithi008 ceph-mon[40743]: 5.1d scrub starts 2024-02-10T07:43:02.198 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:01 smithi008 ceph-mon[40743]: 5.1d scrub ok 2024-02-10T07:43:02.198 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:01 smithi008 ceph-mon[40743]: 6.1c scrub starts 2024-02-10T07:43:02.198 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:01 smithi008 ceph-mon[40743]: 6.1c scrub ok 2024-02-10T07:43:02.198 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:01 smithi008 ceph-mon[40743]: osdmap e270: 8 total, 8 up, 8 in 2024-02-10T07:43:02.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:01 smithi170 ceph-mon[33394]: 3.2 scrub starts 2024-02-10T07:43:02.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:01 smithi170 ceph-mon[33394]: 3.2 scrub ok 2024-02-10T07:43:02.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:01 smithi170 ceph-mon[33394]: 5.1d scrub starts 2024-02-10T07:43:02.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:01 smithi170 ceph-mon[33394]: 5.1d scrub ok 2024-02-10T07:43:02.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:01 smithi170 ceph-mon[33394]: 6.1c scrub starts 2024-02-10T07:43:02.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:01 smithi170 ceph-mon[33394]: 6.1c scrub ok 2024-02-10T07:43:02.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:01 smithi170 ceph-mon[33394]: osdmap e270: 8 total, 8 up, 8 in 2024-02-10T07:43:02.260 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:02.292 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:02.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:02.295 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-10T07:43:02.393 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:02.427 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:02.428 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:02.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:02.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:02.456 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:43:02.504 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:43:02.529 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:02.543 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:43:02.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:02.584 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:02.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:02.587 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:43:02.634 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:43:02.660 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:43:02.674 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:43:02.685 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:02.718 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:43:02.725 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:02.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:02.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:02.759 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:43:02.769 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:43:02.769 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:43:02.769 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:43:02.769 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:43:02.769 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:43:02.769 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:43:02.769 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:43:02.769 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:43:02.774 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 42 16 75298 75282 112.006 61.875 0.005306 0.0089205 2024-02-10T07:43:02.827 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:02.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:02.878 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:02.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:02.979 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:03.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:03.018 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:03.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:03.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:02 smithi008 ceph-mon[29816]: 5.17 scrub starts 2024-02-10T07:43:03.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:02 smithi008 ceph-mon[29816]: 5.17 scrub ok 2024-02-10T07:43:03.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:02 smithi008 ceph-mon[29816]: pgmap v506: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1010 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-10T07:43:03.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:02 smithi008 ceph-mon[29816]: 6.4 scrub starts 2024-02-10T07:43:03.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:02 smithi008 ceph-mon[29816]: 6.4 scrub ok 2024-02-10T07:43:03.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:02 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1421462549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:03.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:02 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:03.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:02 smithi008 ceph-mon[40743]: 5.17 scrub starts 2024-02-10T07:43:03.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:02 smithi008 ceph-mon[40743]: 5.17 scrub ok 2024-02-10T07:43:03.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:02 smithi008 ceph-mon[40743]: pgmap v506: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1010 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-10T07:43:03.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:02 smithi008 ceph-mon[40743]: 6.4 scrub starts 2024-02-10T07:43:03.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:02 smithi008 ceph-mon[40743]: 6.4 scrub ok 2024-02-10T07:43:03.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:02 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1421462549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:03.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:02 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:03.119 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:02 smithi170 ceph-mon[33394]: 5.17 scrub starts 2024-02-10T07:43:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:02 smithi170 ceph-mon[33394]: 5.17 scrub ok 2024-02-10T07:43:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:02 smithi170 ceph-mon[33394]: pgmap v506: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1010 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-10T07:43:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:02 smithi170 ceph-mon[33394]: 6.4 scrub starts 2024-02-10T07:43:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:02 smithi170 ceph-mon[33394]: 6.4 scrub ok 2024-02-10T07:43:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:02 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1421462549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:02 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:03.153 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:03.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:03.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:03.255 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:03.291 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:03.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:03.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:03.393 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:03.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:03.433 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:03.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:03.534 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:03.578 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:03.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:03.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:03.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:43:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:43:03.680 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:03.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:03.725 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:03.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:03.774 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 43 16 76460 76444 111.091 72.625 0.0228538 0.00899632 2024-02-10T07:43:03.826 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:03.869 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:03.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:03.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:03.970 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:04.004 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:04.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:04.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:04.050 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:04.105 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:04.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:03 smithi170 ceph-mon[33394]: 6.b scrub starts 2024-02-10T07:43:04.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:03 smithi170 ceph-mon[33394]: 4.15 scrub starts 2024-02-10T07:43:04.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:03 smithi170 ceph-mon[33394]: 4.15 scrub ok 2024-02-10T07:43:04.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:03 smithi170 ceph-mon[33394]: 6.b scrub ok 2024-02-10T07:43:04.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:03 smithi170 ceph-mon[33394]: 3.d scrub starts 2024-02-10T07:43:04.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:03 smithi170 ceph-mon[33394]: 3.d scrub ok 2024-02-10T07:43:04.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:03 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:04.138 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:04.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:04.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:04.240 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:04.244 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:03 smithi008 ceph-mon[29816]: 6.b scrub starts 2024-02-10T07:43:04.245 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:03 smithi008 ceph-mon[29816]: 4.15 scrub starts 2024-02-10T07:43:04.245 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:03 smithi008 ceph-mon[29816]: 4.15 scrub ok 2024-02-10T07:43:04.245 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:03 smithi008 ceph-mon[29816]: 6.b scrub ok 2024-02-10T07:43:04.245 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:03 smithi008 ceph-mon[29816]: 3.d scrub starts 2024-02-10T07:43:04.245 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:03 smithi008 ceph-mon[29816]: 3.d scrub ok 2024-02-10T07:43:04.245 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:03 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:04.245 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:03 smithi008 ceph-mon[40743]: 6.b scrub starts 2024-02-10T07:43:04.245 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:03 smithi008 ceph-mon[40743]: 4.15 scrub starts 2024-02-10T07:43:04.245 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:03 smithi008 ceph-mon[40743]: 4.15 scrub ok 2024-02-10T07:43:04.245 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:03 smithi008 ceph-mon[40743]: 6.b scrub ok 2024-02-10T07:43:04.245 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:03 smithi008 ceph-mon[40743]: 3.d scrub starts 2024-02-10T07:43:04.245 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:03 smithi008 ceph-mon[40743]: 3.d scrub ok 2024-02-10T07:43:04.245 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:03 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:04.279 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:04.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:04.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:04.381 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:04.417 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:04.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:04.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:04.519 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:04.558 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:04.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:04.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:04.691 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:04.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:04.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:04.774 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 44 16 77620 77604 110.213 72.5 0.0106183 0.00906803 2024-02-10T07:43:04.793 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:04.833 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:04.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:04.935 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:04.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:04.977 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:04.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:05.078 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:05.113 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:05.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:05.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:05.215 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:05.256 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:05.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:05.321 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:04 smithi008 ceph-mon[29816]: pgmap v507: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 849 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-02-10T07:43:05.321 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:04 smithi008 ceph-mon[29816]: 5.1e scrub starts 2024-02-10T07:43:05.321 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:04 smithi008 ceph-mon[29816]: 5.1e scrub ok 2024-02-10T07:43:05.321 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:04 smithi008 ceph-mon[29816]: 3.13 scrub starts 2024-02-10T07:43:05.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:04 smithi008 ceph-mon[29816]: 3.13 scrub ok 2024-02-10T07:43:05.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:04 smithi008 ceph-mon[40743]: pgmap v507: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 849 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-02-10T07:43:05.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:04 smithi008 ceph-mon[40743]: 5.1e scrub starts 2024-02-10T07:43:05.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:04 smithi008 ceph-mon[40743]: 5.1e scrub ok 2024-02-10T07:43:05.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:04 smithi008 ceph-mon[40743]: 3.13 scrub starts 2024-02-10T07:43:05.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:04 smithi008 ceph-mon[40743]: 3.13 scrub ok 2024-02-10T07:43:05.358 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:04 smithi170 ceph-mon[33394]: pgmap v507: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 849 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-02-10T07:43:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:04 smithi170 ceph-mon[33394]: 5.1e scrub starts 2024-02-10T07:43:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:04 smithi170 ceph-mon[33394]: 5.1e scrub ok 2024-02-10T07:43:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:04 smithi170 ceph-mon[33394]: 3.13 scrub starts 2024-02-10T07:43:05.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:04 smithi170 ceph-mon[33394]: 3.13 scrub ok 2024-02-10T07:43:05.399 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:05.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:05.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:05.501 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:05.541 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:05.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:05.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:05.642 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:05.682 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:05.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:05.774 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 45 16 78848 78832 109.47 76.75 0.00552266 0.00913053 2024-02-10T07:43:05.783 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:05.820 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:05.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:05.923 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:05.959 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:05.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:05.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:06.061 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:06.089 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:43:06.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:06.098 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:06.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:05 smithi170 ceph-mon[33394]: 4.1f scrub starts 2024-02-10T07:43:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:05 smithi170 ceph-mon[33394]: 4.1f scrub ok 2024-02-10T07:43:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:05 smithi170 ceph-mon[33394]: 5.6 scrub starts 2024-02-10T07:43:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:05 smithi170 ceph-mon[33394]: 5.6 scrub ok 2024-02-10T07:43:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:05 smithi170 ceph-mon[33394]: 3.6 scrub starts 2024-02-10T07:43:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:05 smithi170 ceph-mon[33394]: 3.6 scrub ok 2024-02-10T07:43:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:05 smithi170 ceph-mon[33394]: 3.f scrub starts 2024-02-10T07:43:06.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:05 smithi170 ceph-mon[33394]: 3.f scrub ok 2024-02-10T07:43:06.199 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:06.199 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[29816]: 4.1f scrub starts 2024-02-10T07:43:06.199 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[29816]: 4.1f scrub ok 2024-02-10T07:43:06.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[29816]: 5.6 scrub starts 2024-02-10T07:43:06.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[29816]: 5.6 scrub ok 2024-02-10T07:43:06.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[29816]: 3.6 scrub starts 2024-02-10T07:43:06.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[29816]: 3.6 scrub ok 2024-02-10T07:43:06.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[29816]: 3.f scrub starts 2024-02-10T07:43:06.200 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[29816]: 3.f scrub ok 2024-02-10T07:43:06.200 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[40743]: 4.1f scrub starts 2024-02-10T07:43:06.200 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[40743]: 4.1f scrub ok 2024-02-10T07:43:06.200 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[40743]: 5.6 scrub starts 2024-02-10T07:43:06.200 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[40743]: 5.6 scrub ok 2024-02-10T07:43:06.200 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[40743]: 3.6 scrub starts 2024-02-10T07:43:06.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[40743]: 3.6 scrub ok 2024-02-10T07:43:06.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[40743]: 3.f scrub starts 2024-02-10T07:43:06.201 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:05 smithi008 ceph-mon[40743]: 3.f scrub ok 2024-02-10T07:43:06.236 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:06.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:06.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:06.338 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:06.378 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:06.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:06.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:06.443 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:43:06.443 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:43:06.480 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:06.517 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:06.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:06.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:06.619 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:06.660 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:06.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:06.761 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:06.774 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 46 16 79948 79932 108.584 68.75 0.0055992 0.00920381 2024-02-10T07:43:06.804 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:06.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:06.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:06.906 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:06.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:06.939 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:06.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:07.040 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:07.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:07.075 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:07.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:07.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:06 smithi008 ceph-mon[29816]: pgmap v508: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-02-10T07:43:07.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:06 smithi008 ceph-mon[29816]: 3.15 scrub starts 2024-02-10T07:43:07.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:06 smithi008 ceph-mon[29816]: 3.15 scrub ok 2024-02-10T07:43:07.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:06 smithi008 ceph-mon[40743]: pgmap v508: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-02-10T07:43:07.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:06 smithi008 ceph-mon[40743]: 3.15 scrub starts 2024-02-10T07:43:07.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:06 smithi008 ceph-mon[40743]: 3.15 scrub ok 2024-02-10T07:43:07.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:06 smithi170 ceph-mon[33394]: pgmap v508: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-02-10T07:43:07.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:06 smithi170 ceph-mon[33394]: 3.15 scrub starts 2024-02-10T07:43:07.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:06 smithi170 ceph-mon[33394]: 3.15 scrub ok 2024-02-10T07:43:07.176 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:07.178 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-10T07:43:07.222 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:07.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:07.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:07.323 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:07.354 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:07.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:07.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:07.368 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:07.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:07.502 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:07.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:07.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:07.541 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:43:07.605 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:07.640 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:07.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:07.742 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:07.774 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 47 16 81008 80992 107.684 66.25 0.0180552 0.00928132 2024-02-10T07:43:07.782 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:07.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:07.884 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:07.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:07.920 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:07.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:07.980 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.1 to 1 (10000) 2024-02-10T07:43:07.989 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:43:08.020 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:08.060 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:08.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:08.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:07 smithi008 ceph-mon[29816]: 4.10 scrub starts 2024-02-10T07:43:08.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:07 smithi008 ceph-mon[29816]: 4.10 scrub ok 2024-02-10T07:43:08.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:07 smithi008 ceph-mon[29816]: 6.d scrub starts 2024-02-10T07:43:08.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:07 smithi008 ceph-mon[29816]: 6.d scrub ok 2024-02-10T07:43:08.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:07 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1431765977' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:08.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:07 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:08.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:07 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2603236309' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:43:08.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:07 smithi008 ceph-mon[40743]: 4.10 scrub starts 2024-02-10T07:43:08.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:07 smithi008 ceph-mon[40743]: 4.10 scrub ok 2024-02-10T07:43:08.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:07 smithi008 ceph-mon[40743]: 6.d scrub starts 2024-02-10T07:43:08.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:07 smithi008 ceph-mon[40743]: 6.d scrub ok 2024-02-10T07:43:08.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:07 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1431765977' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:08.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:07 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:08.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:07 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2603236309' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:43:08.162 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:08.199 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:08.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:08.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:07 smithi170 ceph-mon[33394]: 4.10 scrub starts 2024-02-10T07:43:08.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:07 smithi170 ceph-mon[33394]: 4.10 scrub ok 2024-02-10T07:43:08.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:07 smithi170 ceph-mon[33394]: 6.d scrub starts 2024-02-10T07:43:08.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:07 smithi170 ceph-mon[33394]: 6.d scrub ok 2024-02-10T07:43:08.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:07 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1431765977' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:08.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:07 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:08.253 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:07 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2603236309' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:43:08.302 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:08.336 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:08.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:08.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:08.438 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:08.474 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:08.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:08.575 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:08.604 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:08.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:08.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:08.706 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:08.744 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:08.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:08.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:08.774 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 48 16 82272 82256 107.086 79 0.00988139 0.00933306 2024-02-10T07:43:08.846 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:08.888 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:08.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:08.974 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:43:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:43:08.974 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: 4.a scrub starts 2024-02-10T07:43:08.990 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:09.026 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:09.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:09.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:09.128 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: 4.a scrub starts 2024-02-10T07:43:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: 4.a scrub ok 2024-02-10T07:43:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: pgmap v509: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-10T07:43:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: 5.d scrub starts 2024-02-10T07:43:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: 3.19 scrub starts 2024-02-10T07:43:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:43:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:43:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: 3.19 scrub ok 2024-02-10T07:43:09.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: 5.d scrub ok 2024-02-10T07:43:09.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:43:09.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2603236309' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:43:09.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: osdmap e271: 8 total, 8 up, 8 in 2024-02-10T07:43:09.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: 4.17 scrub starts 2024-02-10T07:43:09.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: 4.17 scrub ok 2024-02-10T07:43:09.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: 5.9 scrub starts 2024-02-10T07:43:09.134 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:08 smithi170 ceph-mon[33394]: 5.9 scrub ok 2024-02-10T07:43:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:09.172 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:09.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:09.251 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: 4.a scrub starts 2024-02-10T07:43:09.251 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: 4.a scrub ok 2024-02-10T07:43:09.251 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: pgmap v509: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-10T07:43:09.251 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: 5.d scrub starts 2024-02-10T07:43:09.251 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: 3.19 scrub starts 2024-02-10T07:43:09.251 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:43:09.251 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:43:09.251 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: 3.19 scrub ok 2024-02-10T07:43:09.251 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: 5.d scrub ok 2024-02-10T07:43:09.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:43:09.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2603236309' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:43:09.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: osdmap e271: 8 total, 8 up, 8 in 2024-02-10T07:43:09.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: 4.17 scrub starts 2024-02-10T07:43:09.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: 4.17 scrub ok 2024-02-10T07:43:09.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: 5.9 scrub starts 2024-02-10T07:43:09.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[29816]: 5.9 scrub ok 2024-02-10T07:43:09.252 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: 4.a scrub ok 2024-02-10T07:43:09.252 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: pgmap v509: 148 pgs: 148 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-10T07:43:09.252 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: 5.d scrub starts 2024-02-10T07:43:09.252 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: 3.19 scrub starts 2024-02-10T07:43:09.252 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:43:09.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:43:09.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: 3.19 scrub ok 2024-02-10T07:43:09.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: 5.d scrub ok 2024-02-10T07:43:09.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:43:09.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2603236309' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:43:09.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: osdmap e271: 8 total, 8 up, 8 in 2024-02-10T07:43:09.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: 4.17 scrub starts 2024-02-10T07:43:09.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: 4.17 scrub ok 2024-02-10T07:43:09.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: 5.9 scrub starts 2024-02-10T07:43:09.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:08 smithi008 ceph-mon[40743]: 5.9 scrub ok 2024-02-10T07:43:09.273 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:09.311 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:09.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:09.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:09.413 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:09.443 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:09.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:09.545 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:09.578 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:09.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:09.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:09.680 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:09.715 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:09.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:09.739 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 2 1 2024-02-10T07:43:09.775 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 49 16 83510 83494 106.478 77.375 0.0208558 0.00938649 2024-02-10T07:43:09.812 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:43:09.812 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:43:09.817 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:09.856 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:09.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:09.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:09.958 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:09.984 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:09.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:10.086 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:10.136 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:10.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:10.238 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:10.288 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:10.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:10.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:10.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:09 smithi008 ceph-mon[29816]: 5.2 scrub starts 2024-02-10T07:43:10.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[29816]: 5.2 scrub ok 2024-02-10T07:43:10.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[29816]: 3.0 scrub starts 2024-02-10T07:43:10.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[29816]: 3.0 scrub ok 2024-02-10T07:43:10.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[29816]: 6.15 scrub starts 2024-02-10T07:43:10.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[29816]: 6.15 scrub ok 2024-02-10T07:43:10.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:09 smithi008 ceph-mon[40743]: 5.2 scrub starts 2024-02-10T07:43:10.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[40743]: 5.2 scrub ok 2024-02-10T07:43:10.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[40743]: 3.0 scrub starts 2024-02-10T07:43:10.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[40743]: 3.0 scrub ok 2024-02-10T07:43:10.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[40743]: 6.15 scrub starts 2024-02-10T07:43:10.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[40743]: 6.15 scrub ok 2024-02-10T07:43:10.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:09 smithi170 ceph-mon[33394]: 5.2 scrub starts 2024-02-10T07:43:10.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:09 smithi170 ceph-mon[33394]: 5.2 scrub ok 2024-02-10T07:43:10.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:09 smithi170 ceph-mon[33394]: 3.0 scrub starts 2024-02-10T07:43:10.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:09 smithi170 ceph-mon[33394]: 3.0 scrub ok 2024-02-10T07:43:10.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:09 smithi170 ceph-mon[33394]: 6.15 scrub starts 2024-02-10T07:43:10.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:09 smithi170 ceph-mon[33394]: 6.15 scrub ok 2024-02-10T07:43:10.390 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:10.428 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:10.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:10.530 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:10.541 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-10T07:43:10.570 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:10.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:10.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:10.672 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:10.717 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:10.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:10.775 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 50 16 84586 84570 105.694 67.25 0.0122984 0.00945589 2024-02-10T07:43:10.819 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:10.872 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:10.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:10.928 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:43:10.973 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:11.015 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:11.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:11.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[29816]: 3.1d scrub starts 2024-02-10T07:43:11.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[29816]: 3.1d scrub ok 2024-02-10T07:43:11.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[29816]: pgmap v511: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-10T07:43:11.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[29816]: 3.b deep-scrub starts 2024-02-10T07:43:11.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[29816]: 3.b deep-scrub ok 2024-02-10T07:43:11.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[29816]: 4.3 deep-scrub starts 2024-02-10T07:43:11.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[29816]: 4.3 deep-scrub ok 2024-02-10T07:43:11.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[29816]: 6.e scrub starts 2024-02-10T07:43:11.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[29816]: 6.e scrub ok 2024-02-10T07:43:11.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[40743]: 3.1d scrub starts 2024-02-10T07:43:11.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[40743]: 3.1d scrub ok 2024-02-10T07:43:11.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[40743]: pgmap v511: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-10T07:43:11.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[40743]: 3.b deep-scrub starts 2024-02-10T07:43:11.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[40743]: 3.b deep-scrub ok 2024-02-10T07:43:11.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[40743]: 4.3 deep-scrub starts 2024-02-10T07:43:11.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[40743]: 4.3 deep-scrub ok 2024-02-10T07:43:11.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[40743]: 6.e scrub starts 2024-02-10T07:43:11.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:10 smithi008 ceph-mon[40743]: 6.e scrub ok 2024-02-10T07:43:11.116 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:11.135 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:43:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:11.140 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:11.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:11.241 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:11.287 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:11.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:11.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:11.340 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:10 smithi170 ceph-mon[33394]: 3.1d scrub starts 2024-02-10T07:43:11.340 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:10 smithi170 ceph-mon[33394]: 3.1d scrub ok 2024-02-10T07:43:11.340 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:10 smithi170 ceph-mon[33394]: pgmap v511: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-10T07:43:11.340 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:10 smithi170 ceph-mon[33394]: 3.b deep-scrub starts 2024-02-10T07:43:11.340 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:10 smithi170 ceph-mon[33394]: 3.b deep-scrub ok 2024-02-10T07:43:11.340 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:10 smithi170 ceph-mon[33394]: 4.3 deep-scrub starts 2024-02-10T07:43:11.340 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:10 smithi170 ceph-mon[33394]: 4.3 deep-scrub ok 2024-02-10T07:43:11.340 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:10 smithi170 ceph-mon[33394]: 6.e scrub starts 2024-02-10T07:43:11.340 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:10 smithi170 ceph-mon[33394]: 6.e scrub ok 2024-02-10T07:43:11.389 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:11.414 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:11.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:11.516 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:11.551 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:11.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:11.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:11.653 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:11.689 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:11.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:11.775 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 51 16 85656 85640 104.932 66.875 0.0121137 0.00952525 2024-02-10T07:43:11.793 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:11.831 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:11.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:11.933 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:11.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:11.971 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:11.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:12.072 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:12.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:12.105 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:12.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:12.206 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:12.230 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:12.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:12.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:12.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:11 smithi008 ceph-mon[29816]: 3.1c scrub starts 2024-02-10T07:43:12.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[29816]: 3.1c scrub ok 2024-02-10T07:43:12.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[29816]: 2.1 scrub starts 2024-02-10T07:43:12.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[29816]: 2.1 scrub ok 2024-02-10T07:43:12.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[29816]: 4.1 scrub starts 2024-02-10T07:43:12.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[29816]: 4.1 scrub ok 2024-02-10T07:43:12.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:11 smithi008 ceph-mon[40743]: 3.1c scrub starts 2024-02-10T07:43:12.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[40743]: 3.1c scrub ok 2024-02-10T07:43:12.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[40743]: 2.1 scrub starts 2024-02-10T07:43:12.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[40743]: 2.1 scrub ok 2024-02-10T07:43:12.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[40743]: 4.1 scrub starts 2024-02-10T07:43:12.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[40743]: 4.1 scrub ok 2024-02-10T07:43:12.333 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:12.350 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:11 smithi170 ceph-mon[33394]: 3.1c scrub starts 2024-02-10T07:43:12.350 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:11 smithi170 ceph-mon[33394]: 3.1c scrub ok 2024-02-10T07:43:12.350 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:11 smithi170 ceph-mon[33394]: 2.1 scrub starts 2024-02-10T07:43:12.350 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:11 smithi170 ceph-mon[33394]: 2.1 scrub ok 2024-02-10T07:43:12.350 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:11 smithi170 ceph-mon[33394]: 4.1 scrub starts 2024-02-10T07:43:12.350 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:11 smithi170 ceph-mon[33394]: 4.1 scrub ok 2024-02-10T07:43:12.369 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:12.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:12.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:12.471 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:12.511 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:12.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:12.613 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:12.650 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:12.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:12.751 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:12.775 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 52 16 86764 86748 104.246 69.25 0.00890372 0.00958791 2024-02-10T07:43:12.787 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:12.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:12.883 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:43:12.883 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:43:12.890 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:12.945 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:12.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:12.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:13.047 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:13.049 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[29816]: 6.11 scrub starts 2024-02-10T07:43:13.050 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[29816]: pgmap v512: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-02-10T07:43:13.050 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[29816]: 6.11 scrub ok 2024-02-10T07:43:13.050 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[29816]: 3.17 scrub starts 2024-02-10T07:43:13.050 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[29816]: 3.17 scrub ok 2024-02-10T07:43:13.050 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[29816]: 5.1a scrub starts 2024-02-10T07:43:13.050 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[29816]: 5.1a scrub ok 2024-02-10T07:43:13.050 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[40743]: 6.11 scrub starts 2024-02-10T07:43:13.050 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[40743]: pgmap v512: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-02-10T07:43:13.050 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[40743]: 6.11 scrub ok 2024-02-10T07:43:13.051 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[40743]: 3.17 scrub starts 2024-02-10T07:43:13.051 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[40743]: 3.17 scrub ok 2024-02-10T07:43:13.051 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[40743]: 5.1a scrub starts 2024-02-10T07:43:13.051 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:12 smithi008 ceph-mon[40743]: 5.1a scrub ok 2024-02-10T07:43:13.080 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:13.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:13.182 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:13.204 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:13.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:13.252 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:13.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:13.318 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:12 smithi170 ceph-mon[33394]: 6.11 scrub starts 2024-02-10T07:43:13.318 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:12 smithi170 ceph-mon[33394]: pgmap v512: 148 pgs: 148 active+clean; 5.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-02-10T07:43:13.318 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:12 smithi170 ceph-mon[33394]: 6.11 scrub ok 2024-02-10T07:43:13.318 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:12 smithi170 ceph-mon[33394]: 3.17 scrub starts 2024-02-10T07:43:13.318 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:12 smithi170 ceph-mon[33394]: 3.17 scrub ok 2024-02-10T07:43:13.318 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:12 smithi170 ceph-mon[33394]: 5.1a scrub starts 2024-02-10T07:43:13.319 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:12 smithi170 ceph-mon[33394]: 5.1a scrub ok 2024-02-10T07:43:13.353 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:13.394 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:13.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:13.481 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:13.496 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:13.544 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:13.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:13.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:13.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:43:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:43:13.572 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-10T07:43:13.596 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:43:13.632 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:43:13.646 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:13.667 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:43:13.687 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:13.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:13.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:13.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:43:13.751 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:43:13.775 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 53 16 87863 87847 103.575 68.6875 0.0154348 0.00964904 2024-02-10T07:43:13.789 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:13.801 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:43:13.836 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:13.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:13.841 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:43:13.884 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:43:13.889 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:43:13.889 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:43:13.889 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:43:13.890 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:43:13.890 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:43:13.890 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:43:13.890 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:43:13.890 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:43:13.938 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:13.952 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:43:13.991 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:13.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:13.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:14.093 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:14.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:14.145 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:14.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:14.246 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:14.267 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:13 smithi170 ceph-mon[33394]: 2.2 scrub starts 2024-02-10T07:43:14.267 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:13 smithi170 ceph-mon[33394]: 2.2 scrub ok 2024-02-10T07:43:14.268 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:13 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:14.268 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:13 smithi170 ceph-mon[33394]: 1.0 scrub starts 2024-02-10T07:43:14.268 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:13 smithi170 ceph-mon[33394]: 1.0 scrub ok 2024-02-10T07:43:14.268 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:13 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2519059271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:14.286 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:14.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:14.303 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:13 smithi008 ceph-mon[29816]: 2.2 scrub starts 2024-02-10T07:43:14.304 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:13 smithi008 ceph-mon[29816]: 2.2 scrub ok 2024-02-10T07:43:14.304 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:13 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:14.304 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:13 smithi008 ceph-mon[29816]: 1.0 scrub starts 2024-02-10T07:43:14.304 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:13 smithi008 ceph-mon[29816]: 1.0 scrub ok 2024-02-10T07:43:14.304 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:13 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2519059271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:14.304 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:13 smithi008 ceph-mon[40743]: 2.2 scrub starts 2024-02-10T07:43:14.304 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:13 smithi008 ceph-mon[40743]: 2.2 scrub ok 2024-02-10T07:43:14.304 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:13 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:14.304 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:13 smithi008 ceph-mon[40743]: 1.0 scrub starts 2024-02-10T07:43:14.304 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:13 smithi008 ceph-mon[40743]: 1.0 scrub ok 2024-02-10T07:43:14.305 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:13 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2519059271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:14.388 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:14.427 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:14.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:14.528 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:14.567 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:14.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:14.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:14.669 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:14.703 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:14.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:14.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:14.776 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 54 16 89083 89067 103.069 76.25 0.00591953 0.00969696 2024-02-10T07:43:14.805 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:14.842 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:14.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:14.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:14.944 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:14.985 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:14.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:15.055 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:15 smithi008 ceph-mon[40743]: pgmap v513: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-10T07:43:15.056 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:15 smithi008 ceph-mon[40743]: 6.5 scrub starts 2024-02-10T07:43:15.056 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:15 smithi008 ceph-mon[40743]: 6.5 scrub ok 2024-02-10T07:43:15.056 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:15 smithi008 ceph-mon[29816]: pgmap v513: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-10T07:43:15.056 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:15 smithi008 ceph-mon[29816]: 6.5 scrub starts 2024-02-10T07:43:15.056 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:15 smithi008 ceph-mon[29816]: 6.5 scrub ok 2024-02-10T07:43:15.086 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:15.126 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:15.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:15.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:15.228 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:15.267 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:15.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:15.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:15.369 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:15.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:15 smithi170 ceph-mon[33394]: pgmap v513: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-10T07:43:15.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:15 smithi170 ceph-mon[33394]: 6.5 scrub starts 2024-02-10T07:43:15.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:15 smithi170 ceph-mon[33394]: 6.5 scrub ok 2024-02-10T07:43:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:15.407 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:15.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:15.508 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:15.541 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:15.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:15.643 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:15.682 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:15.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:15.775 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 55 16 90316 90300 102.596 77.0625 0.0143909 0.00974235 2024-02-10T07:43:15.784 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:15.818 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:15.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:15.923 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:15.969 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:15.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:16.000 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:43:16.000 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2024-02-10T07:43:16.000 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2024-02-10T07:43:16.000 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:43:16.000 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.000 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210702 7.12 7:49611789:::benchmark_data_smithi008_82046_object95564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.058811+0000", 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.93738959299998, 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0023876279999999, 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210702 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058811+0000", 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.001 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058811+0000", 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058812+0000", 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6799999999999999e-07 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058815+0000", 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.881e-06 2024-02-10T07:43:16.002 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058824+0000", 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.2650000000000002e-06 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058830+0000", 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.242e-06 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058883+0000", 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3631999999999997e-05 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.003 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058893+0000", 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0258000000000001e-05 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.042033+0000", 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.98313994000000005 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.042128+0000", 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4141000000000005e-05 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.004 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.042439+0000", 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00031155199999999999 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.045767+0000", 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033280839999999998 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.048185+0000", 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0024177080000000002 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.005 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061099+0000", 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012914583 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061157+0000", 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7429000000000002e-05 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061199+0000", 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2045000000000001e-05 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.006 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210713 7.12 7:48995235:::benchmark_data_smithi008_82046_object95575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.069890+0000", 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.92631109299998, 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.99147127099999999, 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210713 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.007 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069890+0000", 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069890+0000", 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069891+0000", 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5880000000000001e-06 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.008 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069896+0000", 2024-02-10T07:43:16.009 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9250000000000003e-06 2024-02-10T07:43:16.009 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.009 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.009 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.009 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069909+0000", 2024-02-10T07:43:16.009 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2471e-05 2024-02-10T07:43:16.009 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.009 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.009 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.009 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069917+0000", 2024-02-10T07:43:16.009 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1850000000000004e-06 2024-02-10T07:43:16.009 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.009 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.009 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.070014+0000", 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6675999999999994e-05 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.070022+0000", 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7499999999999992e-06 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.044238+0000", 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.97421551399999995 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.010 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.011 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.044356+0000", 2024-02-10T07:43:16.011 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000117617 2024-02-10T07:43:16.011 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.011 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.011 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:43:16.011 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.044601+0000", 2024-02-10T07:43:16.011 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024531800000000001 2024-02-10T07:43:16.011 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.011 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.011 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.011 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049272+0000", 2024-02-10T07:43:16.011 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0046710420000000002 2024-02-10T07:43:16.011 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.012 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.012 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.012 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050384+0000", 2024-02-10T07:43:16.012 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001111721 2024-02-10T07:43:16.012 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.012 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.012 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.012 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061261+0000", 2024-02-10T07:43:16.012 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010877045 2024-02-10T07:43:16.012 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.012 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.012 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.012 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061319+0000", 2024-02-10T07:43:16.013 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8415999999999998e-05 2024-02-10T07:43:16.013 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.013 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.013 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.013 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061361+0000", 2024-02-10T07:43:16.013 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2002999999999997e-05 2024-02-10T07:43:16.013 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210719 7.12 7:4a954bf6:::benchmark_data_smithi008_82046_object95581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.075516+0000", 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.92068485499999, 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.98756840400000001, 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210719 2024-02-10T07:43:16.014 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075516+0000", 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075516+0000", 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075517+0000", 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4219999999999999e-06 2024-02-10T07:43:16.015 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075522+0000", 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7779999999999998e-06 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075534+0000", 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.173e-05 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075544+0000", 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0105000000000001e-05 2024-02-10T07:43:16.016 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.017 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.017 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.017 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075592+0000", 2024-02-10T07:43:16.017 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8173000000000001e-05 2024-02-10T07:43:16.017 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.017 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.017 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.017 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075601+0000", 2024-02-10T07:43:16.017 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5890000000000003e-06 2024-02-10T07:43:16.017 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.017 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.017 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.017 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.045802+0000", 2024-02-10T07:43:16.017 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.97020084299999998 2024-02-10T07:43:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.045871+0000", 2024-02-10T07:43:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9725000000000001e-05 2024-02-10T07:43:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:43:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.046021+0000", 2024-02-10T07:43:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014953899999999999 2024-02-10T07:43:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.018 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049278+0000", 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0032569140000000001 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050430+0000", 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0011520949999999999 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.062999+0000", 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012568823999999999 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.019 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063050+0000", 2024-02-10T07:43:16.020 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0788999999999998e-05 2024-02-10T07:43:16.020 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.020 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.020 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.020 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063084+0000", 2024-02-10T07:43:16.020 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4878e-05 2024-02-10T07:43:16.020 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.020 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.020 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.020 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.020 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.020 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210730 7.12 7:4fc3a9f7:::benchmark_data_smithi008_82046_object95592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:43:16.020 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.081362+0000", 2024-02-10T07:43:16.020 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.91483857700001, 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.98183658399999996, 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210730 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081362+0000", 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.021 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081362+0000", 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081364+0000", 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4219999999999999e-06 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081368+0000", 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5770000000000004e-06 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.022 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081381+0000", 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3118e-05 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081389+0000", 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.379e-06 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081443+0000", 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4475000000000003e-05 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.023 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081452+0000", 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.32e-06 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.046907+0000", 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.96545532300000003 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.047023+0000", 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000115832 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:43:16.024 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.047228+0000", 2024-02-10T07:43:16.025 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020531600000000001 2024-02-10T07:43:16.025 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.025 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.025 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.025 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050394+0000", 2024-02-10T07:43:16.025 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0031658239999999998 2024-02-10T07:43:16.025 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.025 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.025 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.025 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055109+0000", 2024-02-10T07:43:16.025 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0047147710000000004 2024-02-10T07:43:16.025 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.025 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.025 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063136+0000", 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0080273949999999997 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063176+0000", 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0142999999999998e-05 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063199+0000", 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2688999999999999e-05 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.026 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.027 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.027 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.027 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210739 7.12 7:4d4f1063:::benchmark_data_smithi008_82046_object95601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:43:16.027 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.090931+0000", 2024-02-10T07:43:16.027 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.90526967699998, 2024-02-10T07:43:16.027 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.97239903000000005, 2024-02-10T07:43:16.027 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.027 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.027 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.027 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.027 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.027 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210739 2024-02-10T07:43:16.027 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.027 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090931+0000", 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090931+0000", 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090932+0000", 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2309999999999999e-06 2024-02-10T07:43:16.028 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.029 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.029 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.029 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090937+0000", 2024-02-10T07:43:16.029 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5650000000000003e-06 2024-02-10T07:43:16.029 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.029 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.029 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.029 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090954+0000", 2024-02-10T07:43:16.029 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6515e-05 2024-02-10T07:43:16.029 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.029 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.029 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.029 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090962+0000", 2024-02-10T07:43:16.029 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0600000000000008e-06 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.091039+0000", 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7632999999999993e-05 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.091050+0000", 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0491e-05 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.048220+0000", 2024-02-10T07:43:16.030 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.95717007499999995 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.048294+0000", 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4536000000000006e-05 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.048444+0000", 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014942699999999999 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.051558+0000", 2024-02-10T07:43:16.031 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0031144889999999998 2024-02-10T07:43:16.032 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.032 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.032 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.032 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.058790+0000", 2024-02-10T07:43:16.032 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0072319430000000002 2024-02-10T07:43:16.032 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.032 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.032 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.032 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063249+0000", 2024-02-10T07:43:16.032 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0044584890000000004 2024-02-10T07:43:16.032 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.032 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.032 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.032 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063296+0000", 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7327999999999998e-05 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063330+0000", 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4248000000000001e-05 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210750 7.12 7:4d3f77f1:::benchmark_data_smithi008_82046_object95612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.097423+0000", 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.89877749099998, 2024-02-10T07:43:16.033 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.96606917000000003, 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210750 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097423+0000", 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.034 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097423+0000", 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097425+0000", 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9080000000000002e-06 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097430+0000", 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2390000000000004e-06 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.035 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097441+0000", 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1257000000000001e-05 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097449+0000", 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7430000000000002e-06 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097502+0000", 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3361e-05 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.036 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097510+0000", 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.4449999999999996e-06 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049307+0000", 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.95179659999999999 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049373+0000", 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5612999999999997e-05 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:43:16.037 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049521+0000", 2024-02-10T07:43:16.038 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014814799999999999 2024-02-10T07:43:16.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.038 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.038 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.038 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.053835+0000", 2024-02-10T07:43:16.038 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0043143499999999998 2024-02-10T07:43:16.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.038 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.038 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.038 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061401+0000", 2024-02-10T07:43:16.038 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0075663919999999999 2024-02-10T07:43:16.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.038 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.038 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.039 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063378+0000", 2024-02-10T07:43:16.039 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0019766499999999999 2024-02-10T07:43:16.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.039 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.039 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.040 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063453+0000", 2024-02-10T07:43:16.040 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4935e-05 2024-02-10T07:43:16.040 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.040 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.040 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.040 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063493+0000", 2024-02-10T07:43:16.040 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9529e-05 2024-02-10T07:43:16.040 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.040 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.040 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.040 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210807 7.12 7:4f675a45:::benchmark_data_smithi008_82046_object95669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.148770+0000", 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.84743085500003, 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.91486412100000003, 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210807 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.041 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148770+0000", 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148770+0000", 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148771+0000", 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1710000000000001e-06 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.042 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148775+0000", 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1420000000000003e-06 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148788+0000", 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2897e-05 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148797+0000", 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2689999999999999e-06 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.043 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148849+0000", 2024-02-10T07:43:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2095000000000002e-05 2024-02-10T07:43:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148857+0000", 2024-02-10T07:43:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9389999999999997e-06 2024-02-10T07:43:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050456+0000", 2024-02-10T07:43:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.90159990199999995 2024-02-10T07:43:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.044 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050527+0000", 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0321000000000007e-05 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050675+0000", 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000147725 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055123+0000", 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0044484379999999999 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.045 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061450+0000", 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0063272709999999998 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063551+0000", 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0021005830000000001 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063603+0000", 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2441000000000002e-05 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.046 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063634+0000", 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0926999999999997e-05 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210838 7.12 7:4d53824f:::benchmark_data_smithi008_82046_object95700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.176589+0000", 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.81961215000001, 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.88719636599999996, 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.047 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210838 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176589+0000", 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176589+0000", 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.048 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176590+0000", 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.198e-06 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176594+0000", 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8840000000000004e-06 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176607+0000", 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2855e-05 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.049 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.050 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.050 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176616+0000", 2024-02-10T07:43:16.050 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1339999999999993e-06 2024-02-10T07:43:16.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.050 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.050 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.050 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176667+0000", 2024-02-10T07:43:16.050 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0887999999999998e-05 2024-02-10T07:43:16.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.050 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.050 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.050 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176676+0000", 2024-02-10T07:43:16.050 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1009999999999998e-06 2024-02-10T07:43:16.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.051582+0000", 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.87490583200000005 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.051647+0000", 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5252999999999993e-05 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.051803+0000", 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000156336 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.051 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055129+0000", 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033258559999999999 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.062510+0000", 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0073805119999999997 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063680+0000", 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001170508 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.052 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063745+0000", 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5273999999999999e-05 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063785+0000", 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9734999999999998e-05 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210862 7.12 7:48875dbe:::benchmark_data_smithi008_82046_object95724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:43:16.053 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.256071+0000", 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.74012948799998, 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.81128573100000001, 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210862 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256071+0000", 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.054 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256071+0000", 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256072+0000", 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5199999999999995e-07 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256076+0000", 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7910000000000002e-06 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.055 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.056 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.056 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256084+0000", 2024-02-10T07:43:16.056 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0860000000000002e-06 2024-02-10T07:43:16.056 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.056 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.056 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.056 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256089+0000", 2024-02-10T07:43:16.056 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0549999999999999e-06 2024-02-10T07:43:16.056 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.056 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.056 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.056 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256126+0000", 2024-02-10T07:43:16.056 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6480000000000003e-05 2024-02-10T07:43:16.056 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.057 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.057 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.057 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256131+0000", 2024-02-10T07:43:16.057 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7259999999999999e-06 2024-02-10T07:43:16.057 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.057 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.057 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.057 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.052653+0000", 2024-02-10T07:43:16.057 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.79652150600000005 2024-02-10T07:43:16.057 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.057 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.057 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.057 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.052764+0000", 2024-02-10T07:43:16.057 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000111022 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.052966+0000", 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000202442 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057531+0000", 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0045644800000000001 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.062542+0000", 2024-02-10T07:43:16.058 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0050112000000000004 2024-02-10T07:43:16.059 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.059 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.059 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.059 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.067300+0000", 2024-02-10T07:43:16.059 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0047581560000000004 2024-02-10T07:43:16.059 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.059 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.059 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.059 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.067333+0000", 2024-02-10T07:43:16.059 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2437999999999997e-05 2024-02-10T07:43:16.059 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.059 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.059 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.059 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.067357+0000", 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4496999999999999e-05 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210891 7.12 7:4e0bd40b:::benchmark_data_smithi008_82046_object95753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.296428+0000", 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.69977302799998, 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.77191008299999997, 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.060 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210891 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296428+0000", 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296428+0000", 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.061 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296430+0000", 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9659999999999999e-06 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296435+0000", 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8509999999999998e-06 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296452+0000", 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6888000000000002e-05 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296461+0000", 2024-02-10T07:43:16.062 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4429999999999992e-06 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296628+0000", 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000166675 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296636+0000", 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6770000000000001e-06 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.053881+0000", 2024-02-10T07:43:16.063 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.75724445799999995 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.053969+0000", 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7972000000000004e-05 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.054161+0000", 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019222400000000001 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057538+0000", 2024-02-10T07:43:16.064 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033767060000000002 2024-02-10T07:43:16.065 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.065 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.065 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.065 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.067399+0000", 2024-02-10T07:43:16.065 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0098609119999999995 2024-02-10T07:43:16.065 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.065 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.065 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.065 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.068281+0000", 2024-02-10T07:43:16.065 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000881857 2024-02-10T07:43:16.065 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.065 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.065 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.065 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.068312+0000", 2024-02-10T07:43:16.066 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.1659999999999998e-05 2024-02-10T07:43:16.066 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.066 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.066 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.066 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.068338+0000", 2024-02-10T07:43:16.066 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5794000000000001e-05 2024-02-10T07:43:16.066 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.066 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.066 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.066 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.066 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.066 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210902 7.12 7:4cdd2df1:::benchmark_data_smithi008_82046_object95764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:43:16.066 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.308067+0000", 2024-02-10T07:43:16.066 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.688133768, 2024-02-10T07:43:16.067 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.76995323400000004, 2024-02-10T07:43:16.067 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.067 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.067 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.067 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.067 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.067 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210902 2024-02-10T07:43:16.067 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.067 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.067 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.067 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.067 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308067+0000", 2024-02-10T07:43:16.067 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.068 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.068 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.068 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.068 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308067+0000", 2024-02-10T07:43:16.068 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.068 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.068 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.068 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.068 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308068+0000", 2024-02-10T07:43:16.068 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.077e-06 2024-02-10T07:43:16.068 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.068 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.068 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308072+0000", 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8020000000000002e-06 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308083+0000", 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0743e-05 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308091+0000", 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.824e-06 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.069 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.070 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308147+0000", 2024-02-10T07:43:16.070 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5998000000000002e-05 2024-02-10T07:43:16.070 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.070 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.070 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.070 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308157+0000", 2024-02-10T07:43:16.070 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9250000000000007e-06 2024-02-10T07:43:16.070 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.070 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.070 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.070 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055146+0000", 2024-02-10T07:43:16.070 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.74698950799999997 2024-02-10T07:43:16.070 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.070 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.071 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.071 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055221+0000", 2024-02-10T07:43:16.071 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4597000000000004e-05 2024-02-10T07:43:16.071 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.071 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.071 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:43:16.071 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055375+0000", 2024-02-10T07:43:16.071 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015408300000000001 2024-02-10T07:43:16.071 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.071 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.071 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.071 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.058799+0000", 2024-02-10T07:43:16.071 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0034239380000000001 2024-02-10T07:43:16.072 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.072 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.072 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.072 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.068376+0000", 2024-02-10T07:43:16.072 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0095769779999999999 2024-02-10T07:43:16.072 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.072 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.072 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.072 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.077940+0000", 2024-02-10T07:43:16.072 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0095647360000000008 2024-02-10T07:43:16.072 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.072 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.072 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.072 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.077981+0000", 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0688000000000001e-05 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078020+0000", 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9337e-05 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210911 7.12 7:4f38fa58:::benchmark_data_smithi008_82046_object95773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.318835+0000", 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.67736577199997, 2024-02-10T07:43:16.073 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.75930091600000005, 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210911 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318835+0000", 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.074 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318835+0000", 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318836+0000", 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1510000000000001e-06 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318840+0000", 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1309999999999999e-06 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.075 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318850+0000", 2024-02-10T07:43:16.076 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6690000000000001e-06 2024-02-10T07:43:16.076 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.076 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.076 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.076 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318855+0000", 2024-02-10T07:43:16.076 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3410000000000002e-06 2024-02-10T07:43:16.076 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.076 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.076 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.076 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318904+0000", 2024-02-10T07:43:16.076 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8712999999999999e-05 2024-02-10T07:43:16.076 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.076 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.076 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318913+0000", 2024-02-10T07:43:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.8650000000000007e-06 2024-02-10T07:43:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.056341+0000", 2024-02-10T07:43:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.73742814899999998 2024-02-10T07:43:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.056452+0000", 2024-02-10T07:43:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000110511 2024-02-10T07:43:16.077 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.056655+0000", 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020361199999999999 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061057+0000", 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.004401861 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.076677+0000", 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015619687 2024-02-10T07:43:16.078 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.079 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078061+0000", 2024-02-10T07:43:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0013845089999999999 2024-02-10T07:43:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078102+0000", 2024-02-10T07:43:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.108e-05 2024-02-10T07:43:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.080 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078136+0000", 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3637000000000003e-05 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210950 7.12 7:4bea578c:::benchmark_data_smithi008_82046_object95812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.363086+0000", 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.63311480099998, 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.71519846200000003, 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.081 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210950 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363086+0000", 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363086+0000", 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.082 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363087+0000", 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.232e-06 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363091+0000", 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7780000000000001e-06 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363101+0000", 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9580000000000002e-06 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.083 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.084 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363108+0000", 2024-02-10T07:43:16.084 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6239999999999996e-06 2024-02-10T07:43:16.084 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.084 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.084 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.084 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363163+0000", 2024-02-10T07:43:16.084 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4987000000000001e-05 2024-02-10T07:43:16.084 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.084 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.084 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.084 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363173+0000", 2024-02-10T07:43:16.084 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0338e-05 2024-02-10T07:43:16.084 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.084 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057579+0000", 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.69440612899999998 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057654+0000", 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4476999999999999e-05 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057814+0000", 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000160216 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.085 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061207+0000", 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033935850000000002 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.076729+0000", 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015521517 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078191+0000", 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0014615749999999999 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078243+0000", 2024-02-10T07:43:16.086 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2627000000000003e-05 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078285+0000", 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1418999999999998e-05 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210966 7.12 7:4dc81df4:::benchmark_data_smithi008_82046_object95828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.386841+0000", 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.60935974199998, 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.69155063900000002, 2024-02-10T07:43:16.087 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210966 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386841+0000", 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.088 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386841+0000", 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386842+0000", 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3499999999999996e-07 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386844+0000", 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3709999999999998e-06 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386851+0000", 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2490000000000004e-06 2024-02-10T07:43:16.089 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386856+0000", 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5600000000000004e-06 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386907+0000", 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0946000000000003e-05 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386916+0000", 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0450000000000006e-06 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.090 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.058828+0000", 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.67191242600000001 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.058892+0000", 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3441999999999993e-05 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.059060+0000", 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000167808 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.091 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061212+0000", 2024-02-10T07:43:16.092 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0021523620000000001 2024-02-10T07:43:16.092 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.092 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.092 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.092 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.076780+0000", 2024-02-10T07:43:16.092 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015567713 2024-02-10T07:43:16.092 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.092 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.092 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.092 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078326+0000", 2024-02-10T07:43:16.092 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0015468789999999999 2024-02-10T07:43:16.092 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.092 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.092 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078357+0000", 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0905000000000003e-05 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078392+0000", 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4397999999999998e-05 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211133 7.19 7:9c7b01de:::benchmark_data_smithi008_82046_object95995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.029346+0000", 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 339.96685449900002, 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.75181978400000005, 2024-02-10T07:43:16.093 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211133 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029346+0000", 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029346+0000", 2024-02-10T07:43:16.094 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029348+0000", 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.347e-06 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029352+0000", 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4689999999999999e-06 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029364+0000", 2024-02-10T07:43:16.095 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1712999999999999e-05 2024-02-10T07:43:16.096 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.096 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.096 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.096 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029372+0000", 2024-02-10T07:43:16.096 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8869999999999992e-06 2024-02-10T07:43:16.096 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.096 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.096 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.096 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029414+0000", 2024-02-10T07:43:16.096 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2472000000000003e-05 2024-02-10T07:43:16.096 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.096 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.096 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029421+0000", 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5880000000000003e-06 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.774874+0000", 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.74545264499999997 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.774967+0000", 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3955000000000004e-05 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.097 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:43:16.098 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.775347+0000", 2024-02-10T07:43:16.098 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00037925300000000002 2024-02-10T07:43:16.098 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.098 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.098 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.098 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.780874+0000", 2024-02-10T07:43:16.098 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0055271779999999998 2024-02-10T07:43:16.098 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.098 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.098 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.098 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.780892+0000", 2024-02-10T07:43:16.098 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8397999999999999e-05 2024-02-10T07:43:16.098 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.098 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781061+0000", 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016846299999999999 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781118+0000", 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7451000000000003e-05 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781166+0000", 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7964999999999999e-05 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.099 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.100 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.100 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.100 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.100 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211195 7.19 7:98ba502a:::benchmark_data_smithi008_82046_object96057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:43:16.100 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.123282+0000", 2024-02-10T07:43:16.100 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 339.87291886999998, 2024-02-10T07:43:16.100 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.65998720099999997, 2024-02-10T07:43:16.100 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.100 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.100 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.100 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.100 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.100 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211195 2024-02-10T07:43:16.100 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.101 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.101 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.101 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.101 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123282+0000", 2024-02-10T07:43:16.101 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.101 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.101 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.101 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.101 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123282+0000", 2024-02-10T07:43:16.101 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.101 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.101 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.101 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.101 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123283+0000", 2024-02-10T07:43:16.102 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9099999999999991e-07 2024-02-10T07:43:16.102 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.102 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.102 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.102 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123287+0000", 2024-02-10T07:43:16.102 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6839999999999998e-06 2024-02-10T07:43:16.102 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.102 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.102 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.102 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123297+0000", 2024-02-10T07:43:16.102 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0237e-05 2024-02-10T07:43:16.102 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.102 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.102 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123302+0000", 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4909999999999996e-06 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123356+0000", 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3748000000000001e-05 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123366+0000", 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0139000000000001e-05 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.103 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.777248+0000", 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.65388193900000002 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.777361+0000", 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000112723 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.777593+0000", 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00023236400000000001 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.104 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.105 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781245+0000", 2024-02-10T07:43:16.105 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0036513639999999998 2024-02-10T07:43:16.105 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.105 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.105 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.105 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781258+0000", 2024-02-10T07:43:16.105 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3458999999999999e-05 2024-02-10T07:43:16.105 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.105 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.105 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.105 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783159+0000", 2024-02-10T07:43:16.105 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001900484 2024-02-10T07:43:16.105 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.106 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.106 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.106 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783222+0000", 2024-02-10T07:43:16.106 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3149000000000001e-05 2024-02-10T07:43:16.106 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.106 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.106 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.106 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783269+0000", 2024-02-10T07:43:16.106 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7429000000000003e-05 2024-02-10T07:43:16.106 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.106 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.106 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.106 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.106 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.107 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211237 7.19 7:9803a4c9:::benchmark_data_smithi008_82046_object96099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:43:16.107 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.157249+0000", 2024-02-10T07:43:16.107 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 339.838951479, 2024-02-10T07:43:16.107 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.62624425500000003, 2024-02-10T07:43:16.107 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.107 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.107 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.107 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.107 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.107 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211237 2024-02-10T07:43:16.107 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.107 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.107 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.107 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157249+0000", 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157249+0000", 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157250+0000", 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.47e-07 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.108 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.109 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157255+0000", 2024-02-10T07:43:16.109 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2889999999999999e-06 2024-02-10T07:43:16.109 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.109 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.109 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.109 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157265+0000", 2024-02-10T07:43:16.109 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0158e-05 2024-02-10T07:43:16.109 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.109 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.109 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.109 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157272+0000", 2024-02-10T07:43:16.109 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3010000000000001e-06 2024-02-10T07:43:16.109 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.109 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.110 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.110 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157324+0000", 2024-02-10T07:43:16.110 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1997999999999999e-05 2024-02-10T07:43:16.110 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.110 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.110 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.110 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157332+0000", 2024-02-10T07:43:16.110 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4839999999999998e-06 2024-02-10T07:43:16.110 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.110 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.110 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.110 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.778870+0000", 2024-02-10T07:43:16.110 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.62153816500000003 2024-02-10T07:43:16.111 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.111 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.111 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.111 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.778991+0000", 2024-02-10T07:43:16.111 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000121272 2024-02-10T07:43:16.111 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.111 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.111 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:43:16.111 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.779232+0000", 2024-02-10T07:43:16.111 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024073399999999999 2024-02-10T07:43:16.111 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.111 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.111 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.112 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781264+0000", 2024-02-10T07:43:16.112 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020323899999999998 2024-02-10T07:43:16.112 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.112 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.112 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.112 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781322+0000", 2024-02-10T07:43:16.112 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8e-05 2024-02-10T07:43:16.112 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.112 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.112 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.112 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783335+0000", 2024-02-10T07:43:16.112 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020125339999999999 2024-02-10T07:43:16.112 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.113 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.113 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.113 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783454+0000", 2024-02-10T07:43:16.113 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011938300000000001 2024-02-10T07:43:16.113 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.113 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.113 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.113 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783494+0000", 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.96e-05 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211239 7.19 7:9b3bb0be:::benchmark_data_smithi008_82046_object96101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.158720+0000", 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 339.83748100399998, 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.62931068000000001, 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.114 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211239 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158720+0000", 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158720+0000", 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.115 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158721+0000", 2024-02-10T07:43:16.116 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1289999999999999e-06 2024-02-10T07:43:16.116 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.116 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.116 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.116 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158726+0000", 2024-02-10T07:43:16.116 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7010000000000004e-06 2024-02-10T07:43:16.116 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.116 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.116 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.116 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158747+0000", 2024-02-10T07:43:16.116 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1362e-05 2024-02-10T07:43:16.116 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.116 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.116 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158755+0000", 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7680000000000008e-06 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158807+0000", 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2522000000000003e-05 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158817+0000", 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3870000000000001e-06 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.117 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781362+0000", 2024-02-10T07:43:16.118 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.62254494699999996 2024-02-10T07:43:16.118 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.118 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.118 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.118 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781463+0000", 2024-02-10T07:43:16.118 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000101592 2024-02-10T07:43:16.118 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.118 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.118 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:43:16.118 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781718+0000", 2024-02-10T07:43:16.118 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00025504199999999997 2024-02-10T07:43:16.118 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.118 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.118 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.119 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.785814+0000", 2024-02-10T07:43:16.119 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0040961110000000004 2024-02-10T07:43:16.119 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.120 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.120 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.120 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787830+0000", 2024-02-10T07:43:16.120 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020153760000000001 2024-02-10T07:43:16.120 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.120 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.120 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.120 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787925+0000", 2024-02-10T07:43:16.120 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5309000000000005e-05 2024-02-10T07:43:16.120 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.120 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.120 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.120 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787989+0000", 2024-02-10T07:43:16.120 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3578999999999996e-05 2024-02-10T07:43:16.121 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.121 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.121 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.121 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.788031+0000", 2024-02-10T07:43:16.121 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1854999999999998e-05 2024-02-10T07:43:16.121 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.121 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.121 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.121 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.121 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.121 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211270 7.19 7:9a9ecfcf:::benchmark_data_smithi008_82046_object96132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:43:16.121 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.185809+0000", 2024-02-10T07:43:16.121 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 339.81039196299997, 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.60674339600000005, 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211270 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185809+0000", 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.122 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185809+0000", 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185810+0000", 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.215e-06 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185814+0000", 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3259999999999997e-06 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.123 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.124 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185827+0000", 2024-02-10T07:43:16.124 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2669e-05 2024-02-10T07:43:16.124 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.124 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.124 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.124 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185835+0000", 2024-02-10T07:43:16.124 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4769999999999997e-06 2024-02-10T07:43:16.124 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.124 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.124 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.124 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.186136+0000", 2024-02-10T07:43:16.124 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000301155 2024-02-10T07:43:16.124 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.124 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.186151+0000", 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5316e-05 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783531+0000", 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.59737997799999998 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783650+0000", 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000118968 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:43:16.125 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783954+0000", 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00030440799999999999 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787866+0000", 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0039120379999999996 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787875+0000", 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1249999999999993e-06 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792479+0000", 2024-02-10T07:43:16.126 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0046047429999999997 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792525+0000", 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5262000000000001e-05 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792552+0000", 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7716e-05 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.127 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211303 7.19 7:9bbbc725:::benchmark_data_smithi008_82046_object96165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:43:16.128 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.210320+0000", 2024-02-10T07:43:16.128 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 339.78588135899997, 2024-02-10T07:43:16.128 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.58484913400000005, 2024-02-10T07:43:16.128 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:16.128 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:16.128 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:16.128 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:16.128 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:16.128 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211303 2024-02-10T07:43:16.128 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.128 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:16.128 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.128 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:16.128 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210320+0000", 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210320+0000", 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210320+0000", 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1200000000000002e-07 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:16.129 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210324+0000", 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2459999999999999e-06 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210344+0000", 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0020000000000001e-05 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210352+0000", 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0430000000000008e-06 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210431+0000", 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9482999999999998e-05 2024-02-10T07:43:16.130 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210439+0000", 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0909999999999993e-06 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.785850+0000", 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.57541028900000002 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.785977+0000", 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012783599999999999 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.131 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.786232+0000", 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00025462599999999998 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792371+0000", 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0061389529999999999 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792433+0000", 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2088999999999995e-05 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.132 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.795068+0000", 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0026348130000000002 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.795124+0000", 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6365e-05 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.795169+0000", 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4468000000000002e-05 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:16.133 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:16.134 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:43:16.158 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:16.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:16.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:16.260 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:16.270 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[40743]: 3.5 scrub starts 2024-02-10T07:43:16.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[40743]: 3.5 scrub ok 2024-02-10T07:43:16.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:43:16.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:43:16.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[40743]: 5.0 scrub starts 2024-02-10T07:43:16.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[40743]: 5.0 scrub ok 2024-02-10T07:43:16.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[40743]: osdmap e272: 8 total, 8 up, 8 in 2024-02-10T07:43:16.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[40743]: 4.7 scrub starts 2024-02-10T07:43:16.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[40743]: 4.7 scrub ok 2024-02-10T07:43:16.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[40743]: 2.0 scrub starts 2024-02-10T07:43:16.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[40743]: 2.0 scrub ok 2024-02-10T07:43:16.272 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[29816]: 3.5 scrub starts 2024-02-10T07:43:16.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[29816]: 3.5 scrub ok 2024-02-10T07:43:16.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:43:16.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:43:16.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[29816]: 5.0 scrub starts 2024-02-10T07:43:16.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[29816]: 5.0 scrub ok 2024-02-10T07:43:16.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[29816]: osdmap e272: 8 total, 8 up, 8 in 2024-02-10T07:43:16.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[29816]: 4.7 scrub starts 2024-02-10T07:43:16.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[29816]: 4.7 scrub ok 2024-02-10T07:43:16.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[29816]: 2.0 scrub starts 2024-02-10T07:43:16.273 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:16 smithi008 ceph-mon[29816]: 2.0 scrub ok 2024-02-10T07:43:16.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:16.299 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:16.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:16.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:16 smithi170 ceph-mon[33394]: 3.5 scrub starts 2024-02-10T07:43:16.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:16 smithi170 ceph-mon[33394]: 3.5 scrub ok 2024-02-10T07:43:16.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:16 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:43:16.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:16 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:43:16.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:16 smithi170 ceph-mon[33394]: 5.0 scrub starts 2024-02-10T07:43:16.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:16 smithi170 ceph-mon[33394]: 5.0 scrub ok 2024-02-10T07:43:16.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:16 smithi170 ceph-mon[33394]: osdmap e272: 8 total, 8 up, 8 in 2024-02-10T07:43:16.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:16 smithi170 ceph-mon[33394]: 4.7 scrub starts 2024-02-10T07:43:16.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:16 smithi170 ceph-mon[33394]: 4.7 scrub ok 2024-02-10T07:43:16.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:16 smithi170 ceph-mon[33394]: 2.0 scrub starts 2024-02-10T07:43:16.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:16 smithi170 ceph-mon[33394]: 2.0 scrub ok 2024-02-10T07:43:16.400 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:16.425 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:16.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:16.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:16.526 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:16.568 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:16.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:16.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:16.670 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:16.709 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:16.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:16.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:16.776 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 56 16 91440 91424 102.018 70.25 0.0120111 0.00979734 2024-02-10T07:43:16.811 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:16.838 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:16.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:16.939 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:16.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:16.978 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:16.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:17.079 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:17.120 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:17.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:17.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:17.222 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:17.254 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:17.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:17.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:17 smithi008 ceph-mon[40743]: 5.1b scrub starts 2024-02-10T07:43:17.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:17 smithi008 ceph-mon[40743]: 5.1b scrub ok 2024-02-10T07:43:17.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:17 smithi008 ceph-mon[40743]: pgmap v515: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-10T07:43:17.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:17 smithi008 ceph-mon[40743]: 3.1a scrub starts 2024-02-10T07:43:17.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:17 smithi008 ceph-mon[40743]: 3.1a scrub ok 2024-02-10T07:43:17.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:17 smithi008 ceph-mon[40743]: 4.1d scrub starts 2024-02-10T07:43:17.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:17 smithi008 ceph-mon[40743]: 4.1d scrub ok 2024-02-10T07:43:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:17 smithi008 ceph-mon[29816]: 5.1b scrub starts 2024-02-10T07:43:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:17 smithi008 ceph-mon[29816]: 5.1b scrub ok 2024-02-10T07:43:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:17 smithi008 ceph-mon[29816]: pgmap v515: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-10T07:43:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:17 smithi008 ceph-mon[29816]: 3.1a scrub starts 2024-02-10T07:43:17.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:17 smithi008 ceph-mon[29816]: 3.1a scrub ok 2024-02-10T07:43:17.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:17 smithi008 ceph-mon[29816]: 4.1d scrub starts 2024-02-10T07:43:17.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:17 smithi008 ceph-mon[29816]: 4.1d scrub ok 2024-02-10T07:43:17.356 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:17.380 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:17.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:17.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:17 smithi170 ceph-mon[33394]: 5.1b scrub starts 2024-02-10T07:43:17.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:17 smithi170 ceph-mon[33394]: 5.1b scrub ok 2024-02-10T07:43:17.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:17 smithi170 ceph-mon[33394]: pgmap v515: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-10T07:43:17.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:17 smithi170 ceph-mon[33394]: 3.1a scrub starts 2024-02-10T07:43:17.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:17 smithi170 ceph-mon[33394]: 3.1a scrub ok 2024-02-10T07:43:17.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:17 smithi170 ceph-mon[33394]: 4.1d scrub starts 2024-02-10T07:43:17.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:17 smithi170 ceph-mon[33394]: 4.1d scrub ok 2024-02-10T07:43:17.482 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:17.524 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:17.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:17.625 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:17.656 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:17.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:17.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:17.758 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:17.776 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 57 16 92503 92487 101.394 66.4375 0.0138786 0.00985782 2024-02-10T07:43:17.796 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:17.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:17.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:17.898 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:17.938 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:17.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:17.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:18.040 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:18.081 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:18.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:18.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:18.183 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:18.221 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:18.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:18.323 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:43:18.323 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:18.385 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:18.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:18.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:18.487 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:18.533 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:18.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:18.634 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:18.691 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:18.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:18.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:18.776 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 58 15 93493 93478 100.714 61.9375 0.00819527 0.00992412 2024-02-10T07:43:18.793 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:18.828 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:18.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:18.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:18.843 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-10T07:43:18.930 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:18.979 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:18.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:19.081 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:19.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:43:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:43:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:43:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:43:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:18 smithi008 ceph-mon[29816]: pgmap v516: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-10T07:43:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:18 smithi008 ceph-mon[29816]: 4.19 scrub starts 2024-02-10T07:43:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:18 smithi008 ceph-mon[29816]: 4.19 scrub ok 2024-02-10T07:43:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:43:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:18 smithi008 ceph-mon[40743]: pgmap v516: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-10T07:43:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:18 smithi008 ceph-mon[40743]: 4.19 scrub starts 2024-02-10T07:43:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:18 smithi008 ceph-mon[40743]: 4.19 scrub ok 2024-02-10T07:43:19.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:19.128 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:19.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:43:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:18 smithi170 ceph-mon[33394]: pgmap v516: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-10T07:43:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:18 smithi170 ceph-mon[33394]: 4.19 scrub starts 2024-02-10T07:43:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:18 smithi170 ceph-mon[33394]: 4.19 scrub ok 2024-02-10T07:43:19.229 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:19.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:19.269 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:19.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:19.302 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:43:19.369 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:19.407 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:19.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:19.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:19.509 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:19.564 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:19.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:19.666 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:19.712 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:19.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:19.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:19.776 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 59 16 94643 94627 100.224 71.8125 0.0119141 0.00997344 2024-02-10T07:43:19.814 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:19.861 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:19.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:19.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:19.872 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:19 smithi008 ceph-mon[29816]: 5.12 scrub starts 2024-02-10T07:43:19.872 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:19 smithi008 ceph-mon[29816]: 5.12 scrub ok 2024-02-10T07:43:19.872 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:19 smithi008 ceph-mon[29816]: 3.e scrub starts 2024-02-10T07:43:19.872 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:19 smithi008 ceph-mon[29816]: 3.e scrub ok 2024-02-10T07:43:19.873 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:19 smithi008 ceph-mon[40743]: 5.12 scrub starts 2024-02-10T07:43:19.873 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:19 smithi008 ceph-mon[40743]: 5.12 scrub ok 2024-02-10T07:43:19.873 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:19 smithi008 ceph-mon[40743]: 3.e scrub starts 2024-02-10T07:43:19.873 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:19 smithi008 ceph-mon[40743]: 3.e scrub ok 2024-02-10T07:43:19.962 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:20.001 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:20.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:20.103 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:20.132 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:20.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:20.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:19 smithi170 ceph-mon[33394]: 5.12 scrub starts 2024-02-10T07:43:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:19 smithi170 ceph-mon[33394]: 5.12 scrub ok 2024-02-10T07:43:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:19 smithi170 ceph-mon[33394]: 3.e scrub starts 2024-02-10T07:43:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:19 smithi170 ceph-mon[33394]: 3.e scrub ok 2024-02-10T07:43:20.234 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:20.268 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:20.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:20.370 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:20.413 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:20.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:20.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:20.515 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:20.546 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:20.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:20.555 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:20.648 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:20.704 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:20.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:20.769 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:43:20.776 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:43:20.777091+0000 min lat: 0.001562 max lat: 0.194217 avg lat: 0.0100374 2024-02-10T07:43:20.776 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:43:20.776 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 60 16 95621 95605 99.5722 61.125 0.0406268 0.0100374 2024-02-10T07:43:20.805 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:20.846 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:20.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:20.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:20.948 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:20.972 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:20.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:20.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:21.073 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:21.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[29816]: 3.12 scrub starts 2024-02-10T07:43:21.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[29816]: 3.12 scrub ok 2024-02-10T07:43:21.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[29816]: 6.c scrub starts 2024-02-10T07:43:21.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[29816]: 6.c scrub ok 2024-02-10T07:43:21.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[29816]: pgmap v517: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1022 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-02-10T07:43:21.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[29816]: 9.4 scrub starts 2024-02-10T07:43:21.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1075840509' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:43:21.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:43:21.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[40743]: 3.12 scrub starts 2024-02-10T07:43:21.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[40743]: 3.12 scrub ok 2024-02-10T07:43:21.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[40743]: 6.c scrub starts 2024-02-10T07:43:21.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[40743]: 6.c scrub ok 2024-02-10T07:43:21.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[40743]: pgmap v517: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1022 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-02-10T07:43:21.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[40743]: 9.4 scrub starts 2024-02-10T07:43:21.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1075840509' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:43:21.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:20 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:43:21.120 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:21.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:21.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:20 smithi170 ceph-mon[33394]: 3.12 scrub starts 2024-02-10T07:43:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:20 smithi170 ceph-mon[33394]: 3.12 scrub ok 2024-02-10T07:43:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:20 smithi170 ceph-mon[33394]: 6.c scrub starts 2024-02-10T07:43:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:20 smithi170 ceph-mon[33394]: 6.c scrub ok 2024-02-10T07:43:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:20 smithi170 ceph-mon[33394]: pgmap v517: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1022 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-02-10T07:43:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:20 smithi170 ceph-mon[33394]: 9.4 scrub starts 2024-02-10T07:43:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:20 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1075840509' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:43:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:20 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:43:21.222 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:21.264 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:21.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:21.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:21.366 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:21.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:21.409 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:21.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:21.510 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:21.551 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:21.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:21.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:21.607 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:43:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:43:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.25283.0:96507 9.9 e273/253 9:9ad29f9d:::benchmark_data_smithi008_134920_object96506:head v 273'6072, mlcod=273'6069)", 2024-02-10T07:43:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:43:21.600112+0000", 2024-02-10T07:43:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.005419643, 2024-02-10T07:43:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0054525559999999999, 2024-02-10T07:43:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:43:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:21.607 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.600112+0000", 2024-02-10T07:43:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.600112+0000", 2024-02-10T07:43:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.600113+0000", 2024-02-10T07:43:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0999999999999998e-07 2024-02-10T07:43:21.608 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.600173+0000", 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9706000000000001e-05 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.600184+0000", 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1943000000000001e-05 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.600191+0000", 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5690000000000001e-06 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.609 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.600235+0000", 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3553999999999997e-05 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.600256+0000", 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1608999999999999e-05 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.25283.0:96510 9.a e273/253 9:5d375b73:::benchmark_data_smithi008_134920_object96509:head v 273'5885, mlcod=273'5883)", 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:43:21.602608+0000", 2024-02-10T07:43:21.610 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.0029241390000000001, 2024-02-10T07:43:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0030247189999999999, 2024-02-10T07:43:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:43:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.602608+0000", 2024-02-10T07:43:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.602608+0000", 2024-02-10T07:43:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:21.611 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.602609+0000", 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.850000000000001e-07 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.602632+0000", 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3604999999999998e-05 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.602651+0000", 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8474999999999998e-05 2024-02-10T07:43:21.612 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.602659+0000", 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2730000000000002e-06 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.602712+0000", 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2658999999999998e-05 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.602730+0000", 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8363e-05 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:21.613 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.25283.0:96506 9.1 e273/253 9:87ef4337:::benchmark_data_smithi008_134920_object96505:head v 273'5974, mlcod=273'5971)", 2024-02-10T07:43:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:43:21.603822+0000", 2024-02-10T07:43:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.001709473, 2024-02-10T07:43:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0018679670000000001, 2024-02-10T07:43:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:43:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.603822+0000", 2024-02-10T07:43:21.614 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.603822+0000", 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.603823+0000", 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3099999999999996e-07 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.603869+0000", 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5556999999999998e-05 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.615 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.603884+0000", 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.558e-05 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.603892+0000", 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0209999999999999e-06 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.603970+0000", 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8127000000000006e-05 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:21.616 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:21.617 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:21.617 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:43:21.603991+0000", 2024-02-10T07:43:21.617 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0363000000000001e-05 2024-02-10T07:43:21.617 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:21.617 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:21.617 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:21.617 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:21.617 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2024-02-10T07:43:21.617 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 3 2024-02-10T07:43:21.617 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:43:21.653 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:21.693 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:21.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:21.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:21.764 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.2 to 1 (10000) 2024-02-10T07:43:21.776 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 61 16 96725 96709 99.0709 69 0.0289362 0.010089 2024-02-10T07:43:21.795 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:21.828 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:21.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:21.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:21.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:21 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:43:21.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:21 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:43:21.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:21 smithi170 ceph-mon[33394]: osdmap e273: 8 total, 8 up, 8 in 2024-02-10T07:43:21.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:21 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3863589766' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:43:21.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:21 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:43:21.930 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:21.968 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:21.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:21.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:22.015 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:21 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:43:22.016 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:21 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:43:22.016 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:21 smithi008 ceph-mon[29816]: osdmap e273: 8 total, 8 up, 8 in 2024-02-10T07:43:22.016 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:21 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3863589766' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:43:22.016 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:21 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:43:22.016 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:21 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:43:22.016 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:21 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:43:22.016 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:21 smithi008 ceph-mon[40743]: osdmap e273: 8 total, 8 up, 8 in 2024-02-10T07:43:22.016 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:21 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3863589766' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:43:22.016 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:21 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:43:22.070 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:22.115 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:22.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:22.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:22.217 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:22.263 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:22.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:22.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:22.365 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:22.402 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:22.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:22.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:22.503 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:22.547 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-10T07:43:22.553 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:22.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:22.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:22.655 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:22.697 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:22.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:22.776 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 62 16 97934 97918 98.6917 75.5625 0.0170907 0.010128 2024-02-10T07:43:22.798 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:22.818 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 3 1 2024-02-10T07:43:22.846 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:22.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:22.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:22.948 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:22.957 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:43:22.990 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:22.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:22.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:23.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:22 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:43:23.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:22 smithi008 ceph-mon[40743]: osdmap e274: 8 total, 8 up, 8 in 2024-02-10T07:43:23.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:22 smithi008 ceph-mon[40743]: pgmap v520: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 738 B/s rd, 51 MiB/s wr, 823 op/s 2024-02-10T07:43:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:22 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:43:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:22 smithi008 ceph-mon[29816]: osdmap e274: 8 total, 8 up, 8 in 2024-02-10T07:43:23.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:22 smithi008 ceph-mon[29816]: pgmap v520: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 738 B/s rd, 51 MiB/s wr, 823 op/s 2024-02-10T07:43:23.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:22 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:43:23.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:22 smithi170 ceph-mon[33394]: osdmap e274: 8 total, 8 up, 8 in 2024-02-10T07:43:23.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:22 smithi170 ceph-mon[33394]: pgmap v520: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 738 B/s rd, 51 MiB/s wr, 823 op/s 2024-02-10T07:43:23.092 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:23.124 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:23.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:23.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:23.226 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:23.263 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:23.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:23.365 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:23.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:43:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:43:23.405 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:23.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:23.506 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:23.543 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:23.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:23.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:23.645 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:23.699 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:23.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:23.776 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 63 16 99102 99086 98.2837 73 0.0196378 0.0101698 2024-02-10T07:43:23.801 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:23.845 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:23.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:23.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:23.947 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:23.969 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:23.981 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:23.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:23.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:24.015 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:43:24.015 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:23 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:24.016 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:23 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:24.059 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:43:24.082 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:24.112 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:43:24.122 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:24.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:24.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:24.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:23 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:24.169 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:43:24.208 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:43:24.224 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:24.248 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:43:24.260 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:24.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:24.293 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:43:24.334 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:43:24.339 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:43:24.340 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:43:24.340 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:43:24.340 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:43:24.340 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:43:24.340 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:43:24.340 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:43:24.340 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:43:24.362 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:24.397 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:24.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:24.499 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:24.548 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:24.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:24.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:24.650 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:24.696 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:24.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:24.776 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 64 16 100271 100255 97.8896 73.0625 0.0158037 0.0102109 2024-02-10T07:43:24.797 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:24.842 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:24.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:24.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:24.944 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:24.981 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:24.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:25.083 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:25.124 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:25.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:24 smithi170 ceph-mon[33394]: pgmap v521: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-10T07:43:25.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:24 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/481582908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:25.225 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:25.265 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:25.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:25.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:24 smithi008 ceph-mon[29816]: pgmap v521: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-10T07:43:25.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:24 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/481582908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:25.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:24 smithi008 ceph-mon[40743]: pgmap v521: 148 pgs: 148 active+clean; 5.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-10T07:43:25.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:24 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/481582908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:25.367 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:25.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:25.416 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:25.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:25.426 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:43:25.426 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:43:25.517 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:25.556 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:25.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:25.657 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:25.690 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:43:25.692 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:25.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:25.777 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 65 15 101468 101453 97.5354 74.875 0.0191663 0.0102477 2024-02-10T07:43:25.794 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:25.831 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:25.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:25.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:25.933 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:25.986 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:25.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:26.087 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:26.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:26.131 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:26.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:26.149 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-10T07:43:26.232 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:26.282 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:26.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:26.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:26.384 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:26.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:26.425 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:26.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:26.526 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:26.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:26.568 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:26.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:26.601 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:43:26.669 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:26.699 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:26.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:26.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:26.729 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:26.778 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 66 16 102583 102567 97.1119 69.625 0.0123028 0.0102922 2024-02-10T07:43:26.802 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:26.839 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:26.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:26.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:26.941 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:26.964 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:26.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:27.065 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:27.108 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:27.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:27.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:27.187 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:27 smithi008 ceph-mon[29816]: pgmap v522: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-10T07:43:27.187 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:27 smithi008 ceph-mon[40743]: pgmap v522: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-10T07:43:27.210 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:27.244 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:27.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:27.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:27.346 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:27.377 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:27.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:27 smithi170 ceph-mon[33394]: pgmap v522: 148 pgs: 148 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-10T07:43:27.479 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:27.515 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:27.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:27.617 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:27.654 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:27.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:27.755 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:27.777 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 67 16 103698 103682 96.7025 69.6875 0.0114878 0.0103362 2024-02-10T07:43:27.792 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:27.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:27.894 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:27.936 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:27.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:27.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:28.014 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.3 to 1 (10000) 2024-02-10T07:43:28.038 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:28.082 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:28.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:28.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:28.093 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:43:28.185 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:28.216 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:28.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:28.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:28.318 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:28 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1875648623' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:43:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:28 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:43:28.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:28 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1875648623' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:43:28.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:28 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:43:28.380 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:28.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:28.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:28 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1875648623' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:43:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:28 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:43:28.482 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:28.777 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 68 16 104853 104837 96.3419 72.1875 0.00512932 0.0103748 2024-02-10T07:43:28.867 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:28.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:28.969 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:29.010 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:29.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:29.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:29.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:43:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:43:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:43:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:29 smithi008 ceph-mon[29816]: pgmap v523: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.02k op/s 2024-02-10T07:43:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:29 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:43:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:29 smithi008 ceph-mon[29816]: osdmap e275: 8 total, 8 up, 8 in 2024-02-10T07:43:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:29 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:43:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:29 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:43:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:29 smithi008 ceph-mon[40743]: pgmap v523: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.02k op/s 2024-02-10T07:43:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:29 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:43:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:29 smithi008 ceph-mon[40743]: osdmap e275: 8 total, 8 up, 8 in 2024-02-10T07:43:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:29 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:43:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:29 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:43:29.112 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:29.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:29.145 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:29.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:29.246 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:29.265 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:29 smithi170 ceph-mon[33394]: pgmap v523: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.02k op/s 2024-02-10T07:43:29.265 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:29 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:43:29.265 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:29 smithi170 ceph-mon[33394]: osdmap e275: 8 total, 8 up, 8 in 2024-02-10T07:43:29.265 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:29 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:43:29.265 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:29 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:43:29.289 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:29.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:29.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:29.391 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:29.430 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:29.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:29.531 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:29.571 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:29.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:29.672 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:29.713 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:29.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:29.740 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:43:29.741 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:43:29.778 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 69 16 105982 105966 95.9672 70.5625 0.023121 0.0104148 2024-02-10T07:43:29.815 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:29.849 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:29.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:29.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:29.852 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:29.952 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:30.004 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:30.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:30.102 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:30 smithi170 ceph-mon[33394]: osdmap e276: 8 total, 8 up, 8 in 2024-02-10T07:43:30.106 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:30.147 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:30.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:30.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:30.248 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:30.288 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:30.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:30.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:30.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:30 smithi008 ceph-mon[29816]: osdmap e276: 8 total, 8 up, 8 in 2024-02-10T07:43:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:30 smithi008 ceph-mon[40743]: osdmap e276: 8 total, 8 up, 8 in 2024-02-10T07:43:30.389 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:30.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:30.435 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:30.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:30.442 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 4 1 2024-02-10T07:43:30.475 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T07:43:30.536 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:30.574 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:30.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:30.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:30.676 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:30.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:30.727 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:30.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:30.778 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 70 16 107064 107048 95.562 67.625 0.0106311 0.0104594 2024-02-10T07:43:30.828 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:30.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:30.870 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:30.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:30.889 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:43:30.972 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:31.014 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:31.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:31.061 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:43:31.118 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:31.133 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:43:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=cleanup t=2024-02-10T07:43:30.849237204Z level=info msg="Completed cleanup jobs" duration=4.584376ms 2024-02-10T07:43:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:31 smithi170 ceph-mon[33394]: pgmap v526: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-02-10T07:43:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:31 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2343338704' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:31.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:31 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:31.154 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:31.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:31.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:31.256 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:31.290 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:31.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:31.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:31.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:31 smithi008 ceph-mon[29816]: pgmap v526: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-02-10T07:43:31.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:31 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2343338704' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:31.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:31 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:31.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:31 smithi008 ceph-mon[40743]: pgmap v526: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-02-10T07:43:31.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:31 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2343338704' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:31.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:31 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:31.392 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:31.439 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:31.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:31.539 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:31.572 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:31.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:31.674 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:31.714 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:31.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:31.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:31.778 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 71 16 108180 108164 95.1984 69.75 0.0140299 0.0104997 2024-02-10T07:43:31.816 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:31.854 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:31.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:31.955 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:32.008 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:32.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:32.110 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:32.145 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:32.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:32.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:32.247 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:32.283 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:32.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:32.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:32 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:43:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:32 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:43:32.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:32 smithi008 ceph-mon[29816]: osdmap e277: 8 total, 8 up, 8 in 2024-02-10T07:43:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:32 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:43:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:32 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:43:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:32 smithi008 ceph-mon[40743]: osdmap e277: 8 total, 8 up, 8 in 2024-02-10T07:43:32.374 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:32 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:43:32.374 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:32 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:43:32.374 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:32 smithi170 ceph-mon[33394]: osdmap e277: 8 total, 8 up, 8 in 2024-02-10T07:43:32.385 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:32.424 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:32.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:32.525 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:32.565 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:32.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:32.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:32.667 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:32.703 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:32.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:32.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:32.794 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 72 16 109329 109313 94.8533 71.8125 0.011697 0.0105381 2024-02-10T07:43:32.808 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:32.854 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:32.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:32.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:32.885 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:43:32.885 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:43:32.957 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:32.993 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:32.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:32.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:33.095 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:33.109 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:33.127 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:33.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:33.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:33.180 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:43:33.227 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:43:33.229 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:33.261 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:33.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:33.269 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:43:33.313 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:43:33.314 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:33 smithi008 ceph-mon[29816]: pgmap v528: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 960 op/s 2024-02-10T07:43:33.314 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:43:33.314 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:33.315 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:33 smithi008 ceph-mon[40743]: pgmap v528: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 960 op/s 2024-02-10T07:43:33.315 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:43:33.315 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:33.344 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:33 smithi170 ceph-mon[33394]: pgmap v528: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 960 op/s 2024-02-10T07:43:33.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:43:33.345 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:33.361 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:43:33.362 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:33.401 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:43:33.411 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:33.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:33.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:33.429 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:43:33.452 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:43:33.453 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:43:33.453 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:43:33.453 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:43:33.453 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:43:33.453 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:43:33.453 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:43:33.453 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:43:33.453 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:43:33.513 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:33.553 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:33.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:33.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:33.596 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:43:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:43:33.645 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-10T07:43:33.655 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:33.697 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:33.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:33.794 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 73 16 110307 110291 94.3913 61.125 0.0157126 0.0105888 2024-02-10T07:43:33.799 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:33.837 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:33.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:33.939 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:33.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:33.994 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:33.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:34.032 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:43:34.095 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:34.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:34.142 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:34.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:34.193 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:34 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:43:34.193 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:34 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:43:34.193 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:34 smithi008 ceph-mon[29816]: osdmap e278: 8 total, 8 up, 8 in 2024-02-10T07:43:34.193 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:34 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1438583184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:34.193 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:34 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:34.194 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:34 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:43:34.194 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:34 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:43:34.194 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:34 smithi008 ceph-mon[40743]: osdmap e278: 8 total, 8 up, 8 in 2024-02-10T07:43:34.194 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:34 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1438583184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:34.194 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:34 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:34.242 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:34.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:34.290 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:34.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:34.358 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:34 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:43:34.358 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:34 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:43:34.358 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:34 smithi170 ceph-mon[33394]: osdmap e278: 8 total, 8 up, 8 in 2024-02-10T07:43:34.358 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:34 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1438583184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:34.358 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:34 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:34.390 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:34.426 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:34.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:34.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:34.486 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:43:34.527 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:34.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:34.559 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:34.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:34.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:34.693 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:34.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:34.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:34.794 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 74 16 111331 111315 93.9806 64 0.00851272 0.0106357 2024-02-10T07:43:34.794 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:34.829 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:34.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:34.931 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:34.983 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:34.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:35.084 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:35.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:35.127 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:35.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:35.161 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:35 smithi008 ceph-mon[29816]: pgmap v530: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 862 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-10T07:43:35.161 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:35 smithi008 ceph-mon[40743]: pgmap v530: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 862 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-10T07:43:35.228 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:35.266 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:35.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:35.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:35.368 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:35 smithi170 ceph-mon[33394]: pgmap v530: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 862 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-10T07:43:35.416 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:35.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:35.519 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:35.560 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:35.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:35.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:35.662 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:35.703 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:35.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:35.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:35.794 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 75 16 112499 112483 93.7009 73 0.0115737 0.0106673 2024-02-10T07:43:35.805 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:35.872 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:35.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:35.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:35.929 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:35.974 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:36.015 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:36.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:36.021 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:43:36.021 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2024-02-10T07:43:36.021 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2024-02-10T07:43:36.021 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2024-02-10T07:43:36.021 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:43:36.116 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:36.153 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:36.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:36.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:36.255 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:36.295 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:36.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:36.397 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:36.438 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:36.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:36.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:36.540 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:36.575 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:36.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:36.676 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:36.723 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:36.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:36.726 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-10T07:43:36.794 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 76 16 113680 113664 93.4392 73.8125 0.0105793 0.0106977 2024-02-10T07:43:36.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:36.863 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:36.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:36.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:36.965 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:37.003 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:37.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:37.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:37.105 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:37.147 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:37.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:37.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:37.187 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:43:37.250 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:37.297 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:37 smithi008 ceph-mon[29816]: pgmap v531: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 740 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-10T07:43:37.298 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:37 smithi008 ceph-mon[40743]: pgmap v531: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 740 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-10T07:43:37.310 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:37.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:37.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:37.353 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:43:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:37 smithi170 ceph-mon[33394]: pgmap v531: 148 pgs: 148 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 740 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-10T07:43:37.412 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:37.455 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:37.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:37.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:37.559 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:37.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:37.593 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:37.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:37.694 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:37.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:37.755 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:37.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:37.794 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 77 16 114776 114760 93.1153 68.5 0.0169051 0.0107341 2024-02-10T07:43:37.856 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:37.892 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:37.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:37.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:37.994 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:38.040 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:38.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:38.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:38.094 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:38.114 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.4 to 1 (10000) 2024-02-10T07:43:38.143 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:38.191 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:38.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:38.246 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:38 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1407651553' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:43:38.246 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:38 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1407651553' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:43:38.293 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:38.332 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:38.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:38.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:38 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1407651553' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:43:38.434 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:38.481 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:38.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:38.582 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:38.606 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:38.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:38.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:38.708 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:38.740 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:38.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:38.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:38.795 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 78 16 115832 115816 92.7677 66 0.0125203 0.0107747 2024-02-10T07:43:38.842 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:38.877 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:38.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:38.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:38.979 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:39.003 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:39.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:39.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:39.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:43:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:43:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:43:39.106 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:39.114 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:43:39.145 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:39.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:39.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:39.247 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:39.269 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:39.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:39.371 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:39 smithi170 ceph-mon[33394]: pgmap v532: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 994 op/s 2024-02-10T07:43:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:39 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1407651553' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:43:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:39 smithi170 ceph-mon[33394]: osdmap e279: 8 total, 8 up, 8 in 2024-02-10T07:43:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:39 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2374122819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:43:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:39 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:43:39.411 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:39.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:39.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:39.513 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:39.552 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:39.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:39.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:39 smithi008 ceph-mon[29816]: pgmap v532: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 994 op/s 2024-02-10T07:43:39.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:39 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1407651553' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:43:39.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:39 smithi008 ceph-mon[29816]: osdmap e279: 8 total, 8 up, 8 in 2024-02-10T07:43:39.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:39 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2374122819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:43:39.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:39 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:43:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:39 smithi008 ceph-mon[40743]: pgmap v532: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 994 op/s 2024-02-10T07:43:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:39 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1407651553' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:43:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:39 smithi008 ceph-mon[40743]: osdmap e279: 8 total, 8 up, 8 in 2024-02-10T07:43:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:39 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2374122819' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:43:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:39 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:43:39.653 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:39.679 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:39.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:39.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:39.781 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:39.794 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 79 16 116856 116840 92.4036 64 0.0198029 0.0108178 2024-02-10T07:43:39.817 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:39.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:39.919 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:39.956 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:39.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:40.058 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:40.082 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:40.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:40.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:40.184 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:40.221 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:40.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:40.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:40.323 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:40.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:40.353 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:40.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:40 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:43:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:40 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:43:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:40 smithi170 ceph-mon[33394]: osdmap e280: 8 total, 8 up, 8 in 2024-02-10T07:43:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:40 smithi170 ceph-mon[33394]: 4.b scrub starts 2024-02-10T07:43:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:40 smithi170 ceph-mon[33394]: 4.b scrub ok 2024-02-10T07:43:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:40 smithi170 ceph-mon[33394]: 5.4 scrub starts 2024-02-10T07:43:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:40 smithi170 ceph-mon[33394]: 5.4 scrub ok 2024-02-10T07:43:40.453 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:40.485 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:40.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:40.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:40 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:43:40.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:40 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:43:40.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:40 smithi008 ceph-mon[29816]: osdmap e280: 8 total, 8 up, 8 in 2024-02-10T07:43:40.561 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:40 smithi008 ceph-mon[29816]: 4.b scrub starts 2024-02-10T07:43:40.561 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:40 smithi008 ceph-mon[29816]: 4.b scrub ok 2024-02-10T07:43:40.561 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:40 smithi008 ceph-mon[29816]: 5.4 scrub starts 2024-02-10T07:43:40.561 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:40 smithi008 ceph-mon[29816]: 5.4 scrub ok 2024-02-10T07:43:40.561 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:40 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:43:40.561 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:40 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:43:40.561 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:40 smithi008 ceph-mon[40743]: osdmap e280: 8 total, 8 up, 8 in 2024-02-10T07:43:40.561 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:40 smithi008 ceph-mon[40743]: 4.b scrub starts 2024-02-10T07:43:40.561 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:40 smithi008 ceph-mon[40743]: 4.b scrub ok 2024-02-10T07:43:40.561 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:40 smithi008 ceph-mon[40743]: 5.4 scrub starts 2024-02-10T07:43:40.561 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:40 smithi008 ceph-mon[40743]: 5.4 scrub ok 2024-02-10T07:43:40.587 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:40.623 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:40.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:40.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:40.725 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:40.750 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:40.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:40.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:40.795 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:43:40.795618+0000 min lat: 0.001562 max lat: 0.194217 avg lat: 0.0108566 2024-02-10T07:43:40.795 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:43:40.795 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 80 15 117906 117891 92.0697 65.6875 0.0110699 0.0108566 2024-02-10T07:43:40.852 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:40.891 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:40.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:40.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:41.061 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:41.094 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:41.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:41.196 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:41.230 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:41.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:41.331 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:41.357 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:41.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:41 smithi170 ceph-mon[33394]: pgmap v535: 148 pgs: 148 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 744 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-10T07:43:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:41 smithi170 ceph-mon[33394]: 6.1 scrub starts 2024-02-10T07:43:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:41 smithi170 ceph-mon[33394]: 6.1 scrub ok 2024-02-10T07:43:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:41 smithi170 ceph-mon[33394]: 6.9 scrub starts 2024-02-10T07:43:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:41 smithi170 ceph-mon[33394]: 6.9 scrub ok 2024-02-10T07:43:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:41 smithi170 ceph-mon[33394]: 6.1a scrub starts 2024-02-10T07:43:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:41 smithi170 ceph-mon[33394]: 6.1a scrub ok 2024-02-10T07:43:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:41 smithi170 ceph-mon[33394]: 4.11 scrub starts 2024-02-10T07:43:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:41 smithi170 ceph-mon[33394]: 4.11 scrub ok 2024-02-10T07:43:41.459 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:41.493 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:41.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:41.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[29816]: pgmap v535: 148 pgs: 148 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 744 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-10T07:43:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[29816]: 6.1 scrub starts 2024-02-10T07:43:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[29816]: 6.1 scrub ok 2024-02-10T07:43:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[29816]: 6.9 scrub starts 2024-02-10T07:43:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[29816]: 6.9 scrub ok 2024-02-10T07:43:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[29816]: 6.1a scrub starts 2024-02-10T07:43:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[29816]: 6.1a scrub ok 2024-02-10T07:43:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[29816]: 4.11 scrub starts 2024-02-10T07:43:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[29816]: 4.11 scrub ok 2024-02-10T07:43:41.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[40743]: pgmap v535: 148 pgs: 148 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 744 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-10T07:43:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[40743]: 6.1 scrub starts 2024-02-10T07:43:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[40743]: 6.1 scrub ok 2024-02-10T07:43:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[40743]: 6.9 scrub starts 2024-02-10T07:43:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[40743]: 6.9 scrub ok 2024-02-10T07:43:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[40743]: 6.1a scrub starts 2024-02-10T07:43:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[40743]: 6.1a scrub ok 2024-02-10T07:43:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[40743]: 4.11 scrub starts 2024-02-10T07:43:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:41 smithi008 ceph-mon[40743]: 4.11 scrub ok 2024-02-10T07:43:41.595 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:41.625 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:41.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:41.726 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:41.767 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:41.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:41.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:41.794 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 81 16 118835 118819 91.6491 58 0.0216013 0.0109063 2024-02-10T07:43:41.869 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:41.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:41.908 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:41.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:42.010 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:42.153 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:42.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:42.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:42.199 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:42 smithi170 ceph-mon[33394]: 6.a scrub starts 2024-02-10T07:43:42.199 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:42 smithi170 ceph-mon[33394]: 6.a scrub ok 2024-02-10T07:43:42.199 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:42 smithi170 ceph-mon[33394]: 9.5 scrub starts 2024-02-10T07:43:42.199 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:42 smithi170 ceph-mon[33394]: 9.e scrub starts 2024-02-10T07:43:42.199 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:42 smithi170 ceph-mon[33394]: 6.6 scrub starts 2024-02-10T07:43:42.199 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:42 smithi170 ceph-mon[33394]: 6.6 scrub ok 2024-02-10T07:43:42.248 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:43:42.248 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2024-02-10T07:43:42.248 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2024-02-10T07:43:42.248 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:43:42.248 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.248 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:154826 7.0 7.577e9d00 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:43:42.249 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:32.885749+0000", 2024-02-10T07:43:42.249 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 429.35913223300003, 2024-02-10T07:43:42.249 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.86189586600000001, 2024-02-10T07:43:42.249 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.249 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:43:42.249 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:42.249 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:43:42.249 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:43:42.249 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 154826 2024-02-10T07:43:42.249 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.249 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.249 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.249 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.249 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885749+0000", 2024-02-10T07:43:42.250 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.250 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.250 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885749+0000", 2024-02-10T07:43:42.250 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.250 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.250 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885751+0000", 2024-02-10T07:43:42.250 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2109999999999999e-06 2024-02-10T07:43:42.250 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.250 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.250 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885755+0000", 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7960000000000003e-06 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885765+0000", 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0101999999999999e-05 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885775+0000", 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9359999999999994e-06 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.251 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885836+0000", 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0615999999999997e-05 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885856+0000", 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0418999999999999e-05 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:33.747631+0000", 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.86177504800000004 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.252 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.253 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:33.747645+0000", 2024-02-10T07:43:42.253 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3738e-05 2024-02-10T07:43:42.253 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.253 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.253 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.253 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.253 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.253 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:144626 8.4 8:271d8ba5:::benchmark_data_smithi008_109566_object19539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-02-10T07:43:42.253 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.472241+0000", 2024-02-10T07:43:42.253 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 208.772640599, 2024-02-10T07:43:42.253 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12407452200000001, 2024-02-10T07:43:42.253 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.253 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.253 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 144626 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472241+0000", 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472241+0000", 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.254 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472242+0000", 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0999999999999997e-07 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472245+0000", 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.394e-06 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472253+0000", 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4460000000000004e-06 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.255 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472258+0000", 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9049999999999996e-06 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472286+0000", 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7982000000000001e-05 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472361+0000", 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5145e-05 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.256 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.472438+0000", 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7408000000000002e-05 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.475157+0000", 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002719244 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.475595+0000", 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00043725699999999998 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.257 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596109+0000", 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12051481 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596197+0000", 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7805000000000004e-05 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596316+0000", 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000118316 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.258 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15087.0:144623 8.0 e232/200 8:07a3f5f8:::benchmark_data_smithi008_109566_object19536:head v 232'8942, mlcod=232'8939)", 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.476238+0000", 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 208.768643962, 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12355805, 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476238+0000", 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.259 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476238+0000", 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476238+0000", 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2399999999999998e-07 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476244+0000", 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9610000000000002e-06 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.260 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476253+0000", 2024-02-10T07:43:42.261 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.4950000000000008e-06 2024-02-10T07:43:42.261 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.261 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.261 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.261 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476258+0000", 2024-02-10T07:43:42.261 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1760000000000001e-06 2024-02-10T07:43:42.261 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.261 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:42.262 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.262 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.262 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476297+0000", 2024-02-10T07:43:42.262 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8695999999999997e-05 2024-02-10T07:43:42.262 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.262 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476310+0000", 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3533000000000001e-05 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.599757+0000", 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.123446587 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.599796+0000", 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8977999999999998e-05 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.263 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.264 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.264 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.264 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.264 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:144631 8.4 8:2f461f31:::benchmark_data_smithi008_109566_object19544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-02-10T07:43:42.264 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.476357+0000", 2024-02-10T07:43:42.264 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 208.768524614, 2024-02-10T07:43:42.264 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12011643399999999, 2024-02-10T07:43:42.264 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.264 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.264 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:42.264 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:43:42.264 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:43:42.264 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 144631 2024-02-10T07:43:42.264 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476357+0000", 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476357+0000", 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476358+0000", 2024-02-10T07:43:42.265 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.16e-07 2024-02-10T07:43:42.266 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.266 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.266 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.266 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476360+0000", 2024-02-10T07:43:42.266 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5229999999999998e-06 2024-02-10T07:43:42.266 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.266 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.266 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.266 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476367+0000", 2024-02-10T07:43:42.266 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.703e-06 2024-02-10T07:43:42.266 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.266 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.266 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.266 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476372+0000", 2024-02-10T07:43:42.267 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5199999999999999e-06 2024-02-10T07:43:42.267 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.267 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.267 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.267 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476407+0000", 2024-02-10T07:43:42.267 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5549000000000001e-05 2024-02-10T07:43:42.267 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.267 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.267 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.267 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476478+0000", 2024-02-10T07:43:42.267 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0757e-05 2024-02-10T07:43:42.267 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.267 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.267 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:43:42.268 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476546+0000", 2024-02-10T07:43:42.268 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.7724000000000004e-05 2024-02-10T07:43:42.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.268 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.268 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.268 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596071+0000", 2024-02-10T07:43:42.268 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.119525567 2024-02-10T07:43:42.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.268 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.268 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.268 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596359+0000", 2024-02-10T07:43:42.268 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00028778899999999999 2024-02-10T07:43:42.268 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.269 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:42.269 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596371+0000", 2024-02-10T07:43:42.269 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1768e-05 2024-02-10T07:43:42.269 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.269 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.269 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596440+0000", 2024-02-10T07:43:42.269 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8759999999999999e-05 2024-02-10T07:43:42.269 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.269 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.269 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.269 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596473+0000", 2024-02-10T07:43:42.269 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3858e-05 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15087.0:144625 8.0 e232/200 8:0c0904a8:::benchmark_data_smithi008_109566_object19538:head v 232'8943, mlcod=232'8939)", 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.477420+0000", 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 208.767461916, 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12644592499999999, 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.270 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477420+0000", 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477420+0000", 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477420+0000", 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1999999999999999e-07 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477426+0000", 2024-02-10T07:43:42.271 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6230000000000002e-06 2024-02-10T07:43:42.272 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.272 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.272 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.272 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477435+0000", 2024-02-10T07:43:42.272 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.9879999999999993e-06 2024-02-10T07:43:42.272 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.272 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.272 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.272 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477441+0000", 2024-02-10T07:43:42.272 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1990000000000003e-06 2024-02-10T07:43:42.272 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.272 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.272 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.272 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592745+0000", 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11530351899999999 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592769+0000", 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4640000000000001e-05 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.603827+0000", 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011057255 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.273 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.603866+0000", 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9081000000000001e-05 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:144633 8.8 8:1f89b4bf:::benchmark_data_smithi008_109566_object19546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.477783+0000", 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 208.76709902100001, 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12616041, 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 144633 2024-02-10T07:43:42.274 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477783+0000", 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477783+0000", 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477783+0000", 2024-02-10T07:43:42.275 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8999999999999996e-07 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477787+0000", 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.41e-06 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477796+0000", 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0839999999999998e-06 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.477805+0000", 2024-02-10T07:43:42.276 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5629999999999998e-06 2024-02-10T07:43:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592422+0000", 2024-02-10T07:43:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.114616788 2024-02-10T07:43:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592586+0000", 2024-02-10T07:43:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016402499999999999 2024-02-10T07:43:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:43:42.277 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592728+0000", 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014208999999999999 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.602229+0000", 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0095005639999999995 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.602240+0000", 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1272e-05 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.603861+0000", 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0016212139999999999 2024-02-10T07:43:42.278 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.603915+0000", 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3671999999999997e-05 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.603943+0000", 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8038000000000002e-05 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.279 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15087.0:144632 8.a e232/200 8:58d39958:::benchmark_data_smithi008_109566_object19545:head v 232'9186, mlcod=232'9184)", 2024-02-10T07:43:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.478417+0000", 2024-02-10T07:43:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 208.76646481399999, 2024-02-10T07:43:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11883959199999999, 2024-02-10T07:43:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.478417+0000", 2024-02-10T07:43:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.280 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.478417+0000", 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.478417+0000", 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5999999999999999e-07 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.478423+0000", 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2850000000000002e-06 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592761+0000", 2024-02-10T07:43:42.281 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.114338115 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592768+0000", 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5410000000000003e-06 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592813+0000", 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5006999999999998e-05 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592828+0000", 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4491000000000001e-05 2024-02-10T07:43:42.282 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.597220+0000", 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.004392544 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.597256+0000", 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6149000000000002e-05 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.283 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:144636 8.4 8:2934f2c0:::benchmark_data_smithi008_109566_object19549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.480560+0000", 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 208.76432181199999, 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11838391299999999, 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 144636 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480560+0000", 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.284 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480560+0000", 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480560+0000", 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4499999999999997e-07 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480564+0000", 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5039999999999998e-06 2024-02-10T07:43:42.285 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592816+0000", 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11225207399999999 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592831+0000", 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5068999999999999e-05 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592888+0000", 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7164999999999999e-05 2024-02-10T07:43:42.286 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.592998+0000", 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000109948 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.593110+0000", 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000111422 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.596481+0000", 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033717399999999998 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.287 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.288 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.288 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.598790+0000", 2024-02-10T07:43:42.288 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002308531 2024-02-10T07:43:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.598824+0000", 2024-02-10T07:43:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4162999999999998e-05 2024-02-10T07:43:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.598906+0000", 2024-02-10T07:43:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1717000000000004e-05 2024-02-10T07:43:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.289 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.598944+0000", 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7935000000000002e-05 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:168459 8.d 8:b99d57f7:::benchmark_data_smithi008_109566_object43372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:34.586096+0000", 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 187.65878537099999, 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.21918052700000001, 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:43:42.290 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 168459 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.586096+0000", 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.586096+0000", 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.291 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.586098+0000", 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.291e-06 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.586101+0000", 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6459999999999999e-06 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.586336+0000", 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00023513200000000001 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.292 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.586346+0000", 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5750000000000007e-06 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.588181+0000", 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001835218 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.588297+0000", 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000115851 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:43:42.293 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.588414+0000", 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011712199999999999 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.597335+0000", 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0089213580000000008 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.599845+0000", 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0025098989999999999 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.294 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805192+0000", 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.20534719500000001 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805231+0000", 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8674000000000003e-05 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805277+0000", 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5565999999999997e-05 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.295 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:168468 8.4 8:2f095741:::benchmark_data_smithi008_109566_object43381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:34.587949+0000", 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 187.656932764, 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.21746917700000001, 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 168468 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.296 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.587949+0000", 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.587949+0000", 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.587950+0000", 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.8199999999999998e-07 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.297 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.587954+0000", 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5689999999999998e-06 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.587967+0000", 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2918999999999999e-05 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.587975+0000", 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2179999999999999e-06 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.588053+0000", 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7847000000000002e-05 2024-02-10T07:43:42.298 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.588185+0000", 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000132123 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.588303+0000", 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000117546 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.597168+0000", 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0088654700000000003 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.299 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.598139+0000", 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00097050100000000002 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805343+0000", 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.207204164 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805384+0000", 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1090000000000001e-05 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.300 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.301 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.301 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805418+0000", 2024-02-10T07:43:42.301 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3847999999999998e-05 2024-02-10T07:43:42.302 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:168471 8.d 8:b73a9cff:::benchmark_data_smithi008_109566_object43384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:34.591099+0000", 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 187.653783065, 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.21437766799999999, 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 168471 2024-02-10T07:43:42.303 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591099+0000", 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591099+0000", 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591100+0000", 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1340000000000001e-06 2024-02-10T07:43:42.304 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591103+0000", 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.129e-06 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591111+0000", 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2810000000000008e-06 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591117+0000", 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4870000000000002e-06 2024-02-10T07:43:42.305 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591394+0000", 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00027699099999999998 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591520+0000", 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012602199999999999 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591642+0000", 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012226899999999999 2024-02-10T07:43:42.306 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.597340+0000", 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0056981710000000001 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.599885+0000", 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002544883 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805391+0000", 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.20550612700000001 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.307 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805440+0000", 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8715999999999999e-05 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805476+0000", 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6458000000000002e-05 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:168493 8.4 8:2f97658c:::benchmark_data_smithi008_109566_object43406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-10T07:43:42.308 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:34.612764+0000", 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 187.632117722, 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.19627367900000001, 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 168493 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612764+0000", 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.309 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612764+0000", 2024-02-10T07:43:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612765+0000", 2024-02-10T07:43:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.483e-06 2024-02-10T07:43:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612771+0000", 2024-02-10T07:43:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6169999999999998e-06 2024-02-10T07:43:42.310 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612783+0000", 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2041999999999999e-05 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612791+0000", 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6839999999999996e-06 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612849+0000", 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8146000000000002e-05 2024-02-10T07:43:42.311 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612960+0000", 2024-02-10T07:43:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000111319 2024-02-10T07:43:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:43:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.613089+0000", 2024-02-10T07:43:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012928899999999999 2024-02-10T07:43:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.616007+0000", 2024-02-10T07:43:42.312 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0029177740000000001 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.619638+0000", 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0036306519999999998 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.808936+0000", 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.189297875 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.808996+0000", 2024-02-10T07:43:42.313 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9920999999999998e-05 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.809038+0000", 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1876999999999999e-05 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15087.0:172620 8.a e238/200 8:56569598:::benchmark_data_smithi008_109566_object47533:head v 238'10967, mlcod=238'10965)", 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:38.752162+0000", 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 183.49271920699999, 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12993287100000001, 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.314 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.752162+0000", 2024-02-10T07:43:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.752162+0000", 2024-02-10T07:43:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.315 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.316 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.752163+0000", 2024-02-10T07:43:42.316 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9800000000000003e-07 2024-02-10T07:43:42.316 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.316 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.316 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.316 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.752169+0000", 2024-02-10T07:43:42.316 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8130000000000001e-06 2024-02-10T07:43:42.316 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.316 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.316 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.316 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.752179+0000", 2024-02-10T07:43:42.316 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.7459999999999995e-06 2024-02-10T07:43:42.316 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.752184+0000", 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4820000000000002e-06 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.752219+0000", 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4521999999999999e-05 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.752231+0000", 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2057e-05 2024-02-10T07:43:42.317 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.882051+0000", 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.129820773 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.882095+0000", 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3779999999999998e-05 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.318 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:172622 8.d 8:b9980fd8:::benchmark_data_smithi008_109566_object47535:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:38.753034+0000", 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 183.49184785400001, 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.14670670899999999, 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 172622 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753034+0000", 2024-02-10T07:43:42.319 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753034+0000", 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753035+0000", 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1459999999999999e-06 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753039+0000", 2024-02-10T07:43:42.320 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4100000000000001e-06 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753049+0000", 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9240000000000002e-06 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753055+0000", 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4960000000000004e-06 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753104+0000", 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9364999999999999e-05 2024-02-10T07:43:42.321 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753182+0000", 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8361000000000004e-05 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.881819+0000", 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12863662100000001 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.891604+0000", 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0097851339999999992 2024-02-10T07:43:42.322 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.891619+0000", 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4630000000000001e-05 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.899625+0000", 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0080057779999999999 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.899695+0000", 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0203000000000005e-05 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.323 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.899740+0000", 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5640999999999999e-05 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15087.0:172623 8.a e238/200 8:597fdf1a:::benchmark_data_smithi008_109566_object47536:head v 238'10968, mlcod=238'10965)", 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:38.755516+0000", 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 183.489365526, 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.13541141400000001, 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.324 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.755516+0000", 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.755516+0000", 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.755517+0000", 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9500000000000002e-07 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.325 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.755521+0000", 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6979999999999997e-06 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.881761+0000", 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12623997300000001 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.881775+0000", 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3318e-05 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.326 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.882195+0000", 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000420832 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.882215+0000", 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0021999999999999e-05 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.890691+0000", 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0084751050000000001 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.327 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.328 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.890927+0000", 2024-02-10T07:43:42.328 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00023687099999999999 2024-02-10T07:43:42.328 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.328 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.328 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.328 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.328 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.329 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15087.0:172626 8.0 e238/200 8:01f8f047:::benchmark_data_smithi008_109566_object47539:head v 238'10681, mlcod=238'10679)", 2024-02-10T07:43:42.330 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:38.756354+0000", 2024-02-10T07:43:42.330 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 183.48852752499999, 2024-02-10T07:43:42.330 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.13500615499999999, 2024-02-10T07:43:42.330 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.330 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.330 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.330 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.330 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.330 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.756354+0000", 2024-02-10T07:43:42.330 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.330 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.330 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.330 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.756354+0000", 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.756355+0000", 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5600000000000005e-07 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.756360+0000", 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1139999999999997e-06 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.331 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.884620+0000", 2024-02-10T07:43:42.332 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.128259926 2024-02-10T07:43:42.332 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.332 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.332 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.332 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.884632+0000", 2024-02-10T07:43:42.332 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1763000000000001e-05 2024-02-10T07:43:42.332 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.332 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.332 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.332 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.884692+0000", 2024-02-10T07:43:42.332 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0733999999999998e-05 2024-02-10T07:43:42.332 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.332 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.332 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.884713+0000", 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0599e-05 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.891319+0000", 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.006605985 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.891360+0000", 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1378000000000003e-05 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.333 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:227061 8.8 8:11027b6b:::benchmark_data_smithi008_109566_object101974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:41:28.165801+0000", 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 134.07908070799999, 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11876985399999999, 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 227061 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.334 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:28.165801+0000", 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:28.165801+0000", 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:28.165802+0000", 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5999999999999991e-07 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.335 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.336 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.336 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:28.165810+0000", 2024-02-10T07:43:42.336 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7729999999999999e-06 2024-02-10T07:43:42.336 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.336 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.336 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.336 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:28.165820+0000", 2024-02-10T07:43:42.336 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0304e-05 2024-02-10T07:43:42.336 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.336 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.336 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.336 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:28.165825+0000", 2024-02-10T07:43:42.336 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3580000000000002e-06 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:28.165863+0000", 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7803999999999999e-05 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:28.165941+0000", 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8213000000000005e-05 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:28.166012+0000", 2024-02-10T07:43:42.337 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.1064999999999998e-05 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:28.169554+0000", 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0035418519999999998 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:28.169890+0000", 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00033532899999999999 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:28.284502+0000", 2024-02-10T07:43:42.338 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.114612584 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:28.284536+0000", 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4366999999999998e-05 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:28.284571+0000", 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4245000000000002e-05 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.339 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:228487 8.8 8:18bea569:::benchmark_data_smithi008_109566_object103400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e248)", 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:41:29.717003+0000", 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 132.527878907, 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.135159689, 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 228487 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:29.717003+0000", 2024-02-10T07:43:42.340 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:29.717003+0000", 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:29.717004+0000", 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.378e-06 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:29.717010+0000", 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0220000000000001e-06 2024-02-10T07:43:42.341 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:29.717022+0000", 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2387e-05 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:29.717030+0000", 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9619999999999993e-06 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:29.717222+0000", 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000191638 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.342 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:29.717342+0000", 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000120346 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:29.717458+0000", 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000115566 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:29.721673+0000", 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0042154649999999998 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.343 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:29.721704+0000", 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0414000000000001e-05 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:29.852099+0000", 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.130395225 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:29.852137+0000", 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7630999999999999e-05 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.344 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:29.852162+0000", 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5655000000000001e-05 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:232817 8.8 8:1bd7dab4:::benchmark_data_smithi008_109566_object107730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:41:34.311685+0000", 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 127.933196862, 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.13338533299999999, 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:43:42.345 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 232817 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:34.311685+0000", 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:34.311685+0000", 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.346 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:34.311686+0000", 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4900000000000004e-07 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:34.311690+0000", 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7870000000000001e-06 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:34.311700+0000", 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0020000000000001e-05 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.347 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:34.311708+0000", 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.1300000000000003e-06 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:34.311764+0000", 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.605e-05 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:34.311893+0000", 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000129703 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:43:42.348 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:34.312015+0000", 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000121507 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:34.314779+0000", 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0027639499999999998 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:34.314962+0000", 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00018336099999999999 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.349 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:34.444975+0000", 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.13001319 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:34.445031+0000", 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5544999999999998e-05 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:34.445070+0000", 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9140999999999997e-05 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.350 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:233587 8.8 8:1d6f89a0:::benchmark_data_smithi008_109566_object108500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:41:35.116033+0000", 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 127.12884810600001, 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.119584235, 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 233587 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:35.116033+0000", 2024-02-10T07:43:42.351 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:35.116033+0000", 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:35.116035+0000", 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.11e-06 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:35.116039+0000", 2024-02-10T07:43:42.352 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6689999999999997e-06 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:35.116050+0000", 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0907000000000001e-05 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:35.116057+0000", 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.1899999999999998e-06 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:35.117656+0000", 2024-02-10T07:43:42.353 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001598588 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:35.117777+0000", 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000120831 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:35.118072+0000", 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00029505999999999998 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:35.124562+0000", 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0064904100000000003 2024-02-10T07:43:42.354 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:35.124658+0000", 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5600999999999995e-05 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:35.235505+0000", 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.110846954 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:35.235573+0000", 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8083000000000005e-05 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:43:42.355 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:43:42.356 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:43:42.356 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:41:35.235618+0000", 2024-02-10T07:43:42.356 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4832e-05 2024-02-10T07:43:42.356 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.356 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.356 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.356 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:43:42.356 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:43:42.356 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:43:42.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:42.357 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:42.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:42.458 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:42.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:42.491 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:42.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:42.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:42 smithi008 ceph-mon[29816]: 6.a scrub starts 2024-02-10T07:43:42.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:42 smithi008 ceph-mon[29816]: 6.a scrub ok 2024-02-10T07:43:42.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:42 smithi008 ceph-mon[29816]: 9.5 scrub starts 2024-02-10T07:43:42.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:42 smithi008 ceph-mon[29816]: 9.e scrub starts 2024-02-10T07:43:42.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:42 smithi008 ceph-mon[29816]: 6.6 scrub starts 2024-02-10T07:43:42.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:42 smithi008 ceph-mon[29816]: 6.6 scrub ok 2024-02-10T07:43:42.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:42 smithi008 ceph-mon[40743]: 6.a scrub starts 2024-02-10T07:43:42.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:42 smithi008 ceph-mon[40743]: 6.a scrub ok 2024-02-10T07:43:42.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:42 smithi008 ceph-mon[40743]: 9.5 scrub starts 2024-02-10T07:43:42.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:42 smithi008 ceph-mon[40743]: 9.e scrub starts 2024-02-10T07:43:42.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:42 smithi008 ceph-mon[40743]: 6.6 scrub starts 2024-02-10T07:43:42.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:42 smithi008 ceph-mon[40743]: 6.6 scrub ok 2024-02-10T07:43:42.592 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:42.628 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:42.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:42.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:42.730 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:42.768 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:42.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:42.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:42.795 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 82 16 119757 119741 91.2342 57.625 0.00346299 0.0109558 2024-02-10T07:43:42.870 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:42.905 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:42.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:43.006 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:43.037 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:43.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:43.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:43.139 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:43.179 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:43.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:43.281 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:43.332 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-10T07:43:43.332 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:43.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:43.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:43.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:43:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:43:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:43 smithi170 ceph-mon[33394]: 5.1f scrub starts 2024-02-10T07:43:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:43 smithi170 ceph-mon[33394]: 5.1f scrub ok 2024-02-10T07:43:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:43 smithi170 ceph-mon[33394]: pgmap v536: 148 pgs: 148 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 862 op/s 2024-02-10T07:43:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:43 smithi170 ceph-mon[33394]: 6.12 scrub starts 2024-02-10T07:43:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:43 smithi170 ceph-mon[33394]: 6.12 scrub ok 2024-02-10T07:43:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:43 smithi170 ceph-mon[33394]: 6.13 scrub starts 2024-02-10T07:43:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:43 smithi170 ceph-mon[33394]: 6.13 scrub ok 2024-02-10T07:43:43.435 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:43.462 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:43 smithi008 ceph-mon[29816]: 5.1f scrub starts 2024-02-10T07:43:43.462 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:43 smithi008 ceph-mon[29816]: 5.1f scrub ok 2024-02-10T07:43:43.462 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:43 smithi008 ceph-mon[29816]: pgmap v536: 148 pgs: 148 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 862 op/s 2024-02-10T07:43:43.462 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:43 smithi008 ceph-mon[29816]: 6.12 scrub starts 2024-02-10T07:43:43.462 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:43 smithi008 ceph-mon[29816]: 6.12 scrub ok 2024-02-10T07:43:43.462 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:43 smithi008 ceph-mon[29816]: 6.13 scrub starts 2024-02-10T07:43:43.463 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:43 smithi008 ceph-mon[29816]: 6.13 scrub ok 2024-02-10T07:43:43.463 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:43 smithi008 ceph-mon[40743]: 5.1f scrub starts 2024-02-10T07:43:43.463 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:43 smithi008 ceph-mon[40743]: 5.1f scrub ok 2024-02-10T07:43:43.463 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:43 smithi008 ceph-mon[40743]: pgmap v536: 148 pgs: 148 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 862 op/s 2024-02-10T07:43:43.463 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:43 smithi008 ceph-mon[40743]: 6.12 scrub starts 2024-02-10T07:43:43.463 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:43 smithi008 ceph-mon[40743]: 6.12 scrub ok 2024-02-10T07:43:43.463 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:43 smithi008 ceph-mon[40743]: 6.13 scrub starts 2024-02-10T07:43:43.463 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:43 smithi008 ceph-mon[40743]: 6.13 scrub ok 2024-02-10T07:43:43.469 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:43.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:43.571 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:43.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:43.609 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:43.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:43.709 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:43.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:43.748 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:43.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:43.795 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 83 16 120678 120662 90.8285 57.5625 0.0200382 0.0110036 2024-02-10T07:43:43.849 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:43.884 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:43.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:43.985 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:44.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:44.028 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:44.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:44.129 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:44.166 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:44.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:44.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:44.269 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:44.307 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:44.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:44.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:44 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:43:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:44 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:43:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:44 smithi170 ceph-mon[33394]: 4.1c scrub starts 2024-02-10T07:43:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:44 smithi170 ceph-mon[33394]: 4.1c scrub ok 2024-02-10T07:43:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:44 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:44 smithi170 ceph-mon[33394]: 6.2 scrub starts 2024-02-10T07:43:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:44 smithi170 ceph-mon[33394]: 6.2 scrub ok 2024-02-10T07:43:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:44 smithi170 ceph-mon[33394]: 9.3 scrub starts 2024-02-10T07:43:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:44 smithi170 ceph-mon[33394]: osdmap e281: 8 total, 8 up, 8 in 2024-02-10T07:43:44.409 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:44.444 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:44.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:44.546 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:44.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:43:44.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:43:44.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[29816]: 4.1c scrub starts 2024-02-10T07:43:44.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[29816]: 4.1c scrub ok 2024-02-10T07:43:44.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:44.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[29816]: 6.2 scrub starts 2024-02-10T07:43:44.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[29816]: 6.2 scrub ok 2024-02-10T07:43:44.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[29816]: 9.3 scrub starts 2024-02-10T07:43:44.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[29816]: osdmap e281: 8 total, 8 up, 8 in 2024-02-10T07:43:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:43:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:43:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[40743]: 4.1c scrub starts 2024-02-10T07:43:44.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[40743]: 4.1c scrub ok 2024-02-10T07:43:44.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:44.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[40743]: 6.2 scrub starts 2024-02-10T07:43:44.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[40743]: 6.2 scrub ok 2024-02-10T07:43:44.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[40743]: 9.3 scrub starts 2024-02-10T07:43:44.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:44 smithi008 ceph-mon[40743]: osdmap e281: 8 total, 8 up, 8 in 2024-02-10T07:43:44.588 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:44.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:44.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:44.690 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:44.750 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:44.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:44.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:44.795 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 84 16 121713 121697 90.5173 64.6875 0.00644812 0.0110427 2024-02-10T07:43:44.852 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:44.898 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:44.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:44.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:45.000 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:45.041 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:45.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:45.143 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:45.183 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:45.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:45.284 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:45.323 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:45.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:45.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:45 smithi008 ceph-mon[29816]: 9.4 scrub starts 2024-02-10T07:43:45.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:45 smithi008 ceph-mon[29816]: pgmap v538: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-10T07:43:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:45 smithi008 ceph-mon[29816]: 9.e scrub ok 2024-02-10T07:43:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:45 smithi008 ceph-mon[29816]: 9.5 scrub ok 2024-02-10T07:43:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:45 smithi008 ceph-mon[29816]: 5.10 scrub starts 2024-02-10T07:43:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:45 smithi008 ceph-mon[29816]: 5.10 scrub ok 2024-02-10T07:43:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:45 smithi008 ceph-mon[29816]: 6.1f scrub starts 2024-02-10T07:43:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:45 smithi008 ceph-mon[40743]: 9.4 scrub starts 2024-02-10T07:43:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:45 smithi008 ceph-mon[40743]: pgmap v538: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-10T07:43:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:45 smithi008 ceph-mon[40743]: 9.e scrub ok 2024-02-10T07:43:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:45 smithi008 ceph-mon[40743]: 9.5 scrub ok 2024-02-10T07:43:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:45 smithi008 ceph-mon[40743]: 5.10 scrub starts 2024-02-10T07:43:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:45 smithi008 ceph-mon[40743]: 5.10 scrub ok 2024-02-10T07:43:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:45 smithi008 ceph-mon[40743]: 6.1f scrub starts 2024-02-10T07:43:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:45 smithi170 ceph-mon[33394]: 9.4 scrub starts 2024-02-10T07:43:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:45 smithi170 ceph-mon[33394]: pgmap v538: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-10T07:43:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:45 smithi170 ceph-mon[33394]: 9.e scrub ok 2024-02-10T07:43:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:45 smithi170 ceph-mon[33394]: 9.5 scrub ok 2024-02-10T07:43:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:45 smithi170 ceph-mon[33394]: 5.10 scrub starts 2024-02-10T07:43:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:45 smithi170 ceph-mon[33394]: 5.10 scrub ok 2024-02-10T07:43:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:45 smithi170 ceph-mon[33394]: 6.1f scrub starts 2024-02-10T07:43:45.426 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:45.473 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:45.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:45.575 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:45.618 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:45.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:45.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:45.720 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:45.759 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:45.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:45.795 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 85 15 122738 122723 90.2069 64.125 0.0150381 0.0110811 2024-02-10T07:43:45.847 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:43:45.860 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:45.887 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:45.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:45.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:45.889 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 5 1 2024-02-10T07:43:45.896 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:43:45.904 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:45.948 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:43:45.990 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:45.996 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:43:46.036 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:43:46.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:46.073 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:46.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:46.076 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:43:46.121 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:43:46.162 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:43:46.168 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:43:46.169 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:43:46.169 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:43:46.169 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:43:46.169 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:43:46.169 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:43:46.169 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:43:46.169 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:43:46.174 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:46.206 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:46.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:46.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:46.307 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:46.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:46 smithi008 ceph-mon[29816]: 6.1f scrub ok 2024-02-10T07:43:46.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:46 smithi008 ceph-mon[29816]: 4.1b scrub starts 2024-02-10T07:43:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:46 smithi008 ceph-mon[29816]: 4.1b scrub ok 2024-02-10T07:43:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:46 smithi008 ceph-mon[29816]: 6.3 scrub starts 2024-02-10T07:43:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:46 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/619435975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:46.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:46 smithi008 ceph-mon[40743]: 6.1f scrub ok 2024-02-10T07:43:46.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:46 smithi008 ceph-mon[40743]: 4.1b scrub starts 2024-02-10T07:43:46.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:46 smithi008 ceph-mon[40743]: 4.1b scrub ok 2024-02-10T07:43:46.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:46 smithi008 ceph-mon[40743]: 6.3 scrub starts 2024-02-10T07:43:46.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:46 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/619435975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:46.342 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:46.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:46.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:46.444 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:46.487 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:46.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:46.588 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:46.623 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:46.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:46 smithi170 ceph-mon[33394]: 6.1f scrub ok 2024-02-10T07:43:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:46 smithi170 ceph-mon[33394]: 4.1b scrub starts 2024-02-10T07:43:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:46 smithi170 ceph-mon[33394]: 4.1b scrub ok 2024-02-10T07:43:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:46 smithi170 ceph-mon[33394]: 6.3 scrub starts 2024-02-10T07:43:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:46 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/619435975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:43:46.725 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:46.763 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:46.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:46.795 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 86 16 123763 123747 89.9021 64 0.0062426 0.0111184 2024-02-10T07:43:46.865 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:46.906 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:46.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:46.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:47.008 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:47.046 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:47.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:47.093 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:43:47.149 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:47.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:47.182 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:47.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:47.283 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:47.320 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:47.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:47.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[29816]: 6.3 scrub ok 2024-02-10T07:43:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[29816]: 9.3 scrub ok 2024-02-10T07:43:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[29816]: 9.4 scrub ok 2024-02-10T07:43:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[29816]: pgmap v539: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 652 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-10T07:43:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[29816]: 9.9 scrub starts 2024-02-10T07:43:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[29816]: 5.b scrub starts 2024-02-10T07:43:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[29816]: 5.b scrub ok 2024-02-10T07:43:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[29816]: 6.17 scrub starts 2024-02-10T07:43:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[29816]: 9.d scrub starts 2024-02-10T07:43:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[29816]: 5.13 scrub starts 2024-02-10T07:43:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[29816]: 5.13 scrub ok 2024-02-10T07:43:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[29816]: 3.10 scrub starts 2024-02-10T07:43:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[29816]: 3.10 scrub ok 2024-02-10T07:43:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[40743]: 6.3 scrub ok 2024-02-10T07:43:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[40743]: 9.3 scrub ok 2024-02-10T07:43:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[40743]: 9.4 scrub ok 2024-02-10T07:43:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[40743]: pgmap v539: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 652 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-10T07:43:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[40743]: 9.9 scrub starts 2024-02-10T07:43:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[40743]: 5.b scrub starts 2024-02-10T07:43:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[40743]: 5.b scrub ok 2024-02-10T07:43:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[40743]: 6.17 scrub starts 2024-02-10T07:43:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[40743]: 9.d scrub starts 2024-02-10T07:43:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[40743]: 5.13 scrub starts 2024-02-10T07:43:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[40743]: 5.13 scrub ok 2024-02-10T07:43:47.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[40743]: 3.10 scrub starts 2024-02-10T07:43:47.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:47 smithi008 ceph-mon[40743]: 3.10 scrub ok 2024-02-10T07:43:47.422 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:47.457 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:47.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:47.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:47.559 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:47.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:47.605 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:47.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:47 smithi170 ceph-mon[33394]: 6.3 scrub ok 2024-02-10T07:43:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:47 smithi170 ceph-mon[33394]: 9.3 scrub ok 2024-02-10T07:43:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:47 smithi170 ceph-mon[33394]: 9.4 scrub ok 2024-02-10T07:43:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:47 smithi170 ceph-mon[33394]: pgmap v539: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 652 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-02-10T07:43:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:47 smithi170 ceph-mon[33394]: 9.9 scrub starts 2024-02-10T07:43:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:47 smithi170 ceph-mon[33394]: 5.b scrub starts 2024-02-10T07:43:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:47 smithi170 ceph-mon[33394]: 5.b scrub ok 2024-02-10T07:43:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:47 smithi170 ceph-mon[33394]: 6.17 scrub starts 2024-02-10T07:43:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:47 smithi170 ceph-mon[33394]: 9.d scrub starts 2024-02-10T07:43:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:47 smithi170 ceph-mon[33394]: 5.13 scrub starts 2024-02-10T07:43:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:47 smithi170 ceph-mon[33394]: 5.13 scrub ok 2024-02-10T07:43:47.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:47 smithi170 ceph-mon[33394]: 3.10 scrub starts 2024-02-10T07:43:47.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:47 smithi170 ceph-mon[33394]: 3.10 scrub ok 2024-02-10T07:43:47.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:47.743 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:47.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:47.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:47.795 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 87 16 124862 124846 89.6583 68.6875 0.0233342 0.0111485 2024-02-10T07:43:47.844 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:47.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:47.917 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:47.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:48.018 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:48.063 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:48.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:48.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:48.165 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:48.198 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:48.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:48.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:48.300 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:48.397 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:48.407 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:48.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:48.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:48.465 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:48.510 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:48.549 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:48.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:48.553 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:48 smithi008 ceph-mon[29816]: 6.17 scrub ok 2024-02-10T07:43:48.553 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:48 smithi008 ceph-mon[29816]: 5.1 scrub starts 2024-02-10T07:43:48.553 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:48 smithi008 ceph-mon[29816]: 5.1 scrub ok 2024-02-10T07:43:48.553 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:48 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:48.553 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:48 smithi008 ceph-mon[40743]: 6.17 scrub ok 2024-02-10T07:43:48.554 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:48 smithi008 ceph-mon[40743]: 5.1 scrub starts 2024-02-10T07:43:48.554 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:48 smithi008 ceph-mon[40743]: 5.1 scrub ok 2024-02-10T07:43:48.554 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:48 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:48 smithi170 ceph-mon[33394]: 6.17 scrub ok 2024-02-10T07:43:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:48 smithi170 ceph-mon[33394]: 5.1 scrub starts 2024-02-10T07:43:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:48 smithi170 ceph-mon[33394]: 5.1 scrub ok 2024-02-10T07:43:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:48 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:43:48.649 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:48.685 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:48.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:48.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:48.795 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 88 16 125874 125858 89.3582 63.25 0.0141643 0.0111864 2024-02-10T07:43:48.819 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:48.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:48.831 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:43:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:43:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:43:48.921 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:48.952 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:48.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:49.055 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:49.097 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:49.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:49.199 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:49.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:49.240 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:49.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:49.341 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:49.351 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[29816]: 3.4 scrub starts 2024-02-10T07:43:49.351 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[29816]: 3.4 scrub ok 2024-02-10T07:43:49.351 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[29816]: pgmap v540: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 60 MiB/s wr, 964 op/s 2024-02-10T07:43:49.351 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[29816]: 6.8 scrub starts 2024-02-10T07:43:49.351 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[29816]: 6.8 scrub ok 2024-02-10T07:43:49.351 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[29816]: 9.9 scrub ok 2024-02-10T07:43:49.351 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[29816]: 6.1b scrub starts 2024-02-10T07:43:49.351 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[29816]: 6.1b scrub ok 2024-02-10T07:43:49.352 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[40743]: 3.4 scrub starts 2024-02-10T07:43:49.352 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[40743]: 3.4 scrub ok 2024-02-10T07:43:49.352 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[40743]: pgmap v540: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 60 MiB/s wr, 964 op/s 2024-02-10T07:43:49.352 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[40743]: 6.8 scrub starts 2024-02-10T07:43:49.352 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[40743]: 6.8 scrub ok 2024-02-10T07:43:49.352 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[40743]: 9.9 scrub ok 2024-02-10T07:43:49.352 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[40743]: 6.1b scrub starts 2024-02-10T07:43:49.352 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:49 smithi008 ceph-mon[40743]: 6.1b scrub ok 2024-02-10T07:43:49.385 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:49.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:49.488 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:49.521 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:49.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:49.623 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:49 smithi170 ceph-mon[33394]: 3.4 scrub starts 2024-02-10T07:43:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:49 smithi170 ceph-mon[33394]: 3.4 scrub ok 2024-02-10T07:43:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:49 smithi170 ceph-mon[33394]: pgmap v540: 148 pgs: 3 active+clean+scrubbing, 145 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 60 MiB/s wr, 964 op/s 2024-02-10T07:43:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:49 smithi170 ceph-mon[33394]: 6.8 scrub starts 2024-02-10T07:43:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:49 smithi170 ceph-mon[33394]: 6.8 scrub ok 2024-02-10T07:43:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:49 smithi170 ceph-mon[33394]: 9.9 scrub ok 2024-02-10T07:43:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:49 smithi170 ceph-mon[33394]: 6.1b scrub starts 2024-02-10T07:43:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:49 smithi170 ceph-mon[33394]: 6.1b scrub ok 2024-02-10T07:43:49.663 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:49.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:49.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:49.664 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:43:49.664 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:43:49.765 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:49.796 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 89 16 127048 127032 89.1779 73.375 0.00642178 0.0112089 2024-02-10T07:43:49.803 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:49.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:49.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:49.904 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:49.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:49.943 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:49.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:50.044 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:50.081 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:50.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:50.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:50.183 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:50.219 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:50.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:50.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:50.320 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:50 smithi008 ceph-mon[29816]: 9.d scrub ok 2024-02-10T07:43:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:50 smithi008 ceph-mon[29816]: 5.11 scrub starts 2024-02-10T07:43:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:50 smithi008 ceph-mon[29816]: 5.11 scrub ok 2024-02-10T07:43:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:50 smithi008 ceph-mon[40743]: 9.d scrub ok 2024-02-10T07:43:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:50 smithi008 ceph-mon[40743]: 5.11 scrub starts 2024-02-10T07:43:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:50 smithi008 ceph-mon[40743]: 5.11 scrub ok 2024-02-10T07:43:50.356 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:43:50.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:50.357 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:50.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:50.458 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:50.497 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-10T07:43:50.507 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:50.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:50.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:50.609 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:50 smithi170 ceph-mon[33394]: 9.d scrub ok 2024-02-10T07:43:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:50 smithi170 ceph-mon[33394]: 5.11 scrub starts 2024-02-10T07:43:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:50 smithi170 ceph-mon[33394]: 5.11 scrub ok 2024-02-10T07:43:50.639 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:50.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:50.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:50.741 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:50.777 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:50.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:50.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:50.796 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Total time run: 90.011 2024-02-10T07:43:50.796 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Total writes made: 128174 2024-02-10T07:43:50.796 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Write size: 65536 2024-02-10T07:43:50.796 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Object size: 65536 2024-02-10T07:43:50.796 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Bandwidth (MB/sec): 88.9989 2024-02-10T07:43:50.796 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev Bandwidth: 32.9738 2024-02-10T07:43:50.797 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max bandwidth (MB/sec): 199.438 2024-02-10T07:43:50.797 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min bandwidth (MB/sec): 57.5625 2024-02-10T07:43:50.797 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Average IOPS: 1423 2024-02-10T07:43:50.797 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev IOPS: 527.581 2024-02-10T07:43:50.797 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max IOPS: 3191 2024-02-10T07:43:50.797 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min IOPS: 921 2024-02-10T07:43:50.797 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Average Latency(s): 0.011232 2024-02-10T07:43:50.797 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev Latency(s): 0.00810819 2024-02-10T07:43:50.797 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max latency(s): 0.194217 2024-02-10T07:43:50.797 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min latency(s): 0.001562 2024-02-10T07:43:50.800 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Cleaning up (deleting benchmark objects) 2024-02-10T07:43:50.879 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:50.917 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:50.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:50.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:51.019 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:51.051 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:51.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:51.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:51.153 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:51.191 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:51.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:51.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:51.292 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:51.332 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:51.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:51.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:51.435 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:51.467 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:51.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:51.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:51.569 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:51 smithi008 ceph-mon[29816]: pgmap v541: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-10T07:43:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:51 smithi008 ceph-mon[29816]: 9.1 scrub starts 2024-02-10T07:43:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:51 smithi008 ceph-mon[29816]: 3.a scrub starts 2024-02-10T07:43:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:51 smithi008 ceph-mon[29816]: 3.a scrub ok 2024-02-10T07:43:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:51 smithi008 ceph-mon[40743]: pgmap v541: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-10T07:43:51.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:51 smithi008 ceph-mon[40743]: 9.1 scrub starts 2024-02-10T07:43:51.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:51 smithi008 ceph-mon[40743]: 3.a scrub starts 2024-02-10T07:43:51.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:51 smithi008 ceph-mon[40743]: 3.a scrub ok 2024-02-10T07:43:51.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:51.600 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:51.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:51 smithi170 ceph-mon[33394]: pgmap v541: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-10T07:43:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:51 smithi170 ceph-mon[33394]: 9.1 scrub starts 2024-02-10T07:43:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:51 smithi170 ceph-mon[33394]: 3.a scrub starts 2024-02-10T07:43:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:51 smithi170 ceph-mon[33394]: 3.a scrub ok 2024-02-10T07:43:51.701 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:51.735 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:51.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:51.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:51.836 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:51.869 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:51.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:51.971 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:52.004 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:52.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:52.106 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:52.131 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:52.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:52.232 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:52.280 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:52.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:52.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:52.382 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:52.417 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:52.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:52.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:52.519 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:52.554 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:52.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:52.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:52 smithi008 ceph-mon[29816]: 5.a scrub starts 2024-02-10T07:43:52.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:52 smithi008 ceph-mon[29816]: 5.a scrub ok 2024-02-10T07:43:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:52 smithi008 ceph-mon[29816]: 5.18 scrub starts 2024-02-10T07:43:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:52 smithi008 ceph-mon[29816]: 5.18 scrub ok 2024-02-10T07:43:52.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:52 smithi008 ceph-mon[40743]: 5.a scrub starts 2024-02-10T07:43:52.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:52 smithi008 ceph-mon[40743]: 5.a scrub ok 2024-02-10T07:43:52.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:52 smithi008 ceph-mon[40743]: 5.18 scrub starts 2024-02-10T07:43:52.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:52 smithi008 ceph-mon[40743]: 5.18 scrub ok 2024-02-10T07:43:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:52 smithi170 ceph-mon[33394]: 5.a scrub starts 2024-02-10T07:43:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:52 smithi170 ceph-mon[33394]: 5.a scrub ok 2024-02-10T07:43:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:52 smithi170 ceph-mon[33394]: 5.18 scrub starts 2024-02-10T07:43:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:52 smithi170 ceph-mon[33394]: 5.18 scrub ok 2024-02-10T07:43:52.656 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:52.694 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:52.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:52.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:52.795 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:52.825 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:52.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:52.927 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:52.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:52.960 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:52.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:53.061 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:53.096 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:53.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:53.197 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:53.229 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:53.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:53.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:53.331 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:53.364 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:53.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:53.466 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:53.510 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:53.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:53.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:53.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:53 smithi008 ceph-mon[29816]: pgmap v542: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-10T07:43:53.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:53 smithi008 ceph-mon[29816]: 5.f scrub starts 2024-02-10T07:43:53.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:53 smithi008 ceph-mon[29816]: 5.f scrub ok 2024-02-10T07:43:53.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:53 smithi008 ceph-mon[29816]: 9.2 scrub starts 2024-02-10T07:43:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:53 smithi008 ceph-mon[40743]: pgmap v542: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-10T07:43:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:53 smithi008 ceph-mon[40743]: 5.f scrub starts 2024-02-10T07:43:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:53 smithi008 ceph-mon[40743]: 5.f scrub ok 2024-02-10T07:43:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:53 smithi008 ceph-mon[40743]: 9.2 scrub starts 2024-02-10T07:43:53.612 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:53.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:43:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:43:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:53 smithi170 ceph-mon[33394]: pgmap v542: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-10T07:43:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:53 smithi170 ceph-mon[33394]: 5.f scrub starts 2024-02-10T07:43:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:53 smithi170 ceph-mon[33394]: 5.f scrub ok 2024-02-10T07:43:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:53 smithi170 ceph-mon[33394]: 9.2 scrub starts 2024-02-10T07:43:53.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:53.653 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:53.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:53.754 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:53.786 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:53.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:53.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:53.888 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:53.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:53.912 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:53.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:43:54.013 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:54.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:54.048 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:54.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:54.149 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:54.180 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:54.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:54.282 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:54.327 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:54.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:54.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:54.429 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:54.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:54.464 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:54.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:54.565 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:54.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:54 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:54.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:54 smithi008 ceph-mon[29816]: 9.1 scrub ok 2024-02-10T07:43:54.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:54 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:54.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:54 smithi008 ceph-mon[40743]: 9.1 scrub ok 2024-02-10T07:43:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:54.597 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:54.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:54 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:43:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:54 smithi170 ceph-mon[33394]: 9.1 scrub ok 2024-02-10T07:43:54.699 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:54.731 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:54.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:54.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:54.834 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:54.884 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:54.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:54.989 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:55.022 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:55.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:55.124 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:55.175 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:55.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:55.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:55.276 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:55.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:55.318 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:55.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:55.325 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:55 smithi008 ceph-mon[29816]: pgmap v543: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 54 MiB/s wr, 1.04k op/s 2024-02-10T07:43:55.325 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:55 smithi008 ceph-mon[40743]: pgmap v543: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 54 MiB/s wr, 1.04k op/s 2024-02-10T07:43:55.420 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:55.462 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:55.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:55.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:55.564 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:55.617 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:55.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:55.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:55 smithi170 ceph-mon[33394]: pgmap v543: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 54 MiB/s wr, 1.04k op/s 2024-02-10T07:43:55.719 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:55.771 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:55.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:55.873 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:55.902 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:55.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:55.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:55.905 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:55.920 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:43:56.003 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:56.031 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:56.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:56.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:56.133 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:56.193 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:56.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:56.242 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:43:56.294 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:56.356 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:56.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:56.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:56.491 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:56.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:56.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[29816]: 5.19 scrub starts 2024-02-10T07:43:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[29816]: 5.19 scrub ok 2024-02-10T07:43:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[29816]: 4.8 scrub starts 2024-02-10T07:43:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[29816]: 4.8 scrub ok 2024-02-10T07:43:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:43:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[29816]: 9.2 scrub ok 2024-02-10T07:43:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[29816]: 5.14 scrub starts 2024-02-10T07:43:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[29816]: 5.14 scrub ok 2024-02-10T07:43:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/944167289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:56.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:56.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[40743]: 5.19 scrub starts 2024-02-10T07:43:56.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[40743]: 5.19 scrub ok 2024-02-10T07:43:56.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[40743]: 4.8 scrub starts 2024-02-10T07:43:56.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[40743]: 4.8 scrub ok 2024-02-10T07:43:56.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:43:56.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[40743]: 9.2 scrub ok 2024-02-10T07:43:56.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[40743]: 5.14 scrub starts 2024-02-10T07:43:56.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[40743]: 5.14 scrub ok 2024-02-10T07:43:56.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/944167289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:56.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:56 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:56.592 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:56 smithi170 ceph-mon[33394]: 5.19 scrub starts 2024-02-10T07:43:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:56 smithi170 ceph-mon[33394]: 5.19 scrub ok 2024-02-10T07:43:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:56 smithi170 ceph-mon[33394]: 4.8 scrub starts 2024-02-10T07:43:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:56 smithi170 ceph-mon[33394]: 4.8 scrub ok 2024-02-10T07:43:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:56 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:43:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:56 smithi170 ceph-mon[33394]: 9.2 scrub ok 2024-02-10T07:43:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:56 smithi170 ceph-mon[33394]: 5.14 scrub starts 2024-02-10T07:43:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:56 smithi170 ceph-mon[33394]: 5.14 scrub ok 2024-02-10T07:43:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:56 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/944167289' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:56 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:43:56.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:56.639 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:56.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:56.740 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:56.788 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:56.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:56.889 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:56.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:56.939 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:56.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:57.040 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:57.079 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:57.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:57.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:57.181 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:43:57.218 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:57.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:43:57.233 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.5 to 1 (10000) 2024-02-10T07:43:57.320 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:57.363 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:57.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:57.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:57.465 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:43:57.497 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:57.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:57.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:43:57.555 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: pgmap v544: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 938 B/s rd, 48 MiB/s wr, 989 op/s 2024-02-10T07:43:57.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: 5.8 scrub starts 2024-02-10T07:43:57.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: 5.8 scrub ok 2024-02-10T07:43:57.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:43:57.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:43:57.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:43:57.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: osdmap e282: 8 total, 8 up, 8 in 2024-02-10T07:43:57.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: 9.0 deep-scrub starts 2024-02-10T07:43:57.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: 4.14 scrub starts 2024-02-10T07:43:57.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: 4.14 scrub ok 2024-02-10T07:43:57.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/757678943' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:43:57.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:43:57.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: 9.8 scrub starts 2024-02-10T07:43:57.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:43:57.557 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:43:57.557 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:43:57.557 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:43:57.557 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:43:57.557 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:43:57.557 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:43:57.557 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:43:57.557 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: pgmap v544: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 938 B/s rd, 48 MiB/s wr, 989 op/s 2024-02-10T07:43:57.557 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: 5.8 scrub starts 2024-02-10T07:43:57.557 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: 5.8 scrub ok 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: osdmap e282: 8 total, 8 up, 8 in 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: 9.0 deep-scrub starts 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: 4.14 scrub starts 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: 4.14 scrub ok 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/757678943' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: 9.8 scrub starts 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:43:57.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:43:57.559 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:43:57.559 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:57 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:43:57.599 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:57.630 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:57.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:57.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: pgmap v544: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 938 B/s rd, 48 MiB/s wr, 989 op/s 2024-02-10T07:43:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: 5.8 scrub starts 2024-02-10T07:43:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: 5.8 scrub ok 2024-02-10T07:43:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:43:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:43:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:43:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: osdmap e282: 8 total, 8 up, 8 in 2024-02-10T07:43:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: 9.0 deep-scrub starts 2024-02-10T07:43:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: 4.14 scrub starts 2024-02-10T07:43:57.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: 4.14 scrub ok 2024-02-10T07:43:57.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/757678943' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:43:57.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:43:57.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: 9.8 scrub starts 2024-02-10T07:43:57.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:43:57.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:43:57.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:43:57.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T07:43:57.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:43:57.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:43:57.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:43:57.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:57 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:43:57.732 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:57.777 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:57.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:57.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:57.879 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:57.912 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:57.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:57.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:58.014 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:58.050 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:58.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:58.152 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:43:58.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:58.186 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:58.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:43:58.287 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:58.324 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:58.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:58.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:58.426 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:58.456 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:58.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:58.557 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:58 smithi008 ceph-mon[29816]: Adjusting osd_memory_target on smithi008 to 3249M 2024-02-10T07:43:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:58 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:43:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:58 smithi008 ceph-mon[29816]: osdmap e283: 8 total, 8 up, 8 in 2024-02-10T07:43:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:58 smithi008 ceph-mon[29816]: 9.c scrub starts 2024-02-10T07:43:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:58 smithi008 ceph-mon[29816]: 4.5 scrub starts 2024-02-10T07:43:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:58 smithi008 ceph-mon[29816]: 4.5 scrub ok 2024-02-10T07:43:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:58 smithi008 ceph-mon[40743]: Adjusting osd_memory_target on smithi008 to 3249M 2024-02-10T07:43:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:58 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:43:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:58 smithi008 ceph-mon[40743]: osdmap e283: 8 total, 8 up, 8 in 2024-02-10T07:43:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:58 smithi008 ceph-mon[40743]: 9.c scrub starts 2024-02-10T07:43:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:58 smithi008 ceph-mon[40743]: 4.5 scrub starts 2024-02-10T07:43:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:58 smithi008 ceph-mon[40743]: 4.5 scrub ok 2024-02-10T07:43:58.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:58.596 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:58.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:58 smithi170 ceph-mon[33394]: Adjusting osd_memory_target on smithi008 to 3249M 2024-02-10T07:43:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:58 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:43:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:58 smithi170 ceph-mon[33394]: osdmap e283: 8 total, 8 up, 8 in 2024-02-10T07:43:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:58 smithi170 ceph-mon[33394]: 9.c scrub starts 2024-02-10T07:43:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:58 smithi170 ceph-mon[33394]: 4.5 scrub starts 2024-02-10T07:43:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:58 smithi170 ceph-mon[33394]: 4.5 scrub ok 2024-02-10T07:43:58.697 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:58.740 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:58.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:58.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:58.843 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:58.904 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:58.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:58.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:59.006 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:43:59.035 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:59.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:59.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:43:59.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:43:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:43:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:43:59.136 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:59.165 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:59.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:59.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:59.266 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:59.352 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:59.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:59.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:59.454 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:43:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:59.493 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:59.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:43:59.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:59 smithi170 ceph-mon[33394]: pgmap v547: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 974 op/s 2024-02-10T07:43:59.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:59 smithi170 ceph-mon[33394]: 4.f scrub starts 2024-02-10T07:43:59.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:43:59 smithi170 ceph-mon[33394]: 4.f scrub ok 2024-02-10T07:43:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:59 smithi008 ceph-mon[29816]: pgmap v547: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 974 op/s 2024-02-10T07:43:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:59 smithi008 ceph-mon[29816]: 4.f scrub starts 2024-02-10T07:43:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:43:59 smithi008 ceph-mon[29816]: 4.f scrub ok 2024-02-10T07:43:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:59 smithi008 ceph-mon[40743]: pgmap v547: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 974 op/s 2024-02-10T07:43:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:59 smithi008 ceph-mon[40743]: 4.f scrub starts 2024-02-10T07:43:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:43:59 smithi008 ceph-mon[40743]: 4.f scrub ok 2024-02-10T07:43:59.596 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:43:59.631 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:59.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:43:59.733 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:43:59.773 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:43:59.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:59.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:43:59.875 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:43:59.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:43:59.910 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:43:59.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:00.011 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:00.047 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:00.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:00.148 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:00.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:00.181 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:00.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:00.282 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:00.319 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:00.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:00.420 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:00.448 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:00.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:00.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:00.549 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:00.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[29816]: 6.f scrub starts 2024-02-10T07:44:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[29816]: 9.b scrub starts 2024-02-10T07:44:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[29816]: 6.f scrub ok 2024-02-10T07:44:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[29816]: 3.c scrub starts 2024-02-10T07:44:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[29816]: 3.c scrub ok 2024-02-10T07:44:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[29816]: 9.8 scrub ok 2024-02-10T07:44:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[29816]: 3.18 scrub starts 2024-02-10T07:44:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[29816]: 3.18 scrub ok 2024-02-10T07:44:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[29816]: 6.0 scrub starts 2024-02-10T07:44:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[29816]: 6.0 scrub ok 2024-02-10T07:44:00.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[40743]: 6.f scrub starts 2024-02-10T07:44:00.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[40743]: 9.b scrub starts 2024-02-10T07:44:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[40743]: 6.f scrub ok 2024-02-10T07:44:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[40743]: 3.c scrub starts 2024-02-10T07:44:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[40743]: 3.c scrub ok 2024-02-10T07:44:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[40743]: 9.8 scrub ok 2024-02-10T07:44:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[40743]: 3.18 scrub starts 2024-02-10T07:44:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[40743]: 3.18 scrub ok 2024-02-10T07:44:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[40743]: 6.0 scrub starts 2024-02-10T07:44:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:00 smithi008 ceph-mon[40743]: 6.0 scrub ok 2024-02-10T07:44:00.588 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:00.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:00 smithi170 ceph-mon[33394]: 6.f scrub starts 2024-02-10T07:44:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:00 smithi170 ceph-mon[33394]: 9.b scrub starts 2024-02-10T07:44:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:00 smithi170 ceph-mon[33394]: 6.f scrub ok 2024-02-10T07:44:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:00 smithi170 ceph-mon[33394]: 3.c scrub starts 2024-02-10T07:44:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:00 smithi170 ceph-mon[33394]: 3.c scrub ok 2024-02-10T07:44:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:00 smithi170 ceph-mon[33394]: 9.8 scrub ok 2024-02-10T07:44:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:00 smithi170 ceph-mon[33394]: 3.18 scrub starts 2024-02-10T07:44:00.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:00 smithi170 ceph-mon[33394]: 3.18 scrub ok 2024-02-10T07:44:00.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:00 smithi170 ceph-mon[33394]: 6.0 scrub starts 2024-02-10T07:44:00.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:00 smithi170 ceph-mon[33394]: 6.0 scrub ok 2024-02-10T07:44:00.690 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:00.730 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:00.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:00.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:00.832 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:00.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:00.871 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:00.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:00.972 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:01.000 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:01.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:01.102 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:01.136 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:01.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:01.238 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:01.261 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:01.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:01.363 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:01.397 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:01.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:01.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:01.499 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:01.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:01.536 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:01.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:01.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:01 smithi008 ceph-mon[29816]: pgmap v548: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 1.32k op/s 2024-02-10T07:44:01.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:01 smithi008 ceph-mon[29816]: 6.16 scrub starts 2024-02-10T07:44:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:01 smithi008 ceph-mon[29816]: 6.16 scrub ok 2024-02-10T07:44:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:01 smithi008 ceph-mon[29816]: 6.19 scrub starts 2024-02-10T07:44:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:01 smithi008 ceph-mon[29816]: 6.19 scrub ok 2024-02-10T07:44:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:01 smithi008 ceph-mon[29816]: 9.c scrub ok 2024-02-10T07:44:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:01 smithi008 ceph-mon[40743]: pgmap v548: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 1.32k op/s 2024-02-10T07:44:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:01 smithi008 ceph-mon[40743]: 6.16 scrub starts 2024-02-10T07:44:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:01 smithi008 ceph-mon[40743]: 6.16 scrub ok 2024-02-10T07:44:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:01 smithi008 ceph-mon[40743]: 6.19 scrub starts 2024-02-10T07:44:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:01 smithi008 ceph-mon[40743]: 6.19 scrub ok 2024-02-10T07:44:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:01 smithi008 ceph-mon[40743]: 9.c scrub ok 2024-02-10T07:44:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:01 smithi170 ceph-mon[33394]: pgmap v548: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 22 MiB/s wr, 1.32k op/s 2024-02-10T07:44:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:01 smithi170 ceph-mon[33394]: 6.16 scrub starts 2024-02-10T07:44:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:01 smithi170 ceph-mon[33394]: 6.16 scrub ok 2024-02-10T07:44:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:01 smithi170 ceph-mon[33394]: 6.19 scrub starts 2024-02-10T07:44:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:01 smithi170 ceph-mon[33394]: 6.19 scrub ok 2024-02-10T07:44:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:01 smithi170 ceph-mon[33394]: 9.c scrub ok 2024-02-10T07:44:01.637 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:01.661 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:01.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:01.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:01.762 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:01.798 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:01.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:01.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:01.900 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:01.930 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:01.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:01.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:02.032 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:02.074 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:02.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:02.175 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:02.215 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:02.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:02.316 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:02.339 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:02.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:02.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:02.440 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:02.479 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:02.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:02.581 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:02 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:44:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:02 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:44:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:02 smithi008 ceph-mon[29816]: osdmap e284: 8 total, 8 up, 8 in 2024-02-10T07:44:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:02 smithi008 ceph-mon[29816]: 3.1b scrub starts 2024-02-10T07:44:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:02 smithi008 ceph-mon[29816]: 3.1b scrub ok 2024-02-10T07:44:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:02 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:44:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:02 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:44:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:02 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:44:02.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:02 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:44:02.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:02 smithi008 ceph-mon[40743]: osdmap e284: 8 total, 8 up, 8 in 2024-02-10T07:44:02.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:02 smithi008 ceph-mon[40743]: 3.1b scrub starts 2024-02-10T07:44:02.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:02 smithi008 ceph-mon[40743]: 3.1b scrub ok 2024-02-10T07:44:02.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:02 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:44:02.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:02 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:44:02.617 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:02.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:02.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:02 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:44:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:02 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:44:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:02 smithi170 ceph-mon[33394]: osdmap e284: 8 total, 8 up, 8 in 2024-02-10T07:44:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:02 smithi170 ceph-mon[33394]: 3.1b scrub starts 2024-02-10T07:44:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:02 smithi170 ceph-mon[33394]: 3.1b scrub ok 2024-02-10T07:44:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:02 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:44:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:02 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:44:02.718 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:02.752 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:02.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:02.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:02.854 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:02.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:02.886 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:02.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:02.987 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:03.022 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:03.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:03.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:03.124 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:03.158 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:03.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:03.260 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:03.285 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:03.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:03.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:03.387 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:03.429 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:03.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:03.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:03.531 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:03.554 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:03.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:03.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:44:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:44:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:03 smithi170 ceph-mon[33394]: 4.2 scrub starts 2024-02-10T07:44:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:03 smithi170 ceph-mon[33394]: 4.2 scrub ok 2024-02-10T07:44:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:03 smithi170 ceph-mon[33394]: pgmap v550: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 838 op/s 2024-02-10T07:44:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:03 smithi170 ceph-mon[33394]: 9.b scrub ok 2024-02-10T07:44:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:03 smithi170 ceph-mon[33394]: osdmap e285: 8 total, 8 up, 8 in 2024-02-10T07:44:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:03 smithi170 ceph-mon[33394]: 4.13 scrub starts 2024-02-10T07:44:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:03 smithi170 ceph-mon[33394]: 4.13 scrub ok 2024-02-10T07:44:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:44:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:44:03.656 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:03.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:03.692 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:03.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:03.793 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:03.830 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:03.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:03.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[29816]: 4.2 scrub starts 2024-02-10T07:44:03.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[29816]: 4.2 scrub ok 2024-02-10T07:44:03.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[29816]: pgmap v550: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 838 op/s 2024-02-10T07:44:03.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[29816]: 9.b scrub ok 2024-02-10T07:44:03.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[29816]: osdmap e285: 8 total, 8 up, 8 in 2024-02-10T07:44:03.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[29816]: 4.13 scrub starts 2024-02-10T07:44:03.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[29816]: 4.13 scrub ok 2024-02-10T07:44:03.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:44:03.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:44:03.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[40743]: 4.2 scrub starts 2024-02-10T07:44:03.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[40743]: 4.2 scrub ok 2024-02-10T07:44:03.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[40743]: pgmap v550: 148 pgs: 2 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 145 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 838 op/s 2024-02-10T07:44:03.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[40743]: 9.b scrub ok 2024-02-10T07:44:03.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[40743]: osdmap e285: 8 total, 8 up, 8 in 2024-02-10T07:44:03.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[40743]: 4.13 scrub starts 2024-02-10T07:44:03.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[40743]: 4.13 scrub ok 2024-02-10T07:44:03.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:44:03.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:44:03.932 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:03.960 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:03.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:04.062 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:04.104 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:04.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:04.205 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:04.228 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:04.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:04.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:04.330 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:04.366 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:04.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:04.468 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:04.506 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:04.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:04.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:04.608 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:04.645 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:04.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:04.746 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:04.781 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:04.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:04.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:04 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:44:04.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:04 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:44:04.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:04 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:04.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:04 smithi008 ceph-mon[29816]: 5.c scrub starts 2024-02-10T07:44:04.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:04 smithi008 ceph-mon[29816]: 5.c scrub ok 2024-02-10T07:44:04.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:04 smithi008 ceph-mon[29816]: osdmap e286: 8 total, 8 up, 8 in 2024-02-10T07:44:04.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:04 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:44:04.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:04 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:44:04.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:04 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:04.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:04 smithi008 ceph-mon[40743]: 5.c scrub starts 2024-02-10T07:44:04.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:04 smithi008 ceph-mon[40743]: 5.c scrub ok 2024-02-10T07:44:04.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:04 smithi008 ceph-mon[40743]: osdmap e286: 8 total, 8 up, 8 in 2024-02-10T07:44:04.882 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:04 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:44:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:04 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:44:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:04 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:04 smithi170 ceph-mon[33394]: 5.c scrub starts 2024-02-10T07:44:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:04 smithi170 ceph-mon[33394]: 5.c scrub ok 2024-02-10T07:44:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:04 smithi170 ceph-mon[33394]: osdmap e286: 8 total, 8 up, 8 in 2024-02-10T07:44:04.993 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:04.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:04.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:05.095 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:05.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:05.129 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:05.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:05.230 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:05.264 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:05.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:05.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:05.366 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:05.400 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:05.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:05.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:05.502 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:05.527 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:05.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:05.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:05.629 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:05.644 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:05 smithi008 ceph-mon[29816]: pgmap v553: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 955 op/s 2024-02-10T07:44:05.645 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:05 smithi008 ceph-mon[29816]: 5.5 scrub starts 2024-02-10T07:44:05.645 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:05 smithi008 ceph-mon[29816]: 5.5 scrub ok 2024-02-10T07:44:05.645 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:05 smithi008 ceph-mon[29816]: 3.14 scrub starts 2024-02-10T07:44:05.645 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:05 smithi008 ceph-mon[29816]: 3.14 scrub ok 2024-02-10T07:44:05.645 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:05 smithi008 ceph-mon[40743]: pgmap v553: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 955 op/s 2024-02-10T07:44:05.645 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:05 smithi008 ceph-mon[40743]: 5.5 scrub starts 2024-02-10T07:44:05.645 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:05 smithi008 ceph-mon[40743]: 5.5 scrub ok 2024-02-10T07:44:05.645 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:05 smithi008 ceph-mon[40743]: 3.14 scrub starts 2024-02-10T07:44:05.645 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:05 smithi008 ceph-mon[40743]: 3.14 scrub ok 2024-02-10T07:44:05.681 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:05.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:05.783 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:05.814 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:05.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:05.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:05 smithi170 ceph-mon[33394]: pgmap v553: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 955 op/s 2024-02-10T07:44:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:05 smithi170 ceph-mon[33394]: 5.5 scrub starts 2024-02-10T07:44:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:05 smithi170 ceph-mon[33394]: 5.5 scrub ok 2024-02-10T07:44:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:05 smithi170 ceph-mon[33394]: 3.14 scrub starts 2024-02-10T07:44:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:05 smithi170 ceph-mon[33394]: 3.14 scrub ok 2024-02-10T07:44:05.915 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:05.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:05.976 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:05.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:06.077 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:06.109 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:06.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:06.210 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:06.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:06.245 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:06.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:06.288 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:44:06.347 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:06.347 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:44:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:06.389 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:06.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:06.393 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:44:06.438 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:44:06.475 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:44:06.490 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:06.517 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:44:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:06.535 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:06.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:06.563 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:44:06.563 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:06 smithi008 ceph-mon[29816]: 6.1d deep-scrub starts 2024-02-10T07:44:06.563 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:06 smithi008 ceph-mon[29816]: 6.1d deep-scrub ok 2024-02-10T07:44:06.563 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:06 smithi008 ceph-mon[29816]: 3.7 scrub starts 2024-02-10T07:44:06.563 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:06 smithi008 ceph-mon[29816]: 3.7 scrub ok 2024-02-10T07:44:06.563 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:06 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2808508601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:44:06.564 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:06 smithi008 ceph-mon[40743]: 6.1d deep-scrub starts 2024-02-10T07:44:06.564 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:06 smithi008 ceph-mon[40743]: 6.1d deep-scrub ok 2024-02-10T07:44:06.564 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:06 smithi008 ceph-mon[40743]: 3.7 scrub starts 2024-02-10T07:44:06.564 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:06 smithi008 ceph-mon[40743]: 3.7 scrub ok 2024-02-10T07:44:06.564 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:06 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2808508601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:44:06.604 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:44:06.610 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:44:06.610 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:44:06.611 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:44:06.611 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:44:06.611 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:44:06.611 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:44:06.611 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:44:06.611 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:44:06.636 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:06.669 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:06.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:06.770 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:06.815 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:06.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:06.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:06.819 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 6 1 2024-02-10T07:44:06.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:06 smithi170 ceph-mon[33394]: 6.1d deep-scrub starts 2024-02-10T07:44:06.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:06 smithi170 ceph-mon[33394]: 6.1d deep-scrub ok 2024-02-10T07:44:06.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:06 smithi170 ceph-mon[33394]: 3.7 scrub starts 2024-02-10T07:44:06.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:06 smithi170 ceph-mon[33394]: 3.7 scrub ok 2024-02-10T07:44:06.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:06 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2808508601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:44:06.917 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:06.952 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:06.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:07.055 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:07.084 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:07.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:07.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:07.129 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:44:07.129 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:44:07.129 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-02-10T07:44:07.129 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-02-10T07:44:07.129 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:44:07.186 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:07.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:07.231 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:07.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:07.332 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:07.364 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:07.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:07.465 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:07.496 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:07.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:07.598 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:07.635 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:07.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:07.710 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:44:07.736 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:07.756 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:07 smithi008 ceph-mon[29816]: pgmap v554: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 662 op/s 2024-02-10T07:44:07.757 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:07 smithi008 ceph-mon[29816]: 4.1e scrub starts 2024-02-10T07:44:07.757 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:07 smithi008 ceph-mon[29816]: 4.1e scrub ok 2024-02-10T07:44:07.757 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:07 smithi008 ceph-mon[29816]: 4.4 scrub starts 2024-02-10T07:44:07.757 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:07 smithi008 ceph-mon[29816]: 4.4 scrub ok 2024-02-10T07:44:07.757 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:07 smithi008 ceph-mon[40743]: pgmap v554: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 662 op/s 2024-02-10T07:44:07.757 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:07 smithi008 ceph-mon[40743]: 4.1e scrub starts 2024-02-10T07:44:07.757 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:07 smithi008 ceph-mon[40743]: 4.1e scrub ok 2024-02-10T07:44:07.757 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:07 smithi008 ceph-mon[40743]: 4.4 scrub starts 2024-02-10T07:44:07.757 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:07 smithi008 ceph-mon[40743]: 4.4 scrub ok 2024-02-10T07:44:07.802 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:07.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:07.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:07.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:07 smithi170 ceph-mon[33394]: pgmap v554: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 662 op/s 2024-02-10T07:44:07.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:07 smithi170 ceph-mon[33394]: 4.1e scrub starts 2024-02-10T07:44:07.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:07 smithi170 ceph-mon[33394]: 4.1e scrub ok 2024-02-10T07:44:07.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:07 smithi170 ceph-mon[33394]: 4.4 scrub starts 2024-02-10T07:44:07.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:07 smithi170 ceph-mon[33394]: 4.4 scrub ok 2024-02-10T07:44:07.904 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:07.956 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:07.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:08.058 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:08.100 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:08.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:08.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:08.228 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:08.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:08.330 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:08.367 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:08.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:08.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:08.469 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:08.502 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:08.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:08.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:08.604 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:08.639 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:08.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:08.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:08.740 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:08.778 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:08.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:08.789 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-mon[29816]: 3.1 scrub starts 2024-02-10T07:44:08.789 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-mon[29816]: 3.1 scrub ok 2024-02-10T07:44:08.789 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-mon[29816]: 9.6 scrub starts 2024-02-10T07:44:08.789 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:44:08.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:44:08.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-mon[29816]: 4.0 scrub starts 2024-02-10T07:44:08.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-mon[29816]: 4.0 scrub ok 2024-02-10T07:44:08.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-mon[40743]: 3.1 scrub starts 2024-02-10T07:44:08.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-mon[40743]: 3.1 scrub ok 2024-02-10T07:44:08.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-mon[40743]: 9.6 scrub starts 2024-02-10T07:44:08.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:44:08.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:44:08.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-mon[40743]: 4.0 scrub starts 2024-02-10T07:44:08.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-mon[40743]: 4.0 scrub ok 2024-02-10T07:44:08.878 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-10T07:44:08.879 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:08.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:08 smithi170 ceph-mon[33394]: 3.1 scrub starts 2024-02-10T07:44:08.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:08 smithi170 ceph-mon[33394]: 3.1 scrub ok 2024-02-10T07:44:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:08 smithi170 ceph-mon[33394]: 9.6 scrub starts 2024-02-10T07:44:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:08 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:44:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:08 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:44:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:08 smithi170 ceph-mon[33394]: 4.0 scrub starts 2024-02-10T07:44:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:08 smithi170 ceph-mon[33394]: 4.0 scrub ok 2024-02-10T07:44:08.965 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:08.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:08.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:09.066 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:09.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:44:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:44:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:44:09.123 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:09.154 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:09.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:09.173 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:09.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:09.273 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:09.326 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:09.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:09.427 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:09.467 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:09.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:09.568 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:09.611 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:09.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:09.639 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:09 smithi008 ceph-mon[29816]: pgmap v555: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 925 op/s 2024-02-10T07:44:09.639 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:09 smithi008 ceph-mon[29816]: 4.16 scrub starts 2024-02-10T07:44:09.640 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:09 smithi008 ceph-mon[29816]: 4.16 scrub ok 2024-02-10T07:44:09.640 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:09 smithi008 ceph-mon[29816]: 3.9 scrub starts 2024-02-10T07:44:09.640 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:09 smithi008 ceph-mon[29816]: 3.9 scrub ok 2024-02-10T07:44:09.640 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:09 smithi008 ceph-mon[29816]: osdmap e287: 8 total, 8 up, 8 in 2024-02-10T07:44:09.640 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:09 smithi008 ceph-mon[40743]: pgmap v555: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 925 op/s 2024-02-10T07:44:09.640 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:09 smithi008 ceph-mon[40743]: 4.16 scrub starts 2024-02-10T07:44:09.640 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:09 smithi008 ceph-mon[40743]: 4.16 scrub ok 2024-02-10T07:44:09.640 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:09 smithi008 ceph-mon[40743]: 3.9 scrub starts 2024-02-10T07:44:09.640 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:09 smithi008 ceph-mon[40743]: 3.9 scrub ok 2024-02-10T07:44:09.640 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:09 smithi008 ceph-mon[40743]: osdmap e287: 8 total, 8 up, 8 in 2024-02-10T07:44:09.712 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:09.750 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:09.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:09.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:09.852 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:09.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:09 smithi170 ceph-mon[33394]: pgmap v555: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 925 op/s 2024-02-10T07:44:09.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:09 smithi170 ceph-mon[33394]: 4.16 scrub starts 2024-02-10T07:44:09.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:09 smithi170 ceph-mon[33394]: 4.16 scrub ok 2024-02-10T07:44:09.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:09 smithi170 ceph-mon[33394]: 3.9 scrub starts 2024-02-10T07:44:09.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:09 smithi170 ceph-mon[33394]: 3.9 scrub ok 2024-02-10T07:44:09.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:09 smithi170 ceph-mon[33394]: osdmap e287: 8 total, 8 up, 8 in 2024-02-10T07:44:09.893 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:09.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:09.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:09.995 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:10.043 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:10.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:10.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:10.144 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:10.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:10.196 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:10.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:10.296 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:10.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:10.332 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:10.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:10.433 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:10.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:10.486 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:10.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:10.587 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:10.620 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:10.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:10.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:10.722 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:10.741 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:10 smithi008 ceph-mon[29816]: 4.12 scrub starts 2024-02-10T07:44:10.742 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:10 smithi008 ceph-mon[29816]: 4.12 scrub ok 2024-02-10T07:44:10.742 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:10 smithi008 ceph-mon[29816]: 5.3 scrub starts 2024-02-10T07:44:10.742 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:10 smithi008 ceph-mon[29816]: 5.3 scrub ok 2024-02-10T07:44:10.742 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:10 smithi008 ceph-mon[29816]: 9.6 scrub ok 2024-02-10T07:44:10.742 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:10 smithi008 ceph-mon[29816]: pgmap v557: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-10T07:44:10.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:10 smithi008 ceph-mon[40743]: 4.12 scrub starts 2024-02-10T07:44:10.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:10 smithi008 ceph-mon[40743]: 4.12 scrub ok 2024-02-10T07:44:10.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:10 smithi008 ceph-mon[40743]: 5.3 scrub starts 2024-02-10T07:44:10.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:10 smithi008 ceph-mon[40743]: 5.3 scrub ok 2024-02-10T07:44:10.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:10 smithi008 ceph-mon[40743]: 9.6 scrub ok 2024-02-10T07:44:10.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:10 smithi008 ceph-mon[40743]: pgmap v557: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-10T07:44:10.750 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:10.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:10.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:10.853 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:10.882 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:10.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:10.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:10 smithi170 ceph-mon[33394]: 4.12 scrub starts 2024-02-10T07:44:10.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:10 smithi170 ceph-mon[33394]: 4.12 scrub ok 2024-02-10T07:44:10.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:10 smithi170 ceph-mon[33394]: 5.3 scrub starts 2024-02-10T07:44:10.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:10 smithi170 ceph-mon[33394]: 5.3 scrub ok 2024-02-10T07:44:10.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:10 smithi170 ceph-mon[33394]: 9.6 scrub ok 2024-02-10T07:44:10.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:10 smithi170 ceph-mon[33394]: pgmap v557: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-10T07:44:10.985 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:11.023 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:11.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:11.120 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:44:11.126 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:11.157 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:11.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:11.243 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:11.258 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:11.287 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:11.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:11.389 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:11.453 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:11.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:11.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:11.555 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:11.587 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:11.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:11.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:11.784 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:11.820 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:11.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:11.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:11.922 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:11.965 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:11.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:12.066 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:12.096 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:12.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:12.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:12.198 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:12.232 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:12.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:12.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:12.334 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:12.373 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:12.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:12.475 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:12.503 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:12.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:12.605 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:12.638 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:12.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:12.739 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:12.770 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:12.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:12.872 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:12.905 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:12.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:13.007 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:13.037 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:13.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:13.139 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:13.171 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:13.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:13.240 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:12 smithi008 ceph-mon[29816]: pgmap v558: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 632 B/s rd, 0 B/s wr, 743 op/s 2024-02-10T07:44:13.240 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:12 smithi008 ceph-mon[40743]: pgmap v558: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 632 B/s rd, 0 B/s wr, 743 op/s 2024-02-10T07:44:13.273 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:13.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:13.329 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:13.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:13.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:12 smithi170 ceph-mon[33394]: pgmap v558: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 632 B/s rd, 0 B/s wr, 743 op/s 2024-02-10T07:44:13.430 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:13.460 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:13.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:13.562 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:13.592 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:13.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:13.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:13.693 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:13.723 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:13.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:13.825 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:13.868 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:13.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:13.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:44:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:44:13.935 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:44:13.969 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:13.998 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:13.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:13.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:14.100 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:14.129 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:14.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:14.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:14.184 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[29816]: 9.f deep-scrub starts 2024-02-10T07:44:14.184 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[29816]: 5.9 scrub starts 2024-02-10T07:44:14.184 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[29816]: 5.9 scrub ok 2024-02-10T07:44:14.184 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[29816]: 4.18 deep-scrub starts 2024-02-10T07:44:14.184 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:14.185 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[29816]: 4.18 deep-scrub ok 2024-02-10T07:44:14.185 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1459569768' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:44:14.185 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:44:14.185 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[40743]: 9.f deep-scrub starts 2024-02-10T07:44:14.185 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[40743]: 5.9 scrub starts 2024-02-10T07:44:14.185 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[40743]: 5.9 scrub ok 2024-02-10T07:44:14.185 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[40743]: 4.18 deep-scrub starts 2024-02-10T07:44:14.185 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:14.185 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[40743]: 4.18 deep-scrub ok 2024-02-10T07:44:14.185 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1459569768' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:44:14.185 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:13 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:44:14.231 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:14.274 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:14.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:14.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:14.376 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:13 smithi170 ceph-mon[33394]: 9.f deep-scrub starts 2024-02-10T07:44:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:13 smithi170 ceph-mon[33394]: 5.9 scrub starts 2024-02-10T07:44:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:13 smithi170 ceph-mon[33394]: 5.9 scrub ok 2024-02-10T07:44:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:13 smithi170 ceph-mon[33394]: 4.18 deep-scrub starts 2024-02-10T07:44:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:13 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:13 smithi170 ceph-mon[33394]: 4.18 deep-scrub ok 2024-02-10T07:44:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:13 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1459569768' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:44:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:13 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:44:14.415 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:14.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:14.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:14.516 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:14.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:14.562 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:14.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:14.663 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:14.699 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:14.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:14.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:14.801 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:14.824 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:14.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:14.925 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:14.960 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:14.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:14.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:15.062 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:15.085 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:15.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:15.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:15.217 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:15.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:15.319 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:15.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:14 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:44:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:14 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:44:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:14 smithi008 ceph-mon[29816]: osdmap e288: 8 total, 8 up, 8 in 2024-02-10T07:44:15.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:14 smithi008 ceph-mon[29816]: pgmap v560: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:44:15.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:14 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:44:15.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:14 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:44:15.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:14 smithi008 ceph-mon[40743]: osdmap e288: 8 total, 8 up, 8 in 2024-02-10T07:44:15.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:14 smithi008 ceph-mon[40743]: pgmap v560: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:44:15.348 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:15.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:15.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:14 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:44:15.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:14 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:44:15.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:14 smithi170 ceph-mon[33394]: osdmap e288: 8 total, 8 up, 8 in 2024-02-10T07:44:15.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:14 smithi170 ceph-mon[33394]: pgmap v560: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:44:15.449 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:15.484 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:15.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:15.585 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:15.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:15.614 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:15.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:15.715 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:15.750 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:15.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:15.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:15.851 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:15.879 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:15.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:15.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:15.981 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:16.014 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:16.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:16.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:16.115 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:16.156 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:16.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:16.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:16.257 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:16.295 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:16.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:16.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:16.397 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:16.428 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:16.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:16.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:16.529 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:16.558 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:16.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:16.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:16.660 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:16.699 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:16.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:16.800 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:16.834 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:16.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:16.936 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:16.966 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:16.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:16.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:17.067 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:17.098 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:17.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:17.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:17.200 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:17.236 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:17.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:17.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:17.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:17 smithi008 ceph-mon[29816]: pgmap v561: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 929 op/s 2024-02-10T07:44:17.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:17 smithi008 ceph-mon[40743]: pgmap v561: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 929 op/s 2024-02-10T07:44:17.337 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:17.364 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:17.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:17.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:17.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:17 smithi170 ceph-mon[33394]: pgmap v561: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 929 op/s 2024-02-10T07:44:17.466 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:17.497 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:17.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:17.599 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:17.628 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:17.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:17.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:17.730 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:17.756 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:17.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:17.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:17.858 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:17.887 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:17.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:17.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:17.989 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:18.021 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:18.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:18.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:18.123 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:18.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:18.161 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:18.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:18.261 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:18.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:18.297 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:18.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:18.398 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:18.438 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:18.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:18.539 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:18.568 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:18.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:18.670 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:18.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:18.700 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:18.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:18.801 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:18.848 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.6 to 1 (10000) 2024-02-10T07:44:18.849 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:18.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:18.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:18.951 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:18.984 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:18.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:18.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:44:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:44:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:18 smithi008 ceph-mon[29816]: pgmap v562: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 949 op/s 2024-02-10T07:44:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:18 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3643151720' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:44:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:18 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:44:19.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:44:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:44:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:44:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:44:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:44:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:18 smithi008 ceph-mon[40743]: pgmap v562: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 949 op/s 2024-02-10T07:44:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:18 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3643151720' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:44:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:18 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:44:19.086 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:19.119 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:19.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:19.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:44:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:44:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:18 smithi170 ceph-mon[33394]: pgmap v562: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 949 op/s 2024-02-10T07:44:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:18 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3643151720' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:44:19.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:18 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:44:19.220 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:19.259 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:19.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:19.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:19.361 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:19.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:19.404 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:19.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:19.505 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:19.538 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:19.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:19.639 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:19.668 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:19.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:19.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:19.770 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:19.799 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:19.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:19.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:19.902 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:19.941 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:19.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:19.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:19.965 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:44:19.965 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-02-10T07:44:19.965 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-02-10T07:44:19.965 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:44:19.965 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.965 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127184 7.9 7:930e04c1:::benchmark_data_smithi008_82046_object12046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-10T07:44:19.965 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:55.989856+0000", 2024-02-10T07:44:19.965 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 503.970830541, 2024-02-10T07:44:19.965 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.25485934199999999, 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127184 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989856+0000", 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.966 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989856+0000", 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989857+0000", 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3799999999999996e-07 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989861+0000", 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.703e-06 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.967 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989870+0000", 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0869999999999996e-06 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:55.989877+0000", 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.153e-06 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.015176+0000", 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.025298960999999998 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.968 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:19.969 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.015291+0000", 2024-02-10T07:44:19.969 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000115123 2024-02-10T07:44:19.969 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.969 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.969 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:44:19.969 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.015667+0000", 2024-02-10T07:44:19.969 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00037586599999999998 2024-02-10T07:44:19.969 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.969 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.969 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:44:19.969 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.020630+0000", 2024-02-10T07:44:19.969 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0049631440000000001 2024-02-10T07:44:19.969 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.969 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.020900+0000", 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026987000000000002 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244581+0000", 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.22368058900000001 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244640+0000", 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9453999999999997e-05 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.970 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.971 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:19.971 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244716+0000", 2024-02-10T07:44:19.971 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5654000000000004e-05 2024-02-10T07:44:19.971 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:19.971 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:19.971 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:19.971 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.971 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.971 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127211 7.9 7:97623005:::benchmark_data_smithi008_82046_object12073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-10T07:44:19.971 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.027194+0000", 2024-02-10T07:44:19.971 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 503.933492446, 2024-02-10T07:44:19.971 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.217660456, 2024-02-10T07:44:19.971 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:19.971 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:19.972 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:44:19.972 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:44:19.972 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:44:19.972 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127211 2024-02-10T07:44:19.972 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.972 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:19.972 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.972 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:19.972 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027194+0000", 2024-02-10T07:44:19.972 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:19.972 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.972 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.972 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:19.972 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027194+0000", 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027195+0000", 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7499999999999998e-07 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027200+0000", 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6829999999999999e-06 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:19.973 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027212+0000", 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1522000000000001e-05 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.027219+0000", 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6979999999999998e-06 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.031348+0000", 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0041287870000000001 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.031490+0000", 2024-02-10T07:44:19.974 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014215199999999999 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.031908+0000", 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00041737399999999998 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.038097+0000", 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0061895500000000003 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.038176+0000", 2024-02-10T07:44:19.975 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8963999999999998e-05 2024-02-10T07:44:19.976 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.976 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.976 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.976 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244761+0000", 2024-02-10T07:44:19.976 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.206584868 2024-02-10T07:44:19.976 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.976 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.976 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:19.976 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244806+0000", 2024-02-10T07:44:19.976 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5368000000000003e-05 2024-02-10T07:44:19.976 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.976 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.976 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:19.976 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.244855+0000", 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8514999999999999e-05 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127243 7.14 7:2d50c885:::benchmark_data_smithi008_82046_object12105:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.076957+0000", 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 503.88372933599999, 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.074139102, 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:44:19.977 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127243 2024-02-10T07:44:19.978 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.978 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:19.978 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.978 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:19.978 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076957+0000", 2024-02-10T07:44:19.978 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:19.978 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.978 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.978 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:19.978 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076957+0000", 2024-02-10T07:44:19.978 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:19.978 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.978 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.978 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076958+0000", 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9300000000000006e-07 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076963+0000", 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2520000000000001e-06 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076974+0000", 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0808999999999999e-05 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.979 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.076981+0000", 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.391e-06 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.077036+0000", 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5529999999999999e-05 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.077172+0000", 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013559100000000001 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.980 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,3,4,5", 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.077478+0000", 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00030572100000000001 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.080684+0000", 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032062390000000001 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.081086+0000", 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00040183900000000003 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.081152+0000", 2024-02-10T07:44:19.981 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6185000000000004e-05 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.083905+0000", 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027525420000000002 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.090838+0000", 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0069333470000000003 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.233892+0000", 2024-02-10T07:44:19.982 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.143053929 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.233903+0000", 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1412e-05 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.151012+0000", 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.91710869900000003 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.151097+0000", 2024-02-10T07:44:19.983 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4623000000000006e-05 2024-02-10T07:44:19.984 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:19.984 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:19.984 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:19.984 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.984 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.984 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127271 7.17 7:ee10418c:::benchmark_data_smithi008_82046_object12133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:44:19.984 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.283167+0000", 2024-02-10T07:44:19.984 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 503.67751979899998, 2024-02-10T07:44:19.984 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.87381412300000005, 2024-02-10T07:44:19.984 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:19.984 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:19.984 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:44:19.984 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:44:19.984 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127271 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283167+0000", 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283167+0000", 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283168+0000", 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6199999999999996e-07 2024-02-10T07:44:19.985 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283173+0000", 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6589999999999998e-06 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283184+0000", 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1175999999999999e-05 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283192+0000", 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1049999999999995e-06 2024-02-10T07:44:19.986 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283249+0000", 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7491000000000002e-05 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.283259+0000", 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2790000000000004e-06 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.145677+0000", 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.86241815099999997 2024-02-10T07:44:19.987 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.145793+0000", 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011595 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,3,6,7", 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.146056+0000", 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026355900000000002 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.147787+0000", 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0017307329999999999 2024-02-10T07:44:19.988 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.155806+0000", 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0080188640000000005 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.155945+0000", 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013933700000000001 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156537+0000", 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00059228199999999996 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.989 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.990 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156801+0000", 2024-02-10T07:44:19.990 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026387999999999997 2024-02-10T07:44:19.990 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.990 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.990 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:19.990 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156860+0000", 2024-02-10T07:44:19.990 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8900999999999997e-05 2024-02-10T07:44:19.991 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.991 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.991 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:19.991 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156981+0000", 2024-02-10T07:44:19.991 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000120894 2024-02-10T07:44:19.991 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:19.991 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:19.991 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:19.991 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.991 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.991 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:127281 7.17 7:ec58e9f8:::benchmark_data_smithi008_82046_object12143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:44:19.991 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.333207+0000", 2024-02-10T07:44:19.991 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 503.62747949499999, 2024-02-10T07:44:19.991 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.82508648100000004, 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127281 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333207+0000", 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.992 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333207+0000", 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333209+0000", 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.415e-06 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333213+0000", 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2230000000000001e-06 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:19.993 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333224+0000", 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1483e-05 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333232+0000", 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.4970000000000004e-06 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333278+0000", 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6362000000000001e-05 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:44:19.994 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.333287+0000", 2024-02-10T07:44:19.995 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6230000000000003e-06 2024-02-10T07:44:19.995 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.995 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.995 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:19.995 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.147857+0000", 2024-02-10T07:44:19.995 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.81456973200000005 2024-02-10T07:44:19.995 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.995 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.995 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:19.995 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.147984+0000", 2024-02-10T07:44:19.995 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012703200000000001 2024-02-10T07:44:19.995 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.995 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.996 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,3,6,7", 2024-02-10T07:44:19.996 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.148252+0000", 2024-02-10T07:44:19.996 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000268278 2024-02-10T07:44:19.996 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.996 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.996 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.996 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.155857+0000", 2024-02-10T07:44:19.996 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0076047550000000004 2024-02-10T07:44:19.996 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.996 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.996 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.996 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.155901+0000", 2024-02-10T07:44:19.996 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4283000000000003e-05 2024-02-10T07:44:19.996 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.997 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.997 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156520+0000", 2024-02-10T07:44:19.997 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00061886199999999997 2024-02-10T07:44:19.997 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.997 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:44:19.997 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.156552+0000", 2024-02-10T07:44:19.997 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1786999999999999e-05 2024-02-10T07:44:19.997 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.997 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:19.997 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.158034+0000", 2024-02-10T07:44:19.997 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001482141 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.158140+0000", 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010587 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.158294+0000", 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015413800000000001 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.998 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24356.0:1373 5.3 5:c609908c:::notify.5:head [watch ping cookie 93827140438272] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.411579+0000", 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 503.54910770399999, 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.71068318600000002, 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24356", 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1933562162", 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 1373 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:19.999 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411579+0000", 2024-02-10T07:44:20.000 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.000 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.000 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.000 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.000 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411579+0000", 2024-02-10T07:44:20.000 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.000 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.000 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.000 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.000 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411581+0000", 2024-02-10T07:44:20.000 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4839999999999999e-06 2024-02-10T07:44:20.000 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.000 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.000 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411585+0000", 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8799999999999999e-06 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411598+0000", 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2388999999999999e-05 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411606+0000", 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4659999999999999e-06 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.001 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411704+0000", 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8095000000000002e-05 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.411714+0000", 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0044000000000001e-05 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.115532+0000", 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.70381718199999999 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.002 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.115689+0000", 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015699900000000001 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.122262+0000", 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0065736470000000002 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24356.0:1383 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 93827140438272 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:35:56.412062+0000", 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 503.54862467100003, 2024-02-10T07:44:20.003 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.725469484, 2024-02-10T07:44:20.004 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:20.004 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:44:20.004 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:44:20.004 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24356", 2024-02-10T07:44:20.004 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/1933562162", 2024-02-10T07:44:20.004 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 1383 2024-02-10T07:44:20.004 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.004 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:20.004 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.004 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:20.004 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412062+0000", 2024-02-10T07:44:20.004 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.004 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.004 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412062+0000", 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412063+0000", 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7700000000000004e-07 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412067+0000", 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9330000000000003e-06 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.005 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412081+0000", 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3991e-05 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.412088+0000", 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6459999999999993e-06 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.498465+0000", 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.086376877000000005 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:44:20.006 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:56.498476+0000", 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0975000000000001e-05 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.122295+0000", 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.62381883100000002 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.122389+0000", 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3941000000000001e-05 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.007 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:35:57.137532+0000", 2024-02-10T07:44:20.008 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015142513 2024-02-10T07:44:20.008 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.008 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.008 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.008 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.008 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.008 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:131746 7.9 7:97040099:::benchmark_data_smithi008_82046_object16608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:44:20.008 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:05.798493+0000", 2024-02-10T07:44:20.008 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 494.162193398, 2024-02-10T07:44:20.008 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.26161331199999999, 2024-02-10T07:44:20.008 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:20.008 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:20.008 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:44:20.008 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:44:20.009 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:44:20.009 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 131746 2024-02-10T07:44:20.009 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.009 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:20.009 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.009 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:20.009 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798493+0000", 2024-02-10T07:44:20.009 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.009 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.009 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.009 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.009 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798493+0000", 2024-02-10T07:44:20.009 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.009 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798495+0000", 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2240000000000001e-06 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798499+0000", 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2359999999999998e-06 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798518+0000", 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9519000000000001e-05 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.010 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.798530+0000", 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.184e-05 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.830695+0000", 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.032164618999999998 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.830907+0000", 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00021249600000000001 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.011 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.831383+0000", 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000476016 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.834488+0000", 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0031048759999999999 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.834673+0000", 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00018433600000000001 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.012 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.059995+0000", 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.22532233199999999 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060046+0000", 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1076999999999999e-05 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060107+0000", 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0741e-05 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.013 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:131775 7.19 7:9fe66b4c:::benchmark_data_smithi008_82046_object16637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:05.844733+0000", 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 494.115953671, 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.216159669, 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 131775 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.014 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844733+0000", 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844733+0000", 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844734+0000", 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3340000000000001e-06 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.015 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.016 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844739+0000", 2024-02-10T07:44:20.016 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.065e-06 2024-02-10T07:44:20.016 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.016 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.016 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.016 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844748+0000", 2024-02-10T07:44:20.016 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0960000000000007e-06 2024-02-10T07:44:20.016 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.016 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.016 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.016 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:19 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:44:20.017 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:19 smithi008 ceph-mon[40743]: osdmap e289: 8 total, 8 up, 8 in 2024-02-10T07:44:20.017 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:19 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:44:20.017 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:19 smithi008 ceph-mon[29816]: osdmap e289: 8 total, 8 up, 8 in 2024-02-10T07:44:20.017 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844754+0000", 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6900000000000003e-06 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.844936+0000", 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00018217400000000001 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.845059+0000", 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000122815 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-02-10T07:44:20.018 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.845348+0000", 2024-02-10T07:44:20.019 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00028826999999999998 2024-02-10T07:44:20.019 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.019 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.019 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.019 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.856837+0000", 2024-02-10T07:44:20.019 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011489256999999999 2024-02-10T07:44:20.019 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.019 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.019 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.019 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.856879+0000", 2024-02-10T07:44:20.019 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2268000000000003e-05 2024-02-10T07:44:20.019 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.019 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.019 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.020 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.856919+0000", 2024-02-10T07:44:20.020 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9681000000000002e-05 2024-02-10T07:44:20.020 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.020 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.020 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.020 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.856958+0000", 2024-02-10T07:44:20.020 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9623999999999999e-05 2024-02-10T07:44:20.020 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.020 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.020 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:44:20.020 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:05.860314+0000", 2024-02-10T07:44:20.020 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033553839999999999 2024-02-10T07:44:20.020 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.020 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060424+0000", 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.20010988099999999 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060484+0000", 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0136000000000002e-05 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060893+0000", 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00040899399999999998 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.021 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:131874 7.9 7:90ce5fa2:::benchmark_data_smithi008_82046_object16736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:06.060485+0000", 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 493.90020194700003, 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.22169967700000001, 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 131874 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.022 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060485+0000", 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060485+0000", 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060486+0000", 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7999999999999993e-07 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.023 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.024 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060491+0000", 2024-02-10T07:44:20.024 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.814e-06 2024-02-10T07:44:20.024 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.024 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.024 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.024 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060503+0000", 2024-02-10T07:44:20.024 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1915e-05 2024-02-10T07:44:20.024 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.024 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.024 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.024 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.060511+0000", 2024-02-10T07:44:20.024 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6479999999999992e-06 2024-02-10T07:44:20.024 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.024 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.082347+0000", 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.021835391999999999 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.082588+0000", 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00024096600000000001 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.083045+0000", 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00045718000000000001 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.025 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.026 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.026 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.091368+0000", 2024-02-10T07:44:20.026 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0083230149999999996 2024-02-10T07:44:20.026 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.026 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.026 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.026 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.091406+0000", 2024-02-10T07:44:20.026 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8444e-05 2024-02-10T07:44:20.026 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.026 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.026 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:44:20.026 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.282075+0000", 2024-02-10T07:44:20.026 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.19066877500000001 2024-02-10T07:44:20.026 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.282128+0000", 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3090000000000002e-05 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:06.282185+0000", 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6458000000000001e-05 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:132742 7.19 7:991f4e26:::benchmark_data_smithi008_82046_object17604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:44:20.027 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:07.446202+0000", 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 492.51448508300001, 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.25343312400000001, 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 132742 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446202+0000", 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.028 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446202+0000", 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446203+0000", 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.159e-06 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446207+0000", 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.738e-06 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.029 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446217+0000", 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0518e-05 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446225+0000", 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3760000000000002e-06 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446280+0000", 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5883e-05 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.030 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446423+0000", 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014267300000000001 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446714+0000", 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00029046200000000001 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450612+0000", 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0038986860000000002 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.031 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450626+0000", 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4151999999999999e-05 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450672+0000", 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6119999999999999e-05 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450720+0000", 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7803999999999998e-05 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.032 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450765+0000", 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4594e-05 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699383+0000", 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24861809200000001 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699507+0000", 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000124156 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.033 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699635+0000", 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000127711 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:132743 7.19 7:9cfb864f:::benchmark_data_smithi008_82046_object17605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:07.446450+0000", 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 492.51423677100001, 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.25339128999999999, 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:44:20.034 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 132743 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446450+0000", 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446450+0000", 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.035 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446451+0000", 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8599999999999997e-07 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446455+0000", 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.5839999999999999e-06 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446466+0000", 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1242e-05 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.036 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.446474+0000", 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.2309999999999996e-06 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.448986+0000", 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0025125059999999999 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.449131+0000", 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014447200000000001 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,4,6,7", 2024-02-10T07:44:20.037 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.449392+0000", 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00026119499999999998 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450809+0000", 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0014168469999999999 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450905+0000", 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6237999999999996e-05 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.450997+0000", 2024-02-10T07:44:20.038 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1774999999999995e-05 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.452168+0000", 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00117117 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.452212+0000", 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3548999999999999e-05 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699673+0000", 2024-02-10T07:44:20.039 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24746167399999999 2024-02-10T07:44:20.040 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.040 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.040 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:20.040 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699723+0000", 2024-02-10T07:44:20.040 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9421e-05 2024-02-10T07:44:20.040 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.040 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.040 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.040 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:07.699841+0000", 2024-02-10T07:44:20.040 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011849999999999999 2024-02-10T07:44:20.040 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.040 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.040 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.040 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.041 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.041 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:154815 7.14 7:28f3695e:::benchmark_data_smithi008_82046_object39677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:44:20.041 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:36:32.856040+0000", 2024-02-10T07:44:20.041 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 467.104647156, 2024-02-10T07:44:20.041 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.96816402099999999, 2024-02-10T07:44:20.041 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:20.041 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:20.041 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:44:20.041 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:44:20.041 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:44:20.041 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 154815 2024-02-10T07:44:20.041 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.041 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:20.041 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856040+0000", 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856040+0000", 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856042+0000", 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8700000000000001e-06 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.042 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.043 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:20.044 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856046+0000", 2024-02-10T07:44:20.044 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9649999999999999e-06 2024-02-10T07:44:20.044 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.044 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.044 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.044 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856063+0000", 2024-02-10T07:44:20.044 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6838999999999999e-05 2024-02-10T07:44:20.044 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.044 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.044 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.044 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856083+0000", 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9191e-05 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856152+0000", 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9622999999999994e-05 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:32.856165+0000", 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2604e-05 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.820015+0000", 2024-02-10T07:44:20.045 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.963850031 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.820144+0000", 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012971799999999999 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 1,2,3,4,5", 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.820489+0000", 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00034403500000000002 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.822408+0000", 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001919624 2024-02-10T07:44:20.046 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.822852+0000", 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00044381899999999997 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.822915+0000", 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.279e-05 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.822955+0000", 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0460999999999998e-05 2024-02-10T07:44:20.047 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.823010+0000", 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4537999999999999e-05 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.823994+0000", 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00098379999999999995 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.824060+0000", 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6548e-05 2024-02-10T07:44:20.048 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:36:33.824204+0000", 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014356500000000001 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:212345 7.f 7:f80fa090:::benchmark_data_smithi008_82046_object97207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.578926+0000", 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 402.38176044300002, 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24282195200000001, 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:20.049 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 212345 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578926+0000", 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578926+0000", 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.050 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578928+0000", 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.486e-06 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578932+0000", 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2699999999999998e-06 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578945+0000", 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2703e-05 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.051 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578954+0000", 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8710000000000003e-06 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.579336+0000", 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00038255500000000003 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.579461+0000", 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000124621 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:44:20.052 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.579564+0000", 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000103317 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.582702+0000", 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0031373400000000002 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583326+0000", 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00062478000000000002 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821642+0000", 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.238315844 2024-02-10T07:44:20.053 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821699+0000", 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6635000000000002e-05 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821748+0000", 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9530000000000002e-05 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212347 7.12 e187/185 7:4b6dc533:::benchmark_data_smithi008_82046_object97209:head v 187'10763, mlcod=187'10761)", 2024-02-10T07:44:20.054 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.583166+0000", 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 402.37752073899998, 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24110515900000001, 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583166+0000", 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583166+0000", 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.055 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.056 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.056 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583167+0000", 2024-02-10T07:44:20.056 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1299999999999999e-07 2024-02-10T07:44:20.056 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.056 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.056 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.056 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583172+0000", 2024-02-10T07:44:20.056 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6740000000000002e-06 2024-02-10T07:44:20.056 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.056 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.056 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.056 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583182+0000", 2024-02-10T07:44:20.056 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8130000000000007e-06 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583189+0000", 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9140000000000002e-06 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583624+0000", 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00043515399999999999 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583646+0000", 2024-02-10T07:44:20.057 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1474000000000001e-05 2024-02-10T07:44:20.058 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.058 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.058 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:20.058 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.824236+0000", 2024-02-10T07:44:20.058 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24059001699999999 2024-02-10T07:44:20.058 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.058 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.058 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.058 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.824271+0000", 2024-02-10T07:44:20.058 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.54e-05 2024-02-10T07:44:20.058 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.058 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.058 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.058 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212349 7.e e187/166 7:7df8d3ee:::benchmark_data_smithi008_82046_object97211:head v 187'13215, mlcod=187'13213)", 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.583663+0000", 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 402.377023468, 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23797284099999999, 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583663+0000", 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.059 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.060 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583663+0000", 2024-02-10T07:44:20.060 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.060 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.060 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.060 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.060 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583664+0000", 2024-02-10T07:44:20.060 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3900000000000004e-07 2024-02-10T07:44:20.060 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.060 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.060 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.060 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583670+0000", 2024-02-10T07:44:20.060 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7790000000000001e-06 2024-02-10T07:44:20.060 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.060 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.061 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.061 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583680+0000", 2024-02-10T07:44:20.061 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7419999999999992e-06 2024-02-10T07:44:20.061 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.061 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.061 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.061 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583686+0000", 2024-02-10T07:44:20.061 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5740000000000001e-06 2024-02-10T07:44:20.061 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.061 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.061 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.061 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.584048+0000", 2024-02-10T07:44:20.061 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00036165999999999998 2024-02-10T07:44:20.061 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.584076+0000", 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8121000000000001e-05 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821584+0000", 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.237508318 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821636+0000", 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1907999999999998e-05 2024-02-10T07:44:20.062 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop_reply(client.24740.0:212345 7.f e187/98 ondisk, result = 0)", 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.585629+0000", 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 402.37505813199999, 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23612559999999999, 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585629+0000", 2024-02-10T07:44:20.063 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585629+0000", 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585630+0000", 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0999999999999997e-07 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585635+0000", 2024-02-10T07:44:20.064 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1050000000000003e-06 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585645+0000", 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0484e-05 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585652+0000", 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2570000000000001e-06 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821625+0000", 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23597299399999999 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.065 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.066 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:20.066 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821635+0000", 2024-02-10T07:44:20.066 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9240000000000002e-06 2024-02-10T07:44:20.066 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.066 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.066 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.066 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821754+0000", 2024-02-10T07:44:20.066 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011892600000000001 2024-02-10T07:44:20.066 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.066 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.066 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.066 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.066 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.066 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDPGPushReply(7.14 187/156 [PushReplyOp(7:2f8c8aed:::benchmark_data_smithi008_82046_object99852:head)])", 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 402.37249668999999, 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23338334399999999, 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.067 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2499999999999999e-07 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588193+0000", 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.881e-06 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588200+0000", 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.703e-06 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.068 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.069 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.069 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588206+0000", 2024-02-10T07:44:20.069 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4869999999999999e-06 2024-02-10T07:44:20.069 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.069 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.069 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.069 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588865+0000", 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00065820199999999998 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821573+0000", 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.232708846 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212353 7.12 e187/185 7:4af281cd:::benchmark_data_smithi008_82046_object97215:head v 187'10764, mlcod=187'10761)", 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.589498+0000", 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 402.37118895399999, 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23719131800000001, 2024-02-10T07:44:20.070 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589498+0000", 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589498+0000", 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589499+0000", 2024-02-10T07:44:20.071 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8500000000000001e-07 2024-02-10T07:44:20.072 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.072 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.072 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.072 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589505+0000", 2024-02-10T07:44:20.072 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4060000000000002e-06 2024-02-10T07:44:20.072 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.072 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.072 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.072 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821581+0000", 2024-02-10T07:44:20.072 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23207629699999999 2024-02-10T07:44:20.072 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.072 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.072 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.072 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821593+0000", 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1387e-05 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821991+0000", 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00039868400000000001 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.822013+0000", 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1191000000000001e-05 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.826650+0000", 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046371650000000004 2024-02-10T07:44:20.073 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.826689+0000", 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9502999999999998e-05 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212363 7.5 e187/135 7:a6287109:::benchmark_data_smithi008_82046_object97225:head v 187'10252, mlcod=187'10250)", 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.594086+0000", 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 402.36660118200001, 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23397765000000001, 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:44:20.074 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594086+0000", 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594086+0000", 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594087+0000", 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.595e-06 2024-02-10T07:44:20.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594097+0000", 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4560000000000006e-06 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821570+0000", 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.22747376999999999 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821582+0000", 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1093e-05 2024-02-10T07:44:20.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:44:20.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821684+0000", 2024-02-10T07:44:20.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010225699999999999 2024-02-10T07:44:20.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:44:20.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821708+0000", 2024-02-10T07:44:20.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4641e-05 2024-02-10T07:44:20.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:44:20.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.828021+0000", 2024-02-10T07:44:20.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0063127080000000002 2024-02-10T07:44:20.078 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:44:20.078 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:44:20.078 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:44:20.078 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.828063+0000", 2024-02-10T07:44:20.078 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2129999999999998e-05 2024-02-10T07:44:20.078 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.078 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.078 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.078 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:44:20.078 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:44:20.078 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:44:20.106 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:20.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:20.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:19 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:44:20.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:19 smithi170 ceph-mon[33394]: osdmap e289: 8 total, 8 up, 8 in 2024-02-10T07:44:20.208 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:20.248 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:20.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:20.349 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:20.384 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:20.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:20.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:20.485 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:20.522 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:20.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:20.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:20.624 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:20.665 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:20.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:20.767 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:20.807 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:20.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:20.819 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 7 1 2024-02-10T07:44:20.909 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:20.936 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:20.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:20.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:21.038 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:21.070 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:21.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:21.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:21.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:20 smithi008 ceph-mon[29816]: pgmap v564: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-10T07:44:21.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:20 smithi008 ceph-mon[40743]: pgmap v564: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-10T07:44:21.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:20 smithi170 ceph-mon[33394]: pgmap v564: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-10T07:44:21.173 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:21.201 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:21.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:21.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:21.203 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:21.303 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:21.347 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:21.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:21.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:21.418 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:44:21.449 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:21.474 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:21.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:21.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:21.575 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:21.601 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:21.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:21.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:21.703 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:21.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:21.757 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:21.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:21.858 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:21.889 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:21.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:21.991 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:22.021 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:22.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:22.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:22.099 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-10T07:44:22.123 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:22.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:22.225 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:22.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:22.326 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:22.373 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:22.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:22.474 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:22.514 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:22.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:22.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:22.615 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:22.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:22.658 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:22.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:22.759 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:22.796 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:22.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:22.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:22.884 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:22.898 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:22.930 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:22.981 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:22.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:22.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:23.044 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:22 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:44:23.044 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:22 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:44:23.044 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:22 smithi008 ceph-mon[29816]: osdmap e290: 8 total, 8 up, 8 in 2024-02-10T07:44:23.044 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:22 smithi008 ceph-mon[29816]: pgmap v566: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 622 op/s 2024-02-10T07:44:23.044 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:22 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:44:23.044 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:22 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:44:23.045 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:22 smithi008 ceph-mon[40743]: osdmap e290: 8 total, 8 up, 8 in 2024-02-10T07:44:23.045 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:22 smithi008 ceph-mon[40743]: pgmap v566: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 622 op/s 2024-02-10T07:44:23.083 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:23.117 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:23.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:23.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:22 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:44:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:22 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:44:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:22 smithi170 ceph-mon[33394]: osdmap e290: 8 total, 8 up, 8 in 2024-02-10T07:44:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:22 smithi170 ceph-mon[33394]: pgmap v566: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 622 op/s 2024-02-10T07:44:23.219 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:23.251 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:23.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:23.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:23.353 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:23.390 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:23.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:23.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:23.492 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:23.526 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:23.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:23.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:23.628 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:23.668 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:23.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:23.769 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:23.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:23.813 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:23.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:23.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:44:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:44:23.914 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:23.953 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:23.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:24.054 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:24.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:23 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:24.060 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:23 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:24.096 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:24.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:24.198 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:24.234 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:24.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:24.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:24.336 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:24.368 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:24.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:24.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:23 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:24.470 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:24.506 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:24.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:24.608 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:24.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:24.644 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:24.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:24.745 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:24.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:24.787 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:24.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:24.888 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:24.918 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:24.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:24.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:25.020 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:25.057 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:25.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:25.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:25.159 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:25.199 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:25.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:25.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:25.300 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:25.322 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:24 smithi008 ceph-mon[29816]: pgmap v567: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 837 op/s 2024-02-10T07:44:25.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:24 smithi008 ceph-mon[40743]: pgmap v567: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 837 op/s 2024-02-10T07:44:25.330 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:25.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:25.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:24 smithi170 ceph-mon[33394]: pgmap v567: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 837 op/s 2024-02-10T07:44:25.432 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:25.463 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:25.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:25.564 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:25.592 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:25.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:25.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:25.653 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:44:25.693 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:25.694 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:44:25.726 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:44:25.728 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:25.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:25.762 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:44:25.810 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:44:25.831 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:25.858 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:44:25.858 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:25.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:25.899 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:44:25.942 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:44:25.946 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:44:25.946 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:44:25.946 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:44:25.946 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:44:25.947 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:44:25.947 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:44:25.947 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:44:25.947 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:44:25.960 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:25.991 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:25.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:26.093 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:26.127 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:26.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:26.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:26.228 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:26.269 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:26.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:26.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:26.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:25 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3953358259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:44:26.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:25 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3953358259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:44:26.371 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:26.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:25 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3953358259' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:44:26.412 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:26.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:26.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:26.514 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:26.558 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:26.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:26.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:26.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:26.694 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:26.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:26.795 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:26.831 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:26.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:26.932 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:26.970 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:26.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:27.071 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:27.099 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:27.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:27.202 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:27.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:27.239 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:27.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:27.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:26 smithi008 ceph-mon[29816]: pgmap v568: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 755 op/s 2024-02-10T07:44:27.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:26 smithi008 ceph-mon[40743]: pgmap v568: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 755 op/s 2024-02-10T07:44:27.337 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:27.340 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:27.380 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:27.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:26 smithi170 ceph-mon[33394]: pgmap v568: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 755 op/s 2024-02-10T07:44:27.481 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:27.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:27.525 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:27.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:27.626 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:27.666 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:27.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:27.767 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:27.799 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:27.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:27.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:27.903 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:27.930 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:27.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:27.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:27.955 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:44:28.011 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:27 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2555740800' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:44:28.011 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:27 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:44:28.012 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:27 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2555740800' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:44:28.012 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:27 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:44:28.032 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:28.071 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:28.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:28.174 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:28.214 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:28.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:28.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:28.316 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:28.351 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:28.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:28.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:27 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2555740800' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:44:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:27 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:44:28.453 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:28.490 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:28.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:28.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:28.592 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:28.627 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:28.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:28.729 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:28.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:28.764 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:28.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:28.777 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:44:28.865 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:28.896 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:28.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:28.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:28.998 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:29.026 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:29.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:29.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:29.048 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:44:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:44:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:44:29.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:28 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:44:29.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:28 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:44:29.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:28 smithi008 ceph-mon[29816]: osdmap e291: 8 total, 8 up, 8 in 2024-02-10T07:44:29.048 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:28 smithi008 ceph-mon[29816]: pgmap v570: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 700 op/s 2024-02-10T07:44:29.049 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:28 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:44:29.049 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:28 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:44:29.049 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:28 smithi008 ceph-mon[40743]: osdmap e291: 8 total, 8 up, 8 in 2024-02-10T07:44:29.049 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:28 smithi008 ceph-mon[40743]: pgmap v570: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 700 op/s 2024-02-10T07:44:29.127 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:29.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:29.184 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:29.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:29.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:28 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:44:29.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:28 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:44:29.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:28 smithi170 ceph-mon[33394]: osdmap e291: 8 total, 8 up, 8 in 2024-02-10T07:44:29.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:28 smithi170 ceph-mon[33394]: pgmap v570: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 700 op/s 2024-02-10T07:44:29.285 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:29.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:29.322 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:29.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:29.423 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:29.454 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:29.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:29.556 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:29.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:29.602 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:29.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:29.702 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:29.745 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:29.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:29.846 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:29.889 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:29.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:29.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:29.986 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.7 to 1 (10000) 2024-02-10T07:44:29.991 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:30.029 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:30.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:30.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:30.131 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:29 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1455426465' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:44:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:29 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:44:30.167 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:30.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:30.271 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:30.306 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:30.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:30.321 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:29 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1455426465' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:44:30.321 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:29 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:44:30.321 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:29 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1455426465' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:44:30.321 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:29 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:44:30.408 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:30.445 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:30.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:30.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:30.547 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:30.581 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:30.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:30.683 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:30.720 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:30.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:30.822 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:30.852 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:30.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:30.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:30.954 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:30.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:30.991 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:30.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:31.036 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-10T07:44:31.036 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:44:31.049 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:44:31.093 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:31.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:31.129 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:31.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:31.230 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:31.255 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:30 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:44:31.255 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:30 smithi170 ceph-mon[33394]: osdmap e292: 8 total, 8 up, 8 in 2024-02-10T07:44:31.255 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:30 smithi170 ceph-mon[33394]: pgmap v572: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-02-10T07:44:31.260 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:31.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:31.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:31.304 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:30 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:44:31.304 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:30 smithi008 ceph-mon[29816]: osdmap e292: 8 total, 8 up, 8 in 2024-02-10T07:44:31.304 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:30 smithi008 ceph-mon[29816]: pgmap v572: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-02-10T07:44:31.305 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:30 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:44:31.306 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:30 smithi008 ceph-mon[40743]: osdmap e292: 8 total, 8 up, 8 in 2024-02-10T07:44:31.306 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:30 smithi008 ceph-mon[40743]: pgmap v572: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-02-10T07:44:31.362 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:31.390 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:31.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:31.492 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:31.559 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:31.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:31.611 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:31.662 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:31.684 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:44:31.684 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:44:31.691 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:31.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:31.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:31.714 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:31.793 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:31.826 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:31.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:31.928 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:31.972 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:31.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:31.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:32.075 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:32.109 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:32.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:32.210 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:32.255 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:32.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:32.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:32.357 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:32.389 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:32.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:32.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:32.491 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:32.532 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:32.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:32.634 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:32.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:32.670 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:32.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:32.771 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:32.804 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:32.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:32.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:32.858 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-10T07:44:32.905 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:32.949 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:32.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:33.049 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:33.077 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:32 smithi008 ceph-mon[29816]: pgmap v573: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 644 op/s 2024-02-10T07:44:33.078 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:44:33.078 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:32 smithi008 ceph-mon[40743]: pgmap v573: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 644 op/s 2024-02-10T07:44:33.078 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:44:33.092 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:33.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:33.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:33.194 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:33.224 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:33.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:33.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:33.326 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:33.379 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:33.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:32 smithi170 ceph-mon[33394]: pgmap v573: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 644 op/s 2024-02-10T07:44:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:44:33.480 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:33.518 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:33.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:33.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:33.620 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:33.663 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:33.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:33.764 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:33.798 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:33.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:33.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:33.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:44:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:44:33.900 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:33.937 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:33.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:33.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:34.039 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:34.077 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:34.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:34.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:34.178 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:34.217 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:34.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:34.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:34.319 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:34.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:33 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:34.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:33 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:34.354 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:34.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:34.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:33 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:34.455 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:34.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:34.495 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:34.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:34.597 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:34.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:34.658 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:34.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:34.749 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:34.758 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:34.789 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:34.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:34.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:34.891 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:34.933 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:34.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:35.030 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:34 smithi008 ceph-mon[29816]: pgmap v574: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-02-10T07:44:35.030 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:34 smithi008 ceph-mon[40743]: pgmap v574: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-02-10T07:44:35.034 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:35.065 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:35.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:35.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:35.167 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:35.198 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:35.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:35.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:35.300 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:35.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:35.339 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:35.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:35.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:34 smithi170 ceph-mon[33394]: pgmap v574: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-02-10T07:44:35.440 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:35.480 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:35.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:35.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:35.583 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:35.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:35.617 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:35.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:35.718 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:35.754 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:35.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:35.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:35.856 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:35.889 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:35.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:35.991 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:36.035 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:36.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:36.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:36.137 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:36.192 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:36.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:36.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:36.253 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:44:36.295 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:36.303 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:44:36.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:36.338 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:36.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:36.343 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:44:36.398 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:44:36.439 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:36.442 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:44:36.474 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:36.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:36.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:36.485 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:44:36.523 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:44:36.565 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:44:36.570 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:44:36.570 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:44:36.570 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:44:36.570 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:44:36.571 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:44:36.571 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:44:36.571 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:44:36.571 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:44:36.575 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:36.618 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:36.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:36.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:36.719 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:36.753 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:36.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:36.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:36.854 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:36.900 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:36.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:36.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:37.002 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:37.035 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:37.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:37.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:37.138 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:37.175 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:37.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:37.277 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:37.313 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:37.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:37.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:37 smithi008 ceph-mon[29816]: pgmap v575: 148 pgs: 148 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 636 B/s rd, 0 B/s wr, 1.04k op/s 2024-02-10T07:44:37.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:37 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2332004269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:44:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:37 smithi008 ceph-mon[40743]: pgmap v575: 148 pgs: 148 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 636 B/s rd, 0 B/s wr, 1.04k op/s 2024-02-10T07:44:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:37 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2332004269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:44:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:37 smithi170 ceph-mon[33394]: pgmap v575: 148 pgs: 148 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 636 B/s rd, 0 B/s wr, 1.04k op/s 2024-02-10T07:44:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:37 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2332004269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:44:37.415 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:37.456 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:37.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:37.557 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:37.593 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:37.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:37.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:37.694 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:37.726 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:37.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:37.828 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:37.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:37.858 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:37.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:37.959 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:38.012 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:38.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:38.113 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:38.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:38.157 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:38.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:38.258 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:38.295 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:38.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:38.397 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:38.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:38.437 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:38.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:38.538 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:38.574 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:38.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:38.667 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:38.676 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:38.709 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:38.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:38.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:38.811 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:38.847 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:38.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:38.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:38.949 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:38.991 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:38.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:39.035 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:44:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:44:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:44:39.093 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:39.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:39.130 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:39.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:39.230 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:39.262 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:39.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:39.291 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:39 smithi008 ceph-mon[29816]: pgmap v576: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-02-10T07:44:39.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:39 smithi008 ceph-mon[40743]: pgmap v576: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-02-10T07:44:39.364 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:39 smithi170 ceph-mon[33394]: pgmap v576: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-02-10T07:44:39.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:39.401 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:39.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:39.502 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:39.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:39.540 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:39.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:39.641 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:39.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:39.675 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:39.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:39.776 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:39.812 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:39.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:39.913 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:39.949 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:39.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:39.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:40.050 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:40.084 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:40.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:40.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:40.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:40.212 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:40.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:40.263 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:44:40.315 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:40.353 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:40.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:40.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:40.455 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:40.493 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:40.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:40.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:40.595 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:40.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:40.632 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:40.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:40.733 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:40.769 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:40.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:40.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:40.871 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:40.904 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:40.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:41.006 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:41.049 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:41.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:41.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:41.151 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:41.175 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:41.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:41.277 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:41.315 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:41.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:41.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:41.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:41 smithi008 ceph-mon[29816]: pgmap v577: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-10T07:44:41.317 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:41 smithi008 ceph-mon[40743]: pgmap v577: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-10T07:44:41.353 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:41 smithi170 ceph-mon[33394]: pgmap v577: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-10T07:44:41.416 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:41.464 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:41.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:41.566 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:41.603 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:41.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:41.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:41.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:41.739 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:41.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:41.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:41.841 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:41.873 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:41.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:41.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:41.975 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:42.010 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:42.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:42.112 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:42.142 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:42.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:42.243 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:42.275 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:42.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:42.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:42.377 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:42.421 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:42.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:42.523 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:42.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:42.553 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:42.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:42.635 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:44:42.645 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:44:42.654 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:42.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:42.701 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:42.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:42.802 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:42.833 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:42.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:42.935 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:42.966 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:42.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:43.067 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:43.107 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:43.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:43.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:43.208 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:43.212 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:42 smithi008 ceph-mon[29816]: pgmap v578: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-02-10T07:44:43.212 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:42 smithi008 ceph-mon[40743]: pgmap v578: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-02-10T07:44:43.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:43.244 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:43.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:43.345 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:42 smithi170 ceph-mon[33394]: pgmap v578: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-02-10T07:44:43.387 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:43.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:43.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:43.489 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:43.527 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:43.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:43.629 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:43.664 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:43.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:43.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:43.766 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:43.815 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:43.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:43.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:43.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:44:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:44:43.916 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:43.941 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:43.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:43.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:43.956 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:44:44.044 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:44.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:44.080 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:44.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:44.164 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:43 smithi008 ceph-mon[29816]: from='client.25147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:44:44.165 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:43 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/386370410' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:44:44.165 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:43 smithi008 ceph-mon[40743]: from='client.25147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:44:44.165 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:43 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/386370410' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:44:44.181 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:44.220 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:44.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:44.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:44.322 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:44.357 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:44.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:44.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:44.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:43 smithi170 ceph-mon[33394]: from='client.25147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:44:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:43 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/386370410' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:44:44.459 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:44.490 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:44.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:44.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:44.592 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:44.630 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:44.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:44.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:44.732 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:44.767 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:44.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:44.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:44.869 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:44.906 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:44.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:44.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:45.008 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:45.032 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:45.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:45.133 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:45.166 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:45.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:45.268 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:45.298 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:45.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:45.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:45.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:44 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:45.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:44 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:44:45.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:44 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/386370410' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:44:45.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:44 smithi008 ceph-mon[29816]: osdmap e293: 8 total, 8 up, 8 in 2024-02-10T07:44:45.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:44 smithi008 ceph-mon[29816]: pgmap v580: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-02-10T07:44:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:44 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:45.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:44 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:44:45.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:44 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/386370410' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:44:45.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:44 smithi008 ceph-mon[40743]: osdmap e293: 8 total, 8 up, 8 in 2024-02-10T07:44:45.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:44 smithi008 ceph-mon[40743]: pgmap v580: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-02-10T07:44:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:44 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:44 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:44:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:44 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/386370410' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:44:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:44 smithi170 ceph-mon[33394]: osdmap e293: 8 total, 8 up, 8 in 2024-02-10T07:44:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:44 smithi170 ceph-mon[33394]: pgmap v580: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-02-10T07:44:45.400 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:45.433 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:45.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:45.491 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":578,"stamp":"2024-02-10T07:44:41.985183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5425535348,"num_objects":82977,"num_object_clones":0,"num_object_copies":248931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82977,"num_whiteouts":0,"num_read":2119,"num_read_kb":1965,"num_write":174472,"num_write_kb":8205598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":1772204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84272,"ondisk_log_size":84272,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19469640,"kb_used_data":16072456,"kb_used_omap":15,"kb_used_meta":3396976,"kb_avail":730491576,"statfs":{"total":767960285184,"available":748023373824,"internally_reserved":0,"allocated":16458194944,"data_stored":16455781708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15602,"internal_metadata":3478504206},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,10,4,7,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-801570816,"num_objects":-12231,"num_object_clones":0,"num_object_copies":-36693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12231,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12231,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003429"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":503,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.988398+0000","last_change":"2024-02-10T07:43:40.436091+0000","last_active":"2024-02-10T07:44:29.988398+0000","last_peered":"2024-02-10T07:44:29.988398+0000","last_clean":"2024-02-10T07:44:29.988398+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:44:29.988398+0000","last_undegraded":"2024-02-10T07:44:29.988398+0000","last_fullsized":"2024-02-10T07:44:29.988398+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:40.435939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:40.435939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:43.353937+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":513,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.981392+0000","last_change":"2024-02-10T07:43:54.220526+0000","last_active":"2024-02-10T07:44:29.981392+0000","last_peered":"2024-02-10T07:44:29.981392+0000","last_clean":"2024-02-10T07:44:29.981392+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:44:29.981392+0000","last_undegraded":"2024-02-10T07:44:29.981392+0000","last_fullsized":"2024-02-10T07:44:29.981392+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:54.220406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:54.220406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:20.964933+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":521,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.981359+0000","last_change":"2024-02-10T07:42:21.808051+0000","last_active":"2024-02-10T07:44:29.981359+0000","last_peered":"2024-02-10T07:44:29.981359+0000","last_clean":"2024-02-10T07:44:29.981359+0000","last_became_active":"2024-02-10T07:36:33.760016+0000","last_became_peered":"2024-02-10T07:36:33.760016+0000","last_unstale":"2024-02-10T07:44:29.981359+0000","last_undegraded":"2024-02-10T07:44:29.981359+0000","last_fullsized":"2024-02-10T07:44:29.981359+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:42:21.807909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:42:21.807909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:34.341894+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"287'19","reported_seq":555,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.987528+0000","last_change":"2024-02-10T07:44:13.403141+0000","last_active":"2024-02-10T07:44:29.987528+0000","last_peered":"2024-02-10T07:44:29.987528+0000","last_clean":"2024-02-10T07:44:29.987528+0000","last_became_active":"2024-02-10T07:38:34.790379+0000","last_became_peered":"2024-02-10T07:38:34.790379+0000","last_unstale":"2024-02-10T07:44:29.987528+0000","last_undegraded":"2024-02-10T07:44:29.987528+0000","last_fullsized":"2024-02-10T07:44:29.987528+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'19","last_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_deep_scrub":"287'19","last_deep_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_clean_scrub_stamp":"2024-02-10T07:44:13.402994+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:31.292454+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":494,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.970108+0000","last_change":"2024-02-10T07:43:48.894294+0000","last_active":"2024-02-10T07:44:29.970108+0000","last_peered":"2024-02-10T07:44:29.970108+0000","last_clean":"2024-02-10T07:44:29.970108+0000","last_became_active":"2024-02-10T07:31:57.300056+0000","last_became_peered":"2024-02-10T07:31:57.300056+0000","last_unstale":"2024-02-10T07:44:29.970108+0000","last_undegraded":"2024-02-10T07:44:29.970108+0000","last_fullsized":"2024-02-10T07:44:29.970108+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:48.894116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:48.894116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:05.267308+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":506,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.987442+0000","last_change":"2024-02-10T07:43:51.422878+0000","last_active":"2024-02-10T07:44:29.987442+0000","last_peered":"2024-02-10T07:44:29.987442+0000","last_clean":"2024-02-10T07:44:29.987442+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:44:29.987442+0000","last_undegraded":"2024-02-10T07:44:29.987442+0000","last_fullsized":"2024-02-10T07:44:29.987442+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:51.422763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:51.422763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:16.973118+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":499,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.969892+0000","last_change":"2024-02-10T07:42:16.139633+0000","last_active":"2024-02-10T07:44:29.969892+0000","last_peered":"2024-02-10T07:44:29.969892+0000","last_clean":"2024-02-10T07:44:29.969892+0000","last_became_active":"2024-02-10T07:31:51.323331+0000","last_became_peered":"2024-02-10T07:31:51.323331+0000","last_unstale":"2024-02-10T07:44:29.969892+0000","last_undegraded":"2024-02-10T07:44:29.969892+0000","last_fullsized":"2024-02-10T07:44:29.969892+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:16.139516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:42:16.139516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"250'27","reported_seq":545,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.970343+0000","last_change":"2024-02-10T07:43:17.985386+0000","last_active":"2024-02-10T07:44:29.970343+0000","last_peered":"2024-02-10T07:44:29.970343+0000","last_clean":"2024-02-10T07:44:29.970343+0000","last_became_active":"2024-02-10T07:31:53.286957+0000","last_became_peered":"2024-02-10T07:31:53.286957+0000","last_unstale":"2024-02-10T07:44:29.970343+0000","last_undegraded":"2024-02-10T07:44:29.970343+0000","last_fullsized":"2024-02-10T07:44:29.970343+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:43:17.984918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:17.984918+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":493,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.982991+0000","last_change":"2024-02-10T07:42:16.835823+0000","last_active":"2024-02-10T07:44:29.982991+0000","last_peered":"2024-02-10T07:44:29.982991+0000","last_clean":"2024-02-10T07:44:29.982991+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:44:29.982991+0000","last_undegraded":"2024-02-10T07:44:29.982991+0000","last_fullsized":"2024-02-10T07:44:29.982991+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:16.835727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:42:16.835727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:34.341954+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":466,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.971248+0000","last_change":"2024-02-10T07:43:15.186714+0000","last_active":"2024-02-10T07:44:29.971248+0000","last_peered":"2024-02-10T07:44:29.971248+0000","last_clean":"2024-02-10T07:44:29.971248+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:44:29.971248+0000","last_undegraded":"2024-02-10T07:44:29.971248+0000","last_fullsized":"2024-02-10T07:44:29.971248+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:15.186469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:15.186469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:31.184297+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":447,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.976870+0000","last_change":"2024-02-10T07:43:09.236406+0000","last_active":"2024-02-10T07:44:29.976870+0000","last_peered":"2024-02-10T07:44:29.976870+0000","last_clean":"2024-02-10T07:44:29.976870+0000","last_became_active":"2024-02-10T07:36:33.771340+0000","last_became_peered":"2024-02-10T07:36:33.771340+0000","last_unstale":"2024-02-10T07:44:29.976870+0000","last_undegraded":"2024-02-10T07:44:29.976870+0000","last_fullsized":"2024-02-10T07:44:29.976870+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:43:09.236268+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:34:49.349360+0000","last_clean_scrub_stamp":"2024-02-10T07:43:09.236268+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:31.184197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"250'15","reported_seq":527,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.988604+0000","last_change":"2024-02-10T07:42:06.021547+0000","last_active":"2024-02-10T07:44:29.988604+0000","last_peered":"2024-02-10T07:44:29.988604+0000","last_clean":"2024-02-10T07:44:29.988604+0000","last_became_active":"2024-02-10T07:31:53.287858+0000","last_became_peered":"2024-02-10T07:31:53.287858+0000","last_unstale":"2024-02-10T07:44:29.988604+0000","last_undegraded":"2024-02-10T07:44:29.988604+0000","last_fullsized":"2024-02-10T07:44:29.988604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:42:06.021407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:42:06.021407+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:33.290626+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":466,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.979696+0000","last_change":"2024-02-10T07:44:00.364614+0000","last_active":"2024-02-10T07:44:29.979696+0000","last_peered":"2024-02-10T07:44:29.979696+0000","last_clean":"2024-02-10T07:44:29.979696+0000","last_became_active":"2024-02-10T07:36:33.757073+0000","last_became_peered":"2024-02-10T07:36:33.757073+0000","last_unstale":"2024-02-10T07:44:29.979696+0000","last_undegraded":"2024-02-10T07:44:29.979696+0000","last_fullsized":"2024-02-10T07:44:29.979696+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:00.364505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:44:00.364505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:06.001177+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":506,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.974487+0000","last_change":"2024-02-10T07:43:12.387217+0000","last_active":"2024-02-10T07:44:29.974487+0000","last_peered":"2024-02-10T07:44:29.974487+0000","last_clean":"2024-02-10T07:44:29.974487+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:44:29.974487+0000","last_undegraded":"2024-02-10T07:44:29.974487+0000","last_fullsized":"2024-02-10T07:44:29.974487+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:12.387113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:12.387113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:36.181815+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":470,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.971228+0000","last_change":"2024-02-10T07:43:10.261985+0000","last_active":"2024-02-10T07:44:29.971228+0000","last_peered":"2024-02-10T07:44:29.971228+0000","last_clean":"2024-02-10T07:44:29.971228+0000","last_became_active":"2024-02-10T07:36:33.767970+0000","last_became_peered":"2024-02-10T07:36:33.767970+0000","last_unstale":"2024-02-10T07:44:29.971228+0000","last_undegraded":"2024-02-10T07:44:29.971228+0000","last_fullsized":"2024-02-10T07:44:29.971228+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:10.261904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.473686+0000","last_clean_scrub_stamp":"2024-02-10T07:43:10.261904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:31.184233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"250'7","reported_seq":515,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.989200+0000","last_change":"2024-02-10T07:43:45.444220+0000","last_active":"2024-02-10T07:44:29.989200+0000","last_peered":"2024-02-10T07:44:29.989200+0000","last_clean":"2024-02-10T07:44:29.989200+0000","last_became_active":"2024-02-10T07:31:53.292417+0000","last_became_peered":"2024-02-10T07:31:53.292417+0000","last_unstale":"2024-02-10T07:44:29.989200+0000","last_undegraded":"2024-02-10T07:44:29.989200+0000","last_fullsized":"2024-02-10T07:44:29.989200+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7","last_scrub_stamp":"2024-02-10T07:43:45.444068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:45.444068+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:05.997143+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":504,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.994178+0000","last_change":"2024-02-10T07:41:23.894038+0000","last_active":"2024-02-10T07:44:29.994178+0000","last_peered":"2024-02-10T07:44:29.994178+0000","last_clean":"2024-02-10T07:44:29.994178+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:44:29.994178+0000","last_undegraded":"2024-02-10T07:44:29.994178+0000","last_fullsized":"2024-02-10T07:44:29.994178+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:41:23.893959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:41:23.893959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:33.290507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":504,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.980206+0000","last_change":"2024-02-10T07:43:01.010982+0000","last_active":"2024-02-10T07:44:29.980206+0000","last_peered":"2024-02-10T07:44:29.980206+0000","last_clean":"2024-02-10T07:44:29.980206+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:44:29.980206+0000","last_undegraded":"2024-02-10T07:44:29.980206+0000","last_fullsized":"2024-02-10T07:44:29.980206+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:01.010877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:01.010877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:33.070160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":513,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.980911+0000","last_change":"2024-02-10T07:44:01.315570+0000","last_active":"2024-02-10T07:44:29.980911+0000","last_peered":"2024-02-10T07:44:29.980911+0000","last_clean":"2024-02-10T07:44:29.980911+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:44:29.980911+0000","last_undegraded":"2024-02-10T07:44:29.980911+0000","last_fullsized":"2024-02-10T07:44:29.980911+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:01.315205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:44:01.315205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:18.186519+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"250'34","reported_seq":559,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.980116+0000","last_change":"2024-02-10T07:43:43.279558+0000","last_active":"2024-02-10T07:44:29.980116+0000","last_peered":"2024-02-10T07:44:29.980116+0000","last_clean":"2024-02-10T07:44:29.980116+0000","last_became_active":"2024-02-10T07:31:53.295527+0000","last_became_peered":"2024-02-10T07:31:53.295527+0000","last_unstale":"2024-02-10T07:44:29.980116+0000","last_undegraded":"2024-02-10T07:44:29.980116+0000","last_fullsized":"2024-02-10T07:44:29.980116+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:43:43.279421+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:43:43.279421+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:57.100334+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":496,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.971298+0000","last_change":"2024-02-10T07:43:45.077379+0000","last_active":"2024-02-10T07:44:29.971298+0000","last_peered":"2024-02-10T07:44:29.971298+0000","last_clean":"2024-02-10T07:44:29.971298+0000","last_became_active":"2024-02-10T07:36:33.754341+0000","last_became_peered":"2024-02-10T07:36:33.754341+0000","last_unstale":"2024-02-10T07:44:29.971298+0000","last_undegraded":"2024-02-10T07:44:29.971298+0000","last_fullsized":"2024-02-10T07:44:29.971298+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:45.077266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:45.077266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:53.398106+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":498,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.992238+0000","last_change":"2024-02-10T07:41:28.853312+0000","last_active":"2024-02-10T07:44:29.992238+0000","last_peered":"2024-02-10T07:44:29.992238+0000","last_clean":"2024-02-10T07:44:29.992238+0000","last_became_active":"2024-02-10T07:31:55.298474+0000","last_became_peered":"2024-02-10T07:31:55.298474+0000","last_unstale":"2024-02-10T07:44:29.992238+0000","last_undegraded":"2024-02-10T07:44:29.992238+0000","last_fullsized":"2024-02-10T07:44:29.992238+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_clean_scrub_stamp":"2024-02-10T07:41:28.853233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:33.290562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":514,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.988706+0000","last_change":"2024-02-10T07:43:16.451188+0000","last_active":"2024-02-10T07:44:29.988706+0000","last_peered":"2024-02-10T07:44:29.988706+0000","last_clean":"2024-02-10T07:44:29.988706+0000","last_became_active":"2024-02-10T07:31:51.321237+0000","last_became_peered":"2024-02-10T07:31:51.321237+0000","last_unstale":"2024-02-10T07:44:29.988706+0000","last_undegraded":"2024-02-10T07:44:29.988706+0000","last_fullsized":"2024-02-10T07:44:29.988706+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:16.451067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:16.451067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:44.108579+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"250'18","reported_seq":534,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.969792+0000","last_change":"2024-02-10T07:43:16.944761+0000","last_active":"2024-02-10T07:44:29.969792+0000","last_peered":"2024-02-10T07:44:29.969792+0000","last_clean":"2024-02-10T07:44:29.969792+0000","last_became_active":"2024-02-10T07:31:53.294914+0000","last_became_peered":"2024-02-10T07:31:53.294914+0000","last_unstale":"2024-02-10T07:44:29.969792+0000","last_undegraded":"2024-02-10T07:44:29.969792+0000","last_fullsized":"2024-02-10T07:44:29.969792+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'18","last_scrub_stamp":"2024-02-10T07:43:16.944627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:16.944627+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:33.834115+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":504,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.974037+0000","last_change":"2024-02-10T07:43:01.353222+0000","last_active":"2024-02-10T07:44:29.974037+0000","last_peered":"2024-02-10T07:44:29.974037+0000","last_clean":"2024-02-10T07:44:29.974037+0000","last_became_active":"2024-02-10T07:36:33.751352+0000","last_became_peered":"2024-02-10T07:36:33.751352+0000","last_unstale":"2024-02-10T07:44:29.974037+0000","last_undegraded":"2024-02-10T07:44:29.974037+0000","last_fullsized":"2024-02-10T07:44:29.974037+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:43:01.352980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:01.352980+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:32.454849+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":687,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:31.522652+0000","last_change":"2024-02-10T07:43:41.166832+0000","last_active":"2024-02-10T07:44:31.522652+0000","last_peered":"2024-02-10T07:44:31.522652+0000","last_clean":"2024-02-10T07:44:31.522652+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:44:31.522652+0000","last_undegraded":"2024-02-10T07:44:31.522652+0000","last_fullsized":"2024-02-10T07:44:31.522652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:43:41.166747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:41.166747+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:07.711794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":513,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.981481+0000","last_change":"2024-02-10T07:43:07.962862+0000","last_active":"2024-02-10T07:44:29.981481+0000","last_peered":"2024-02-10T07:44:29.981481+0000","last_clean":"2024-02-10T07:44:29.981481+0000","last_became_active":"2024-02-10T07:31:51.282077+0000","last_became_peered":"2024-02-10T07:31:51.282077+0000","last_unstale":"2024-02-10T07:44:29.981481+0000","last_undegraded":"2024-02-10T07:44:29.981481+0000","last_fullsized":"2024-02-10T07:44:29.981481+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:43:07.962748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:07.962748+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:31.437618+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"250'25","reported_seq":553,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.980911+0000","last_change":"2024-02-10T07:44:06.180569+0000","last_active":"2024-02-10T07:44:29.980911+0000","last_peered":"2024-02-10T07:44:29.980911+0000","last_clean":"2024-02-10T07:44:29.980911+0000","last_became_active":"2024-02-10T07:31:53.292936+0000","last_became_peered":"2024-02-10T07:31:53.292936+0000","last_unstale":"2024-02-10T07:44:29.980911+0000","last_undegraded":"2024-02-10T07:44:29.980911+0000","last_fullsized":"2024-02-10T07:44:29.980911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:44:06.180439+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-02-10T07:42:17.820959+0000","last_clean_scrub_stamp":"2024-02-10T07:44:06.180439+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:09.980808+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":509,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.980348+0000","last_change":"2024-02-10T07:44:05.558893+0000","last_active":"2024-02-10T07:44:29.980348+0000","last_peered":"2024-02-10T07:44:29.980348+0000","last_clean":"2024-02-10T07:44:29.980348+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:44:29.980348+0000","last_undegraded":"2024-02-10T07:44:29.980348+0000","last_fullsized":"2024-02-10T07:44:29.980348+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_clean_scrub_stamp":"2024-02-10T07:44:05.558659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:22.648251+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":498,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.981177+0000","last_change":"2024-02-10T07:43:04.051025+0000","last_active":"2024-02-10T07:44:29.981177+0000","last_peered":"2024-02-10T07:44:29.981177+0000","last_clean":"2024-02-10T07:44:29.981177+0000","last_became_active":"2024-02-10T07:31:55.297595+0000","last_became_peered":"2024-02-10T07:31:55.297595+0000","last_unstale":"2024-02-10T07:44:29.981177+0000","last_undegraded":"2024-02-10T07:44:29.981177+0000","last_fullsized":"2024-02-10T07:44:29.981177+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:04.050906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:04.050906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:31.685094+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":506,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.969595+0000","last_change":"2024-02-10T07:43:59.897845+0000","last_active":"2024-02-10T07:44:29.969595+0000","last_peered":"2024-02-10T07:44:29.969595+0000","last_clean":"2024-02-10T07:44:29.969595+0000","last_became_active":"2024-02-10T07:31:51.285405+0000","last_became_peered":"2024-02-10T07:31:51.285405+0000","last_unstale":"2024-02-10T07:44:29.969595+0000","last_undegraded":"2024-02-10T07:44:29.969595+0000","last_fullsized":"2024-02-10T07:44:29.969595+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:59.897726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:59.897726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:17.362848+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"250'19","reported_seq":550,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.984229+0000","last_change":"2024-02-10T07:43:04.399390+0000","last_active":"2024-02-10T07:44:29.984229+0000","last_peered":"2024-02-10T07:44:29.984229+0000","last_clean":"2024-02-10T07:44:29.984229+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:44:29.984229+0000","last_undegraded":"2024-02-10T07:44:29.984229+0000","last_fullsized":"2024-02-10T07:44:29.984229+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:43:04.399223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:04.399223+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:31.032513+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":502,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.974000+0000","last_change":"2024-02-10T07:43:42.484221+0000","last_active":"2024-02-10T07:44:29.974000+0000","last_peered":"2024-02-10T07:44:29.974000+0000","last_clean":"2024-02-10T07:44:29.974000+0000","last_became_active":"2024-02-10T07:31:57.306776+0000","last_became_peered":"2024-02-10T07:31:57.306776+0000","last_unstale":"2024-02-10T07:44:29.974000+0000","last_undegraded":"2024-02-10T07:44:29.974000+0000","last_fullsized":"2024-02-10T07:44:29.974000+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:42.484093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:43:42.484093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:05.722649+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":513,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.983587+0000","last_change":"2024-02-10T07:43:50.101849+0000","last_active":"2024-02-10T07:44:29.983587+0000","last_peered":"2024-02-10T07:44:29.983587+0000","last_clean":"2024-02-10T07:44:29.983587+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:44:29.983587+0000","last_undegraded":"2024-02-10T07:44:29.983587+0000","last_fullsized":"2024-02-10T07:44:29.983587+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:50.101724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:50.101724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:00.993819+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":522,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.981557+0000","last_change":"2024-02-10T07:43:12.131490+0000","last_active":"2024-02-10T07:44:29.981557+0000","last_peered":"2024-02-10T07:44:29.981557+0000","last_clean":"2024-02-10T07:44:29.981557+0000","last_became_active":"2024-02-10T07:36:33.760213+0000","last_became_peered":"2024-02-10T07:36:33.760213+0000","last_unstale":"2024-02-10T07:44:29.981557+0000","last_undegraded":"2024-02-10T07:44:29.981557+0000","last_fullsized":"2024-02-10T07:44:29.981557+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:12.131342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:12.131342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:34.341979+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"250'10","reported_seq":508,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.969676+0000","last_change":"2024-02-10T07:43:06.022311+0000","last_active":"2024-02-10T07:44:29.969676+0000","last_peered":"2024-02-10T07:44:29.969676+0000","last_clean":"2024-02-10T07:44:29.969676+0000","last_became_active":"2024-02-10T07:31:53.295520+0000","last_became_peered":"2024-02-10T07:31:53.295520+0000","last_unstale":"2024-02-10T07:44:29.969676+0000","last_undegraded":"2024-02-10T07:44:29.969676+0000","last_fullsized":"2024-02-10T07:44:29.969676+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'10","last_scrub_stamp":"2024-02-10T07:43:06.022190+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:43:06.022190+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:33.135813+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":494,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.970296+0000","last_change":"2024-02-10T07:43:43.019843+0000","last_active":"2024-02-10T07:44:29.970296+0000","last_peered":"2024-02-10T07:44:29.970296+0000","last_clean":"2024-02-10T07:44:29.970296+0000","last_became_active":"2024-02-10T07:31:57.315738+0000","last_became_peered":"2024-02-10T07:31:57.315738+0000","last_unstale":"2024-02-10T07:44:29.970296+0000","last_undegraded":"2024-02-10T07:44:29.970296+0000","last_fullsized":"2024-02-10T07:44:29.970296+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:43.019550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:43.019550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:01.353357+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":506,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.973408+0000","last_change":"2024-02-10T07:43:44.490369+0000","last_active":"2024-02-10T07:44:29.973408+0000","last_peered":"2024-02-10T07:44:29.973408+0000","last_clean":"2024-02-10T07:44:29.973408+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:44:29.973408+0000","last_undegraded":"2024-02-10T07:44:29.973408+0000","last_fullsized":"2024-02-10T07:44:29.973408+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:44.490246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:44.490246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:58.669303+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":430,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.981197+0000","last_change":"2024-02-10T07:42:21.106187+0000","last_active":"2024-02-10T07:44:29.981197+0000","last_peered":"2024-02-10T07:44:29.981197+0000","last_clean":"2024-02-10T07:44:29.981197+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:44:29.981197+0000","last_undegraded":"2024-02-10T07:44:29.981197+0000","last_fullsized":"2024-02-10T07:44:29.981197+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:21.106090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:42:21.106090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:31.184135+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"250'19","reported_seq":533,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.970037+0000","last_change":"2024-02-10T07:43:40.896052+0000","last_active":"2024-02-10T07:44:29.970037+0000","last_peered":"2024-02-10T07:44:29.970037+0000","last_clean":"2024-02-10T07:44:29.970037+0000","last_became_active":"2024-02-10T07:31:53.287823+0000","last_became_peered":"2024-02-10T07:31:53.287823+0000","last_unstale":"2024-02-10T07:44:29.970037+0000","last_undegraded":"2024-02-10T07:44:29.970037+0000","last_fullsized":"2024-02-10T07:44:29.970037+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:43:40.895108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:40.895108+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:09.724187+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":494,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.982689+0000","last_change":"2024-02-10T07:42:12.805668+0000","last_active":"2024-02-10T07:44:29.982689+0000","last_peered":"2024-02-10T07:44:29.982689+0000","last_clean":"2024-02-10T07:44:29.982689+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:44:29.982689+0000","last_undegraded":"2024-02-10T07:44:29.982689+0000","last_fullsized":"2024-02-10T07:44:29.982689+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:12.805543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:42:12.805543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:34.341858+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":498,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.970117+0000","last_change":"2024-02-10T07:43:46.833839+0000","last_active":"2024-02-10T07:44:29.970117+0000","last_peered":"2024-02-10T07:44:29.970117+0000","last_clean":"2024-02-10T07:44:29.970117+0000","last_became_active":"2024-02-10T07:31:55.296670+0000","last_became_peered":"2024-02-10T07:31:55.296670+0000","last_unstale":"2024-02-10T07:44:29.970117+0000","last_undegraded":"2024-02-10T07:44:29.970117+0000","last_fullsized":"2024-02-10T07:44:29.970117+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:46.833613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:46.833613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:59.787265+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":514,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.979647+0000","last_change":"2024-02-10T07:43:06.386801+0000","last_active":"2024-02-10T07:44:29.979647+0000","last_peered":"2024-02-10T07:44:29.979647+0000","last_clean":"2024-02-10T07:44:29.979647+0000","last_became_active":"2024-02-10T07:31:51.288165+0000","last_became_peered":"2024-02-10T07:31:51.288165+0000","last_unstale":"2024-02-10T07:44:29.979647+0000","last_undegraded":"2024-02-10T07:44:29.979647+0000","last_fullsized":"2024-02-10T07:44:29.979647+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:06.386654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:06.386654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:32.454961+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"250'19","reported_seq":315,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.979148+0000","last_change":"2024-02-10T07:44:08.503149+0000","last_active":"2024-02-10T07:44:29.979148+0000","last_peered":"2024-02-10T07:44:29.979148+0000","last_clean":"2024-02-10T07:44:29.979148+0000","last_became_active":"2024-02-10T07:37:06.187551+0000","last_became_peered":"2024-02-10T07:37:06.187551+0000","last_unstale":"2024-02-10T07:44:29.979148+0000","last_undegraded":"2024-02-10T07:44:29.979148+0000","last_fullsized":"2024-02-10T07:44:29.979148+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:44:08.503058+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-02-10T07:39:32.934269+0000","last_clean_scrub_stamp":"2024-02-10T07:44:08.503058+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:37.824115+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":495,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.969715+0000","last_change":"2024-02-10T07:43:11.935665+0000","last_active":"2024-02-10T07:44:29.969715+0000","last_peered":"2024-02-10T07:44:29.969715+0000","last_clean":"2024-02-10T07:44:29.969715+0000","last_became_active":"2024-02-10T07:36:33.750030+0000","last_became_peered":"2024-02-10T07:36:33.750030+0000","last_unstale":"2024-02-10T07:44:29.969715+0000","last_undegraded":"2024-02-10T07:44:29.969715+0000","last_fullsized":"2024-02-10T07:44:29.969715+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:11.935253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:11.935253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:40.667135+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":513,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.979807+0000","last_change":"2024-02-10T07:43:19.008513+0000","last_active":"2024-02-10T07:44:29.979807+0000","last_peered":"2024-02-10T07:44:29.979807+0000","last_clean":"2024-02-10T07:44:29.979807+0000","last_became_active":"2024-02-10T07:36:33.752418+0000","last_became_peered":"2024-02-10T07:36:33.752418+0000","last_unstale":"2024-02-10T07:44:29.979807+0000","last_undegraded":"2024-02-10T07:44:29.979807+0000","last_fullsized":"2024-02-10T07:44:29.979807+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:19.008018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:19.008018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:34.016777+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":514,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.992038+0000","last_change":"2024-02-10T07:44:04.423152+0000","last_active":"2024-02-10T07:44:29.992038+0000","last_peered":"2024-02-10T07:44:29.992038+0000","last_clean":"2024-02-10T07:44:29.992038+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:44:29.992038+0000","last_undegraded":"2024-02-10T07:44:29.992038+0000","last_fullsized":"2024-02-10T07:44:29.992038+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:04.423015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:44:04.423015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:09.200801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"250'24","reported_seq":564,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.989097+0000","last_change":"2024-02-10T07:44:02.402382+0000","last_active":"2024-02-10T07:44:29.989097+0000","last_peered":"2024-02-10T07:44:29.989097+0000","last_clean":"2024-02-10T07:44:29.989097+0000","last_became_active":"2024-02-10T07:32:18.389564+0000","last_became_peered":"2024-02-10T07:32:18.389564+0000","last_unstale":"2024-02-10T07:44:29.989097+0000","last_undegraded":"2024-02-10T07:44:29.989097+0000","last_fullsized":"2024-02-10T07:44:29.989097+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:44:02.402293+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:44:02.402293+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:28.724385+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":501,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.980610+0000","last_change":"2024-02-10T07:44:00.234087+0000","last_active":"2024-02-10T07:44:29.980610+0000","last_peered":"2024-02-10T07:44:29.980610+0000","last_clean":"2024-02-10T07:44:29.980610+0000","last_became_active":"2024-02-10T07:31:57.318009+0000","last_became_peered":"2024-02-10T07:31:57.318009+0000","last_unstale":"2024-02-10T07:44:29.980610+0000","last_undegraded":"2024-02-10T07:44:29.980610+0000","last_fullsized":"2024-02-10T07:44:29.980610+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:00.233946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:44:00.233946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:16.097710+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"158'10","reported_seq":577,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:31.523131+0000","last_change":"2024-02-10T07:42:15.162807+0000","last_active":"2024-02-10T07:44:31.523131+0000","last_peered":"2024-02-10T07:44:31.523131+0000","last_clean":"2024-02-10T07:44:31.523131+0000","last_became_active":"2024-02-10T07:36:33.774773+0000","last_became_peered":"2024-02-10T07:36:33.774773+0000","last_unstale":"2024-02-10T07:44:31.523131+0000","last_undegraded":"2024-02-10T07:44:31.523131+0000","last_fullsized":"2024-02-10T07:44:31.523131+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:42:15.162654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:42:15.162654+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:37.299201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":514,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.973608+0000","last_change":"2024-02-10T07:43:04.293059+0000","last_active":"2024-02-10T07:44:29.973608+0000","last_peered":"2024-02-10T07:44:29.973608+0000","last_clean":"2024-02-10T07:44:29.973608+0000","last_became_active":"2024-02-10T07:31:51.320644+0000","last_became_peered":"2024-02-10T07:31:51.320644+0000","last_unstale":"2024-02-10T07:44:29.973608+0000","last_undegraded":"2024-02-10T07:44:29.973608+0000","last_fullsized":"2024-02-10T07:44:29.973608+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:04.292917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:43:04.292917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:32.454816+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"250'25","reported_seq":355,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.971091+0000","last_change":"2024-02-10T07:43:56.304513+0000","last_active":"2024-02-10T07:44:29.971091+0000","last_peered":"2024-02-10T07:44:29.971091+0000","last_clean":"2024-02-10T07:44:29.971091+0000","last_became_active":"2024-02-10T07:36:33.752317+0000","last_became_peered":"2024-02-10T07:36:33.752317+0000","last_unstale":"2024-02-10T07:44:29.971091+0000","last_undegraded":"2024-02-10T07:44:29.971091+0000","last_fullsized":"2024-02-10T07:44:29.971091+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:43:56.304364+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:43:56.304364+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:24.039285+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":503,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.990385+0000","last_change":"2024-02-10T07:43:46.588398+0000","last_active":"2024-02-10T07:44:29.990385+0000","last_peered":"2024-02-10T07:44:29.990385+0000","last_clean":"2024-02-10T07:44:29.990385+0000","last_became_active":"2024-02-10T07:36:33.766051+0000","last_became_peered":"2024-02-10T07:36:33.766051+0000","last_unstale":"2024-02-10T07:44:29.990385+0000","last_undegraded":"2024-02-10T07:44:29.990385+0000","last_fullsized":"2024-02-10T07:44:29.990385+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:46.588253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:46.588253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:56.838799+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":660,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:31.526359+0000","last_change":"2024-02-10T07:43:55.848459+0000","last_active":"2024-02-10T07:44:31.526359+0000","last_peered":"2024-02-10T07:44:31.526359+0000","last_clean":"2024-02-10T07:44:31.526359+0000","last_became_active":"2024-02-10T07:31:55.294594+0000","last_became_peered":"2024-02-10T07:31:55.294594+0000","last_unstale":"2024-02-10T07:44:31.526359+0000","last_undegraded":"2024-02-10T07:44:31.526359+0000","last_fullsized":"2024-02-10T07:44:31.526359+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:43:55.848231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:55.848231+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:01.223931+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":513,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.978522+0000","last_change":"2024-02-10T07:43:19.230062+0000","last_active":"2024-02-10T07:44:29.978522+0000","last_peered":"2024-02-10T07:44:29.978522+0000","last_clean":"2024-02-10T07:44:29.978522+0000","last_became_active":"2024-02-10T07:31:51.282353+0000","last_became_peered":"2024-02-10T07:31:51.282353+0000","last_unstale":"2024-02-10T07:44:29.978522+0000","last_undegraded":"2024-02-10T07:44:29.978522+0000","last_fullsized":"2024-02-10T07:44:29.978522+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:19.229934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:19.229934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.657808+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"250'15","reported_seq":479,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.977312+0000","last_change":"2024-02-10T07:43:02.231786+0000","last_active":"2024-02-10T07:44:29.977312+0000","last_peered":"2024-02-10T07:44:29.977312+0000","last_clean":"2024-02-10T07:44:29.977312+0000","last_became_active":"2024-02-10T07:36:33.769894+0000","last_became_peered":"2024-02-10T07:36:33.769894+0000","last_unstale":"2024-02-10T07:44:29.977312+0000","last_undegraded":"2024-02-10T07:44:29.977312+0000","last_fullsized":"2024-02-10T07:44:29.977312+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:43:02.231670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:02.231670+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:31.184267+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":509,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.980393+0000","last_change":"2024-02-10T07:42:59.879292+0000","last_active":"2024-02-10T07:44:29.980393+0000","last_peered":"2024-02-10T07:44:29.980393+0000","last_clean":"2024-02-10T07:44:29.980393+0000","last_became_active":"2024-02-10T07:31:57.308615+0000","last_became_peered":"2024-02-10T07:31:57.308615+0000","last_unstale":"2024-02-10T07:44:29.980393+0000","last_undegraded":"2024-02-10T07:44:29.980393+0000","last_fullsized":"2024-02-10T07:44:29.980393+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:59.879171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:42:59.879171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:32.956490+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":490,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.971711+0000","last_change":"2024-02-10T07:43:01.179228+0000","last_active":"2024-02-10T07:44:29.971711+0000","last_peered":"2024-02-10T07:44:29.971711+0000","last_clean":"2024-02-10T07:44:29.971711+0000","last_became_active":"2024-02-10T07:31:55.297431+0000","last_became_peered":"2024-02-10T07:31:55.297431+0000","last_unstale":"2024-02-10T07:44:29.971711+0000","last_undegraded":"2024-02-10T07:44:29.971711+0000","last_fullsized":"2024-02-10T07:44:29.971711+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:01.179025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:43:01.179025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":514,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.983637+0000","last_change":"2024-02-10T07:42:59.403762+0000","last_active":"2024-02-10T07:44:29.983637+0000","last_peered":"2024-02-10T07:44:29.983637+0000","last_clean":"2024-02-10T07:44:29.983637+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:44:29.983637+0000","last_undegraded":"2024-02-10T07:44:29.983637+0000","last_fullsized":"2024-02-10T07:44:29.983637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:59.403332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:42:59.403332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"250'19","reported_seq":546,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.981007+0000","last_change":"2024-02-10T07:44:08.130835+0000","last_active":"2024-02-10T07:44:29.981007+0000","last_peered":"2024-02-10T07:44:29.981007+0000","last_clean":"2024-02-10T07:44:29.981007+0000","last_became_active":"2024-02-10T07:31:53.295353+0000","last_became_peered":"2024-02-10T07:31:53.295353+0000","last_unstale":"2024-02-10T07:44:29.981007+0000","last_undegraded":"2024-02-10T07:44:29.981007+0000","last_fullsized":"2024-02-10T07:44:29.981007+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:44:08.130707+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:44:08.130707+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:27.190414+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":494,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.979543+0000","last_change":"2024-02-10T07:43:09.421682+0000","last_active":"2024-02-10T07:44:29.979543+0000","last_peered":"2024-02-10T07:44:29.979543+0000","last_clean":"2024-02-10T07:44:29.979543+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:44:29.979543+0000","last_undegraded":"2024-02-10T07:44:29.979543+0000","last_fullsized":"2024-02-10T07:44:29.979543+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:09.421372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:09.421372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:32.454936+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":454,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.981281+0000","last_change":"2024-02-10T07:42:07.138453+0000","last_active":"2024-02-10T07:44:29.981281+0000","last_peered":"2024-02-10T07:44:29.981281+0000","last_clean":"2024-02-10T07:44:29.981281+0000","last_became_active":"2024-02-10T07:36:33.752051+0000","last_became_peered":"2024-02-10T07:36:33.752051+0000","last_unstale":"2024-02-10T07:44:29.981281+0000","last_undegraded":"2024-02-10T07:44:29.981281+0000","last_fullsized":"2024-02-10T07:44:29.981281+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:07.138357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:42:07.138357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:31.183946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":530,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.983235+0000","last_change":"2024-02-10T07:43:47.053223+0000","last_active":"2024-02-10T07:44:29.983235+0000","last_peered":"2024-02-10T07:44:29.983235+0000","last_clean":"2024-02-10T07:44:29.983235+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:44:29.983235+0000","last_undegraded":"2024-02-10T07:44:29.983235+0000","last_fullsized":"2024-02-10T07:44:29.983235+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:47.052981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:47.052981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:52.720988+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"250'12","reported_seq":512,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.971615+0000","last_change":"2024-02-10T07:43:08.018116+0000","last_active":"2024-02-10T07:44:29.971615+0000","last_peered":"2024-02-10T07:44:29.971615+0000","last_clean":"2024-02-10T07:44:29.971615+0000","last_became_active":"2024-02-10T07:31:53.296831+0000","last_became_peered":"2024-02-10T07:31:53.296831+0000","last_unstale":"2024-02-10T07:44:29.971615+0000","last_undegraded":"2024-02-10T07:44:29.971615+0000","last_fullsized":"2024-02-10T07:44:29.971615+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'12","last_scrub_stamp":"2024-02-10T07:43:08.017802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:08.017802+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"292'10940","reported_seq":11113,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:38.815178+0000","last_change":"2024-02-10T07:43:44.147560+0000","last_active":"2024-02-10T07:44:38.815178+0000","last_peered":"2024-02-10T07:44:38.815178+0000","last_clean":"2024-02-10T07:44:38.815178+0000","last_became_active":"2024-02-10T07:42:09.176416+0000","last_became_peered":"2024-02-10T07:42:09.176416+0000","last_unstale":"2024-02-10T07:44:38.815178+0000","last_undegraded":"2024-02-10T07:44:38.815178+0000","last_fullsized":"2024-02-10T07:44:38.815178+0000","mapping_epoch":253,"log_start":"270'4900","ondisk_log_start":"270'4900","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'7616","last_scrub_stamp":"2024-02-10T07:43:44.147381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:44.147381+0000","objects_scrubbed":7544,"log_size":6040,"log_dups_size":0,"ondisk_log_size":6040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:48.586825+0000","scrub_duration":2922,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338821120,"num_objects":5170,"num_object_clones":0,"num_object_copies":15510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10940,"num_write_kb":515520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":466,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.971959+0000","last_change":"2024-02-10T07:43:40.413257+0000","last_active":"2024-02-10T07:44:29.971959+0000","last_peered":"2024-02-10T07:44:29.971959+0000","last_clean":"2024-02-10T07:44:29.971959+0000","last_became_active":"2024-02-10T07:36:33.763431+0000","last_became_peered":"2024-02-10T07:36:33.763431+0000","last_unstale":"2024-02-10T07:44:29.971959+0000","last_undegraded":"2024-02-10T07:44:29.971959+0000","last_fullsized":"2024-02-10T07:44:29.971959+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:40.413128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:40.413128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:48.471922+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"245'4","reported_seq":528,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.973933+0000","last_change":"2024-02-10T07:43:05.338877+0000","last_active":"2024-02-10T07:44:29.973933+0000","last_peered":"2024-02-10T07:44:29.973933+0000","last_clean":"2024-02-10T07:44:29.973933+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:44:29.973933+0000","last_undegraded":"2024-02-10T07:44:29.973933+0000","last_fullsized":"2024-02-10T07:44:29.973933+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'4","last_scrub_stamp":"2024-02-10T07:43:05.338740+0000","last_deep_scrub":"245'4","last_deep_scrub_stamp":"2024-02-10T07:41:23.717923+0000","last_clean_scrub_stamp":"2024-02-10T07:43:05.338740+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:32.454880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"250'34","reported_seq":482,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.970876+0000","last_change":"2024-02-10T07:43:55.309700+0000","last_active":"2024-02-10T07:44:29.970876+0000","last_peered":"2024-02-10T07:44:29.970876+0000","last_clean":"2024-02-10T07:44:29.970876+0000","last_became_active":"2024-02-10T07:36:33.754134+0000","last_became_peered":"2024-02-10T07:36:33.754134+0000","last_unstale":"2024-02-10T07:44:29.970876+0000","last_undegraded":"2024-02-10T07:44:29.970876+0000","last_fullsized":"2024-02-10T07:44:29.970876+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:43:55.309563+0000","last_deep_scrub":"246'22","last_deep_scrub_stamp":"2024-02-10T07:41:26.771112+0000","last_clean_scrub_stamp":"2024-02-10T07:43:55.309563+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:59.444281+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":676,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:31.523084+0000","last_change":"2024-02-10T07:44:13.245977+0000","last_active":"2024-02-10T07:44:31.523084+0000","last_peered":"2024-02-10T07:44:31.523084+0000","last_clean":"2024-02-10T07:44:31.523084+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:44:31.523084+0000","last_undegraded":"2024-02-10T07:44:31.523084+0000","last_fullsized":"2024-02-10T07:44:31.523084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:44:13.245900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:44:13.245900+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:31.843130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"292'10952","reported_seq":11148,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:38.914323+0000","last_change":"2024-02-10T07:43:45.786143+0000","last_active":"2024-02-10T07:44:38.914323+0000","last_peered":"2024-02-10T07:44:38.914323+0000","last_clean":"2024-02-10T07:44:38.914323+0000","last_became_active":"2024-02-10T07:42:09.451050+0000","last_became_peered":"2024-02-10T07:42:09.451050+0000","last_unstale":"2024-02-10T07:44:38.914323+0000","last_undegraded":"2024-02-10T07:44:38.914323+0000","last_fullsized":"2024-02-10T07:44:38.914323+0000","mapping_epoch":253,"log_start":"274'6100","ondisk_log_start":"274'6100","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'7684","last_scrub_stamp":"2024-02-10T07:43:45.785994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:45.785994+0000","objects_scrubbed":7601,"log_size":4852,"log_dups_size":0,"ondisk_log_size":4852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:01.148494+0000","scrub_duration":2593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335413248,"num_objects":5118,"num_object_clones":0,"num_object_copies":15354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10952,"num_write_kb":514240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":486,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.970205+0000","last_change":"2024-02-10T07:43:02.251344+0000","last_active":"2024-02-10T07:44:29.970205+0000","last_peered":"2024-02-10T07:44:29.970205+0000","last_clean":"2024-02-10T07:44:29.970205+0000","last_became_active":"2024-02-10T07:31:57.314872+0000","last_became_peered":"2024-02-10T07:31:57.314872+0000","last_unstale":"2024-02-10T07:44:29.970205+0000","last_undegraded":"2024-02-10T07:44:29.970205+0000","last_fullsized":"2024-02-10T07:44:29.970205+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:02.251208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:02.251208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:33.135780+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":514,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.974208+0000","last_change":"2024-02-10T07:43:19.419259+0000","last_active":"2024-02-10T07:44:29.974208+0000","last_peered":"2024-02-10T07:44:29.974208+0000","last_clean":"2024-02-10T07:44:29.974208+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:44:29.974208+0000","last_undegraded":"2024-02-10T07:44:29.974208+0000","last_fullsized":"2024-02-10T07:44:29.974208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:19.419141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:43:19.419141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:48.757225+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"250'23","reported_seq":557,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.988243+0000","last_change":"2024-02-10T07:41:21.884798+0000","last_active":"2024-02-10T07:44:29.988243+0000","last_peered":"2024-02-10T07:44:29.988243+0000","last_clean":"2024-02-10T07:44:29.988243+0000","last_became_active":"2024-02-10T07:32:18.389409+0000","last_became_peered":"2024-02-10T07:32:18.389409+0000","last_unstale":"2024-02-10T07:44:29.988243+0000","last_undegraded":"2024-02-10T07:44:29.988243+0000","last_fullsized":"2024-02-10T07:44:29.988243+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'17","last_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_deep_scrub":"245'17","last_deep_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_clean_scrub_stamp":"2024-02-10T07:41:21.884664+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:33.290478+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":513,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.987555+0000","last_change":"2024-02-10T07:43:56.145450+0000","last_active":"2024-02-10T07:44:29.987555+0000","last_peered":"2024-02-10T07:44:29.987555+0000","last_clean":"2024-02-10T07:44:29.987555+0000","last_became_active":"2024-02-10T07:31:55.303059+0000","last_became_peered":"2024-02-10T07:31:55.303059+0000","last_unstale":"2024-02-10T07:44:29.987555+0000","last_undegraded":"2024-02-10T07:44:29.987555+0000","last_fullsized":"2024-02-10T07:44:29.987555+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:56.145004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:56.145004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:22.618065+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"292'10735","reported_seq":10824,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:37.105018+0000","last_change":"2024-02-10T07:42:09.176546+0000","last_active":"2024-02-10T07:44:37.105018+0000","last_peered":"2024-02-10T07:44:37.105018+0000","last_clean":"2024-02-10T07:44:37.105018+0000","last_became_active":"2024-02-10T07:42:09.175986+0000","last_became_peered":"2024-02-10T07:42:09.175986+0000","last_unstale":"2024-02-10T07:44:37.105018+0000","last_undegraded":"2024-02-10T07:44:37.105018+0000","last_fullsized":"2024-02-10T07:44:37.105018+0000","mapping_epoch":253,"log_start":"271'5400","ondisk_log_start":"271'5400","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:42:08.158083+0000","objects_scrubbed":0,"log_size":5335,"log_dups_size":0,"ondisk_log_size":5335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:37.301418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339148800,"num_objects":5175,"num_object_clones":0,"num_object_copies":15525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10735,"num_write_kb":509120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":510,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.972988+0000","last_change":"2024-02-10T07:43:48.467475+0000","last_active":"2024-02-10T07:44:29.972988+0000","last_peered":"2024-02-10T07:44:29.972988+0000","last_clean":"2024-02-10T07:44:29.972988+0000","last_became_active":"2024-02-10T07:31:57.304971+0000","last_became_peered":"2024-02-10T07:31:57.304971+0000","last_unstale":"2024-02-10T07:44:29.972988+0000","last_undegraded":"2024-02-10T07:44:29.972988+0000","last_fullsized":"2024-02-10T07:44:29.972988+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:48.467367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:36.741855+0000","last_clean_scrub_stamp":"2024-02-10T07:43:48.467367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:01.277638+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":522,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.979985+0000","last_change":"2024-02-10T07:43:46.261370+0000","last_active":"2024-02-10T07:44:29.979985+0000","last_peered":"2024-02-10T07:44:29.979985+0000","last_clean":"2024-02-10T07:44:29.979985+0000","last_became_active":"2024-02-10T07:36:33.754602+0000","last_became_peered":"2024-02-10T07:36:33.754602+0000","last_unstale":"2024-02-10T07:44:29.979985+0000","last_undegraded":"2024-02-10T07:44:29.979985+0000","last_fullsized":"2024-02-10T07:44:29.979985+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:46.261253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:46.261253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:58.623684+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":523,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.979857+0000","last_change":"2024-02-10T07:43:03.294877+0000","last_active":"2024-02-10T07:44:29.979857+0000","last_peered":"2024-02-10T07:44:29.979857+0000","last_clean":"2024-02-10T07:44:29.979857+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:44:29.979857+0000","last_undegraded":"2024-02-10T07:44:29.979857+0000","last_fullsized":"2024-02-10T07:44:29.979857+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:03.294778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:03.294778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"250'35","reported_seq":572,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.983483+0000","last_change":"2024-02-10T07:43:07.470125+0000","last_active":"2024-02-10T07:44:29.983483+0000","last_peered":"2024-02-10T07:44:29.983483+0000","last_clean":"2024-02-10T07:44:29.983483+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:44:29.983483+0000","last_undegraded":"2024-02-10T07:44:29.983483+0000","last_fullsized":"2024-02-10T07:44:29.983483+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'35","last_scrub_stamp":"2024-02-10T07:43:07.469991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:07.469991+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:31.032534+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"292'10941","reported_seq":11109,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:37.110803+0000","last_change":"2024-02-10T07:44:09.296562+0000","last_active":"2024-02-10T07:44:37.110803+0000","last_peered":"2024-02-10T07:44:37.110803+0000","last_clean":"2024-02-10T07:44:37.110803+0000","last_became_active":"2024-02-10T07:42:09.395485+0000","last_became_peered":"2024-02-10T07:42:09.395485+0000","last_unstale":"2024-02-10T07:44:37.110803+0000","last_undegraded":"2024-02-10T07:44:37.110803+0000","last_fullsized":"2024-02-10T07:44:37.110803+0000","mapping_epoch":253,"log_start":"271'5600","ondisk_log_start":"271'5600","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'9251","last_scrub_stamp":"2024-02-10T07:44:09.296369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:44:09.296369+0000","objects_scrubbed":7105,"log_size":5341,"log_dups_size":0,"ondisk_log_size":5341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:35.841544+0000","scrub_duration":2022,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351600640,"num_objects":5365,"num_object_clones":0,"num_object_copies":16095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10941,"num_write_kb":521792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":501,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.980649+0000","last_change":"2024-02-10T07:43:40.302514+0000","last_active":"2024-02-10T07:44:29.980649+0000","last_peered":"2024-02-10T07:44:29.980649+0000","last_clean":"2024-02-10T07:44:29.980649+0000","last_became_active":"2024-02-10T07:31:57.313394+0000","last_became_peered":"2024-02-10T07:31:57.313394+0000","last_unstale":"2024-02-10T07:44:29.980649+0000","last_undegraded":"2024-02-10T07:44:29.980649+0000","last_fullsized":"2024-02-10T07:44:29.980649+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:40.302404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:40.302404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:43.567142+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":513,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.981462+0000","last_change":"2024-02-10T07:43:51.246335+0000","last_active":"2024-02-10T07:44:29.981462+0000","last_peered":"2024-02-10T07:44:29.981462+0000","last_clean":"2024-02-10T07:44:29.981462+0000","last_became_active":"2024-02-10T07:31:55.298622+0000","last_became_peered":"2024-02-10T07:31:55.298622+0000","last_unstale":"2024-02-10T07:44:29.981462+0000","last_undegraded":"2024-02-10T07:44:29.981462+0000","last_fullsized":"2024-02-10T07:44:29.981462+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:51.245746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:51.245746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:56.919752+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":470,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.970951+0000","last_change":"2024-02-10T07:43:59.363625+0000","last_active":"2024-02-10T07:44:29.970951+0000","last_peered":"2024-02-10T07:44:29.970951+0000","last_clean":"2024-02-10T07:44:29.970951+0000","last_became_active":"2024-02-10T07:36:33.762155+0000","last_became_peered":"2024-02-10T07:36:33.762155+0000","last_unstale":"2024-02-10T07:44:29.970951+0000","last_undegraded":"2024-02-10T07:44:29.970951+0000","last_fullsized":"2024-02-10T07:44:29.970951+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:59.363500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:59.363500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:23.913803+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"250'15","reported_seq":534,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.981075+0000","last_change":"2024-02-10T07:43:39.224099+0000","last_active":"2024-02-10T07:44:29.981075+0000","last_peered":"2024-02-10T07:44:29.981075+0000","last_clean":"2024-02-10T07:44:29.981075+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:44:29.981075+0000","last_undegraded":"2024-02-10T07:44:29.981075+0000","last_fullsized":"2024-02-10T07:44:29.981075+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:43:39.223948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:39.223948+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:43.101629+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"292'10927","reported_seq":11103,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:38.790541+0000","last_change":"2024-02-10T07:43:48.784291+0000","last_active":"2024-02-10T07:44:38.790541+0000","last_peered":"2024-02-10T07:44:38.790541+0000","last_clean":"2024-02-10T07:44:38.790541+0000","last_became_active":"2024-02-10T07:42:09.175005+0000","last_became_peered":"2024-02-10T07:42:09.175005+0000","last_unstale":"2024-02-10T07:44:38.790541+0000","last_undegraded":"2024-02-10T07:44:38.790541+0000","last_fullsized":"2024-02-10T07:44:38.790541+0000","mapping_epoch":253,"log_start":"270'4900","ondisk_log_start":"270'4900","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'7895","last_scrub_stamp":"2024-02-10T07:43:48.784116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:48.784116+0000","objects_scrubbed":7808,"log_size":6027,"log_dups_size":0,"ondisk_log_size":6027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:06.007154+0000","scrub_duration":2586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335740928,"num_objects":5123,"num_object_clones":0,"num_object_copies":15369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10927,"num_write_kb":513600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":486,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.970154+0000","last_change":"2024-02-10T07:43:41.991762+0000","last_active":"2024-02-10T07:44:29.970154+0000","last_peered":"2024-02-10T07:44:29.970154+0000","last_clean":"2024-02-10T07:44:29.970154+0000","last_became_active":"2024-02-10T07:31:57.313854+0000","last_became_peered":"2024-02-10T07:31:57.313854+0000","last_unstale":"2024-02-10T07:44:29.970154+0000","last_undegraded":"2024-02-10T07:44:29.970154+0000","last_fullsized":"2024-02-10T07:44:29.970154+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:41.991625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:43:41.991625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:51.962914+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":453,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.971763+0000","last_change":"2024-02-10T07:43:12.274076+0000","last_active":"2024-02-10T07:44:29.971763+0000","last_peered":"2024-02-10T07:44:29.971763+0000","last_clean":"2024-02-10T07:44:29.971763+0000","last_became_active":"2024-02-10T07:36:33.759656+0000","last_became_peered":"2024-02-10T07:36:33.759656+0000","last_unstale":"2024-02-10T07:44:29.971763+0000","last_undegraded":"2024-02-10T07:44:29.971763+0000","last_fullsized":"2024-02-10T07:44:29.971763+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:43:12.273960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:43:12.273960+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":507,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.988995+0000","last_change":"2024-02-10T07:42:02.993202+0000","last_active":"2024-02-10T07:44:29.988995+0000","last_peered":"2024-02-10T07:44:29.988995+0000","last_clean":"2024-02-10T07:44:29.988995+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:44:29.988995+0000","last_undegraded":"2024-02-10T07:44:29.988995+0000","last_fullsized":"2024-02-10T07:44:29.988995+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:02.993090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:42:02.993090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:33.290656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"250'40","reported_seq":729,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:31.527936+0000","last_change":"2024-02-10T07:44:06.496121+0000","last_active":"2024-02-10T07:44:31.527936+0000","last_peered":"2024-02-10T07:44:31.527936+0000","last_clean":"2024-02-10T07:44:31.527936+0000","last_became_active":"2024-02-10T07:31:53.301863+0000","last_became_peered":"2024-02-10T07:31:53.301863+0000","last_unstale":"2024-02-10T07:44:31.527936+0000","last_undegraded":"2024-02-10T07:44:31.527936+0000","last_fullsized":"2024-02-10T07:44:31.527936+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'40","last_scrub_stamp":"2024-02-10T07:44:06.495977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:44:06.495977+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:09.502391+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":514,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.980846+0000","last_change":"2024-02-10T07:44:04.180332+0000","last_active":"2024-02-10T07:44:29.980846+0000","last_peered":"2024-02-10T07:44:29.980846+0000","last_clean":"2024-02-10T07:44:29.980846+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:44:29.980846+0000","last_undegraded":"2024-02-10T07:44:29.980846+0000","last_fullsized":"2024-02-10T07:44:29.980846+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:04.179805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:22.786949+0000","last_clean_scrub_stamp":"2024-02-10T07:44:04.179805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:09.552301+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"292'10553","reported_seq":10641,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:37.110582+0000","last_change":"2024-02-10T07:42:09.176925+0000","last_active":"2024-02-10T07:44:37.110582+0000","last_peered":"2024-02-10T07:44:37.110582+0000","last_clean":"2024-02-10T07:44:37.110582+0000","last_became_active":"2024-02-10T07:42:09.176328+0000","last_became_peered":"2024-02-10T07:42:09.176328+0000","last_unstale":"2024-02-10T07:44:37.110582+0000","last_undegraded":"2024-02-10T07:44:37.110582+0000","last_fullsized":"2024-02-10T07:44:37.110582+0000","mapping_epoch":253,"log_start":"271'5200","ondisk_log_start":"271'5200","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:42:08.158083+0000","objects_scrubbed":0,"log_size":5353,"log_dups_size":0,"ondisk_log_size":5353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:37.299170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345702400,"num_objects":5275,"num_object_clones":0,"num_object_copies":15825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10553,"num_write_kb":506496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":503,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.979998+0000","last_change":"2024-02-10T07:43:14.465370+0000","last_active":"2024-02-10T07:44:29.979998+0000","last_peered":"2024-02-10T07:44:29.979998+0000","last_clean":"2024-02-10T07:44:29.979998+0000","last_became_active":"2024-02-10T07:31:57.299958+0000","last_became_peered":"2024-02-10T07:31:57.299958+0000","last_unstale":"2024-02-10T07:44:29.979998+0000","last_undegraded":"2024-02-10T07:44:29.979998+0000","last_fullsized":"2024-02-10T07:44:29.979998+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:14.465247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:42.730875+0000","last_clean_scrub_stamp":"2024-02-10T07:43:14.465247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:38.370079+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":513,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.981824+0000","last_change":"2024-02-10T07:43:08.955491+0000","last_active":"2024-02-10T07:44:29.981824+0000","last_peered":"2024-02-10T07:44:29.981824+0000","last_clean":"2024-02-10T07:44:29.981824+0000","last_became_active":"2024-02-10T07:31:51.320864+0000","last_became_peered":"2024-02-10T07:31:51.320864+0000","last_unstale":"2024-02-10T07:44:29.981824+0000","last_undegraded":"2024-02-10T07:44:29.981824+0000","last_fullsized":"2024-02-10T07:44:29.981824+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:08.955292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:08.955292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:30.882331+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.979487+0000","last_change":"2024-02-10T07:43:10.998646+0000","last_active":"2024-02-10T07:44:29.979487+0000","last_peered":"2024-02-10T07:44:29.979487+0000","last_clean":"2024-02-10T07:44:29.979487+0000","last_became_active":"2024-02-10T07:31:50.281517+0000","last_became_peered":"2024-02-10T07:31:50.281517+0000","last_unstale":"2024-02-10T07:44:29.979487+0000","last_undegraded":"2024-02-10T07:44:29.979487+0000","last_fullsized":"2024-02-10T07:44:29.979487+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:10.998492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:43:10.998492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:32.956556+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"250'17","reported_seq":302,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.984730+0000","last_change":"2024-02-10T07:43:15.107129+0000","last_active":"2024-02-10T07:44:29.984730+0000","last_peered":"2024-02-10T07:44:29.984730+0000","last_clean":"2024-02-10T07:44:29.984730+0000","last_became_active":"2024-02-10T07:37:06.196787+0000","last_became_peered":"2024-02-10T07:37:06.196787+0000","last_unstale":"2024-02-10T07:44:29.984730+0000","last_undegraded":"2024-02-10T07:44:29.984730+0000","last_fullsized":"2024-02-10T07:44:29.984730+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'17","last_scrub_stamp":"2024-02-10T07:43:15.106978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:15.106978+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:40.379951+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":517,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.979971+0000","last_change":"2024-02-10T07:43:04.918865+0000","last_active":"2024-02-10T07:44:29.979971+0000","last_peered":"2024-02-10T07:44:29.979971+0000","last_clean":"2024-02-10T07:44:29.979971+0000","last_became_active":"2024-02-10T07:36:33.751551+0000","last_became_peered":"2024-02-10T07:36:33.751551+0000","last_unstale":"2024-02-10T07:44:29.979971+0000","last_undegraded":"2024-02-10T07:44:29.979971+0000","last_fullsized":"2024-02-10T07:44:29.979971+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:04.918467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:04.918467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:34.367945+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"292'10952","reported_seq":11126,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:39.314261+0000","last_change":"2024-02-10T07:43:59.609577+0000","last_active":"2024-02-10T07:44:39.314261+0000","last_peered":"2024-02-10T07:44:39.314261+0000","last_clean":"2024-02-10T07:44:39.314261+0000","last_became_active":"2024-02-10T07:42:09.176267+0000","last_became_peered":"2024-02-10T07:42:09.176267+0000","last_unstale":"2024-02-10T07:44:39.314261+0000","last_undegraded":"2024-02-10T07:44:39.314261+0000","last_fullsized":"2024-02-10T07:44:39.314261+0000","mapping_epoch":253,"log_start":"274'6100","ondisk_log_start":"274'6100","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'8548","last_scrub_stamp":"2024-02-10T07:43:59.609439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:59.609439+0000","objects_scrubbed":7542,"log_size":4852,"log_dups_size":0,"ondisk_log_size":4852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:03.912783+0000","scrub_duration":2815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332529664,"num_objects":5074,"num_object_clones":0,"num_object_copies":15222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10952,"num_write_kb":512832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":458,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.972711+0000","last_change":"2024-02-10T07:42:17.190620+0000","last_active":"2024-02-10T07:44:29.972711+0000","last_peered":"2024-02-10T07:44:29.972711+0000","last_clean":"2024-02-10T07:44:29.972711+0000","last_became_active":"2024-02-10T07:36:33.769763+0000","last_became_peered":"2024-02-10T07:36:33.769763+0000","last_unstale":"2024-02-10T07:44:29.972711+0000","last_undegraded":"2024-02-10T07:44:29.972711+0000","last_fullsized":"2024-02-10T07:44:29.972711+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_clean_scrub_stamp":"2024-02-10T07:42:17.190492+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:31.184096+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":760,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.973178+0000","last_change":"2024-02-10T07:43:13.432811+0000","last_active":"2024-02-10T07:44:29.973178+0000","last_peered":"2024-02-10T07:44:29.973178+0000","last_clean":"2024-02-10T07:44:29.973178+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:44:29.973178+0000","last_undegraded":"2024-02-10T07:44:29.973178+0000","last_fullsized":"2024-02-10T07:44:29.973178+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:43:13.432680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:43:13.432680+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:32.454909+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":508,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.970610+0000","last_change":"2024-02-10T07:43:00.146103+0000","last_active":"2024-02-10T07:44:29.970610+0000","last_peered":"2024-02-10T07:44:29.970610+0000","last_clean":"2024-02-10T07:44:29.970610+0000","last_became_active":"2024-02-10T07:36:33.752703+0000","last_became_peered":"2024-02-10T07:36:33.752703+0000","last_unstale":"2024-02-10T07:44:29.970610+0000","last_undegraded":"2024-02-10T07:44:29.970610+0000","last_fullsized":"2024-02-10T07:44:29.970610+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:00.145780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:00.145780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"250'36","reported_seq":720,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:31.522639+0000","last_change":"2024-02-10T07:43:57.949399+0000","last_active":"2024-02-10T07:44:31.522639+0000","last_peered":"2024-02-10T07:44:31.522639+0000","last_clean":"2024-02-10T07:44:31.522639+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:44:31.522639+0000","last_undegraded":"2024-02-10T07:44:31.522639+0000","last_fullsized":"2024-02-10T07:44:31.522639+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'36","last_scrub_stamp":"2024-02-10T07:43:57.949264+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:43:57.949264+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:04.425290+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":516,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.974134+0000","last_change":"2024-02-10T07:43:39.433253+0000","last_active":"2024-02-10T07:44:29.974134+0000","last_peered":"2024-02-10T07:44:29.974134+0000","last_clean":"2024-02-10T07:44:29.974134+0000","last_became_active":"2024-02-10T07:36:33.752711+0000","last_became_peered":"2024-02-10T07:36:33.752711+0000","last_unstale":"2024-02-10T07:44:29.974134+0000","last_undegraded":"2024-02-10T07:44:29.974134+0000","last_fullsized":"2024-02-10T07:44:29.974134+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:39.433049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.736901+0000","last_clean_scrub_stamp":"2024-02-10T07:43:39.433049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:48.100463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"292'10979","reported_seq":11151,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:38.803111+0000","last_change":"2024-02-10T07:44:02.098658+0000","last_active":"2024-02-10T07:44:38.803111+0000","last_peered":"2024-02-10T07:44:38.803111+0000","last_clean":"2024-02-10T07:44:38.803111+0000","last_became_active":"2024-02-10T07:42:09.176217+0000","last_became_peered":"2024-02-10T07:42:09.176217+0000","last_unstale":"2024-02-10T07:44:38.803111+0000","last_undegraded":"2024-02-10T07:44:38.803111+0000","last_fullsized":"2024-02-10T07:44:38.803111+0000","mapping_epoch":253,"log_start":"270'4900","ondisk_log_start":"270'4900","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'8727","last_scrub_stamp":"2024-02-10T07:44:02.098377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:44:02.098377+0000","objects_scrubbed":7438,"log_size":6079,"log_dups_size":0,"ondisk_log_size":6079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:22.096752+0000","scrub_duration":2893,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335609856,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10979,"num_write_kb":515200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":284,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.979385+0000","last_change":"2024-02-10T07:43:02.043769+0000","last_active":"2024-02-10T07:44:29.979385+0000","last_peered":"2024-02-10T07:44:29.979385+0000","last_clean":"2024-02-10T07:44:29.979385+0000","last_became_active":"2024-02-10T07:37:06.202738+0000","last_became_peered":"2024-02-10T07:37:06.202738+0000","last_unstale":"2024-02-10T07:44:29.979385+0000","last_undegraded":"2024-02-10T07:44:29.979385+0000","last_fullsized":"2024-02-10T07:44:29.979385+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:02.043646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:02.043646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:30.882294+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":514,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.980457+0000","last_change":"2024-02-10T07:44:07.146438+0000","last_active":"2024-02-10T07:44:29.980457+0000","last_peered":"2024-02-10T07:44:29.980457+0000","last_clean":"2024-02-10T07:44:29.980457+0000","last_became_active":"2024-02-10T07:31:51.286076+0000","last_became_peered":"2024-02-10T07:31:51.286076+0000","last_unstale":"2024-02-10T07:44:29.980457+0000","last_undegraded":"2024-02-10T07:44:29.980457+0000","last_fullsized":"2024-02-10T07:44:29.980457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:07.146214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:37.574311+0000","last_clean_scrub_stamp":"2024-02-10T07:44:07.146214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:34.549494+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1287,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:36.257097+0000","last_change":"2024-02-10T07:43:15.404193+0000","last_active":"2024-02-10T07:44:36.257097+0000","last_peered":"2024-02-10T07:44:36.257097+0000","last_clean":"2024-02-10T07:44:36.257097+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:44:36.257097+0000","last_undegraded":"2024-02-10T07:44:36.257097+0000","last_fullsized":"2024-02-10T07:44:36.257097+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:43:15.404111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:43:15.404111+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:37.569877+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":760,"num_read_kb":755,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"250'30","reported_seq":543,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.980721+0000","last_change":"2024-02-10T07:42:20.822551+0000","last_active":"2024-02-10T07:44:29.980721+0000","last_peered":"2024-02-10T07:44:29.980721+0000","last_clean":"2024-02-10T07:44:29.980721+0000","last_became_active":"2024-02-10T07:31:53.300687+0000","last_became_peered":"2024-02-10T07:31:53.300687+0000","last_unstale":"2024-02-10T07:44:29.980721+0000","last_undegraded":"2024-02-10T07:44:29.980721+0000","last_fullsized":"2024-02-10T07:44:29.980721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:42:20.822401+0000","last_deep_scrub":"237'18","last_deep_scrub_stamp":"2024-02-10T07:40:31.005376+0000","last_clean_scrub_stamp":"2024-02-10T07:42:20.822401+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:34.341927+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":452,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.972848+0000","last_change":"2024-02-10T07:42:16.148432+0000","last_active":"2024-02-10T07:44:29.972848+0000","last_peered":"2024-02-10T07:44:29.972848+0000","last_clean":"2024-02-10T07:44:29.972848+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:44:29.972848+0000","last_undegraded":"2024-02-10T07:44:29.972848+0000","last_fullsized":"2024-02-10T07:44:29.972848+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:16.148315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:42:16.148315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:31.184165+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"292'10833","reported_seq":11006,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:37.917753+0000","last_change":"2024-02-10T07:43:44.092163+0000","last_active":"2024-02-10T07:44:37.917753+0000","last_peered":"2024-02-10T07:44:37.917753+0000","last_clean":"2024-02-10T07:44:37.917753+0000","last_became_active":"2024-02-10T07:42:09.450389+0000","last_became_peered":"2024-02-10T07:42:09.450389+0000","last_unstale":"2024-02-10T07:44:37.917753+0000","last_undegraded":"2024-02-10T07:44:37.917753+0000","last_fullsized":"2024-02-10T07:44:37.917753+0000","mapping_epoch":253,"log_start":"272'5900","ondisk_log_start":"272'5900","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'7600","last_scrub_stamp":"2024-02-10T07:43:44.092031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:44.092031+0000","objects_scrubbed":7513,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:52.678333+0000","scrub_duration":2666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345440256,"num_objects":5271,"num_object_clones":0,"num_object_copies":15813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10833,"num_write_kb":515328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":501,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.981301+0000","last_change":"2024-02-10T07:43:40.151895+0000","last_active":"2024-02-10T07:44:29.981301+0000","last_peered":"2024-02-10T07:44:29.981301+0000","last_clean":"2024-02-10T07:44:29.981301+0000","last_became_active":"2024-02-10T07:31:57.307615+0000","last_became_peered":"2024-02-10T07:31:57.307615+0000","last_unstale":"2024-02-10T07:44:29.981301+0000","last_undegraded":"2024-02-10T07:44:29.981301+0000","last_fullsized":"2024-02-10T07:44:29.981301+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:40.151648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:40.151648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:54.107957+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":522,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.979632+0000","last_change":"2024-02-10T07:43:47.219926+0000","last_active":"2024-02-10T07:44:29.979632+0000","last_peered":"2024-02-10T07:44:29.979632+0000","last_clean":"2024-02-10T07:44:29.979632+0000","last_became_active":"2024-02-10T07:36:33.750867+0000","last_became_peered":"2024-02-10T07:36:33.750867+0000","last_unstale":"2024-02-10T07:44:29.979632+0000","last_undegraded":"2024-02-10T07:44:29.979632+0000","last_fullsized":"2024-02-10T07:44:29.979632+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:47.219800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:43:47.219800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:16.514612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"271'37","reported_seq":570,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.980821+0000","last_change":"2024-02-10T07:43:10.047505+0000","last_active":"2024-02-10T07:44:29.980821+0000","last_peered":"2024-02-10T07:44:29.980821+0000","last_clean":"2024-02-10T07:44:29.980821+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:44:29.980821+0000","last_undegraded":"2024-02-10T07:44:29.980821+0000","last_fullsized":"2024-02-10T07:44:29.980821+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'37","last_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_deep_scrub":"271'37","last_deep_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_clean_scrub_stamp":"2024-02-10T07:43:10.047405+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:34.342008+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":523,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.983487+0000","last_change":"2024-02-10T07:43:08.422662+0000","last_active":"2024-02-10T07:44:29.983487+0000","last_peered":"2024-02-10T07:44:29.983487+0000","last_clean":"2024-02-10T07:44:29.983487+0000","last_became_active":"2024-02-10T07:36:33.765998+0000","last_became_peered":"2024-02-10T07:36:33.765998+0000","last_unstale":"2024-02-10T07:44:29.983487+0000","last_undegraded":"2024-02-10T07:44:29.983487+0000","last_fullsized":"2024-02-10T07:44:29.983487+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:08.422557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:08.422557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:30.401789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"292'13673","reported_seq":13884,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:37.095026+0000","last_change":"2024-02-10T07:44:28.091404+0000","last_active":"2024-02-10T07:44:37.095026+0000","last_peered":"2024-02-10T07:44:37.095026+0000","last_clean":"2024-02-10T07:44:37.095026+0000","last_became_active":"2024-02-10T07:42:09.176384+0000","last_became_peered":"2024-02-10T07:42:09.176384+0000","last_unstale":"2024-02-10T07:44:37.095026+0000","last_undegraded":"2024-02-10T07:44:37.095026+0000","last_fullsized":"2024-02-10T07:44:37.095026+0000","mapping_epoch":253,"log_start":"281'8403","ondisk_log_start":"281'8403","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:42:08.158083+0000","objects_scrubbed":2936,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:40.300727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348651520,"num_objects":5320,"num_object_clones":0,"num_object_copies":15960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10806,"num_write_kb":516032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":389,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.983160+0000","last_change":"2024-02-10T07:43:59.970168+0000","last_active":"2024-02-10T07:44:29.983160+0000","last_peered":"2024-02-10T07:44:29.983160+0000","last_clean":"2024-02-10T07:44:29.983160+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:44:29.983160+0000","last_undegraded":"2024-02-10T07:44:29.983160+0000","last_fullsized":"2024-02-10T07:44:29.983160+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:59.970065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:59.970065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:07.935598+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":474,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.977030+0000","last_change":"2024-02-10T07:43:14.153897+0000","last_active":"2024-02-10T07:44:29.977030+0000","last_peered":"2024-02-10T07:44:29.977030+0000","last_clean":"2024-02-10T07:44:29.977030+0000","last_became_active":"2024-02-10T07:36:33.779820+0000","last_became_peered":"2024-02-10T07:36:33.779820+0000","last_unstale":"2024-02-10T07:44:29.977030+0000","last_undegraded":"2024-02-10T07:44:29.977030+0000","last_fullsized":"2024-02-10T07:44:29.977030+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:14.153641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:14.153641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:39.040931+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"250'20","reported_seq":319,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.979706+0000","last_change":"2024-02-10T07:44:01.357558+0000","last_active":"2024-02-10T07:44:29.979706+0000","last_peered":"2024-02-10T07:44:29.979706+0000","last_clean":"2024-02-10T07:44:29.979706+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:44:29.979706+0000","last_undegraded":"2024-02-10T07:44:29.979706+0000","last_fullsized":"2024-02-10T07:44:29.979706+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:44:01.357268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:44:01.357268+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:29.526294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"158'10","reported_seq":678,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:31.525356+0000","last_change":"2024-02-10T07:44:09.183908+0000","last_active":"2024-02-10T07:44:31.525356+0000","last_peered":"2024-02-10T07:44:31.525356+0000","last_clean":"2024-02-10T07:44:31.525356+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:44:31.525356+0000","last_undegraded":"2024-02-10T07:44:31.525356+0000","last_fullsized":"2024-02-10T07:44:31.525356+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:44:09.183790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:44:09.183790+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:10.060743+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"292'10873","reported_seq":11043,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:39.309585+0000","last_change":"2024-02-10T07:44:00.802043+0000","last_active":"2024-02-10T07:44:39.309585+0000","last_peered":"2024-02-10T07:44:39.309585+0000","last_clean":"2024-02-10T07:44:39.309585+0000","last_became_active":"2024-02-10T07:42:09.450743+0000","last_became_peered":"2024-02-10T07:42:09.450743+0000","last_unstale":"2024-02-10T07:44:39.309585+0000","last_undegraded":"2024-02-10T07:44:39.309585+0000","last_fullsized":"2024-02-10T07:44:39.309585+0000","mapping_epoch":253,"log_start":"274'6000","ondisk_log_start":"274'6000","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'8606","last_scrub_stamp":"2024-02-10T07:44:00.801833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:44:00.801833+0000","objects_scrubbed":7458,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:26.243602+0000","scrub_duration":3048,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333643776,"num_objects":5091,"num_object_clones":0,"num_object_copies":15273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10873,"num_write_kb":510848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":503,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.973802+0000","last_change":"2024-02-10T07:43:45.565156+0000","last_active":"2024-02-10T07:44:29.973802+0000","last_peered":"2024-02-10T07:44:29.973802+0000","last_clean":"2024-02-10T07:44:29.973802+0000","last_became_active":"2024-02-10T07:31:57.305620+0000","last_became_peered":"2024-02-10T07:31:57.305620+0000","last_unstale":"2024-02-10T07:44:29.973802+0000","last_undegraded":"2024-02-10T07:44:29.973802+0000","last_fullsized":"2024-02-10T07:44:29.973802+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:45.565037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:45.565037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:53.867913+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":514,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.980815+0000","last_change":"2024-02-10T07:43:05.083484+0000","last_active":"2024-02-10T07:44:29.980815+0000","last_peered":"2024-02-10T07:44:29.980815+0000","last_clean":"2024-02-10T07:44:29.980815+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:44:29.980815+0000","last_undegraded":"2024-02-10T07:44:29.980815+0000","last_fullsized":"2024-02-10T07:44:29.980815+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:43:05.083356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:05.083356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:31.346846+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"250'28","reported_seq":554,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.988069+0000","last_change":"2024-02-10T07:43:11.452648+0000","last_active":"2024-02-10T07:44:29.988069+0000","last_peered":"2024-02-10T07:44:29.988069+0000","last_clean":"2024-02-10T07:44:29.988069+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:44:29.988069+0000","last_undegraded":"2024-02-10T07:44:29.988069+0000","last_fullsized":"2024-02-10T07:44:29.988069+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-02-10T07:43:11.452502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:11.452502+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:33.290789+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"227'9","reported_seq":662,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:31.523356+0000","last_change":"2024-02-10T07:43:14.988651+0000","last_active":"2024-02-10T07:44:31.523356+0000","last_peered":"2024-02-10T07:44:31.523356+0000","last_clean":"2024-02-10T07:44:31.523356+0000","last_became_active":"2024-02-10T07:31:55.302863+0000","last_became_peered":"2024-02-10T07:31:55.302863+0000","last_unstale":"2024-02-10T07:44:31.523356+0000","last_undegraded":"2024-02-10T07:44:31.523356+0000","last_fullsized":"2024-02-10T07:44:31.523356+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'9","last_scrub_stamp":"2024-02-10T07:43:14.987456+0000","last_deep_scrub":"227'9","last_deep_scrub_stamp":"2024-02-10T07:39:53.440221+0000","last_clean_scrub_stamp":"2024-02-10T07:43:14.987456+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:41.046267+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"292'10748","reported_seq":10921,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:37.935603+0000","last_change":"2024-02-10T07:43:48.708669+0000","last_active":"2024-02-10T07:44:37.935603+0000","last_peered":"2024-02-10T07:44:37.935603+0000","last_clean":"2024-02-10T07:44:37.935603+0000","last_became_active":"2024-02-10T07:42:09.175677+0000","last_became_peered":"2024-02-10T07:42:09.175677+0000","last_unstale":"2024-02-10T07:44:37.935603+0000","last_undegraded":"2024-02-10T07:44:37.935603+0000","last_fullsized":"2024-02-10T07:44:37.935603+0000","mapping_epoch":253,"log_start":"273'5900","ondisk_log_start":"273'5900","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'7841","last_scrub_stamp":"2024-02-10T07:43:48.708425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:48.708425+0000","objects_scrubbed":7768,"log_size":4848,"log_dups_size":0,"ondisk_log_size":4848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:14.910084+0000","scrub_duration":2197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342294552,"num_objects":5224,"num_object_clones":0,"num_object_copies":15672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5224,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10748,"num_write_kb":511041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":495,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.988473+0000","last_change":"2024-02-10T07:43:43.422664+0000","last_active":"2024-02-10T07:44:29.988473+0000","last_peered":"2024-02-10T07:44:29.988473+0000","last_clean":"2024-02-10T07:44:29.988473+0000","last_became_active":"2024-02-10T07:31:57.316047+0000","last_became_peered":"2024-02-10T07:31:57.316047+0000","last_unstale":"2024-02-10T07:44:29.988473+0000","last_undegraded":"2024-02-10T07:44:29.988473+0000","last_fullsized":"2024-02-10T07:44:29.988473+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:43.422546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:43:43.422546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:46.243272+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":506,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.971438+0000","last_change":"2024-02-10T07:44:06.020845+0000","last_active":"2024-02-10T07:44:29.971438+0000","last_peered":"2024-02-10T07:44:29.971438+0000","last_clean":"2024-02-10T07:44:29.971438+0000","last_became_active":"2024-02-10T07:31:51.285333+0000","last_became_peered":"2024-02-10T07:31:51.285333+0000","last_unstale":"2024-02-10T07:44:29.971438+0000","last_undegraded":"2024-02-10T07:44:29.971438+0000","last_fullsized":"2024-02-10T07:44:29.971438+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:06.020588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:44:06.020588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:21.600571+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"250'30","reported_seq":553,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.969863+0000","last_change":"2024-02-10T07:44:08.032219+0000","last_active":"2024-02-10T07:44:29.969863+0000","last_peered":"2024-02-10T07:44:29.969863+0000","last_clean":"2024-02-10T07:44:29.969863+0000","last_became_active":"2024-02-10T07:31:53.290157+0000","last_became_peered":"2024-02-10T07:31:53.290157+0000","last_unstale":"2024-02-10T07:44:29.969863+0000","last_undegraded":"2024-02-10T07:44:29.969863+0000","last_fullsized":"2024-02-10T07:44:29.969863+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:44:08.032066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:44:08.032066+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:35.903498+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":507,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.987875+0000","last_change":"2024-02-10T07:43:47.483128+0000","last_active":"2024-02-10T07:44:29.987875+0000","last_peered":"2024-02-10T07:44:29.987875+0000","last_clean":"2024-02-10T07:44:29.987875+0000","last_became_active":"2024-02-10T07:31:55.295900+0000","last_became_peered":"2024-02-10T07:31:55.295900+0000","last_unstale":"2024-02-10T07:44:29.987875+0000","last_undegraded":"2024-02-10T07:44:29.987875+0000","last_fullsized":"2024-02-10T07:44:29.987875+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:47.483009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:47.483009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:53.829855+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"292'10994","reported_seq":11170,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:39.315174+0000","last_change":"2024-02-10T07:43:55.810600+0000","last_active":"2024-02-10T07:44:39.315174+0000","last_peered":"2024-02-10T07:44:39.315174+0000","last_clean":"2024-02-10T07:44:39.315174+0000","last_became_active":"2024-02-10T07:42:09.395607+0000","last_became_peered":"2024-02-10T07:42:09.395607+0000","last_unstale":"2024-02-10T07:44:39.315174+0000","last_undegraded":"2024-02-10T07:44:39.315174+0000","last_fullsized":"2024-02-10T07:44:39.315174+0000","mapping_epoch":253,"log_start":"273'6100","ondisk_log_start":"273'6100","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'8402","last_scrub_stamp":"2024-02-10T07:43:55.810372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:55.810372+0000","objects_scrubbed":7820,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:14.377222+0000","scrub_duration":3066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335806464,"num_objects":5124,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10994,"num_write_kb":515776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":452,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.977113+0000","last_change":"2024-02-10T07:43:06.325383+0000","last_active":"2024-02-10T07:44:29.977113+0000","last_peered":"2024-02-10T07:44:29.977113+0000","last_clean":"2024-02-10T07:44:29.977113+0000","last_became_active":"2024-02-10T07:36:33.772094+0000","last_became_peered":"2024-02-10T07:36:33.772094+0000","last_unstale":"2024-02-10T07:44:29.977113+0000","last_undegraded":"2024-02-10T07:44:29.977113+0000","last_fullsized":"2024-02-10T07:44:29.977113+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:06.325264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:49.704024+0000","last_clean_scrub_stamp":"2024-02-10T07:43:06.325264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:35.928074+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"158'10","reported_seq":670,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:31.522873+0000","last_change":"2024-02-10T07:42:17.097563+0000","last_active":"2024-02-10T07:44:31.522873+0000","last_peered":"2024-02-10T07:44:31.522873+0000","last_clean":"2024-02-10T07:44:31.522873+0000","last_became_active":"2024-02-10T07:36:33.755111+0000","last_became_peered":"2024-02-10T07:36:33.755111+0000","last_unstale":"2024-02-10T07:44:31.522873+0000","last_undegraded":"2024-02-10T07:44:31.522873+0000","last_fullsized":"2024-02-10T07:44:31.522873+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:42:17.097480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:42:17.097480+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:39.174185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":498,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.969765+0000","last_change":"2024-02-10T07:42:23.152250+0000","last_active":"2024-02-10T07:44:29.969765+0000","last_peered":"2024-02-10T07:44:29.969765+0000","last_clean":"2024-02-10T07:44:29.969765+0000","last_became_active":"2024-02-10T07:31:51.282137+0000","last_became_peered":"2024-02-10T07:31:51.282137+0000","last_unstale":"2024-02-10T07:44:29.969765+0000","last_undegraded":"2024-02-10T07:44:29.969765+0000","last_fullsized":"2024-02-10T07:44:29.969765+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:23.152057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:42:23.152057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"250'27","reported_seq":559,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.984412+0000","last_change":"2024-02-10T07:43:58.327164+0000","last_active":"2024-02-10T07:44:29.984412+0000","last_peered":"2024-02-10T07:44:29.984412+0000","last_clean":"2024-02-10T07:44:29.984412+0000","last_became_active":"2024-02-10T07:31:53.294218+0000","last_became_peered":"2024-02-10T07:31:53.294218+0000","last_unstale":"2024-02-10T07:44:29.984412+0000","last_undegraded":"2024-02-10T07:44:29.984412+0000","last_fullsized":"2024-02-10T07:44:29.984412+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:43:58.327027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:58.327027+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:12.111184+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"292'10810","reported_seq":10983,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:37.906727+0000","last_change":"2024-02-10T07:43:45.681822+0000","last_active":"2024-02-10T07:44:37.906727+0000","last_peered":"2024-02-10T07:44:37.906727+0000","last_clean":"2024-02-10T07:44:37.906727+0000","last_became_active":"2024-02-10T07:42:09.174538+0000","last_became_peered":"2024-02-10T07:42:09.174538+0000","last_unstale":"2024-02-10T07:44:37.906727+0000","last_undegraded":"2024-02-10T07:44:37.906727+0000","last_fullsized":"2024-02-10T07:44:37.906727+0000","mapping_epoch":253,"log_start":"272'5900","ondisk_log_start":"272'5900","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'7651","last_scrub_stamp":"2024-02-10T07:43:45.681682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:45.681682+0000","objects_scrubbed":7571,"log_size":4910,"log_dups_size":0,"ondisk_log_size":4910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:00.433446+0000","scrub_duration":2221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342753280,"num_objects":5230,"num_object_clones":0,"num_object_copies":15690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10810,"num_write_kb":513280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":503,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.970050+0000","last_change":"2024-02-10T07:43:19.934726+0000","last_active":"2024-02-10T07:44:29.970050+0000","last_peered":"2024-02-10T07:44:29.970050+0000","last_clean":"2024-02-10T07:44:29.970050+0000","last_became_active":"2024-02-10T07:36:33.753921+0000","last_became_peered":"2024-02-10T07:36:33.753921+0000","last_unstale":"2024-02-10T07:44:29.970050+0000","last_undegraded":"2024-02-10T07:44:29.970050+0000","last_fullsized":"2024-02-10T07:44:29.970050+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:19.934260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_clean_scrub_stamp":"2024-02-10T07:43:19.934260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:43.253214+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":434,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.972658+0000","last_change":"2024-02-10T07:43:52.274357+0000","last_active":"2024-02-10T07:44:29.972658+0000","last_peered":"2024-02-10T07:44:29.972658+0000","last_clean":"2024-02-10T07:44:29.972658+0000","last_became_active":"2024-02-10T07:36:33.759566+0000","last_became_peered":"2024-02-10T07:36:33.759566+0000","last_unstale":"2024-02-10T07:44:29.972658+0000","last_undegraded":"2024-02-10T07:44:29.972658+0000","last_fullsized":"2024-02-10T07:44:29.972658+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:52.274260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:52.274260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:54.593440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":514,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.987946+0000","last_change":"2024-02-10T07:44:08.363775+0000","last_active":"2024-02-10T07:44:29.987946+0000","last_peered":"2024-02-10T07:44:29.987946+0000","last_clean":"2024-02-10T07:44:29.987946+0000","last_became_active":"2024-02-10T07:31:51.320715+0000","last_became_peered":"2024-02-10T07:31:51.320715+0000","last_unstale":"2024-02-10T07:44:29.987946+0000","last_undegraded":"2024-02-10T07:44:29.987946+0000","last_fullsized":"2024-02-10T07:44:29.987946+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:08.363673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:44:08.363673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:36.016926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"250'24","reported_seq":574,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.990489+0000","last_change":"2024-02-10T07:42:11.104029+0000","last_active":"2024-02-10T07:44:29.990489+0000","last_peered":"2024-02-10T07:44:29.990489+0000","last_clean":"2024-02-10T07:44:29.990489+0000","last_became_active":"2024-02-10T07:37:06.193386+0000","last_became_peered":"2024-02-10T07:37:06.193386+0000","last_unstale":"2024-02-10T07:44:29.990489+0000","last_undegraded":"2024-02-10T07:44:29.990489+0000","last_fullsized":"2024-02-10T07:44:29.990489+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:42:11.103919+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:42:11.103919+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:33.290689+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"292'16185","reported_seq":16518,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:38.898377+0000","last_change":"2024-02-10T07:44:28.084232+0000","last_active":"2024-02-10T07:44:38.898377+0000","last_peered":"2024-02-10T07:44:38.898377+0000","last_clean":"2024-02-10T07:44:38.898377+0000","last_became_active":"2024-02-10T07:42:09.394893+0000","last_became_peered":"2024-02-10T07:42:09.394893+0000","last_unstale":"2024-02-10T07:44:38.898377+0000","last_undegraded":"2024-02-10T07:44:38.898377+0000","last_fullsized":"2024-02-10T07:44:38.898377+0000","mapping_epoch":253,"log_start":"287'11402","ondisk_log_start":"287'11402","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:42:08.158083+0000","objects_scrubbed":5960,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:42.157264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332005376,"num_objects":5066,"num_object_clones":0,"num_object_copies":15198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10728,"num_write_kb":505408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":509,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.981270+0000","last_change":"2024-02-10T07:43:59.300606+0000","last_active":"2024-02-10T07:44:29.981270+0000","last_peered":"2024-02-10T07:44:29.981270+0000","last_clean":"2024-02-10T07:44:29.981270+0000","last_became_active":"2024-02-10T07:31:57.311427+0000","last_became_peered":"2024-02-10T07:31:57.311427+0000","last_unstale":"2024-02-10T07:44:29.981270+0000","last_undegraded":"2024-02-10T07:44:29.981270+0000","last_fullsized":"2024-02-10T07:44:29.981270+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:59.300487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:43:59.300487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:16.206253+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":504,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.979810+0000","last_change":"2024-02-10T07:44:03.427446+0000","last_active":"2024-02-10T07:44:29.979810+0000","last_peered":"2024-02-10T07:44:29.979810+0000","last_clean":"2024-02-10T07:44:29.979810+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:44:29.979810+0000","last_undegraded":"2024-02-10T07:44:29.979810+0000","last_fullsized":"2024-02-10T07:44:29.979810+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:03.427333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:44:03.427333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:16.605118+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":529,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.983577+0000","last_change":"2024-02-10T07:43:51.058868+0000","last_active":"2024-02-10T07:44:29.983577+0000","last_peered":"2024-02-10T07:44:29.983577+0000","last_clean":"2024-02-10T07:44:29.983577+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:44:29.983577+0000","last_undegraded":"2024-02-10T07:44:29.983577+0000","last_fullsized":"2024-02-10T07:44:29.983577+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:51.058792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:44.067302+0000","last_clean_scrub_stamp":"2024-02-10T07:43:51.058792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:08.489187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"250'39","reported_seq":559,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.989872+0000","last_change":"2024-02-10T07:41:25.892072+0000","last_active":"2024-02-10T07:44:29.989872+0000","last_peered":"2024-02-10T07:44:29.989872+0000","last_clean":"2024-02-10T07:44:29.989872+0000","last_became_active":"2024-02-10T07:31:53.292277+0000","last_became_peered":"2024-02-10T07:31:53.292277+0000","last_unstale":"2024-02-10T07:44:29.989872+0000","last_undegraded":"2024-02-10T07:44:29.989872+0000","last_fullsized":"2024-02-10T07:44:29.989872+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'25","last_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_clean_scrub_stamp":"2024-02-10T07:41:25.891983+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:33.290536+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"292'10800","reported_seq":10975,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:39.286878+0000","last_change":"2024-02-10T07:43:53.588670+0000","last_active":"2024-02-10T07:44:39.286878+0000","last_peered":"2024-02-10T07:44:39.286878+0000","last_clean":"2024-02-10T07:44:39.286878+0000","last_became_active":"2024-02-10T07:42:09.170186+0000","last_became_peered":"2024-02-10T07:42:09.170186+0000","last_unstale":"2024-02-10T07:44:39.286878+0000","last_undegraded":"2024-02-10T07:44:39.286878+0000","last_fullsized":"2024-02-10T07:44:39.286878+0000","mapping_epoch":253,"log_start":"272'5900","ondisk_log_start":"272'5900","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'8104","last_scrub_stamp":"2024-02-10T07:43:53.588443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:53.588443+0000","objects_scrubbed":7832,"log_size":4900,"log_dups_size":0,"ondisk_log_size":4900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:57.545210+0000","scrub_duration":2816,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329777152,"num_objects":5032,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10800,"num_write_kb":506624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":494,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.995187+0000","last_change":"2024-02-10T07:43:10.479711+0000","last_active":"2024-02-10T07:44:29.995187+0000","last_peered":"2024-02-10T07:44:29.995187+0000","last_clean":"2024-02-10T07:44:29.995187+0000","last_became_active":"2024-02-10T07:31:57.308608+0000","last_became_peered":"2024-02-10T07:31:57.308608+0000","last_unstale":"2024-02-10T07:44:29.995187+0000","last_undegraded":"2024-02-10T07:44:29.995187+0000","last_fullsized":"2024-02-10T07:44:29.995187+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:10.479470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:10.479470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"158'10","reported_seq":680,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:31.522665+0000","last_change":"2024-02-10T07:43:07.964022+0000","last_active":"2024-02-10T07:44:31.522665+0000","last_peered":"2024-02-10T07:44:31.522665+0000","last_clean":"2024-02-10T07:44:31.522665+0000","last_became_active":"2024-02-10T07:36:33.776454+0000","last_became_peered":"2024-02-10T07:36:33.776454+0000","last_unstale":"2024-02-10T07:44:31.522665+0000","last_undegraded":"2024-02-10T07:44:31.522665+0000","last_fullsized":"2024-02-10T07:44:31.522665+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:43:07.963897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:07.963897+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:38.910891+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":498,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.969970+0000","last_change":"2024-02-10T07:43:09.950745+0000","last_active":"2024-02-10T07:44:29.969970+0000","last_peered":"2024-02-10T07:44:29.969970+0000","last_clean":"2024-02-10T07:44:29.969970+0000","last_became_active":"2024-02-10T07:31:51.288197+0000","last_became_peered":"2024-02-10T07:31:51.288197+0000","last_unstale":"2024-02-10T07:44:29.969970+0000","last_undegraded":"2024-02-10T07:44:29.969970+0000","last_fullsized":"2024-02-10T07:44:29.969970+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_clean_scrub_stamp":"2024-02-10T07:43:09.950640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:33.135842+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"250'20","reported_seq":532,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-02-10T07:44:29.992159+0000","last_change":"2024-02-10T07:42:10.093762+0000","last_active":"2024-02-10T07:44:29.992159+0000","last_peered":"2024-02-10T07:44:29.992159+0000","last_clean":"2024-02-10T07:44:29.992159+0000","last_became_active":"2024-02-10T07:31:53.287984+0000","last_became_peered":"2024-02-10T07:31:53.287984+0000","last_unstale":"2024-02-10T07:44:29.992159+0000","last_undegraded":"2024-02-10T07:44:29.992159+0000","last_fullsized":"2024-02-10T07:44:29.992159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:42:10.093651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:42:10.093651+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:33.290593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5424939032,"num_objects":82779,"num_object_clones":0,"num_object_copies":248337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173571,"num_write_kb":8203137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16292327424,"data_stored":16292315208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83290,"ondisk_log_size":83290,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":760,"num_read_kb":755,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1122,"num_read_kb":879,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":762,"ondisk_log_size":762,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037918,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3480472,"kb_used_data":3021712,"kb_used_omap":1,"kb_used_meta":458750,"kb_avail":90264680,"statfs":{"total":95995035648,"available":92431032320,"internally_reserved":0,"allocated":3094233088,"data_stored":3093932686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":469760458},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":2.1400000000000001,"15min":1.6659999999999999},"min":{"1min":0.26800000000000002,"5min":0.224,"15min":0.20699999999999999},"max":{"1min":6.008,"5min":36.024000000000001,"15min":52.591999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":2.2210000000000001,"15min":1.5609999999999999},"min":{"1min":0.30599999999999999,"5min":0.24399999999999999,"15min":0.23100000000000001},"max":{"1min":6.681,"5min":35.548000000000002,"15min":52.494},"last":0.314}]},{"osd":1,"last update":"Sat Feb 10 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":2.8450000000000002,"5min":2.5499999999999998,"15min":1.4470000000000001},"min":{"1min":0.318,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":30.228000000000002,"5min":35.459000000000003,"15min":35.459000000000003},"last":2.6040000000000001},{"interface":"front","average":{"1min":2.859,"5min":2.4510000000000001,"15min":1.458},"min":{"1min":0.27400000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":23.239000000000001,"5min":35.225000000000001,"15min":35.225000000000001},"last":3.0920000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.2669999999999999,"15min":1.1020000000000001},"min":{"1min":0.314,"5min":0.29199999999999998,"15min":0.27300000000000002},"max":{"1min":5.9450000000000003,"5min":18.57,"15min":19.710000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":1.117,"5min":1.468,"15min":1.228},"min":{"1min":0.32500000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.343,"5min":18.521000000000001,"15min":19.643000000000001},"last":0.41299999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":4.423,"5min":2.2200000000000002,"15min":1.7210000000000001},"min":{"1min":0.35699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":43.573999999999998,"5min":43.573999999999998,"15min":64.120999999999995},"last":1.6779999999999999},{"interface":"front","average":{"1min":4.4539999999999997,"5min":2.2789999999999999,"15min":1.8700000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":43.451000000000001,"5min":43.451000000000001,"15min":64.400000000000006},"last":0.44900000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.85299999999999998,"15min":0.61399999999999999},"min":{"1min":0.28599999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.895,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.77300000000000002,"15min":0.60299999999999998},"min":{"1min":0.33700000000000002,"5min":0.26600000000000001,"15min":0.24099999999999999},"max":{"1min":1.6699999999999999,"5min":4.6980000000000004,"15min":4.6980000000000004},"last":0.61499999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:43:46 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":0.96099999999999997,"15min":0.79500000000000004},"min":{"1min":0.34300000000000003,"5min":0.245,"15min":0.245},"max":{"1min":6.1050000000000004,"5min":6.1050000000000004,"15min":6.1050000000000004},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.204,"5min":1.0629999999999999,"15min":0.82899999999999996},"min":{"1min":0.33600000000000002,"5min":0.32200000000000001,"15min":0.29999999999999999},"max":{"1min":4.0369999999999999,"5min":4.4939999999999998,"15min":4.4939999999999998},"last":0.54600000000000004}]},{"osd":6,"last update":"Sat Feb 10 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1,"15min":0.73199999999999998},"min":{"1min":0.36499999999999999,"5min":0.30299999999999999,"15min":0.28199999999999997},"max":{"1min":3.2269999999999999,"5min":9.7720000000000002,"15min":9.7720000000000002},"last":0.496},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.873,"15min":0.69499999999999995},"min":{"1min":0.40100000000000002,"5min":0.29799999999999999,"15min":0.27400000000000002},"max":{"1min":1.756,"5min":6.6109999999999998,"15min":6.6109999999999998},"last":0.56000000000000005}]}]},{"osd":6,"up_from":50,"seq":214748364963,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1333124,"kb_used_data":1004860,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92412028,"statfs":{"total":95995035648,"available":94629916672,"internally_reserved":0,"allocated":1028976640,"data_stored":1028696041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":336132550},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.3959999999999999,"15min":1.3700000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.217},"max":{"1min":4.468,"5min":13.728999999999999,"15min":63.405999999999999},"last":2.3610000000000002},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.214,"15min":1.5089999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.0140000000000002,"5min":13.768000000000001,"15min":67.748000000000005},"last":0.34899999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":2.1579999999999999,"15min":1.4239999999999999},"min":{"1min":0.25600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":3.8149999999999999,"5min":37.659999999999997,"15min":37.659999999999997},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":2.1789999999999998,"15min":1.4379999999999999},"min":{"1min":0.27000000000000002,"5min":0.253,"15min":0.24399999999999999},"max":{"1min":4.9749999999999996,"5min":37.75,"15min":37.75},"last":0.70899999999999996}]},{"osd":2,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.145,"15min":1.3839999999999999},"min":{"1min":0.28899999999999998,"5min":0.253,"15min":0.253},"max":{"1min":4.524,"5min":4.556,"15min":65.278999999999996},"last":1.04},{"interface":"front","average":{"1min":1.127,"5min":1.0249999999999999,"15min":1.4279999999999999},"min":{"1min":0.34599999999999997,"5min":0.28399999999999997,"15min":0.25800000000000001},"max":{"1min":5.6779999999999999,"5min":5.6779999999999999,"15min":65.265000000000001},"last":1.722}]},{"osd":3,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.899,"5min":1.8240000000000001,"15min":1.766},"min":{"1min":0.34799999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":11.103999999999999,"5min":20.603999999999999,"15min":62.835999999999999},"last":2.7200000000000002},{"interface":"front","average":{"1min":2.274,"5min":1.925,"15min":1.4259999999999999},"min":{"1min":0.36399999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":11.304,"5min":20.478999999999999,"15min":63.878999999999998},"last":1.3759999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.68799999999999994,"15min":0.57999999999999996},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.502,"5min":4.7489999999999997,"15min":7.7839999999999998},"last":0.432},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.65400000000000003,"15min":0.56799999999999995},"min":{"1min":0.30299999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80900000000000005,"5min":4.3520000000000003,"15min":4.3520000000000003},"last":0.41799999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.76300000000000001,"15min":0.746},"min":{"1min":0.34000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.899,"5min":5.899,"15min":5.899},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.1319999999999999,"5min":0.875,"15min":0.68500000000000005},"min":{"1min":0.33300000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.855,"5min":4.9299999999999997,"15min":4.9299999999999997},"last":0.56299999999999994}]},{"osd":7,"last update":"Sat Feb 10 07:44:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.876,"15min":0.68999999999999995},"min":{"1min":0.32200000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.4279999999999999,"5min":4.641,"15min":7.859},"last":1.008},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.86499999999999999,"15min":0.69599999999999995},"min":{"1min":0.39900000000000002,"5min":0.315,"15min":0.29699999999999999},"max":{"1min":4.4630000000000001,"5min":4.4850000000000003,"15min":4.4850000000000003},"last":1.2210000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575037,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3501168,"kb_used_data":3004904,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":90243984,"statfs":{"total":95995035648,"available":92409839616,"internally_reserved":0,"allocated":3077021696,"data_stored":3076696468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.9570000000000001,"5min":1.7490000000000001,"15min":1.71},"min":{"1min":0.192,"5min":0.182,"15min":0.182},"max":{"1min":4.5830000000000002,"5min":7.2850000000000001,"15min":114.27},"last":1.214},{"interface":"front","average":{"1min":2.0270000000000001,"5min":1.726,"15min":1.8799999999999999},"min":{"1min":0.42299999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":5.4640000000000004,"5min":7.8330000000000002,"15min":119.499},"last":0.94199999999999995}]},{"osd":2,"last update":"Sat Feb 10 07:43:58 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.456,"15min":1.081},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.19600000000000001},"max":{"1min":6.782,"5min":6.782,"15min":8.9209999999999994},"last":2.9900000000000002},{"interface":"front","average":{"1min":1.327,"5min":1.579,"15min":1.1919999999999999},"min":{"1min":0.30399999999999999,"5min":0.25600000000000001,"15min":0.23699999999999999},"max":{"1min":3.976,"5min":8.577,"15min":8.577},"last":0.995}]},{"osd":3,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.8340000000000001,"5min":2.0939999999999999,"15min":1.46},"min":{"1min":0.30299999999999999,"5min":0.222,"15min":0.21099999999999999},"max":{"1min":8.4139999999999997,"5min":11.535,"15min":11.535},"last":2.6030000000000002},{"interface":"front","average":{"1min":2.6379999999999999,"5min":1.8859999999999999,"15min":1.3979999999999999},"min":{"1min":0.59599999999999997,"5min":0.24199999999999999,"15min":0.23799999999999999},"max":{"1min":8.6289999999999996,"5min":9.516,"15min":11.698},"last":1.056}]},{"osd":4,"last update":"Sat Feb 10 07:43:52 2024","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":3.536,"15min":1.8440000000000001},"min":{"1min":0.39300000000000002,"5min":0.32900000000000001,"15min":0.30499999999999999},"max":{"1min":6.8410000000000002,"5min":207.45699999999999,"15min":207.45699999999999},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.377,"5min":1.234,"15min":0.97199999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":4.3769999999999998,"5min":4.6159999999999997,"15min":7.8079999999999998},"last":2.6680000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:43:52 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":2.7789999999999999,"15min":2.0099999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":8.2170000000000005,"5min":98.703000000000003,"15min":98.703000000000003},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.8779999999999999,"5min":2.661,"15min":1.913},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":8.2289999999999992,"5min":98.664000000000001,"15min":98.664000000000001},"last":1.0189999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.6950000000000001,"5min":1.286,"15min":1.0720000000000001},"min":{"1min":0.35399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":6.7869999999999999,"5min":6.7869999999999999,"15min":11.244999999999999},"last":2.6960000000000002},{"interface":"front","average":{"1min":2.335,"5min":1.6399999999999999,"15min":1.2609999999999999},"min":{"1min":0.375,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":8.6020000000000003,"5min":8.6020000000000003,"15min":9.0090000000000003},"last":1.1319999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:43:58 2024","interfaces":[{"interface":"back","average":{"1min":1.869,"5min":1.569,"15min":1.175},"min":{"1min":0.50700000000000001,"5min":0.36399999999999999,"15min":0.36099999999999999},"max":{"1min":7.9500000000000002,"5min":7.9500000000000002,"15min":7.9500000000000002},"last":0.97199999999999998},{"interface":"front","average":{"1min":2.032,"5min":1.4770000000000001,"15min":1.1339999999999999},"min":{"1min":0.65200000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":7.7089999999999996,"5min":7.7089999999999996,"15min":11.212},"last":2.7370000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738562,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1747292,"kb_used_data":1328660,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":91997860,"statfs":{"total":95995035648,"available":94205808640,"internally_reserved":0,"allocated":1360547840,"data_stored":1360243935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:44:30 2024","interfaces":[{"interface":"back","average":{"1min":3.1640000000000001,"5min":3.6339999999999999,"15min":2.0619999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":48.447000000000003,"5min":156.16200000000001,"15min":156.16200000000001},"last":5.2220000000000004},{"interface":"front","average":{"1min":3.371,"5min":3.758,"15min":2.0790000000000002},"min":{"1min":0.19,"5min":0.183,"15min":0.183},"max":{"1min":49.302,"5min":156.941,"15min":156.941},"last":5.79}]},{"osd":2,"last update":"Sat Feb 10 07:44:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.827,"15min":1.4099999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.6950000000000003,"5min":28.163,"15min":28.163},"last":4.8049999999999997},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.375,"15min":1.208},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":4.8090000000000002,"5min":24.059999999999999,"15min":24.059999999999999},"last":5.3010000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:44:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9640000000000004,"5min":3.6070000000000002,"15min":2.2069999999999999},"min":{"1min":0.25700000000000001,"5min":0.222,"15min":0.222},"max":{"1min":97.677000000000007,"5min":97.677000000000007,"15min":97.677000000000007},"last":4.7809999999999997},{"interface":"front","average":{"1min":5.7160000000000002,"5min":2.5670000000000002,"15min":1.784},"min":{"1min":0.32400000000000001,"5min":0.246,"15min":0.246},"max":{"1min":49.344000000000001,"5min":49.344000000000001,"15min":49.344000000000001},"last":5.3789999999999996}]},{"osd":4,"last update":"Sat Feb 10 07:43:51 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.071,"15min":0.95399999999999996},"min":{"1min":0.32800000000000001,"5min":0.29899999999999999,"15min":0.27600000000000002},"max":{"1min":3.5670000000000002,"5min":5.0700000000000003,"15min":5.4260000000000002},"last":5.0700000000000003},{"interface":"front","average":{"1min":1.024,"5min":1.1599999999999999,"15min":1.1599999999999999},"min":{"1min":0.36399999999999999,"5min":0.28399999999999997,"15min":0.26900000000000002},"max":{"1min":3.3260000000000001,"5min":4.8220000000000001,"15min":19.219000000000001},"last":4.6970000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:43:51 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.5900000000000001,"15min":2.6890000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":3.4849999999999999,"5min":23.391999999999999,"15min":33.289999999999999},"last":5.1929999999999996},{"interface":"front","average":{"1min":1.202,"5min":1.5649999999999999,"15min":3.1779999999999999},"min":{"1min":0.42299999999999999,"5min":0.308,"15min":0.308},"max":{"1min":3.1859999999999999,"5min":23.417999999999999,"15min":33.430999999999997},"last":4.7300000000000004}]},{"osd":6,"last update":"Sat Feb 10 07:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.89500000000000002,"15min":1.117},"min":{"1min":0.41099999999999998,"5min":0.33600000000000002,"15min":0.29599999999999999},"max":{"1min":2.286,"5min":4.4109999999999996,"15min":19.16},"last":5.3310000000000004},{"interface":"front","average":{"1min":1.028,"5min":1.248,"15min":1.1579999999999999},"min":{"1min":0.38300000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":4.1059999999999999,"5min":16.431000000000001,"15min":16.431000000000001},"last":5.1509999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:44:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.4450000000000001,"15min":1.5069999999999999},"min":{"1min":0.42199999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.3519999999999999,"5min":16.282,"15min":19.178000000000001},"last":4.7560000000000002},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1120000000000001,"15min":1.1040000000000001},"min":{"1min":0.41499999999999998,"5min":0.379,"15min":0.376},"max":{"1min":3.4630000000000001,"5min":5.5659999999999998,"15min":8.5700000000000003},"last":5.3550000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411511,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1441740,"kb_used_data":1005188,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92303412,"statfs":{"total":95995035648,"available":94518693888,"internally_reserved":0,"allocated":1029312512,"data_stored":1029024207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:44:04 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":2.1659999999999999,"15min":1.9350000000000001},"min":{"1min":0.252,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":3.899,"5min":65.430999999999997,"15min":90.046999999999997},"last":7.1890000000000001},{"interface":"front","average":{"1min":1.758,"5min":1.641,"15min":1.748},"min":{"1min":0.24099999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.2009999999999996,"5min":8.3309999999999995,"15min":90.016000000000005},"last":0.27200000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:44:04 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":2.4470000000000001,"15min":1.556},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.19500000000000001},"max":{"1min":6.8280000000000003,"5min":65.795000000000002,"15min":65.795000000000002},"last":7.3230000000000004},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.8360000000000001,"15min":1.343},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19500000000000001},"max":{"1min":5.1749999999999998,"5min":8.1010000000000009,"15min":9.5630000000000006},"last":6.7279999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":2.3300000000000001,"15min":1.6859999999999999},"min":{"1min":0.32700000000000001,"5min":0.29799999999999999,"15min":0.23400000000000001},"max":{"1min":5.6230000000000002,"5min":65.869,"15min":65.869},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.635,"5min":1.6259999999999999,"15min":1.4830000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.7160000000000002,"5min":8.9710000000000001,"15min":11.739000000000001},"last":6.8970000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:43:47 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":1.9259999999999999,"15min":1.3999999999999999},"min":{"1min":0.32500000000000001,"5min":0.309,"15min":0.29099999999999998},"max":{"1min":1.5189999999999999,"5min":65.494,"15min":65.494},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.92600000000000005,"5min":1.214,"15min":1.234},"min":{"1min":0.318,"5min":0.311,"15min":0.311},"max":{"1min":2.6499999999999999,"5min":6.7510000000000003,"15min":33.554000000000002},"last":6.8259999999999996}]},{"osd":5,"last update":"Sat Feb 10 07:43:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.5980000000000001,"15min":1.3200000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":4.5739999999999998,"5min":7.1109999999999998,"15min":7.1109999999999998},"last":6.915},{"interface":"front","average":{"1min":1.4279999999999999,"5min":2.1699999999999999,"15min":1.786},"min":{"1min":0.47499999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":4.5540000000000003,"5min":65.834000000000003,"15min":65.834000000000003},"last":0.44800000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.9470000000000001,"15min":1.4419999999999999},"min":{"1min":0.46300000000000002,"5min":0.36599999999999999,"15min":0.33800000000000002},"max":{"1min":3.1309999999999998,"5min":65.748000000000005,"15min":65.748000000000005},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.381,"15min":1.1870000000000001},"min":{"1min":0.42999999999999999,"5min":0.39900000000000002,"15min":0.315},"max":{"1min":3.8839999999999999,"5min":7.0359999999999996,"15min":8.1899999999999995},"last":6.7800000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:44:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.5,"15min":1.208},"min":{"1min":0.42399999999999999,"5min":0.38900000000000001,"15min":0.33000000000000002},"max":{"1min":5.5,"5min":7.1779999999999999,"15min":8.2189999999999994},"last":6.8659999999999997},{"interface":"front","average":{"1min":1.2430000000000001,"5min":2.0590000000000002,"15min":1.3979999999999999},"min":{"1min":0.47599999999999998,"5min":0.44600000000000001,"15min":0.38200000000000001},"max":{"1min":5.6849999999999996,"5min":65.539000000000001,"15min":65.539000000000001},"last":7.0469999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3122724,"kb_used_data":2613980,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":90622428,"statfs":{"total":95995035648,"available":92797366272,"internally_reserved":0,"allocated":2676715520,"data_stored":2676402115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":520944071},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,3,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:44:31 2024","interfaces":[{"interface":"back","average":{"1min":2.056,"5min":2.0369999999999999,"15min":1.595},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.17499999999999999},"max":{"1min":7.181,"5min":15.446,"15min":25.277000000000001},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.903,"5min":1.671,"15min":1.4810000000000001},"min":{"1min":0.215,"5min":0.21299999999999999,"15min":0.18099999999999999},"max":{"1min":6.4000000000000004,"5min":9.0459999999999994,"15min":28.390000000000001},"last":0.72499999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:44:31 2024","interfaces":[{"interface":"back","average":{"1min":2.056,"5min":2.0699999999999998,"15min":1.5129999999999999},"min":{"1min":0.36099999999999999,"5min":0.219,"15min":0.20999999999999999},"max":{"1min":6.5289999999999999,"5min":25.105,"15min":25.105},"last":0.55600000000000005},{"interface":"front","average":{"1min":2.4470000000000001,"5min":2.6230000000000002,"15min":1.823},"min":{"1min":0.29599999999999999,"5min":0.251,"15min":0.23300000000000001},"max":{"1min":6.2489999999999997,"5min":28.555,"15min":28.555},"last":0.73999999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:44:31 2024","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.2069999999999999,"15min":2.0790000000000002},"min":{"1min":0.25900000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.9560000000000004,"5min":13.209,"15min":163.441},"last":0.38},{"interface":"front","average":{"1min":1.7589999999999999,"5min":1.865,"15min":2.044},"min":{"1min":0.42499999999999999,"5min":0.313,"15min":0.218},"max":{"1min":5.4630000000000001,"5min":9.3989999999999991,"15min":166.82400000000001},"last":1.3}]},{"osd":4,"last update":"Sat Feb 10 07:43:50 2024","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.3819999999999999,"15min":1.121},"min":{"1min":0.42699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":6.5019999999999998,"5min":9.6609999999999996,"15min":9.6609999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":3.1819999999999999,"5min":1.927,"15min":1.3500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":13.276,"5min":13.276,"15min":13.276},"last":0.45400000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:43:50 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":2.2149999999999999,"15min":1.4650000000000001},"min":{"1min":0.45900000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":62.817,"5min":62.817,"15min":62.817},"last":0.69399999999999995},{"interface":"front","average":{"1min":7.5099999999999998,"5min":2.8250000000000002,"15min":1.75},"min":{"1min":0.499,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":62.865000000000002,"5min":62.865000000000002,"15min":62.865000000000002},"last":0.70799999999999996}]},{"osd":6,"last update":"Sat Feb 10 07:44:33 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":2.1120000000000001,"15min":1.5860000000000001},"min":{"1min":0.33800000000000002,"5min":0.318,"15min":0.318},"max":{"1min":5.9219999999999997,"5min":13.307,"15min":32.048999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.665,"15min":1.3660000000000001},"min":{"1min":0.437,"5min":0.38300000000000001,"15min":0.33600000000000002},"max":{"1min":5.6340000000000003,"5min":10.191000000000001,"15min":25.937000000000001},"last":1.002}]},{"osd":7,"last update":"Sat Feb 10 07:44:03 2024","interfaces":[{"interface":"back","average":{"1min":3.8559999999999999,"5min":1.954,"15min":1.444},"min":{"1min":0.47599999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":31.257999999999999,"5min":31.257999999999999,"15min":31.257999999999999},"last":0.62},{"interface":"front","average":{"1min":5.609,"5min":2.5720000000000001,"15min":1.698},"min":{"1min":0.46999999999999997,"5min":0.39300000000000002,"15min":0.33600000000000002},"max":{"1min":30.5,"5min":30.5,"15min":30.5},"last":0.53300000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953645,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1387872,"kb_used_data":1013784,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92357280,"statfs":{"total":95995035648,"available":94573854720,"internally_reserved":0,"allocated":1038114816,"data_stored":1037805619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:43:48 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":2.633,"15min":1.9399999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":5.2919999999999998,"5min":74.533000000000001,"15min":74.533000000000001},"last":1.6439999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":2.5640000000000001,"15min":1.839},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.7959999999999998,"5min":74.340999999999994,"15min":74.340999999999994},"last":0.27000000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:43:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":3.1339999999999999,"15min":1.704},"min":{"1min":0.34399999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":40.680999999999997,"5min":69.852000000000004,"15min":69.852000000000004},"last":0.626},{"interface":"front","average":{"1min":2.9420000000000002,"5min":3.21,"15min":1.7290000000000001},"min":{"1min":0.29699999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":41.161000000000001,"5min":70.692999999999998,"15min":70.692999999999998},"last":2.5750000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:43:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":1.0029999999999999,"15min":1.1839999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.26800000000000002},"max":{"1min":2.0049999999999999,"5min":15.351000000000001,"15min":50.162999999999997},"last":1.591},{"interface":"front","average":{"1min":1,"5min":1.0900000000000001,"15min":1.1679999999999999},"min":{"1min":0.309,"5min":0.28399999999999997,"15min":0.27700000000000002},"max":{"1min":4.3200000000000003,"5min":8.5129999999999999,"15min":50.453000000000003},"last":0.33600000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:43:48 2024","interfaces":[{"interface":"back","average":{"1min":2.9470000000000001,"5min":2.089,"15min":1.607},"min":{"1min":0.30499999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":40.710000000000001,"5min":69.813000000000002,"15min":69.813000000000002},"last":0.311},{"interface":"front","average":{"1min":2.9580000000000002,"5min":2.319,"15min":1.6619999999999999},"min":{"1min":0.39000000000000001,"5min":0.30099999999999999,"15min":0.27900000000000003},"max":{"1min":40.981000000000002,"5min":69.760999999999996,"15min":69.760999999999996},"last":2.29}]},{"osd":5,"last update":"Sat Feb 10 07:43:48 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.78000000000000003,"15min":0.58299999999999996},"min":{"1min":0.28000000000000003,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.7749999999999999,"5min":7.4969999999999999,"15min":7.4969999999999999},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.133,"5min":0.877,"15min":0.60099999999999998},"min":{"1min":0.32400000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.4129999999999998,"5min":7.101,"15min":7.101},"last":0.41399999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:44:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.70299999999999996,"15min":0.56299999999999994},"min":{"1min":0.28899999999999998,"5min":0.28399999999999997,"15min":0.25900000000000001},"max":{"1min":2.8690000000000002,"5min":3.347,"15min":3.347},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.68999999999999995,"15min":0.57799999999999996},"min":{"1min":0.30299999999999999,"5min":0.26600000000000001,"15min":0.24399999999999999},"max":{"1min":1.744,"5min":3.0219999999999998,"15min":3.7109999999999999},"last":1.625}]},{"osd":7,"last update":"Sat Feb 10 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.69299999999999995,"15min":0.58799999999999997},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.29099999999999998},"max":{"1min":2.8180000000000001,"5min":2.9740000000000002,"15min":2.9740000000000002},"last":1.9610000000000001},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.80700000000000005,"15min":0.65500000000000003},"min":{"1min":0.42899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.8460000000000001,"5min":4.1710000000000003,"15min":7.0359999999999996},"last":1.9750000000000001}]}]},{"osd":5,"up_from":156,"seq":670014898274,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3455248,"kb_used_data":3079368,"kb_used_omap":4,"kb_used_meta":375867,"kb_avail":90289904,"statfs":{"total":95995035648,"available":92456861696,"internally_reserved":0,"allocated":3153272832,"data_stored":3152980637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":384888482},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":2.3490000000000002,"15min":1.833},"min":{"1min":0.29599999999999999,"5min":0.247,"15min":0.20499999999999999},"max":{"1min":1.8140000000000001,"5min":52.938000000000002,"15min":52.938000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.877,"5min":2.6059999999999999,"15min":2.0209999999999999},"min":{"1min":0.30599999999999999,"5min":0.246,"15min":0.246},"max":{"1min":5.6719999999999997,"5min":52.805,"15min":52.805},"last":1.2430000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:43:43 2024","interfaces":[{"interface":"back","average":{"1min":1.859,"5min":2.7109999999999999,"15min":3.1120000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.245},"max":{"1min":5.702,"5min":54.298000000000002,"15min":54.298000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.4450000000000001,"5min":2.5779999999999998,"15min":3.1480000000000001},"min":{"1min":0.27900000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.7949999999999999,"5min":52.969999999999999,"15min":52.969999999999999},"last":1.794}]},{"osd":2,"last update":"Sat Feb 10 07:43:43 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.0529999999999999,"15min":1.5800000000000001},"min":{"1min":0.308,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":5.5519999999999996,"5min":5.5519999999999996,"15min":11.601000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.018,"15min":1.645},"min":{"1min":0.32100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.8639999999999999,"5min":4.6619999999999999,"15min":11.618},"last":0.78100000000000003}]},{"osd":3,"last update":"Sat Feb 10 07:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":2.7490000000000001,"15min":1.9319999999999999},"min":{"1min":0.39100000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":3.2400000000000002,"5min":53.366999999999997,"15min":53.366999999999997},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.9690000000000001,"5min":2.9159999999999999,"15min":2.2069999999999999},"min":{"1min":0.38500000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":6.3289999999999997,"5min":52.731000000000002,"15min":52.731000000000002},"last":0.53200000000000003}]},{"osd":4,"last update":"Sat Feb 10 07:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.78000000000000003,"15min":0.60899999999999999},"min":{"1min":0.27100000000000002,"5min":0.246,"15min":0.246},"max":{"1min":2.8879999999999999,"5min":5.4050000000000002,"15min":5.4050000000000002},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.79200000000000004,"15min":0.57099999999999995},"min":{"1min":0.33200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.4950000000000001,"5min":4.843,"15min":4.843},"last":0.57699999999999996}]},{"osd":6,"last update":"Sat Feb 10 07:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.79600000000000004,"15min":0.623},"min":{"1min":0.27900000000000003,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.7149999999999999,"5min":4.3860000000000001,"15min":4.3860000000000001},"last":0.752},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.81100000000000005,"15min":0.73199999999999998},"min":{"1min":0.34000000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.5899999999999999,"5min":2.839,"15min":4.2839999999999998},"last":0.65000000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:43:43 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.0389999999999999,"15min":0.70899999999999996},"min":{"1min":0.377,"5min":0.312,"15min":0.312},"max":{"1min":8.8059999999999992,"5min":8.8059999999999992,"15min":8.8059999999999992},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.85299999999999998,"15min":0.68899999999999995},"min":{"1min":0.39200000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":3.653,"5min":4.3600000000000003,"15min":4.3600000000000003},"last":0.81200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1345650688,"data_stored":1345650688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3039039488,"data_stored":3039035416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1017839616,"data_stored":1017839616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2666528768,"data_stored":2666528768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1026359296,"data_stored":1026359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3113488384,"data_stored":3113484312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1015021568,"data_stored":1015021568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3068399616,"data_stored":3068395544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:44:45.499 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:44:45.535 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:45.564 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:45.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:45.666 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:45.701 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:45.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:45.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:45.803 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:45.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:45.848 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:45.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:45.949 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:45.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:45.984 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:45.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:46.085 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:46.118 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:46.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:46.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:46.220 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:46.255 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:46.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:46.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:45 smithi008 ceph-mon[29816]: 4.1f scrub starts 2024-02-10T07:44:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:45 smithi008 ceph-mon[29816]: 4.1f scrub ok 2024-02-10T07:44:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:45 smithi008 ceph-mon[29816]: 3.1e scrub starts 2024-02-10T07:44:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:45 smithi008 ceph-mon[29816]: 3.1e scrub ok 2024-02-10T07:44:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:45 smithi008 ceph-mon[29816]: 4.9 scrub starts 2024-02-10T07:44:46.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:45 smithi008 ceph-mon[29816]: 4.9 scrub ok 2024-02-10T07:44:46.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:45 smithi008 ceph-mon[40743]: 4.1f scrub starts 2024-02-10T07:44:46.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:45 smithi008 ceph-mon[40743]: 4.1f scrub ok 2024-02-10T07:44:46.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:45 smithi008 ceph-mon[40743]: 3.1e scrub starts 2024-02-10T07:44:46.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:45 smithi008 ceph-mon[40743]: 3.1e scrub ok 2024-02-10T07:44:46.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:45 smithi008 ceph-mon[40743]: 4.9 scrub starts 2024-02-10T07:44:46.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:45 smithi008 ceph-mon[40743]: 4.9 scrub ok 2024-02-10T07:44:46.357 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:45 smithi170 ceph-mon[33394]: 4.1f scrub starts 2024-02-10T07:44:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:45 smithi170 ceph-mon[33394]: 4.1f scrub ok 2024-02-10T07:44:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:45 smithi170 ceph-mon[33394]: 3.1e scrub starts 2024-02-10T07:44:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:45 smithi170 ceph-mon[33394]: 3.1e scrub ok 2024-02-10T07:44:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:45 smithi170 ceph-mon[33394]: 4.9 scrub starts 2024-02-10T07:44:46.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:45 smithi170 ceph-mon[33394]: 4.9 scrub ok 2024-02-10T07:44:46.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:46.392 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:46.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:46.493 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:46.533 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:46.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:46.635 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:46.675 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:46.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:46.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:46.777 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:46.812 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:46.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:46.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:46.914 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:46.955 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:46.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:47.056 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:47.089 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:47.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:47.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:47.191 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:47.210 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:46 smithi008 ceph-mon[29816]: pgmap v581: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-10T07:44:47.211 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:46 smithi008 ceph-mon[40743]: pgmap v581: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-10T07:44:47.225 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:47.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:47.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:47.327 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:47.365 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:47.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:47.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:47.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:46 smithi170 ceph-mon[33394]: pgmap v581: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-10T07:44:47.467 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:47.500 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:47.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:47.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:47.602 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:47.638 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:47.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:47.740 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:47.805 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:47.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:47.906 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:47.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:47.942 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:47.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:48.043 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:48.084 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:48.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:48.185 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:48.193 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:48.212 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:48.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:48.229 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:47 smithi008 ceph-mon[29816]: 6.14 scrub starts 2024-02-10T07:44:48.229 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:47 smithi008 ceph-mon[29816]: 6.14 scrub ok 2024-02-10T07:44:48.229 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:47 smithi008 ceph-mon[29816]: 9.0 scrub starts 2024-02-10T07:44:48.229 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:47 smithi008 ceph-mon[29816]: 3.13 scrub starts 2024-02-10T07:44:48.230 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:47 smithi008 ceph-mon[29816]: 3.13 scrub ok 2024-02-10T07:44:48.230 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:47 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:44:48.230 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:47 smithi008 ceph-mon[40743]: 6.14 scrub starts 2024-02-10T07:44:48.230 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:47 smithi008 ceph-mon[40743]: 6.14 scrub ok 2024-02-10T07:44:48.230 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:47 smithi008 ceph-mon[40743]: 9.0 scrub starts 2024-02-10T07:44:48.230 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:47 smithi008 ceph-mon[40743]: 3.13 scrub starts 2024-02-10T07:44:48.230 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:47 smithi008 ceph-mon[40743]: 3.13 scrub ok 2024-02-10T07:44:48.230 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:47 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:44:48.314 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:48.348 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:48.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:48.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:47 smithi170 ceph-mon[33394]: 6.14 scrub starts 2024-02-10T07:44:48.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:47 smithi170 ceph-mon[33394]: 6.14 scrub ok 2024-02-10T07:44:48.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:47 smithi170 ceph-mon[33394]: 9.0 scrub starts 2024-02-10T07:44:48.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:47 smithi170 ceph-mon[33394]: 3.13 scrub starts 2024-02-10T07:44:48.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:47 smithi170 ceph-mon[33394]: 3.13 scrub ok 2024-02-10T07:44:48.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:47 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:44:48.449 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:48.495 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:48.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:48.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:48.597 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:48.632 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:48.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:48.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:48.734 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:48.775 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:48.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:48.876 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:48.907 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:48.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:48.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:48.981 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:44:48.981 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:44:48.982 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-02-10T07:44:48.982 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-02-10T07:44:48.982 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:44:49.008 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:49.042 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:49.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:49.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:48 smithi008 ceph-mon[29816]: 9.f scrub starts 2024-02-10T07:44:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:48 smithi008 ceph-mon[29816]: pgmap v582: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-02-10T07:44:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:48 smithi008 ceph-mon[29816]: 6.1e scrub starts 2024-02-10T07:44:49.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:48 smithi008 ceph-mon[29816]: 6.1e scrub ok 2024-02-10T07:44:49.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:44:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:44:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:44:49.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:48 smithi008 ceph-mon[40743]: 9.f scrub starts 2024-02-10T07:44:49.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:48 smithi008 ceph-mon[40743]: pgmap v582: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-02-10T07:44:49.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:48 smithi008 ceph-mon[40743]: 6.1e scrub starts 2024-02-10T07:44:49.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:48 smithi008 ceph-mon[40743]: 6.1e scrub ok 2024-02-10T07:44:49.144 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:49.182 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:49.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:49.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:49.284 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:49.328 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:49.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:48 smithi170 ceph-mon[33394]: 9.f scrub starts 2024-02-10T07:44:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:48 smithi170 ceph-mon[33394]: pgmap v582: 148 pgs: 148 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-02-10T07:44:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:48 smithi170 ceph-mon[33394]: 6.1e scrub starts 2024-02-10T07:44:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:48 smithi170 ceph-mon[33394]: 6.1e scrub ok 2024-02-10T07:44:49.430 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:49.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:49.478 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:49.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:49.579 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:49.617 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:49.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:49.718 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:49.752 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:49.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:49.854 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:49.888 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:49.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:49.896 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:49.990 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:50.029 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:50.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:50.130 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:50.166 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:50.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:50.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:50.232 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:44:50.268 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:50.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[29816]: 4.a scrub starts 2024-02-10T07:44:50.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[29816]: 4.a scrub ok 2024-02-10T07:44:50.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[29816]: 6.10 scrub starts 2024-02-10T07:44:50.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[29816]: 3.8 scrub starts 2024-02-10T07:44:50.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[29816]: 6.10 scrub ok 2024-02-10T07:44:50.271 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[29816]: 3.8 scrub ok 2024-02-10T07:44:50.271 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[29816]: 9.0 scrub ok 2024-02-10T07:44:50.271 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[29816]: 9.f scrub ok 2024-02-10T07:44:50.271 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[29816]: 6.4 scrub starts 2024-02-10T07:44:50.271 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[29816]: 6.4 scrub ok 2024-02-10T07:44:50.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[40743]: 4.a scrub starts 2024-02-10T07:44:50.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[40743]: 4.a scrub ok 2024-02-10T07:44:50.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[40743]: 6.10 scrub starts 2024-02-10T07:44:50.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[40743]: 3.8 scrub starts 2024-02-10T07:44:50.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[40743]: 6.10 scrub ok 2024-02-10T07:44:50.272 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[40743]: 3.8 scrub ok 2024-02-10T07:44:50.272 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[40743]: 9.0 scrub ok 2024-02-10T07:44:50.272 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[40743]: 9.f scrub ok 2024-02-10T07:44:50.272 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[40743]: 6.4 scrub starts 2024-02-10T07:44:50.272 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:49 smithi008 ceph-mon[40743]: 6.4 scrub ok 2024-02-10T07:44:50.332 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:50.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:50.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:49 smithi170 ceph-mon[33394]: 4.a scrub starts 2024-02-10T07:44:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:49 smithi170 ceph-mon[33394]: 4.a scrub ok 2024-02-10T07:44:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:49 smithi170 ceph-mon[33394]: 6.10 scrub starts 2024-02-10T07:44:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:49 smithi170 ceph-mon[33394]: 3.8 scrub starts 2024-02-10T07:44:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:49 smithi170 ceph-mon[33394]: 6.10 scrub ok 2024-02-10T07:44:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:49 smithi170 ceph-mon[33394]: 3.8 scrub ok 2024-02-10T07:44:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:49 smithi170 ceph-mon[33394]: 9.0 scrub ok 2024-02-10T07:44:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:49 smithi170 ceph-mon[33394]: 9.f scrub ok 2024-02-10T07:44:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:49 smithi170 ceph-mon[33394]: 6.4 scrub starts 2024-02-10T07:44:50.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:49 smithi170 ceph-mon[33394]: 6.4 scrub ok 2024-02-10T07:44:50.418 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-10T07:44:50.434 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:50.532 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:50.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:50.635 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:50.664 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:50.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:50.765 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:50.795 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:50.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:50.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:50.896 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:50.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:50.940 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:50.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:51.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:51.074 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:51.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:51.175 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:51.207 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:51.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:51.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:51.309 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:51.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:50 smithi008 ceph-mon[29816]: pgmap v583: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:44:51.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:50 smithi008 ceph-mon[40743]: pgmap v583: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:44:51.348 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:51.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:51.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:51.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:50 smithi170 ceph-mon[33394]: pgmap v583: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:44:51.451 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:51.488 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:51.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:51.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:51.590 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:51.622 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:51.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:51.643 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:51.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:51.744 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:51.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:51.788 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:51.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:51.889 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:51.935 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:51.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:51.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:52.037 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:52.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:52.073 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:52.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:52.075 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:51 smithi008 ceph-mon[29816]: 4.17 scrub starts 2024-02-10T07:44:52.075 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:51 smithi008 ceph-mon[29816]: 4.17 scrub ok 2024-02-10T07:44:52.075 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:51 smithi008 ceph-mon[29816]: 5.1d scrub starts 2024-02-10T07:44:52.075 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:51 smithi008 ceph-mon[29816]: 5.1d scrub ok 2024-02-10T07:44:52.076 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:51 smithi008 ceph-mon[29816]: 6.1c scrub starts 2024-02-10T07:44:52.076 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:51 smithi008 ceph-mon[29816]: 6.1c scrub ok 2024-02-10T07:44:52.076 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:51 smithi008 ceph-mon[40743]: 4.17 scrub starts 2024-02-10T07:44:52.076 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:51 smithi008 ceph-mon[40743]: 4.17 scrub ok 2024-02-10T07:44:52.076 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:51 smithi008 ceph-mon[40743]: 5.1d scrub starts 2024-02-10T07:44:52.076 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:51 smithi008 ceph-mon[40743]: 5.1d scrub ok 2024-02-10T07:44:52.076 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:51 smithi008 ceph-mon[40743]: 6.1c scrub starts 2024-02-10T07:44:52.076 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:51 smithi008 ceph-mon[40743]: 6.1c scrub ok 2024-02-10T07:44:52.174 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:52.206 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:52.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:52.308 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:52.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:52.360 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:52.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:51 smithi170 ceph-mon[33394]: 4.17 scrub starts 2024-02-10T07:44:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:51 smithi170 ceph-mon[33394]: 4.17 scrub ok 2024-02-10T07:44:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:51 smithi170 ceph-mon[33394]: 5.1d scrub starts 2024-02-10T07:44:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:51 smithi170 ceph-mon[33394]: 5.1d scrub ok 2024-02-10T07:44:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:51 smithi170 ceph-mon[33394]: 6.1c scrub starts 2024-02-10T07:44:52.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:51 smithi170 ceph-mon[33394]: 6.1c scrub ok 2024-02-10T07:44:52.461 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:52.503 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:52.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:52.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:52.605 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:52.636 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:52.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:52.738 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:52.776 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:52.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:52.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:52.878 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:52.917 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:52.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:52.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:53.019 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:53.048 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:53.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:53.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:53.149 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:53.181 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:53.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:53.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:53.247 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:53 smithi008 ceph-mon[29816]: 3.1f scrub starts 2024-02-10T07:44:53.247 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:53 smithi008 ceph-mon[29816]: 3.1f scrub ok 2024-02-10T07:44:53.247 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:53 smithi008 ceph-mon[29816]: pgmap v584: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:44:53.247 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:53 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:44:53.247 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:53 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:44:53.248 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:52 smithi008 ceph-mon[40743]: 3.1f scrub starts 2024-02-10T07:44:53.248 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:52 smithi008 ceph-mon[40743]: 3.1f scrub ok 2024-02-10T07:44:53.248 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:53 smithi008 ceph-mon[40743]: pgmap v584: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:44:53.248 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:53 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:44:53.248 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:53 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:44:53.282 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:53.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:53.323 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:53.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:52 smithi170 ceph-mon[33394]: 3.1f scrub starts 2024-02-10T07:44:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:52 smithi170 ceph-mon[33394]: 3.1f scrub ok 2024-02-10T07:44:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:52 smithi170 ceph-mon[33394]: pgmap v584: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:44:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:52 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:44:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:52 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:44:53.425 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:53.461 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:53.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:53.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:53.563 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:53.610 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:53.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:53.694 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:44:53.711 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:53.781 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:53.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:53.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:44:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:44:53.894 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:53.934 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:53.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:54.036 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:54.077 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:54.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:54.179 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:54.217 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:54.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:54.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:54.241 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:54 smithi008 ceph-mon[29816]: 4.6 scrub starts 2024-02-10T07:44:54.241 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:54 smithi008 ceph-mon[29816]: 4.6 scrub ok 2024-02-10T07:44:54.241 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:54 smithi008 ceph-mon[29816]: osdmap e294: 8 total, 8 up, 8 in 2024-02-10T07:44:54.241 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:54 smithi008 ceph-mon[29816]: 3.0 scrub starts 2024-02-10T07:44:54.241 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:54 smithi008 ceph-mon[29816]: 3.0 scrub ok 2024-02-10T07:44:54.241 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:54 smithi008 ceph-mon[29816]: 5.d scrub starts 2024-02-10T07:44:54.241 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:54 smithi008 ceph-mon[29816]: 5.d scrub ok 2024-02-10T07:44:54.241 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:54 smithi008 ceph-mon[40743]: 4.6 scrub starts 2024-02-10T07:44:54.241 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:54 smithi008 ceph-mon[40743]: 4.6 scrub ok 2024-02-10T07:44:54.241 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:54 smithi008 ceph-mon[40743]: osdmap e294: 8 total, 8 up, 8 in 2024-02-10T07:44:54.242 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:54 smithi008 ceph-mon[40743]: 3.0 scrub starts 2024-02-10T07:44:54.242 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:54 smithi008 ceph-mon[40743]: 3.0 scrub ok 2024-02-10T07:44:54.242 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:54 smithi008 ceph-mon[40743]: 5.d scrub starts 2024-02-10T07:44:54.242 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:54 smithi008 ceph-mon[40743]: 5.d scrub ok 2024-02-10T07:44:54.319 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:54.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:54.375 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:54.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:54 smithi170 ceph-mon[33394]: 4.6 scrub starts 2024-02-10T07:44:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:54 smithi170 ceph-mon[33394]: 4.6 scrub ok 2024-02-10T07:44:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:54 smithi170 ceph-mon[33394]: osdmap e294: 8 total, 8 up, 8 in 2024-02-10T07:44:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:54 smithi170 ceph-mon[33394]: 3.0 scrub starts 2024-02-10T07:44:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:54 smithi170 ceph-mon[33394]: 3.0 scrub ok 2024-02-10T07:44:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:54 smithi170 ceph-mon[33394]: 5.d scrub starts 2024-02-10T07:44:54.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:54 smithi170 ceph-mon[33394]: 5.d scrub ok 2024-02-10T07:44:54.477 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:54.514 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:54.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:54.616 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:54.666 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:54.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:54.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:54.779 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:44:54.780 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:44:54.801 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:54.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:54.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:54.818 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:44:54.862 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:44:54.902 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:44:54.903 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:54.944 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:54.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:54.954 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:44:55.001 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:44:55.046 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:55.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[29816]: 5.2 scrub starts 2024-02-10T07:44:55.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[29816]: 5.2 scrub ok 2024-02-10T07:44:55.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[29816]: 6.18 scrub starts 2024-02-10T07:44:55.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[29816]: 6.18 scrub ok 2024-02-10T07:44:55.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:55.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[29816]: pgmap v586: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-10T07:44:55.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[29816]: 3.10 scrub starts 2024-02-10T07:44:55.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[29816]: 3.10 scrub ok 2024-02-10T07:44:55.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[29816]: 3.19 scrub starts 2024-02-10T07:44:55.060 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[29816]: 3.19 scrub ok 2024-02-10T07:44:55.061 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1770408865' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:44:55.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[40743]: 5.2 scrub starts 2024-02-10T07:44:55.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[40743]: 5.2 scrub ok 2024-02-10T07:44:55.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[40743]: 6.18 scrub starts 2024-02-10T07:44:55.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[40743]: 6.18 scrub ok 2024-02-10T07:44:55.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:55.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[40743]: pgmap v586: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-10T07:44:55.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[40743]: 3.10 scrub starts 2024-02-10T07:44:55.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[40743]: 3.10 scrub ok 2024-02-10T07:44:55.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[40743]: 3.19 scrub starts 2024-02-10T07:44:55.061 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[40743]: 3.19 scrub ok 2024-02-10T07:44:55.062 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:55 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1770408865' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:44:55.062 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:44:55.075 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:44:55.075 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:44:55.075 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:44:55.076 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:44:55.076 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:44:55.076 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:44:55.076 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:44:55.076 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:44:55.092 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:55.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:55.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:55.194 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:55.235 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:55.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:55.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:55.339 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:55 smithi170 ceph-mon[33394]: 5.2 scrub starts 2024-02-10T07:44:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:55 smithi170 ceph-mon[33394]: 5.2 scrub ok 2024-02-10T07:44:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:55 smithi170 ceph-mon[33394]: 6.18 scrub starts 2024-02-10T07:44:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:55 smithi170 ceph-mon[33394]: 6.18 scrub ok 2024-02-10T07:44:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:55 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:44:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:55 smithi170 ceph-mon[33394]: pgmap v586: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-10T07:44:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:55 smithi170 ceph-mon[33394]: 3.10 scrub starts 2024-02-10T07:44:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:55 smithi170 ceph-mon[33394]: 3.10 scrub ok 2024-02-10T07:44:55.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:55 smithi170 ceph-mon[33394]: 3.19 scrub starts 2024-02-10T07:44:55.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:55 smithi170 ceph-mon[33394]: 3.19 scrub ok 2024-02-10T07:44:55.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:55 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1770408865' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:44:55.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:55.396 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:55.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:55.501 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:55.502 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:44:55.513 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:44:55.555 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:55.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:55.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:55.657 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:55.695 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:55.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:55.797 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:55.836 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:55.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:55.936 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:55.973 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:55.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:55.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:56.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:56.118 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:56.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:56.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:56.220 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:56.259 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:56.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:56.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:56.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:56 smithi008 ceph-mon[29816]: 3.2 scrub starts 2024-02-10T07:44:56.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:56 smithi008 ceph-mon[29816]: 3.2 scrub ok 2024-02-10T07:44:56.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:56 smithi008 ceph-mon[29816]: 4.d scrub starts 2024-02-10T07:44:56.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:56 smithi008 ceph-mon[29816]: 4.d scrub ok 2024-02-10T07:44:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:56 smithi008 ceph-mon[40743]: 3.2 scrub starts 2024-02-10T07:44:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:56 smithi008 ceph-mon[40743]: 3.2 scrub ok 2024-02-10T07:44:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:56 smithi008 ceph-mon[40743]: 4.d scrub starts 2024-02-10T07:44:56.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:56 smithi008 ceph-mon[40743]: 4.d scrub ok 2024-02-10T07:44:56.362 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:56.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:56 smithi170 ceph-mon[33394]: 3.2 scrub starts 2024-02-10T07:44:56.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:56 smithi170 ceph-mon[33394]: 3.2 scrub ok 2024-02-10T07:44:56.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:56 smithi170 ceph-mon[33394]: 4.d scrub starts 2024-02-10T07:44:56.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:56 smithi170 ceph-mon[33394]: 4.d scrub ok 2024-02-10T07:44:56.396 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:56.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:56.497 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:56.537 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:56.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:56.638 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:44:56.671 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:56.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:56.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:44:56.773 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:56.811 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:56.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:56.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:56.913 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:56.958 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:56.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:57.059 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:57.090 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:57.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:57.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:57.191 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:57.237 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:57.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[29816]: 3.17 scrub starts 2024-02-10T07:44:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[29816]: 3.17 scrub ok 2024-02-10T07:44:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[29816]: 5.16 scrub starts 2024-02-10T07:44:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[29816]: 5.16 scrub ok 2024-02-10T07:44:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[29816]: 5.0 scrub starts 2024-02-10T07:44:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[29816]: 5.0 scrub ok 2024-02-10T07:44:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[29816]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:44:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[29816]: pgmap v587: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-10T07:44:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[29816]: 5.1c scrub starts 2024-02-10T07:44:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[29816]: 5.1c scrub ok 2024-02-10T07:44:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[29816]: 5.12 scrub starts 2024-02-10T07:44:57.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[29816]: 5.12 scrub ok 2024-02-10T07:44:57.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[29816]: 2.1 scrub starts 2024-02-10T07:44:57.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[29816]: 2.1 scrub ok 2024-02-10T07:44:57.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[40743]: 3.17 scrub starts 2024-02-10T07:44:57.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[40743]: 3.17 scrub ok 2024-02-10T07:44:57.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[40743]: 5.16 scrub starts 2024-02-10T07:44:57.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[40743]: 5.16 scrub ok 2024-02-10T07:44:57.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[40743]: 5.0 scrub starts 2024-02-10T07:44:57.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[40743]: 5.0 scrub ok 2024-02-10T07:44:57.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[40743]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:44:57.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[40743]: pgmap v587: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-10T07:44:57.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[40743]: 5.1c scrub starts 2024-02-10T07:44:57.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[40743]: 5.1c scrub ok 2024-02-10T07:44:57.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[40743]: 5.12 scrub starts 2024-02-10T07:44:57.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[40743]: 5.12 scrub ok 2024-02-10T07:44:57.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[40743]: 2.1 scrub starts 2024-02-10T07:44:57.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:57 smithi008 ceph-mon[40743]: 2.1 scrub ok 2024-02-10T07:44:57.339 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:57.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:57.383 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:57.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:57.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:57 smithi170 ceph-mon[33394]: 3.17 scrub starts 2024-02-10T07:44:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:57 smithi170 ceph-mon[33394]: 3.17 scrub ok 2024-02-10T07:44:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:57 smithi170 ceph-mon[33394]: 5.16 scrub starts 2024-02-10T07:44:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:57 smithi170 ceph-mon[33394]: 5.16 scrub ok 2024-02-10T07:44:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:57 smithi170 ceph-mon[33394]: 5.0 scrub starts 2024-02-10T07:44:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:57 smithi170 ceph-mon[33394]: 5.0 scrub ok 2024-02-10T07:44:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:57 smithi170 ceph-mon[33394]: from='client.25159 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:44:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:57 smithi170 ceph-mon[33394]: pgmap v587: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-10T07:44:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:57 smithi170 ceph-mon[33394]: 5.1c scrub starts 2024-02-10T07:44:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:57 smithi170 ceph-mon[33394]: 5.1c scrub ok 2024-02-10T07:44:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:57 smithi170 ceph-mon[33394]: 5.12 scrub starts 2024-02-10T07:44:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:57 smithi170 ceph-mon[33394]: 5.12 scrub ok 2024-02-10T07:44:57.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:57 smithi170 ceph-mon[33394]: 2.1 scrub starts 2024-02-10T07:44:57.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:57 smithi170 ceph-mon[33394]: 2.1 scrub ok 2024-02-10T07:44:57.484 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:57.536 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:57.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:57.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:57.638 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:57.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:57.686 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:57.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:57.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:57.829 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:57.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:57.930 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:57.961 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:57.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:58.063 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:58.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:58 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:44:58.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:58 smithi008 ceph-mon[29816]: 3.f scrub starts 2024-02-10T07:44:58.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:58 smithi008 ceph-mon[29816]: 3.f scrub ok 2024-02-10T07:44:58.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:58 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:44:58.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:58 smithi008 ceph-mon[40743]: 3.f scrub starts 2024-02-10T07:44:58.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:58 smithi008 ceph-mon[40743]: 3.f scrub ok 2024-02-10T07:44:58.093 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:58.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:58.195 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:58.259 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:58.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:58.360 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:58.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:58 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:44:58.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:58 smithi170 ceph-mon[33394]: 3.f scrub starts 2024-02-10T07:44:58.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:58 smithi170 ceph-mon[33394]: 3.f scrub ok 2024-02-10T07:44:58.403 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:58.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:58.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:58.448 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:44:58.505 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:58.555 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:58.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:58.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:58.656 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:58.688 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:58.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:58.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:44:58.790 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:44:58.839 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:58.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:58.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:44:58.941 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:44:58.958 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":586,"stamp":"2024-02-10T07:44:53.990573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4405860724,"num_objects":67418,"num_object_clones":0,"num_object_copies":202254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67418,"num_whiteouts":0,"num_read":2134,"num_read_kb":1980,"num_write":190031,"num_write_kb":8205598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":1772204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84131,"ondisk_log_size":84131,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16432056,"kb_used_data":13064052,"kb_used_omap":15,"kb_used_meta":3367856,"kb_avail":733529160,"statfs":{"total":767960285184,"available":751133859840,"internally_reserved":0,"allocated":13377589248,"data_stored":13375138685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15602,"internal_metadata":3448685326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,5,8,4,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-728104960,"num_objects":-11110,"num_object_clones":0,"num_object_copies":-33330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11110,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11110,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005053"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":505,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.948320+0000","last_change":"2024-02-10T07:43:40.436091+0000","last_active":"2024-02-10T07:44:43.948320+0000","last_peered":"2024-02-10T07:44:43.948320+0000","last_clean":"2024-02-10T07:44:43.948320+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:44:43.948320+0000","last_undegraded":"2024-02-10T07:44:43.948320+0000","last_fullsized":"2024-02-10T07:44:43.948320+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:40.435939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:40.435939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":515,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.953946+0000","last_change":"2024-02-10T07:43:54.220526+0000","last_active":"2024-02-10T07:44:43.953946+0000","last_peered":"2024-02-10T07:44:43.953946+0000","last_clean":"2024-02-10T07:44:43.953946+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:44:43.953946+0000","last_undegraded":"2024-02-10T07:44:43.953946+0000","last_fullsized":"2024-02-10T07:44:43.953946+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:54.220406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:54.220406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:20.964933+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":533,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.013081+0000","last_change":"2024-02-10T07:44:51.181426+0000","last_active":"2024-02-10T07:44:53.013081+0000","last_peered":"2024-02-10T07:44:53.013081+0000","last_clean":"2024-02-10T07:44:53.013081+0000","last_became_active":"2024-02-10T07:36:33.760016+0000","last_became_peered":"2024-02-10T07:36:33.760016+0000","last_unstale":"2024-02-10T07:44:53.013081+0000","last_undegraded":"2024-02-10T07:44:53.013081+0000","last_fullsized":"2024-02-10T07:44:53.013081+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:44:51.181293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:44:51.181293+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:46:01.706140+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"287'19","reported_seq":557,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.947809+0000","last_change":"2024-02-10T07:44:13.403141+0000","last_active":"2024-02-10T07:44:43.947809+0000","last_peered":"2024-02-10T07:44:43.947809+0000","last_clean":"2024-02-10T07:44:43.947809+0000","last_became_active":"2024-02-10T07:38:34.790379+0000","last_became_peered":"2024-02-10T07:38:34.790379+0000","last_unstale":"2024-02-10T07:44:43.947809+0000","last_undegraded":"2024-02-10T07:44:43.947809+0000","last_fullsized":"2024-02-10T07:44:43.947809+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'19","last_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_deep_scrub":"287'19","last_deep_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_clean_scrub_stamp":"2024-02-10T07:44:13.402994+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:31.292454+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":496,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.955132+0000","last_change":"2024-02-10T07:43:48.894294+0000","last_active":"2024-02-10T07:44:43.955132+0000","last_peered":"2024-02-10T07:44:43.955132+0000","last_clean":"2024-02-10T07:44:43.955132+0000","last_became_active":"2024-02-10T07:31:57.300056+0000","last_became_peered":"2024-02-10T07:31:57.300056+0000","last_unstale":"2024-02-10T07:44:43.955132+0000","last_undegraded":"2024-02-10T07:44:43.955132+0000","last_fullsized":"2024-02-10T07:44:43.955132+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:48.894116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:48.894116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:05.267308+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":508,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.947697+0000","last_change":"2024-02-10T07:43:51.422878+0000","last_active":"2024-02-10T07:44:43.947697+0000","last_peered":"2024-02-10T07:44:43.947697+0000","last_clean":"2024-02-10T07:44:43.947697+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:44:43.947697+0000","last_undegraded":"2024-02-10T07:44:43.947697+0000","last_fullsized":"2024-02-10T07:44:43.947697+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:51.422763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:51.422763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:16.973118+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":509,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:44.253406+0000","last_change":"2024-02-10T07:44:44.253443+0000","last_active":"2024-02-10T07:44:44.253406+0000","last_peered":"2024-02-10T07:44:44.253406+0000","last_clean":"2024-02-10T07:44:44.253406+0000","last_became_active":"2024-02-10T07:31:51.323331+0000","last_became_peered":"2024-02-10T07:31:51.323331+0000","last_unstale":"2024-02-10T07:44:44.253406+0000","last_undegraded":"2024-02-10T07:44:44.253406+0000","last_fullsized":"2024-02-10T07:44:44.253406+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:44.253319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:44:44.253319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:45.548617+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"250'27","reported_seq":547,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.954722+0000","last_change":"2024-02-10T07:43:17.985386+0000","last_active":"2024-02-10T07:44:43.954722+0000","last_peered":"2024-02-10T07:44:43.954722+0000","last_clean":"2024-02-10T07:44:43.954722+0000","last_became_active":"2024-02-10T07:31:53.286957+0000","last_became_peered":"2024-02-10T07:31:53.286957+0000","last_unstale":"2024-02-10T07:44:43.954722+0000","last_undegraded":"2024-02-10T07:44:43.954722+0000","last_fullsized":"2024-02-10T07:44:43.954722+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:43:17.984918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:17.984918+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":505,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.302168+0000","last_change":"2024-02-10T07:44:53.302209+0000","last_active":"2024-02-10T07:44:53.302168+0000","last_peered":"2024-02-10T07:44:53.302168+0000","last_clean":"2024-02-10T07:44:53.302168+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:44:53.302168+0000","last_undegraded":"2024-02-10T07:44:53.302168+0000","last_fullsized":"2024-02-10T07:44:53.302168+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:53.302048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:44:53.302048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:46:00.511864+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":468,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.952324+0000","last_change":"2024-02-10T07:43:15.186714+0000","last_active":"2024-02-10T07:44:43.952324+0000","last_peered":"2024-02-10T07:44:43.952324+0000","last_clean":"2024-02-10T07:44:43.952324+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:44:43.952324+0000","last_undegraded":"2024-02-10T07:44:43.952324+0000","last_fullsized":"2024-02-10T07:44:43.952324+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:15.186469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:15.186469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.190010+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":449,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.952428+0000","last_change":"2024-02-10T07:43:09.236406+0000","last_active":"2024-02-10T07:44:43.952428+0000","last_peered":"2024-02-10T07:44:43.952428+0000","last_clean":"2024-02-10T07:44:43.952428+0000","last_became_active":"2024-02-10T07:36:33.771340+0000","last_became_peered":"2024-02-10T07:36:33.771340+0000","last_unstale":"2024-02-10T07:44:43.952428+0000","last_undegraded":"2024-02-10T07:44:43.952428+0000","last_fullsized":"2024-02-10T07:44:43.952428+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:43:09.236268+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:34:49.349360+0000","last_clean_scrub_stamp":"2024-02-10T07:43:09.236268+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.189930+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"250'15","reported_seq":529,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.948534+0000","last_change":"2024-02-10T07:42:06.021547+0000","last_active":"2024-02-10T07:44:43.948534+0000","last_peered":"2024-02-10T07:44:43.948534+0000","last_clean":"2024-02-10T07:44:43.948534+0000","last_became_active":"2024-02-10T07:31:53.287858+0000","last_became_peered":"2024-02-10T07:31:53.287858+0000","last_unstale":"2024-02-10T07:44:43.948534+0000","last_undegraded":"2024-02-10T07:44:43.948534+0000","last_fullsized":"2024-02-10T07:44:43.948534+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:42:06.021407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:42:06.021407+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":468,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.951796+0000","last_change":"2024-02-10T07:44:00.364614+0000","last_active":"2024-02-10T07:44:43.951796+0000","last_peered":"2024-02-10T07:44:43.951796+0000","last_clean":"2024-02-10T07:44:43.951796+0000","last_became_active":"2024-02-10T07:36:33.757073+0000","last_became_peered":"2024-02-10T07:36:33.757073+0000","last_unstale":"2024-02-10T07:44:43.951796+0000","last_undegraded":"2024-02-10T07:44:43.951796+0000","last_fullsized":"2024-02-10T07:44:43.951796+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:00.364505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:44:00.364505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:06.001177+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":508,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.947933+0000","last_change":"2024-02-10T07:43:12.387217+0000","last_active":"2024-02-10T07:44:43.947933+0000","last_peered":"2024-02-10T07:44:43.947933+0000","last_clean":"2024-02-10T07:44:43.947933+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:44:43.947933+0000","last_undegraded":"2024-02-10T07:44:43.947933+0000","last_fullsized":"2024-02-10T07:44:43.947933+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:12.387113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:12.387113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":472,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.952086+0000","last_change":"2024-02-10T07:43:10.261985+0000","last_active":"2024-02-10T07:44:43.952086+0000","last_peered":"2024-02-10T07:44:43.952086+0000","last_clean":"2024-02-10T07:44:43.952086+0000","last_became_active":"2024-02-10T07:36:33.767970+0000","last_became_peered":"2024-02-10T07:36:33.767970+0000","last_unstale":"2024-02-10T07:44:43.952086+0000","last_undegraded":"2024-02-10T07:44:43.952086+0000","last_fullsized":"2024-02-10T07:44:43.952086+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:10.261904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.473686+0000","last_clean_scrub_stamp":"2024-02-10T07:43:10.261904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.189956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"250'7","reported_seq":517,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.949399+0000","last_change":"2024-02-10T07:43:45.444220+0000","last_active":"2024-02-10T07:44:43.949399+0000","last_peered":"2024-02-10T07:44:43.949399+0000","last_clean":"2024-02-10T07:44:43.949399+0000","last_became_active":"2024-02-10T07:31:53.292417+0000","last_became_peered":"2024-02-10T07:31:53.292417+0000","last_unstale":"2024-02-10T07:44:43.949399+0000","last_undegraded":"2024-02-10T07:44:43.949399+0000","last_fullsized":"2024-02-10T07:44:43.949399+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7","last_scrub_stamp":"2024-02-10T07:43:45.444068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:45.444068+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:05.997143+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":514,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:48.094209+0000","last_change":"2024-02-10T07:44:48.094231+0000","last_active":"2024-02-10T07:44:48.094209+0000","last_peered":"2024-02-10T07:44:48.094209+0000","last_clean":"2024-02-10T07:44:48.094209+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:44:48.094209+0000","last_undegraded":"2024-02-10T07:44:48.094209+0000","last_fullsized":"2024-02-10T07:44:48.094209+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:48.094138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:44:48.094138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:52.958018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":506,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.956024+0000","last_change":"2024-02-10T07:43:01.010982+0000","last_active":"2024-02-10T07:44:43.956024+0000","last_peered":"2024-02-10T07:44:43.956024+0000","last_clean":"2024-02-10T07:44:43.956024+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:44:43.956024+0000","last_undegraded":"2024-02-10T07:44:43.956024+0000","last_fullsized":"2024-02-10T07:44:43.956024+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:01.010877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:01.010877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.207026+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":517,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.010785+0000","last_change":"2024-02-10T07:44:01.315570+0000","last_active":"2024-02-10T07:44:53.010785+0000","last_peered":"2024-02-10T07:44:53.010785+0000","last_clean":"2024-02-10T07:44:53.010785+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:44:53.010785+0000","last_undegraded":"2024-02-10T07:44:53.010785+0000","last_fullsized":"2024-02-10T07:44:53.010785+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:01.315205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:44:01.315205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:18.186519+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"250'34","reported_seq":563,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.020604+0000","last_change":"2024-02-10T07:43:43.279558+0000","last_active":"2024-02-10T07:44:53.020604+0000","last_peered":"2024-02-10T07:44:53.020604+0000","last_clean":"2024-02-10T07:44:53.020604+0000","last_became_active":"2024-02-10T07:31:53.295527+0000","last_became_peered":"2024-02-10T07:31:53.295527+0000","last_unstale":"2024-02-10T07:44:53.020604+0000","last_undegraded":"2024-02-10T07:44:53.020604+0000","last_fullsized":"2024-02-10T07:44:53.020604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:43:43.279421+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:43:43.279421+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:57.100334+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":498,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.961082+0000","last_change":"2024-02-10T07:43:45.077379+0000","last_active":"2024-02-10T07:44:43.961082+0000","last_peered":"2024-02-10T07:44:43.961082+0000","last_clean":"2024-02-10T07:44:43.961082+0000","last_became_active":"2024-02-10T07:36:33.754341+0000","last_became_peered":"2024-02-10T07:36:33.754341+0000","last_unstale":"2024-02-10T07:44:43.961082+0000","last_undegraded":"2024-02-10T07:44:43.961082+0000","last_fullsized":"2024-02-10T07:44:43.961082+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:45.077266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:45.077266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:53.398106+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":500,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.949982+0000","last_change":"2024-02-10T07:41:28.853312+0000","last_active":"2024-02-10T07:44:43.949982+0000","last_peered":"2024-02-10T07:44:43.949982+0000","last_clean":"2024-02-10T07:44:43.949982+0000","last_became_active":"2024-02-10T07:31:55.298474+0000","last_became_peered":"2024-02-10T07:31:55.298474+0000","last_unstale":"2024-02-10T07:44:43.949982+0000","last_undegraded":"2024-02-10T07:44:43.949982+0000","last_fullsized":"2024-02-10T07:44:43.949982+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_clean_scrub_stamp":"2024-02-10T07:41:28.853233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":516,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.948668+0000","last_change":"2024-02-10T07:43:16.451188+0000","last_active":"2024-02-10T07:44:43.948668+0000","last_peered":"2024-02-10T07:44:43.948668+0000","last_clean":"2024-02-10T07:44:43.948668+0000","last_became_active":"2024-02-10T07:31:51.321237+0000","last_became_peered":"2024-02-10T07:31:51.321237+0000","last_unstale":"2024-02-10T07:44:43.948668+0000","last_undegraded":"2024-02-10T07:44:43.948668+0000","last_fullsized":"2024-02-10T07:44:43.948668+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:16.451067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:16.451067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:44.108579+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"250'18","reported_seq":536,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.956375+0000","last_change":"2024-02-10T07:43:16.944761+0000","last_active":"2024-02-10T07:44:43.956375+0000","last_peered":"2024-02-10T07:44:43.956375+0000","last_clean":"2024-02-10T07:44:43.956375+0000","last_became_active":"2024-02-10T07:31:53.294914+0000","last_became_peered":"2024-02-10T07:31:53.294914+0000","last_unstale":"2024-02-10T07:44:43.956375+0000","last_undegraded":"2024-02-10T07:44:43.956375+0000","last_fullsized":"2024-02-10T07:44:43.956375+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'18","last_scrub_stamp":"2024-02-10T07:43:16.944627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:16.944627+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":514,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:51.349871+0000","last_change":"2024-02-10T07:44:51.349920+0000","last_active":"2024-02-10T07:44:51.349871+0000","last_peered":"2024-02-10T07:44:51.349871+0000","last_clean":"2024-02-10T07:44:51.349871+0000","last_became_active":"2024-02-10T07:36:33.751352+0000","last_became_peered":"2024-02-10T07:36:33.751352+0000","last_unstale":"2024-02-10T07:44:51.349871+0000","last_undegraded":"2024-02-10T07:44:51.349871+0000","last_fullsized":"2024-02-10T07:44:51.349871+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:44:51.349776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:44:51.349776+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:46:03.551865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":692,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:46.527498+0000","last_change":"2024-02-10T07:43:41.166832+0000","last_active":"2024-02-10T07:44:46.527498+0000","last_peered":"2024-02-10T07:44:46.527498+0000","last_clean":"2024-02-10T07:44:46.527498+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:44:46.527498+0000","last_undegraded":"2024-02-10T07:44:46.527498+0000","last_fullsized":"2024-02-10T07:44:46.527498+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:43:41.166747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:41.166747+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:07.711794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":515,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.954029+0000","last_change":"2024-02-10T07:43:07.962862+0000","last_active":"2024-02-10T07:44:43.954029+0000","last_peered":"2024-02-10T07:44:43.954029+0000","last_clean":"2024-02-10T07:44:43.954029+0000","last_became_active":"2024-02-10T07:31:51.282077+0000","last_became_peered":"2024-02-10T07:31:51.282077+0000","last_unstale":"2024-02-10T07:44:43.954029+0000","last_undegraded":"2024-02-10T07:44:43.954029+0000","last_fullsized":"2024-02-10T07:44:43.954029+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:43:07.962748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:07.962748+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.207086+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"250'25","reported_seq":557,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.015194+0000","last_change":"2024-02-10T07:44:06.180569+0000","last_active":"2024-02-10T07:44:53.015194+0000","last_peered":"2024-02-10T07:44:53.015194+0000","last_clean":"2024-02-10T07:44:53.015194+0000","last_became_active":"2024-02-10T07:31:53.292936+0000","last_became_peered":"2024-02-10T07:31:53.292936+0000","last_unstale":"2024-02-10T07:44:53.015194+0000","last_undegraded":"2024-02-10T07:44:53.015194+0000","last_fullsized":"2024-02-10T07:44:53.015194+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:44:06.180439+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-02-10T07:42:17.820959+0000","last_clean_scrub_stamp":"2024-02-10T07:44:06.180439+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:09.980808+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":511,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.956153+0000","last_change":"2024-02-10T07:44:05.558893+0000","last_active":"2024-02-10T07:44:43.956153+0000","last_peered":"2024-02-10T07:44:43.956153+0000","last_clean":"2024-02-10T07:44:43.956153+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:44:43.956153+0000","last_undegraded":"2024-02-10T07:44:43.956153+0000","last_fullsized":"2024-02-10T07:44:43.956153+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_clean_scrub_stamp":"2024-02-10T07:44:05.558659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:22.648251+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":502,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.015464+0000","last_change":"2024-02-10T07:43:04.051025+0000","last_active":"2024-02-10T07:44:53.015464+0000","last_peered":"2024-02-10T07:44:53.015464+0000","last_clean":"2024-02-10T07:44:53.015464+0000","last_became_active":"2024-02-10T07:31:55.297595+0000","last_became_peered":"2024-02-10T07:31:55.297595+0000","last_unstale":"2024-02-10T07:44:53.015464+0000","last_undegraded":"2024-02-10T07:44:53.015464+0000","last_fullsized":"2024-02-10T07:44:53.015464+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:04.050906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:04.050906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":508,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.963921+0000","last_change":"2024-02-10T07:43:59.897845+0000","last_active":"2024-02-10T07:44:43.963921+0000","last_peered":"2024-02-10T07:44:43.963921+0000","last_clean":"2024-02-10T07:44:43.963921+0000","last_became_active":"2024-02-10T07:31:51.285405+0000","last_became_peered":"2024-02-10T07:31:51.285405+0000","last_unstale":"2024-02-10T07:44:43.963921+0000","last_undegraded":"2024-02-10T07:44:43.963921+0000","last_fullsized":"2024-02-10T07:44:43.963921+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:59.897726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:59.897726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:17.362848+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"250'19","reported_seq":560,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:44.216906+0000","last_change":"2024-02-10T07:44:44.216951+0000","last_active":"2024-02-10T07:44:44.216906+0000","last_peered":"2024-02-10T07:44:44.216906+0000","last_clean":"2024-02-10T07:44:44.216906+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:44:44.216906+0000","last_undegraded":"2024-02-10T07:44:44.216906+0000","last_fullsized":"2024-02-10T07:44:44.216906+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:44:44.216823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:44:44.216823+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:46:14.138040+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":504,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.947773+0000","last_change":"2024-02-10T07:43:42.484221+0000","last_active":"2024-02-10T07:44:43.947773+0000","last_peered":"2024-02-10T07:44:43.947773+0000","last_clean":"2024-02-10T07:44:43.947773+0000","last_became_active":"2024-02-10T07:31:57.306776+0000","last_became_peered":"2024-02-10T07:31:57.306776+0000","last_unstale":"2024-02-10T07:44:43.947773+0000","last_undegraded":"2024-02-10T07:44:43.947773+0000","last_fullsized":"2024-02-10T07:44:43.947773+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:42.484093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:43:42.484093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:05.722649+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":515,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.967483+0000","last_change":"2024-02-10T07:43:50.101849+0000","last_active":"2024-02-10T07:44:43.967483+0000","last_peered":"2024-02-10T07:44:43.967483+0000","last_clean":"2024-02-10T07:44:43.967483+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:44:43.967483+0000","last_undegraded":"2024-02-10T07:44:43.967483+0000","last_fullsized":"2024-02-10T07:44:43.967483+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:50.101724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:50.101724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:00.993819+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":526,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.013312+0000","last_change":"2024-02-10T07:43:12.131490+0000","last_active":"2024-02-10T07:44:53.013312+0000","last_peered":"2024-02-10T07:44:53.013312+0000","last_clean":"2024-02-10T07:44:53.013312+0000","last_became_active":"2024-02-10T07:36:33.760213+0000","last_became_peered":"2024-02-10T07:36:33.760213+0000","last_unstale":"2024-02-10T07:44:53.013312+0000","last_undegraded":"2024-02-10T07:44:53.013312+0000","last_fullsized":"2024-02-10T07:44:53.013312+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:12.131342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:12.131342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"250'10","reported_seq":510,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.964009+0000","last_change":"2024-02-10T07:43:06.022311+0000","last_active":"2024-02-10T07:44:43.964009+0000","last_peered":"2024-02-10T07:44:43.964009+0000","last_clean":"2024-02-10T07:44:43.964009+0000","last_became_active":"2024-02-10T07:31:53.295520+0000","last_became_peered":"2024-02-10T07:31:53.295520+0000","last_unstale":"2024-02-10T07:44:43.964009+0000","last_undegraded":"2024-02-10T07:44:43.964009+0000","last_fullsized":"2024-02-10T07:44:43.964009+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'10","last_scrub_stamp":"2024-02-10T07:43:06.022190+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:43:06.022190+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":496,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.955290+0000","last_change":"2024-02-10T07:43:43.019843+0000","last_active":"2024-02-10T07:44:43.955290+0000","last_peered":"2024-02-10T07:44:43.955290+0000","last_clean":"2024-02-10T07:44:43.955290+0000","last_became_active":"2024-02-10T07:31:57.315738+0000","last_became_peered":"2024-02-10T07:31:57.315738+0000","last_unstale":"2024-02-10T07:44:43.955290+0000","last_undegraded":"2024-02-10T07:44:43.955290+0000","last_fullsized":"2024-02-10T07:44:43.955290+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:43.019550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:43.019550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:01.353357+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":508,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.947445+0000","last_change":"2024-02-10T07:43:44.490369+0000","last_active":"2024-02-10T07:44:43.947445+0000","last_peered":"2024-02-10T07:44:43.947445+0000","last_clean":"2024-02-10T07:44:43.947445+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:44:43.947445+0000","last_undegraded":"2024-02-10T07:44:43.947445+0000","last_fullsized":"2024-02-10T07:44:43.947445+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:44.490246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:44.490246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:58.669303+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":432,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.952381+0000","last_change":"2024-02-10T07:42:21.106187+0000","last_active":"2024-02-10T07:44:43.952381+0000","last_peered":"2024-02-10T07:44:43.952381+0000","last_clean":"2024-02-10T07:44:43.952381+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:44:43.952381+0000","last_undegraded":"2024-02-10T07:44:43.952381+0000","last_fullsized":"2024-02-10T07:44:43.952381+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:21.106090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:42:21.106090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.189876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"250'19","reported_seq":535,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.954629+0000","last_change":"2024-02-10T07:43:40.896052+0000","last_active":"2024-02-10T07:44:43.954629+0000","last_peered":"2024-02-10T07:44:43.954629+0000","last_clean":"2024-02-10T07:44:43.954629+0000","last_became_active":"2024-02-10T07:31:53.287823+0000","last_became_peered":"2024-02-10T07:31:53.287823+0000","last_unstale":"2024-02-10T07:44:43.954629+0000","last_undegraded":"2024-02-10T07:44:43.954629+0000","last_fullsized":"2024-02-10T07:44:43.954629+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:43:40.895108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:40.895108+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:09.724187+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":506,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.007974+0000","last_change":"2024-02-10T07:44:48.276900+0000","last_active":"2024-02-10T07:44:53.007974+0000","last_peered":"2024-02-10T07:44:53.007974+0000","last_clean":"2024-02-10T07:44:53.007974+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:44:53.007974+0000","last_undegraded":"2024-02-10T07:44:53.007974+0000","last_fullsized":"2024-02-10T07:44:53.007974+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:48.276778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:44:48.276778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:53.654741+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":500,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.955047+0000","last_change":"2024-02-10T07:43:46.833839+0000","last_active":"2024-02-10T07:44:43.955047+0000","last_peered":"2024-02-10T07:44:43.955047+0000","last_clean":"2024-02-10T07:44:43.955047+0000","last_became_active":"2024-02-10T07:31:55.296670+0000","last_became_peered":"2024-02-10T07:31:55.296670+0000","last_unstale":"2024-02-10T07:44:43.955047+0000","last_undegraded":"2024-02-10T07:44:43.955047+0000","last_fullsized":"2024-02-10T07:44:43.955047+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:46.833613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:46.833613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:59.787265+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":516,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.948296+0000","last_change":"2024-02-10T07:43:06.386801+0000","last_active":"2024-02-10T07:44:43.948296+0000","last_peered":"2024-02-10T07:44:43.948296+0000","last_clean":"2024-02-10T07:44:43.948296+0000","last_became_active":"2024-02-10T07:31:51.288165+0000","last_became_peered":"2024-02-10T07:31:51.288165+0000","last_unstale":"2024-02-10T07:44:43.948296+0000","last_undegraded":"2024-02-10T07:44:43.948296+0000","last_fullsized":"2024-02-10T07:44:43.948296+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:06.386654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:06.386654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"250'19","reported_seq":317,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.958428+0000","last_change":"2024-02-10T07:44:08.503149+0000","last_active":"2024-02-10T07:44:43.958428+0000","last_peered":"2024-02-10T07:44:43.958428+0000","last_clean":"2024-02-10T07:44:43.958428+0000","last_became_active":"2024-02-10T07:37:06.187551+0000","last_became_peered":"2024-02-10T07:37:06.187551+0000","last_unstale":"2024-02-10T07:44:43.958428+0000","last_undegraded":"2024-02-10T07:44:43.958428+0000","last_fullsized":"2024-02-10T07:44:43.958428+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:44:08.503058+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-02-10T07:39:32.934269+0000","last_clean_scrub_stamp":"2024-02-10T07:44:08.503058+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:37.824115+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":497,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.954328+0000","last_change":"2024-02-10T07:43:11.935665+0000","last_active":"2024-02-10T07:44:43.954328+0000","last_peered":"2024-02-10T07:44:43.954328+0000","last_clean":"2024-02-10T07:44:43.954328+0000","last_became_active":"2024-02-10T07:36:33.750030+0000","last_became_peered":"2024-02-10T07:36:33.750030+0000","last_unstale":"2024-02-10T07:44:43.954328+0000","last_undegraded":"2024-02-10T07:44:43.954328+0000","last_fullsized":"2024-02-10T07:44:43.954328+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:11.935253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:11.935253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":515,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.959225+0000","last_change":"2024-02-10T07:43:19.008513+0000","last_active":"2024-02-10T07:44:43.959225+0000","last_peered":"2024-02-10T07:44:43.959225+0000","last_clean":"2024-02-10T07:44:43.959225+0000","last_became_active":"2024-02-10T07:36:33.752418+0000","last_became_peered":"2024-02-10T07:36:33.752418+0000","last_unstale":"2024-02-10T07:44:43.959225+0000","last_undegraded":"2024-02-10T07:44:43.959225+0000","last_fullsized":"2024-02-10T07:44:43.959225+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:19.008018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:19.008018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.207110+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":516,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.949729+0000","last_change":"2024-02-10T07:44:04.423152+0000","last_active":"2024-02-10T07:44:43.949729+0000","last_peered":"2024-02-10T07:44:43.949729+0000","last_clean":"2024-02-10T07:44:43.949729+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:44:43.949729+0000","last_undegraded":"2024-02-10T07:44:43.949729+0000","last_fullsized":"2024-02-10T07:44:43.949729+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:04.423015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:44:04.423015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:09.200801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"250'24","reported_seq":566,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.949197+0000","last_change":"2024-02-10T07:44:02.402382+0000","last_active":"2024-02-10T07:44:43.949197+0000","last_peered":"2024-02-10T07:44:43.949197+0000","last_clean":"2024-02-10T07:44:43.949197+0000","last_became_active":"2024-02-10T07:32:18.389564+0000","last_became_peered":"2024-02-10T07:32:18.389564+0000","last_unstale":"2024-02-10T07:44:43.949197+0000","last_undegraded":"2024-02-10T07:44:43.949197+0000","last_fullsized":"2024-02-10T07:44:43.949197+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:44:02.402293+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:44:02.402293+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:28.724385+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":505,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.014867+0000","last_change":"2024-02-10T07:44:00.234087+0000","last_active":"2024-02-10T07:44:53.014867+0000","last_peered":"2024-02-10T07:44:53.014867+0000","last_clean":"2024-02-10T07:44:53.014867+0000","last_became_active":"2024-02-10T07:31:57.318009+0000","last_became_peered":"2024-02-10T07:31:57.318009+0000","last_unstale":"2024-02-10T07:44:53.014867+0000","last_undegraded":"2024-02-10T07:44:53.014867+0000","last_fullsized":"2024-02-10T07:44:53.014867+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:00.233946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:44:00.233946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:16.097710+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"158'10","reported_seq":582,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:46.527700+0000","last_change":"2024-02-10T07:42:15.162807+0000","last_active":"2024-02-10T07:44:46.527700+0000","last_peered":"2024-02-10T07:44:46.527700+0000","last_clean":"2024-02-10T07:44:46.527700+0000","last_became_active":"2024-02-10T07:36:33.774773+0000","last_became_peered":"2024-02-10T07:36:33.774773+0000","last_unstale":"2024-02-10T07:44:46.527700+0000","last_undegraded":"2024-02-10T07:44:46.527700+0000","last_fullsized":"2024-02-10T07:44:46.527700+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:42:15.162654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:42:15.162654+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":524,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:47.431110+0000","last_change":"2024-02-10T07:44:47.431145+0000","last_active":"2024-02-10T07:44:47.431110+0000","last_peered":"2024-02-10T07:44:47.431110+0000","last_clean":"2024-02-10T07:44:47.431110+0000","last_became_active":"2024-02-10T07:31:51.320644+0000","last_became_peered":"2024-02-10T07:31:51.320644+0000","last_unstale":"2024-02-10T07:44:47.431110+0000","last_undegraded":"2024-02-10T07:44:47.431110+0000","last_fullsized":"2024-02-10T07:44:47.431110+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:47.431041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:44:47.431041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:46:05.720607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"250'25","reported_seq":357,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.952028+0000","last_change":"2024-02-10T07:43:56.304513+0000","last_active":"2024-02-10T07:44:43.952028+0000","last_peered":"2024-02-10T07:44:43.952028+0000","last_clean":"2024-02-10T07:44:43.952028+0000","last_became_active":"2024-02-10T07:36:33.752317+0000","last_became_peered":"2024-02-10T07:36:33.752317+0000","last_unstale":"2024-02-10T07:44:43.952028+0000","last_undegraded":"2024-02-10T07:44:43.952028+0000","last_fullsized":"2024-02-10T07:44:43.952028+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:43:56.304364+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:43:56.304364+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:24.039285+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":505,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.950837+0000","last_change":"2024-02-10T07:43:46.588398+0000","last_active":"2024-02-10T07:44:43.950837+0000","last_peered":"2024-02-10T07:44:43.950837+0000","last_clean":"2024-02-10T07:44:43.950837+0000","last_became_active":"2024-02-10T07:36:33.766051+0000","last_became_peered":"2024-02-10T07:36:33.766051+0000","last_unstale":"2024-02-10T07:44:43.950837+0000","last_undegraded":"2024-02-10T07:44:43.950837+0000","last_fullsized":"2024-02-10T07:44:43.950837+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:46.588253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:46.588253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:56.838799+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":664,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.956855+0000","last_change":"2024-02-10T07:43:55.848459+0000","last_active":"2024-02-10T07:44:43.956855+0000","last_peered":"2024-02-10T07:44:43.956855+0000","last_clean":"2024-02-10T07:44:43.956855+0000","last_became_active":"2024-02-10T07:31:55.294594+0000","last_became_peered":"2024-02-10T07:31:55.294594+0000","last_unstale":"2024-02-10T07:44:43.956855+0000","last_undegraded":"2024-02-10T07:44:43.956855+0000","last_fullsized":"2024-02-10T07:44:43.956855+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:43:55.848231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:55.848231+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:01.223931+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":517,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.008092+0000","last_change":"2024-02-10T07:43:19.230062+0000","last_active":"2024-02-10T07:44:53.008092+0000","last_peered":"2024-02-10T07:44:53.008092+0000","last_clean":"2024-02-10T07:44:53.008092+0000","last_became_active":"2024-02-10T07:31:51.282353+0000","last_became_peered":"2024-02-10T07:31:51.282353+0000","last_unstale":"2024-02-10T07:44:53.008092+0000","last_undegraded":"2024-02-10T07:44:53.008092+0000","last_fullsized":"2024-02-10T07:44:53.008092+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:19.229934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:19.229934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"250'15","reported_seq":481,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.952798+0000","last_change":"2024-02-10T07:43:02.231786+0000","last_active":"2024-02-10T07:44:43.952798+0000","last_peered":"2024-02-10T07:44:43.952798+0000","last_clean":"2024-02-10T07:44:43.952798+0000","last_became_active":"2024-02-10T07:36:33.769894+0000","last_became_peered":"2024-02-10T07:36:33.769894+0000","last_unstale":"2024-02-10T07:44:43.952798+0000","last_undegraded":"2024-02-10T07:44:43.952798+0000","last_fullsized":"2024-02-10T07:44:43.952798+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:43:02.231670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:02.231670+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.189986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":521,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.020892+0000","last_change":"2024-02-10T07:44:46.781578+0000","last_active":"2024-02-10T07:44:53.020892+0000","last_peered":"2024-02-10T07:44:53.020892+0000","last_clean":"2024-02-10T07:44:53.020892+0000","last_became_active":"2024-02-10T07:31:57.308615+0000","last_became_peered":"2024-02-10T07:31:57.308615+0000","last_unstale":"2024-02-10T07:44:53.020892+0000","last_undegraded":"2024-02-10T07:44:53.020892+0000","last_fullsized":"2024-02-10T07:44:53.020892+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:46.781458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:44:46.781458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:46:00.084375+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":492,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.961542+0000","last_change":"2024-02-10T07:43:01.179228+0000","last_active":"2024-02-10T07:44:43.961542+0000","last_peered":"2024-02-10T07:44:43.961542+0000","last_clean":"2024-02-10T07:44:43.961542+0000","last_became_active":"2024-02-10T07:31:55.297431+0000","last_became_peered":"2024-02-10T07:31:55.297431+0000","last_unstale":"2024-02-10T07:44:43.961542+0000","last_undegraded":"2024-02-10T07:44:43.961542+0000","last_fullsized":"2024-02-10T07:44:43.961542+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:01.179025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:43:01.179025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":516,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.947784+0000","last_change":"2024-02-10T07:42:59.403762+0000","last_active":"2024-02-10T07:44:43.947784+0000","last_peered":"2024-02-10T07:44:43.947784+0000","last_clean":"2024-02-10T07:44:43.947784+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:44:43.947784+0000","last_undegraded":"2024-02-10T07:44:43.947784+0000","last_fullsized":"2024-02-10T07:44:43.947784+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:59.403332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:42:59.403332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"250'19","reported_seq":550,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.015292+0000","last_change":"2024-02-10T07:44:08.130835+0000","last_active":"2024-02-10T07:44:53.015292+0000","last_peered":"2024-02-10T07:44:53.015292+0000","last_clean":"2024-02-10T07:44:53.015292+0000","last_became_active":"2024-02-10T07:31:53.295353+0000","last_became_peered":"2024-02-10T07:31:53.295353+0000","last_unstale":"2024-02-10T07:44:53.015292+0000","last_undegraded":"2024-02-10T07:44:53.015292+0000","last_fullsized":"2024-02-10T07:44:53.015292+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:44:08.130707+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:44:08.130707+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:27.190414+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":496,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.948216+0000","last_change":"2024-02-10T07:43:09.421682+0000","last_active":"2024-02-10T07:44:43.948216+0000","last_peered":"2024-02-10T07:44:43.948216+0000","last_clean":"2024-02-10T07:44:43.948216+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:44:43.948216+0000","last_undegraded":"2024-02-10T07:44:43.948216+0000","last_fullsized":"2024-02-10T07:44:43.948216+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:09.421372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:09.421372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":456,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.952472+0000","last_change":"2024-02-10T07:42:07.138453+0000","last_active":"2024-02-10T07:44:43.952472+0000","last_peered":"2024-02-10T07:44:43.952472+0000","last_clean":"2024-02-10T07:44:43.952472+0000","last_became_active":"2024-02-10T07:36:33.752051+0000","last_became_peered":"2024-02-10T07:36:33.752051+0000","last_unstale":"2024-02-10T07:44:43.952472+0000","last_undegraded":"2024-02-10T07:44:43.952472+0000","last_fullsized":"2024-02-10T07:44:43.952472+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:07.138357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:42:07.138357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.189676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":532,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.969395+0000","last_change":"2024-02-10T07:43:47.053223+0000","last_active":"2024-02-10T07:44:43.969395+0000","last_peered":"2024-02-10T07:44:43.969395+0000","last_clean":"2024-02-10T07:44:43.969395+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:44:43.969395+0000","last_undegraded":"2024-02-10T07:44:43.969395+0000","last_fullsized":"2024-02-10T07:44:43.969395+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:47.052981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:47.052981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:52.720988+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"250'12","reported_seq":514,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.961358+0000","last_change":"2024-02-10T07:43:08.018116+0000","last_active":"2024-02-10T07:44:43.961358+0000","last_peered":"2024-02-10T07:44:43.961358+0000","last_clean":"2024-02-10T07:44:43.961358+0000","last_became_active":"2024-02-10T07:31:53.296831+0000","last_became_peered":"2024-02-10T07:31:53.296831+0000","last_unstale":"2024-02-10T07:44:43.961358+0000","last_undegraded":"2024-02-10T07:44:43.961358+0000","last_fullsized":"2024-02-10T07:44:43.961358+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'12","last_scrub_stamp":"2024-02-10T07:43:08.017802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:08.017802+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"294'12060","reported_seq":12238,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.849622+0000","last_change":"2024-02-10T07:43:44.147560+0000","last_active":"2024-02-10T07:44:53.849622+0000","last_peered":"2024-02-10T07:44:53.849622+0000","last_clean":"2024-02-10T07:44:53.849622+0000","last_became_active":"2024-02-10T07:42:09.176416+0000","last_became_peered":"2024-02-10T07:42:09.176416+0000","last_unstale":"2024-02-10T07:44:53.849622+0000","last_undegraded":"2024-02-10T07:44:53.849622+0000","last_fullsized":"2024-02-10T07:44:53.849622+0000","mapping_epoch":253,"log_start":"272'6000","ondisk_log_start":"272'6000","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'7616","last_scrub_stamp":"2024-02-10T07:43:44.147381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:44.147381+0000","objects_scrubbed":7544,"log_size":6060,"log_dups_size":0,"ondisk_log_size":6060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2922,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265420800,"num_objects":4050,"num_object_clones":0,"num_object_copies":12150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12060,"num_write_kb":515520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":468,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.952537+0000","last_change":"2024-02-10T07:43:40.413257+0000","last_active":"2024-02-10T07:44:43.952537+0000","last_peered":"2024-02-10T07:44:43.952537+0000","last_clean":"2024-02-10T07:44:43.952537+0000","last_became_active":"2024-02-10T07:36:33.763431+0000","last_became_peered":"2024-02-10T07:36:33.763431+0000","last_unstale":"2024-02-10T07:44:43.952537+0000","last_undegraded":"2024-02-10T07:44:43.952537+0000","last_fullsized":"2024-02-10T07:44:43.952537+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:40.413128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:40.413128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:48.471922+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"245'4","reported_seq":530,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.948676+0000","last_change":"2024-02-10T07:43:05.338877+0000","last_active":"2024-02-10T07:44:43.948676+0000","last_peered":"2024-02-10T07:44:43.948676+0000","last_clean":"2024-02-10T07:44:43.948676+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:44:43.948676+0000","last_undegraded":"2024-02-10T07:44:43.948676+0000","last_fullsized":"2024-02-10T07:44:43.948676+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'4","last_scrub_stamp":"2024-02-10T07:43:05.338740+0000","last_deep_scrub":"245'4","last_deep_scrub_stamp":"2024-02-10T07:41:23.717923+0000","last_clean_scrub_stamp":"2024-02-10T07:43:05.338740+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"250'34","reported_seq":484,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.952470+0000","last_change":"2024-02-10T07:43:55.309700+0000","last_active":"2024-02-10T07:44:43.952470+0000","last_peered":"2024-02-10T07:44:43.952470+0000","last_clean":"2024-02-10T07:44:43.952470+0000","last_became_active":"2024-02-10T07:36:33.754134+0000","last_became_peered":"2024-02-10T07:36:33.754134+0000","last_unstale":"2024-02-10T07:44:43.952470+0000","last_undegraded":"2024-02-10T07:44:43.952470+0000","last_fullsized":"2024-02-10T07:44:43.952470+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:43:55.309563+0000","last_deep_scrub":"246'22","last_deep_scrub_stamp":"2024-02-10T07:41:26.771112+0000","last_clean_scrub_stamp":"2024-02-10T07:43:55.309563+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:59.444281+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":681,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:46.527536+0000","last_change":"2024-02-10T07:44:13.245977+0000","last_active":"2024-02-10T07:44:46.527536+0000","last_peered":"2024-02-10T07:44:46.527536+0000","last_clean":"2024-02-10T07:44:46.527536+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:44:46.527536+0000","last_undegraded":"2024-02-10T07:44:46.527536+0000","last_fullsized":"2024-02-10T07:44:46.527536+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:44:13.245900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:44:13.245900+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:31.843130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"293'11651","reported_seq":11849,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:48.938039+0000","last_change":"2024-02-10T07:43:45.786143+0000","last_active":"2024-02-10T07:44:48.938039+0000","last_peered":"2024-02-10T07:44:48.938039+0000","last_clean":"2024-02-10T07:44:48.938039+0000","last_became_active":"2024-02-10T07:42:09.451050+0000","last_became_peered":"2024-02-10T07:42:09.451050+0000","last_unstale":"2024-02-10T07:44:48.938039+0000","last_undegraded":"2024-02-10T07:44:48.938039+0000","last_fullsized":"2024-02-10T07:44:48.938039+0000","mapping_epoch":253,"log_start":"277'6800","ondisk_log_start":"277'6800","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'7684","last_scrub_stamp":"2024-02-10T07:43:45.785994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:45.785994+0000","objects_scrubbed":7601,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:01.148494+0000","scrub_duration":2593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289603584,"num_objects":4419,"num_object_clones":0,"num_object_copies":13257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11651,"num_write_kb":514240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":488,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.955211+0000","last_change":"2024-02-10T07:43:02.251344+0000","last_active":"2024-02-10T07:44:43.955211+0000","last_peered":"2024-02-10T07:44:43.955211+0000","last_clean":"2024-02-10T07:44:43.955211+0000","last_became_active":"2024-02-10T07:31:57.314872+0000","last_became_peered":"2024-02-10T07:31:57.314872+0000","last_unstale":"2024-02-10T07:44:43.955211+0000","last_undegraded":"2024-02-10T07:44:43.955211+0000","last_fullsized":"2024-02-10T07:44:43.955211+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:02.251208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:02.251208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":516,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.950081+0000","last_change":"2024-02-10T07:43:19.419259+0000","last_active":"2024-02-10T07:44:43.950081+0000","last_peered":"2024-02-10T07:44:43.950081+0000","last_clean":"2024-02-10T07:44:43.950081+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:44:43.950081+0000","last_undegraded":"2024-02-10T07:44:43.950081+0000","last_fullsized":"2024-02-10T07:44:43.950081+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:19.419141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:43:19.419141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:48.757225+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"250'23","reported_seq":567,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:45.144214+0000","last_change":"2024-02-10T07:44:45.144260+0000","last_active":"2024-02-10T07:44:45.144214+0000","last_peered":"2024-02-10T07:44:45.144214+0000","last_clean":"2024-02-10T07:44:45.144214+0000","last_became_active":"2024-02-10T07:32:18.389409+0000","last_became_peered":"2024-02-10T07:32:18.389409+0000","last_unstale":"2024-02-10T07:44:45.144214+0000","last_undegraded":"2024-02-10T07:44:45.144214+0000","last_fullsized":"2024-02-10T07:44:45.144214+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-02-10T07:44:45.144119+0000","last_deep_scrub":"245'17","last_deep_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_clean_scrub_stamp":"2024-02-10T07:44:45.144119+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:47.186136+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":517,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.022774+0000","last_change":"2024-02-10T07:43:56.145450+0000","last_active":"2024-02-10T07:44:53.022774+0000","last_peered":"2024-02-10T07:44:53.022774+0000","last_clean":"2024-02-10T07:44:53.022774+0000","last_became_active":"2024-02-10T07:31:55.303059+0000","last_became_peered":"2024-02-10T07:31:55.303059+0000","last_unstale":"2024-02-10T07:44:53.022774+0000","last_undegraded":"2024-02-10T07:44:53.022774+0000","last_fullsized":"2024-02-10T07:44:53.022774+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:56.145004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:56.145004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:22.618065+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"293'11868","reported_seq":11959,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:52.136404+0000","last_change":"2024-02-10T07:42:09.176546+0000","last_active":"2024-02-10T07:44:52.136404+0000","last_peered":"2024-02-10T07:44:52.136404+0000","last_clean":"2024-02-10T07:44:52.136404+0000","last_became_active":"2024-02-10T07:42:09.175986+0000","last_became_peered":"2024-02-10T07:42:09.175986+0000","last_unstale":"2024-02-10T07:44:52.136404+0000","last_undegraded":"2024-02-10T07:44:52.136404+0000","last_fullsized":"2024-02-10T07:44:52.136404+0000","mapping_epoch":253,"log_start":"276'6600","ondisk_log_start":"276'6600","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:42:08.158083+0000","objects_scrubbed":0,"log_size":5268,"log_dups_size":0,"ondisk_log_size":5268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264896512,"num_objects":4042,"num_object_clones":0,"num_object_copies":12126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11868,"num_write_kb":509120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":512,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.946860+0000","last_change":"2024-02-10T07:43:48.467475+0000","last_active":"2024-02-10T07:44:43.946860+0000","last_peered":"2024-02-10T07:44:43.946860+0000","last_clean":"2024-02-10T07:44:43.946860+0000","last_became_active":"2024-02-10T07:31:57.304971+0000","last_became_peered":"2024-02-10T07:31:57.304971+0000","last_unstale":"2024-02-10T07:44:43.946860+0000","last_undegraded":"2024-02-10T07:44:43.946860+0000","last_fullsized":"2024-02-10T07:44:43.946860+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:48.467367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:36.741855+0000","last_clean_scrub_stamp":"2024-02-10T07:43:48.467367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:01.277638+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":526,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.020450+0000","last_change":"2024-02-10T07:43:46.261370+0000","last_active":"2024-02-10T07:44:53.020450+0000","last_peered":"2024-02-10T07:44:53.020450+0000","last_clean":"2024-02-10T07:44:53.020450+0000","last_became_active":"2024-02-10T07:36:33.754602+0000","last_became_peered":"2024-02-10T07:36:33.754602+0000","last_unstale":"2024-02-10T07:44:53.020450+0000","last_undegraded":"2024-02-10T07:44:53.020450+0000","last_fullsized":"2024-02-10T07:44:53.020450+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:46.261253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:46.261253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:58.623684+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":525,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.948655+0000","last_change":"2024-02-10T07:43:03.294877+0000","last_active":"2024-02-10T07:44:43.948655+0000","last_peered":"2024-02-10T07:44:43.948655+0000","last_clean":"2024-02-10T07:44:43.948655+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:44:43.948655+0000","last_undegraded":"2024-02-10T07:44:43.948655+0000","last_fullsized":"2024-02-10T07:44:43.948655+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:03.294778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:03.294778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"250'35","reported_seq":582,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:48.190310+0000","last_change":"2024-02-10T07:44:48.190357+0000","last_active":"2024-02-10T07:44:48.190310+0000","last_peered":"2024-02-10T07:44:48.190310+0000","last_clean":"2024-02-10T07:44:48.190310+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:44:48.190310+0000","last_undegraded":"2024-02-10T07:44:48.190310+0000","last_fullsized":"2024-02-10T07:44:48.190310+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'35","last_scrub_stamp":"2024-02-10T07:44:48.190239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:44:48.190239+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:46:04.445013+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"293'12113","reported_seq":12283,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:52.127164+0000","last_change":"2024-02-10T07:44:09.296562+0000","last_active":"2024-02-10T07:44:52.127164+0000","last_peered":"2024-02-10T07:44:52.127164+0000","last_clean":"2024-02-10T07:44:52.127164+0000","last_became_active":"2024-02-10T07:42:09.395485+0000","last_became_peered":"2024-02-10T07:42:09.395485+0000","last_unstale":"2024-02-10T07:44:52.127164+0000","last_undegraded":"2024-02-10T07:44:52.127164+0000","last_fullsized":"2024-02-10T07:44:52.127164+0000","mapping_epoch":253,"log_start":"276'6800","ondisk_log_start":"276'6800","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'9251","last_scrub_stamp":"2024-02-10T07:44:09.296369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:44:09.296369+0000","objects_scrubbed":7105,"log_size":5313,"log_dups_size":0,"ondisk_log_size":5313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:35.841544+0000","scrub_duration":2022,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274792448,"num_objects":4193,"num_object_clones":0,"num_object_copies":12579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12113,"num_write_kb":521792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":505,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.008370+0000","last_change":"2024-02-10T07:43:40.302514+0000","last_active":"2024-02-10T07:44:53.008370+0000","last_peered":"2024-02-10T07:44:53.008370+0000","last_clean":"2024-02-10T07:44:53.008370+0000","last_became_active":"2024-02-10T07:31:57.313394+0000","last_became_peered":"2024-02-10T07:31:57.313394+0000","last_unstale":"2024-02-10T07:44:53.008370+0000","last_undegraded":"2024-02-10T07:44:53.008370+0000","last_fullsized":"2024-02-10T07:44:53.008370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:40.302404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:40.302404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":517,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.036276+0000","last_change":"2024-02-10T07:43:51.246335+0000","last_active":"2024-02-10T07:44:53.036276+0000","last_peered":"2024-02-10T07:44:53.036276+0000","last_clean":"2024-02-10T07:44:53.036276+0000","last_became_active":"2024-02-10T07:31:55.298622+0000","last_became_peered":"2024-02-10T07:31:55.298622+0000","last_unstale":"2024-02-10T07:44:53.036276+0000","last_undegraded":"2024-02-10T07:44:53.036276+0000","last_fullsized":"2024-02-10T07:44:53.036276+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:51.245746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:51.245746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:56.919752+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":472,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.951925+0000","last_change":"2024-02-10T07:43:59.363625+0000","last_active":"2024-02-10T07:44:43.951925+0000","last_peered":"2024-02-10T07:44:43.951925+0000","last_clean":"2024-02-10T07:44:43.951925+0000","last_became_active":"2024-02-10T07:36:33.762155+0000","last_became_peered":"2024-02-10T07:36:33.762155+0000","last_unstale":"2024-02-10T07:44:43.951925+0000","last_undegraded":"2024-02-10T07:44:43.951925+0000","last_fullsized":"2024-02-10T07:44:43.951925+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:59.363500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:59.363500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:23.913803+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"250'15","reported_seq":538,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.010987+0000","last_change":"2024-02-10T07:43:39.224099+0000","last_active":"2024-02-10T07:44:53.010987+0000","last_peered":"2024-02-10T07:44:53.010987+0000","last_clean":"2024-02-10T07:44:53.010987+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:44:53.010987+0000","last_undegraded":"2024-02-10T07:44:53.010987+0000","last_fullsized":"2024-02-10T07:44:53.010987+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:43:39.223948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:39.223948+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"294'12045","reported_seq":12225,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.811528+0000","last_change":"2024-02-10T07:43:48.784291+0000","last_active":"2024-02-10T07:44:53.811528+0000","last_peered":"2024-02-10T07:44:53.811528+0000","last_clean":"2024-02-10T07:44:53.811528+0000","last_became_active":"2024-02-10T07:42:09.175005+0000","last_became_peered":"2024-02-10T07:42:09.175005+0000","last_unstale":"2024-02-10T07:44:53.811528+0000","last_undegraded":"2024-02-10T07:44:53.811528+0000","last_fullsized":"2024-02-10T07:44:53.811528+0000","mapping_epoch":253,"log_start":"272'6000","ondisk_log_start":"272'6000","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'7895","last_scrub_stamp":"2024-02-10T07:43:48.784116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:48.784116+0000","objects_scrubbed":7808,"log_size":6045,"log_dups_size":0,"ondisk_log_size":6045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:06.007154+0000","scrub_duration":2586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262471680,"num_objects":4005,"num_object_clones":0,"num_object_copies":12015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12045,"num_write_kb":513600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":488,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.956867+0000","last_change":"2024-02-10T07:43:41.991762+0000","last_active":"2024-02-10T07:44:43.956867+0000","last_peered":"2024-02-10T07:44:43.956867+0000","last_clean":"2024-02-10T07:44:43.956867+0000","last_became_active":"2024-02-10T07:31:57.313854+0000","last_became_peered":"2024-02-10T07:31:57.313854+0000","last_unstale":"2024-02-10T07:44:43.956867+0000","last_undegraded":"2024-02-10T07:44:43.956867+0000","last_fullsized":"2024-02-10T07:44:43.956867+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:41.991625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:43:41.991625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:51.962914+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":456,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.952453+0000","last_change":"2024-02-10T07:43:12.274076+0000","last_active":"2024-02-10T07:44:43.952453+0000","last_peered":"2024-02-10T07:44:43.952453+0000","last_clean":"2024-02-10T07:44:43.952453+0000","last_became_active":"2024-02-10T07:36:33.759656+0000","last_became_peered":"2024-02-10T07:36:33.759656+0000","last_unstale":"2024-02-10T07:44:43.952453+0000","last_undegraded":"2024-02-10T07:44:43.952453+0000","last_fullsized":"2024-02-10T07:44:43.952453+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:43:12.273960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:43:12.273960+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":509,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.948999+0000","last_change":"2024-02-10T07:42:02.993202+0000","last_active":"2024-02-10T07:44:43.948999+0000","last_peered":"2024-02-10T07:44:43.948999+0000","last_clean":"2024-02-10T07:44:43.948999+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:44:43.948999+0000","last_undegraded":"2024-02-10T07:44:43.948999+0000","last_fullsized":"2024-02-10T07:44:43.948999+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:02.993090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:42:02.993090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"250'40","reported_seq":733,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.955773+0000","last_change":"2024-02-10T07:44:06.496121+0000","last_active":"2024-02-10T07:44:43.955773+0000","last_peered":"2024-02-10T07:44:43.955773+0000","last_clean":"2024-02-10T07:44:43.955773+0000","last_became_active":"2024-02-10T07:31:53.301863+0000","last_became_peered":"2024-02-10T07:31:53.301863+0000","last_unstale":"2024-02-10T07:44:43.955773+0000","last_undegraded":"2024-02-10T07:44:43.955773+0000","last_fullsized":"2024-02-10T07:44:43.955773+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'40","last_scrub_stamp":"2024-02-10T07:44:06.495977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:44:06.495977+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:09.502391+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":518,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.013411+0000","last_change":"2024-02-10T07:44:04.180332+0000","last_active":"2024-02-10T07:44:53.013411+0000","last_peered":"2024-02-10T07:44:53.013411+0000","last_clean":"2024-02-10T07:44:53.013411+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:44:53.013411+0000","last_undegraded":"2024-02-10T07:44:53.013411+0000","last_fullsized":"2024-02-10T07:44:53.013411+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:04.179805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:22.786949+0000","last_clean_scrub_stamp":"2024-02-10T07:44:04.179805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:09.552301+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"293'11682","reported_seq":11773,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:52.107495+0000","last_change":"2024-02-10T07:42:09.176925+0000","last_active":"2024-02-10T07:44:52.107495+0000","last_peered":"2024-02-10T07:44:52.107495+0000","last_clean":"2024-02-10T07:44:52.107495+0000","last_became_active":"2024-02-10T07:42:09.176328+0000","last_became_peered":"2024-02-10T07:42:09.176328+0000","last_unstale":"2024-02-10T07:44:52.107495+0000","last_undegraded":"2024-02-10T07:44:52.107495+0000","last_fullsized":"2024-02-10T07:44:52.107495+0000","mapping_epoch":253,"log_start":"275'6400","ondisk_log_start":"275'6400","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:42:08.158083+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271712256,"num_objects":4146,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11682,"num_write_kb":506496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":505,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.948845+0000","last_change":"2024-02-10T07:43:14.465370+0000","last_active":"2024-02-10T07:44:43.948845+0000","last_peered":"2024-02-10T07:44:43.948845+0000","last_clean":"2024-02-10T07:44:43.948845+0000","last_became_active":"2024-02-10T07:31:57.299958+0000","last_became_peered":"2024-02-10T07:31:57.299958+0000","last_unstale":"2024-02-10T07:44:43.948845+0000","last_undegraded":"2024-02-10T07:44:43.948845+0000","last_fullsized":"2024-02-10T07:44:43.948845+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:14.465247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:42.730875+0000","last_clean_scrub_stamp":"2024-02-10T07:43:14.465247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":515,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.954442+0000","last_change":"2024-02-10T07:43:08.955491+0000","last_active":"2024-02-10T07:44:43.954442+0000","last_peered":"2024-02-10T07:44:43.954442+0000","last_clean":"2024-02-10T07:44:43.954442+0000","last_became_active":"2024-02-10T07:31:51.320864+0000","last_became_peered":"2024-02-10T07:31:51.320864+0000","last_unstale":"2024-02-10T07:44:43.954442+0000","last_undegraded":"2024-02-10T07:44:43.954442+0000","last_fullsized":"2024-02-10T07:44:43.954442+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:08.955292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:08.955292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.207050+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":527,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.019997+0000","last_change":"2024-02-10T07:43:10.998646+0000","last_active":"2024-02-10T07:44:53.019997+0000","last_peered":"2024-02-10T07:44:53.019997+0000","last_clean":"2024-02-10T07:44:53.019997+0000","last_became_active":"2024-02-10T07:31:50.281517+0000","last_became_peered":"2024-02-10T07:31:50.281517+0000","last_unstale":"2024-02-10T07:44:53.019997+0000","last_undegraded":"2024-02-10T07:44:53.019997+0000","last_fullsized":"2024-02-10T07:44:53.019997+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:10.998492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:43:10.998492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"250'17","reported_seq":304,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.960647+0000","last_change":"2024-02-10T07:43:15.107129+0000","last_active":"2024-02-10T07:44:43.960647+0000","last_peered":"2024-02-10T07:44:43.960647+0000","last_clean":"2024-02-10T07:44:43.960647+0000","last_became_active":"2024-02-10T07:37:06.196787+0000","last_became_peered":"2024-02-10T07:37:06.196787+0000","last_unstale":"2024-02-10T07:44:43.960647+0000","last_undegraded":"2024-02-10T07:44:43.960647+0000","last_fullsized":"2024-02-10T07:44:43.960647+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'17","last_scrub_stamp":"2024-02-10T07:43:15.106978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:15.106978+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.207139+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":521,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.022864+0000","last_change":"2024-02-10T07:43:04.918865+0000","last_active":"2024-02-10T07:44:53.022864+0000","last_peered":"2024-02-10T07:44:53.022864+0000","last_clean":"2024-02-10T07:44:53.022864+0000","last_became_active":"2024-02-10T07:36:33.751551+0000","last_became_peered":"2024-02-10T07:36:33.751551+0000","last_unstale":"2024-02-10T07:44:53.022864+0000","last_undegraded":"2024-02-10T07:44:53.022864+0000","last_fullsized":"2024-02-10T07:44:53.022864+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:04.918467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:04.918467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"293'11700","reported_seq":11876,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:49.360326+0000","last_change":"2024-02-10T07:43:59.609577+0000","last_active":"2024-02-10T07:44:49.360326+0000","last_peered":"2024-02-10T07:44:49.360326+0000","last_clean":"2024-02-10T07:44:49.360326+0000","last_became_active":"2024-02-10T07:42:09.176267+0000","last_became_peered":"2024-02-10T07:42:09.176267+0000","last_unstale":"2024-02-10T07:44:49.360326+0000","last_undegraded":"2024-02-10T07:44:49.360326+0000","last_fullsized":"2024-02-10T07:44:49.360326+0000","mapping_epoch":253,"log_start":"277'6800","ondisk_log_start":"277'6800","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'8548","last_scrub_stamp":"2024-02-10T07:43:59.609439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:59.609439+0000","objects_scrubbed":7542,"log_size":4900,"log_dups_size":0,"ondisk_log_size":4900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:03.912783+0000","scrub_duration":2815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283508736,"num_objects":4326,"num_object_clones":0,"num_object_copies":12978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11700,"num_write_kb":512832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":460,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.954485+0000","last_change":"2024-02-10T07:42:17.190620+0000","last_active":"2024-02-10T07:44:43.954485+0000","last_peered":"2024-02-10T07:44:43.954485+0000","last_clean":"2024-02-10T07:44:43.954485+0000","last_became_active":"2024-02-10T07:36:33.769763+0000","last_became_peered":"2024-02-10T07:36:33.769763+0000","last_unstale":"2024-02-10T07:44:43.954485+0000","last_undegraded":"2024-02-10T07:44:43.954485+0000","last_fullsized":"2024-02-10T07:44:43.954485+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_clean_scrub_stamp":"2024-02-10T07:42:17.190492+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.189851+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":762,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.947047+0000","last_change":"2024-02-10T07:43:13.432811+0000","last_active":"2024-02-10T07:44:43.947047+0000","last_peered":"2024-02-10T07:44:43.947047+0000","last_clean":"2024-02-10T07:44:43.947047+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:44:43.947047+0000","last_undegraded":"2024-02-10T07:44:43.947047+0000","last_fullsized":"2024-02-10T07:44:43.947047+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:43:13.432680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:43:13.432680+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":510,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.953912+0000","last_change":"2024-02-10T07:43:00.146103+0000","last_active":"2024-02-10T07:44:43.953912+0000","last_peered":"2024-02-10T07:44:43.953912+0000","last_clean":"2024-02-10T07:44:43.953912+0000","last_became_active":"2024-02-10T07:36:33.752703+0000","last_became_peered":"2024-02-10T07:36:33.752703+0000","last_unstale":"2024-02-10T07:44:43.953912+0000","last_undegraded":"2024-02-10T07:44:43.953912+0000","last_fullsized":"2024-02-10T07:44:43.953912+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:00.145780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:00.145780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"250'36","reported_seq":725,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:46.527554+0000","last_change":"2024-02-10T07:43:57.949399+0000","last_active":"2024-02-10T07:44:46.527554+0000","last_peered":"2024-02-10T07:44:46.527554+0000","last_clean":"2024-02-10T07:44:46.527554+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:44:46.527554+0000","last_undegraded":"2024-02-10T07:44:46.527554+0000","last_fullsized":"2024-02-10T07:44:46.527554+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'36","last_scrub_stamp":"2024-02-10T07:43:57.949264+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:43:57.949264+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:04.425290+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":518,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.948477+0000","last_change":"2024-02-10T07:43:39.433253+0000","last_active":"2024-02-10T07:44:43.948477+0000","last_peered":"2024-02-10T07:44:43.948477+0000","last_clean":"2024-02-10T07:44:43.948477+0000","last_became_active":"2024-02-10T07:36:33.752711+0000","last_became_peered":"2024-02-10T07:36:33.752711+0000","last_unstale":"2024-02-10T07:44:43.948477+0000","last_undegraded":"2024-02-10T07:44:43.948477+0000","last_fullsized":"2024-02-10T07:44:43.948477+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:39.433049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.736901+0000","last_clean_scrub_stamp":"2024-02-10T07:43:39.433049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:48.100463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"294'12103","reported_seq":12279,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.842667+0000","last_change":"2024-02-10T07:44:02.098658+0000","last_active":"2024-02-10T07:44:53.842667+0000","last_peered":"2024-02-10T07:44:53.842667+0000","last_clean":"2024-02-10T07:44:53.842667+0000","last_became_active":"2024-02-10T07:42:09.176217+0000","last_became_peered":"2024-02-10T07:42:09.176217+0000","last_unstale":"2024-02-10T07:44:53.842667+0000","last_undegraded":"2024-02-10T07:44:53.842667+0000","last_fullsized":"2024-02-10T07:44:53.842667+0000","mapping_epoch":253,"log_start":"274'6100","ondisk_log_start":"274'6100","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'8727","last_scrub_stamp":"2024-02-10T07:44:02.098377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:44:02.098377+0000","objects_scrubbed":7438,"log_size":6003,"log_dups_size":0,"ondisk_log_size":6003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:22.096752+0000","scrub_duration":2893,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261947392,"num_objects":3997,"num_object_clones":0,"num_object_copies":11991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12103,"num_write_kb":515200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":286,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.955353+0000","last_change":"2024-02-10T07:43:02.043769+0000","last_active":"2024-02-10T07:44:43.955353+0000","last_peered":"2024-02-10T07:44:43.955353+0000","last_clean":"2024-02-10T07:44:43.955353+0000","last_became_active":"2024-02-10T07:37:06.202738+0000","last_became_peered":"2024-02-10T07:37:06.202738+0000","last_unstale":"2024-02-10T07:44:43.955353+0000","last_undegraded":"2024-02-10T07:44:43.955353+0000","last_fullsized":"2024-02-10T07:44:43.955353+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:02.043646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:02.043646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.207002+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":518,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.008153+0000","last_change":"2024-02-10T07:44:07.146438+0000","last_active":"2024-02-10T07:44:53.008153+0000","last_peered":"2024-02-10T07:44:53.008153+0000","last_clean":"2024-02-10T07:44:53.008153+0000","last_became_active":"2024-02-10T07:31:51.286076+0000","last_became_peered":"2024-02-10T07:31:51.286076+0000","last_unstale":"2024-02-10T07:44:53.008153+0000","last_undegraded":"2024-02-10T07:44:53.008153+0000","last_fullsized":"2024-02-10T07:44:53.008153+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:07.146214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:37.574311+0000","last_clean_scrub_stamp":"2024-02-10T07:44:07.146214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:34.549494+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1304,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:51.282081+0000","last_change":"2024-02-10T07:43:15.404193+0000","last_active":"2024-02-10T07:44:51.282081+0000","last_peered":"2024-02-10T07:44:51.282081+0000","last_clean":"2024-02-10T07:44:51.282081+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:44:51.282081+0000","last_undegraded":"2024-02-10T07:44:51.282081+0000","last_fullsized":"2024-02-10T07:44:51.282081+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:43:15.404111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:43:15.404111+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":775,"num_read_kb":770,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"250'30","reported_seq":555,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.014968+0000","last_change":"2024-02-10T07:44:52.210159+0000","last_active":"2024-02-10T07:44:53.014968+0000","last_peered":"2024-02-10T07:44:53.014968+0000","last_clean":"2024-02-10T07:44:53.014968+0000","last_became_active":"2024-02-10T07:31:53.300687+0000","last_became_peered":"2024-02-10T07:31:53.300687+0000","last_unstale":"2024-02-10T07:44:53.014968+0000","last_undegraded":"2024-02-10T07:44:53.014968+0000","last_fullsized":"2024-02-10T07:44:53.014968+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:44:52.210019+0000","last_deep_scrub":"237'18","last_deep_scrub_stamp":"2024-02-10T07:40:31.005376+0000","last_clean_scrub_stamp":"2024-02-10T07:44:52.210019+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:52.878456+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":454,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.954614+0000","last_change":"2024-02-10T07:42:16.148432+0000","last_active":"2024-02-10T07:44:43.954614+0000","last_peered":"2024-02-10T07:44:43.954614+0000","last_clean":"2024-02-10T07:44:43.954614+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:44:43.954614+0000","last_undegraded":"2024-02-10T07:44:43.954614+0000","last_fullsized":"2024-02-10T07:44:43.954614+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:42:16.148315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:42:16.148315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.189900+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"293'11959","reported_seq":12134,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:52.957347+0000","last_change":"2024-02-10T07:43:44.092163+0000","last_active":"2024-02-10T07:44:52.957347+0000","last_peered":"2024-02-10T07:44:52.957347+0000","last_clean":"2024-02-10T07:44:52.957347+0000","last_became_active":"2024-02-10T07:42:09.450389+0000","last_became_peered":"2024-02-10T07:42:09.450389+0000","last_unstale":"2024-02-10T07:44:52.957347+0000","last_undegraded":"2024-02-10T07:44:52.957347+0000","last_fullsized":"2024-02-10T07:44:52.957347+0000","mapping_epoch":253,"log_start":"278'7100","ondisk_log_start":"278'7100","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'7600","last_scrub_stamp":"2024-02-10T07:43:44.092031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:44.092031+0000","objects_scrubbed":7513,"log_size":4859,"log_dups_size":0,"ondisk_log_size":4859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271646720,"num_objects":4145,"num_object_clones":0,"num_object_copies":12435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11959,"num_write_kb":515328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":503,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.953856+0000","last_change":"2024-02-10T07:43:40.151895+0000","last_active":"2024-02-10T07:44:43.953856+0000","last_peered":"2024-02-10T07:44:43.953856+0000","last_clean":"2024-02-10T07:44:43.953856+0000","last_became_active":"2024-02-10T07:31:57.307615+0000","last_became_peered":"2024-02-10T07:31:57.307615+0000","last_unstale":"2024-02-10T07:44:43.953856+0000","last_undegraded":"2024-02-10T07:44:43.953856+0000","last_fullsized":"2024-02-10T07:44:43.953856+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:40.151648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:40.151648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:54.107957+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":524,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.955647+0000","last_change":"2024-02-10T07:43:47.219926+0000","last_active":"2024-02-10T07:44:43.955647+0000","last_peered":"2024-02-10T07:44:43.955647+0000","last_clean":"2024-02-10T07:44:43.955647+0000","last_became_active":"2024-02-10T07:36:33.750867+0000","last_became_peered":"2024-02-10T07:36:33.750867+0000","last_unstale":"2024-02-10T07:44:43.955647+0000","last_undegraded":"2024-02-10T07:44:43.955647+0000","last_fullsized":"2024-02-10T07:44:43.955647+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:47.219800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:43:47.219800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:16.514612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"271'37","reported_seq":574,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.010690+0000","last_change":"2024-02-10T07:43:10.047505+0000","last_active":"2024-02-10T07:44:53.010690+0000","last_peered":"2024-02-10T07:44:53.010690+0000","last_clean":"2024-02-10T07:44:53.010690+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:44:53.010690+0000","last_undegraded":"2024-02-10T07:44:53.010690+0000","last_fullsized":"2024-02-10T07:44:53.010690+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'37","last_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_deep_scrub":"271'37","last_deep_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_clean_scrub_stamp":"2024-02-10T07:43:10.047405+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":525,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.967398+0000","last_change":"2024-02-10T07:43:08.422662+0000","last_active":"2024-02-10T07:44:43.967398+0000","last_peered":"2024-02-10T07:44:43.967398+0000","last_clean":"2024-02-10T07:44:43.967398+0000","last_became_active":"2024-02-10T07:36:33.765998+0000","last_became_peered":"2024-02-10T07:36:33.765998+0000","last_unstale":"2024-02-10T07:44:43.967398+0000","last_undegraded":"2024-02-10T07:44:43.967398+0000","last_fullsized":"2024-02-10T07:44:43.967398+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:08.422557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:08.422557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:44.078265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"293'14795","reported_seq":15063,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:52.136443+0000","last_change":"2024-02-10T07:44:48.806944+0000","last_active":"2024-02-10T07:44:52.136443+0000","last_peered":"2024-02-10T07:44:52.136443+0000","last_clean":"2024-02-10T07:44:52.136443+0000","last_became_active":"2024-02-10T07:42:09.176384+0000","last_became_peered":"2024-02-10T07:42:09.176384+0000","last_unstale":"2024-02-10T07:44:52.136443+0000","last_undegraded":"2024-02-10T07:44:52.136443+0000","last_fullsized":"2024-02-10T07:44:52.136443+0000","mapping_epoch":253,"log_start":"287'9503","ondisk_log_start":"287'9503","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'14542","last_scrub_stamp":"2024-02-10T07:44:48.806768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:44:48.806768+0000","objects_scrubbed":4494,"log_size":5292,"log_dups_size":0,"ondisk_log_size":5292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:46:00.047004+0000","scrub_duration":1595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275120128,"num_objects":4198,"num_object_clones":0,"num_object_copies":12594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11928,"num_write_kb":516032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":391,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.969288+0000","last_change":"2024-02-10T07:43:59.970168+0000","last_active":"2024-02-10T07:44:43.969288+0000","last_peered":"2024-02-10T07:44:43.969288+0000","last_clean":"2024-02-10T07:44:43.969288+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:44:43.969288+0000","last_undegraded":"2024-02-10T07:44:43.969288+0000","last_fullsized":"2024-02-10T07:44:43.969288+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:59.970065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:59.970065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:07.935598+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":477,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.952667+0000","last_change":"2024-02-10T07:43:14.153897+0000","last_active":"2024-02-10T07:44:43.952667+0000","last_peered":"2024-02-10T07:44:43.952667+0000","last_clean":"2024-02-10T07:44:43.952667+0000","last_became_active":"2024-02-10T07:36:33.779820+0000","last_became_peered":"2024-02-10T07:36:33.779820+0000","last_unstale":"2024-02-10T07:44:43.952667+0000","last_undegraded":"2024-02-10T07:44:43.952667+0000","last_fullsized":"2024-02-10T07:44:43.952667+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:14.153641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:14.153641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"250'20","reported_seq":321,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.959136+0000","last_change":"2024-02-10T07:44:01.357558+0000","last_active":"2024-02-10T07:44:43.959136+0000","last_peered":"2024-02-10T07:44:43.959136+0000","last_clean":"2024-02-10T07:44:43.959136+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:44:43.959136+0000","last_undegraded":"2024-02-10T07:44:43.959136+0000","last_fullsized":"2024-02-10T07:44:43.959136+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:44:01.357268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:44:01.357268+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:29.526294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"158'10","reported_seq":685,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.011092+0000","last_change":"2024-02-10T07:44:09.183908+0000","last_active":"2024-02-10T07:44:53.011092+0000","last_peered":"2024-02-10T07:44:53.011092+0000","last_clean":"2024-02-10T07:44:53.011092+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:44:53.011092+0000","last_undegraded":"2024-02-10T07:44:53.011092+0000","last_fullsized":"2024-02-10T07:44:53.011092+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:44:09.183790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:44:09.183790+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:10.060743+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"293'11586","reported_seq":11760,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:49.366400+0000","last_change":"2024-02-10T07:44:00.802043+0000","last_active":"2024-02-10T07:44:49.366400+0000","last_peered":"2024-02-10T07:44:49.366400+0000","last_clean":"2024-02-10T07:44:49.366400+0000","last_became_active":"2024-02-10T07:42:09.450743+0000","last_became_peered":"2024-02-10T07:42:09.450743+0000","last_unstale":"2024-02-10T07:44:49.366400+0000","last_undegraded":"2024-02-10T07:44:49.366400+0000","last_fullsized":"2024-02-10T07:44:49.366400+0000","mapping_epoch":253,"log_start":"277'6700","ondisk_log_start":"277'6700","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'8606","last_scrub_stamp":"2024-02-10T07:44:00.801833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:44:00.801833+0000","objects_scrubbed":7458,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:26.243602+0000","scrub_duration":3048,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286916608,"num_objects":4378,"num_object_clones":0,"num_object_copies":13134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11586,"num_write_kb":510848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":505,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.948592+0000","last_change":"2024-02-10T07:43:45.565156+0000","last_active":"2024-02-10T07:44:43.948592+0000","last_peered":"2024-02-10T07:44:43.948592+0000","last_clean":"2024-02-10T07:44:43.948592+0000","last_became_active":"2024-02-10T07:31:57.305620+0000","last_became_peered":"2024-02-10T07:31:57.305620+0000","last_unstale":"2024-02-10T07:44:43.948592+0000","last_undegraded":"2024-02-10T07:44:43.948592+0000","last_fullsized":"2024-02-10T07:44:43.948592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:45.565037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:45.565037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:53.867913+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":519,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.015138+0000","last_change":"2024-02-10T07:43:05.083484+0000","last_active":"2024-02-10T07:44:53.015138+0000","last_peered":"2024-02-10T07:44:53.015138+0000","last_clean":"2024-02-10T07:44:53.015138+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:44:53.015138+0000","last_undegraded":"2024-02-10T07:44:53.015138+0000","last_fullsized":"2024-02-10T07:44:53.015138+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:43:05.083356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:05.083356+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"250'28","reported_seq":556,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.949492+0000","last_change":"2024-02-10T07:43:11.452648+0000","last_active":"2024-02-10T07:44:43.949492+0000","last_peered":"2024-02-10T07:44:43.949492+0000","last_clean":"2024-02-10T07:44:43.949492+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:44:43.949492+0000","last_undegraded":"2024-02-10T07:44:43.949492+0000","last_fullsized":"2024-02-10T07:44:43.949492+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-02-10T07:43:11.452502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:11.452502+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"227'9","reported_seq":666,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.964374+0000","last_change":"2024-02-10T07:43:14.988651+0000","last_active":"2024-02-10T07:44:43.964374+0000","last_peered":"2024-02-10T07:44:43.964374+0000","last_clean":"2024-02-10T07:44:43.964374+0000","last_became_active":"2024-02-10T07:31:55.302863+0000","last_became_peered":"2024-02-10T07:31:55.302863+0000","last_unstale":"2024-02-10T07:44:43.964374+0000","last_undegraded":"2024-02-10T07:44:43.964374+0000","last_fullsized":"2024-02-10T07:44:43.964374+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'9","last_scrub_stamp":"2024-02-10T07:43:14.987456+0000","last_deep_scrub":"227'9","last_deep_scrub_stamp":"2024-02-10T07:39:53.440221+0000","last_clean_scrub_stamp":"2024-02-10T07:43:14.987456+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"293'11794","reported_seq":11972,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:52.893914+0000","last_change":"2024-02-10T07:43:48.708669+0000","last_active":"2024-02-10T07:44:52.893914+0000","last_peered":"2024-02-10T07:44:52.893914+0000","last_clean":"2024-02-10T07:44:52.893914+0000","last_became_active":"2024-02-10T07:42:09.175677+0000","last_became_peered":"2024-02-10T07:42:09.175677+0000","last_unstale":"2024-02-10T07:44:52.893914+0000","last_undegraded":"2024-02-10T07:44:52.893914+0000","last_fullsized":"2024-02-10T07:44:52.893914+0000","mapping_epoch":253,"log_start":"278'6900","ondisk_log_start":"278'6900","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'7841","last_scrub_stamp":"2024-02-10T07:43:48.708425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:48.708425+0000","objects_scrubbed":7768,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:14.910084+0000","scrub_duration":2197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273743896,"num_objects":4178,"num_object_clones":0,"num_object_copies":12534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11794,"num_write_kb":511041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":497,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.948403+0000","last_change":"2024-02-10T07:43:43.422664+0000","last_active":"2024-02-10T07:44:43.948403+0000","last_peered":"2024-02-10T07:44:43.948403+0000","last_clean":"2024-02-10T07:44:43.948403+0000","last_became_active":"2024-02-10T07:31:57.316047+0000","last_became_peered":"2024-02-10T07:31:57.316047+0000","last_unstale":"2024-02-10T07:44:43.948403+0000","last_undegraded":"2024-02-10T07:44:43.948403+0000","last_fullsized":"2024-02-10T07:44:43.948403+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:43.422546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:43:43.422546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:46.243272+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":508,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.961222+0000","last_change":"2024-02-10T07:44:06.020845+0000","last_active":"2024-02-10T07:44:43.961222+0000","last_peered":"2024-02-10T07:44:43.961222+0000","last_clean":"2024-02-10T07:44:43.961222+0000","last_became_active":"2024-02-10T07:31:51.285333+0000","last_became_peered":"2024-02-10T07:31:51.285333+0000","last_unstale":"2024-02-10T07:44:43.961222+0000","last_undegraded":"2024-02-10T07:44:43.961222+0000","last_fullsized":"2024-02-10T07:44:43.961222+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:06.020588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:44:06.020588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:21.600571+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"250'30","reported_seq":555,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.964221+0000","last_change":"2024-02-10T07:44:08.032219+0000","last_active":"2024-02-10T07:44:43.964221+0000","last_peered":"2024-02-10T07:44:43.964221+0000","last_clean":"2024-02-10T07:44:43.964221+0000","last_became_active":"2024-02-10T07:31:53.290157+0000","last_became_peered":"2024-02-10T07:31:53.290157+0000","last_unstale":"2024-02-10T07:44:43.964221+0000","last_undegraded":"2024-02-10T07:44:43.964221+0000","last_fullsized":"2024-02-10T07:44:43.964221+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:44:08.032066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:44:08.032066+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:35.903498+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":509,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.947911+0000","last_change":"2024-02-10T07:43:47.483128+0000","last_active":"2024-02-10T07:44:43.947911+0000","last_peered":"2024-02-10T07:44:43.947911+0000","last_clean":"2024-02-10T07:44:43.947911+0000","last_became_active":"2024-02-10T07:31:55.295900+0000","last_became_peered":"2024-02-10T07:31:55.295900+0000","last_unstale":"2024-02-10T07:44:43.947911+0000","last_undegraded":"2024-02-10T07:44:43.947911+0000","last_fullsized":"2024-02-10T07:44:43.947911+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:47.483009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:47.483009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:53.829855+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"293'11735","reported_seq":11913,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:49.379190+0000","last_change":"2024-02-10T07:43:55.810600+0000","last_active":"2024-02-10T07:44:49.379190+0000","last_peered":"2024-02-10T07:44:49.379190+0000","last_clean":"2024-02-10T07:44:49.379190+0000","last_became_active":"2024-02-10T07:42:09.395607+0000","last_became_peered":"2024-02-10T07:42:09.395607+0000","last_unstale":"2024-02-10T07:44:49.379190+0000","last_undegraded":"2024-02-10T07:44:49.379190+0000","last_fullsized":"2024-02-10T07:44:49.379190+0000","mapping_epoch":253,"log_start":"277'6800","ondisk_log_start":"277'6800","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'8402","last_scrub_stamp":"2024-02-10T07:43:55.810372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:55.810372+0000","objects_scrubbed":7820,"log_size":4935,"log_dups_size":0,"ondisk_log_size":4935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:14.377222+0000","scrub_duration":3066,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287244288,"num_objects":4383,"num_object_clones":0,"num_object_copies":13149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11735,"num_write_kb":515776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":454,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.952703+0000","last_change":"2024-02-10T07:43:06.325383+0000","last_active":"2024-02-10T07:44:43.952703+0000","last_peered":"2024-02-10T07:44:43.952703+0000","last_clean":"2024-02-10T07:44:43.952703+0000","last_became_active":"2024-02-10T07:36:33.772094+0000","last_became_peered":"2024-02-10T07:36:33.772094+0000","last_unstale":"2024-02-10T07:44:43.952703+0000","last_undegraded":"2024-02-10T07:44:43.952703+0000","last_fullsized":"2024-02-10T07:44:43.952703+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:06.325264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:49.704024+0000","last_clean_scrub_stamp":"2024-02-10T07:43:06.325264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:47.190064+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"158'10","reported_seq":675,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:46.527712+0000","last_change":"2024-02-10T07:42:17.097563+0000","last_active":"2024-02-10T07:44:46.527712+0000","last_peered":"2024-02-10T07:44:46.527712+0000","last_clean":"2024-02-10T07:44:46.527712+0000","last_became_active":"2024-02-10T07:36:33.755111+0000","last_became_peered":"2024-02-10T07:36:33.755111+0000","last_unstale":"2024-02-10T07:44:46.527712+0000","last_undegraded":"2024-02-10T07:44:46.527712+0000","last_fullsized":"2024-02-10T07:44:46.527712+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:42:17.097480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:42:17.097480+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":508,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:48.313336+0000","last_change":"2024-02-10T07:44:48.313376+0000","last_active":"2024-02-10T07:44:48.313336+0000","last_peered":"2024-02-10T07:44:48.313336+0000","last_clean":"2024-02-10T07:44:48.313336+0000","last_became_active":"2024-02-10T07:31:51.282137+0000","last_became_peered":"2024-02-10T07:31:51.282137+0000","last_unstale":"2024-02-10T07:44:48.313336+0000","last_undegraded":"2024-02-10T07:44:48.313336+0000","last_fullsized":"2024-02-10T07:44:48.313336+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:48.313261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:44:48.313261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:57.338354+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"250'27","reported_seq":561,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.956953+0000","last_change":"2024-02-10T07:43:58.327164+0000","last_active":"2024-02-10T07:44:43.956953+0000","last_peered":"2024-02-10T07:44:43.956953+0000","last_clean":"2024-02-10T07:44:43.956953+0000","last_became_active":"2024-02-10T07:31:53.294218+0000","last_became_peered":"2024-02-10T07:31:53.294218+0000","last_unstale":"2024-02-10T07:44:43.956953+0000","last_undegraded":"2024-02-10T07:44:43.956953+0000","last_fullsized":"2024-02-10T07:44:43.956953+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:43:58.327027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:58.327027+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:12.111184+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"293'11913","reported_seq":12088,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:52.951391+0000","last_change":"2024-02-10T07:43:45.681822+0000","last_active":"2024-02-10T07:44:52.951391+0000","last_peered":"2024-02-10T07:44:52.951391+0000","last_clean":"2024-02-10T07:44:52.951391+0000","last_became_active":"2024-02-10T07:42:09.174538+0000","last_became_peered":"2024-02-10T07:42:09.174538+0000","last_unstale":"2024-02-10T07:44:52.951391+0000","last_undegraded":"2024-02-10T07:44:52.951391+0000","last_fullsized":"2024-02-10T07:44:52.951391+0000","mapping_epoch":253,"log_start":"278'7000","ondisk_log_start":"278'7000","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'7651","last_scrub_stamp":"2024-02-10T07:43:45.681682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:45.681682+0000","objects_scrubbed":7571,"log_size":4913,"log_dups_size":0,"ondisk_log_size":4913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:00.433446+0000","scrub_duration":2221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270467072,"num_objects":4127,"num_object_clones":0,"num_object_copies":12381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11913,"num_write_kb":513280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":505,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.956627+0000","last_change":"2024-02-10T07:43:19.934726+0000","last_active":"2024-02-10T07:44:43.956627+0000","last_peered":"2024-02-10T07:44:43.956627+0000","last_clean":"2024-02-10T07:44:43.956627+0000","last_became_active":"2024-02-10T07:36:33.753921+0000","last_became_peered":"2024-02-10T07:36:33.753921+0000","last_unstale":"2024-02-10T07:44:43.956627+0000","last_undegraded":"2024-02-10T07:44:43.956627+0000","last_fullsized":"2024-02-10T07:44:43.956627+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:19.934260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_clean_scrub_stamp":"2024-02-10T07:43:19.934260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":436,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.954531+0000","last_change":"2024-02-10T07:43:52.274357+0000","last_active":"2024-02-10T07:44:43.954531+0000","last_peered":"2024-02-10T07:44:43.954531+0000","last_clean":"2024-02-10T07:44:43.954531+0000","last_became_active":"2024-02-10T07:36:33.759566+0000","last_became_peered":"2024-02-10T07:36:33.759566+0000","last_unstale":"2024-02-10T07:44:43.954531+0000","last_undegraded":"2024-02-10T07:44:43.954531+0000","last_fullsized":"2024-02-10T07:44:43.954531+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:52.274260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:52.274260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:54.593440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":517,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.948013+0000","last_change":"2024-02-10T07:44:08.363775+0000","last_active":"2024-02-10T07:44:43.948013+0000","last_peered":"2024-02-10T07:44:43.948013+0000","last_clean":"2024-02-10T07:44:43.948013+0000","last_became_active":"2024-02-10T07:31:51.320715+0000","last_became_peered":"2024-02-10T07:31:51.320715+0000","last_unstale":"2024-02-10T07:44:43.948013+0000","last_undegraded":"2024-02-10T07:44:43.948013+0000","last_fullsized":"2024-02-10T07:44:43.948013+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:08.363673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:44:08.363673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:36.016926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"250'24","reported_seq":576,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.950470+0000","last_change":"2024-02-10T07:42:11.104029+0000","last_active":"2024-02-10T07:44:43.950470+0000","last_peered":"2024-02-10T07:44:43.950470+0000","last_clean":"2024-02-10T07:44:43.950470+0000","last_became_active":"2024-02-10T07:37:06.193386+0000","last_became_peered":"2024-02-10T07:37:06.193386+0000","last_unstale":"2024-02-10T07:44:43.950470+0000","last_undegraded":"2024-02-10T07:44:43.950470+0000","last_fullsized":"2024-02-10T07:44:43.950470+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:42:11.103919+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:42:11.103919+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"293'16898","reported_seq":17285,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:48.935630+0000","last_change":"2024-02-10T07:44:48.775525+0000","last_active":"2024-02-10T07:44:48.935630+0000","last_peered":"2024-02-10T07:44:48.935630+0000","last_clean":"2024-02-10T07:44:48.935630+0000","last_became_active":"2024-02-10T07:42:09.394893+0000","last_became_peered":"2024-02-10T07:42:09.394893+0000","last_unstale":"2024-02-10T07:44:48.935630+0000","last_undegraded":"2024-02-10T07:44:48.935630+0000","last_fullsized":"2024-02-10T07:44:48.935630+0000","mapping_epoch":253,"log_start":"288'12102","ondisk_log_start":"288'12102","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'16884","last_scrub_stamp":"2024-02-10T07:44:48.772883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:44:48.772883+0000","objects_scrubbed":4423,"log_size":4796,"log_dups_size":0,"ondisk_log_size":4796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:46:09.433047+0000","scrub_duration":1537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285278208,"num_objects":4353,"num_object_clones":0,"num_object_copies":13059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11441,"num_write_kb":505408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":513,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.021360+0000","last_change":"2024-02-10T07:43:59.300606+0000","last_active":"2024-02-10T07:44:53.021360+0000","last_peered":"2024-02-10T07:44:53.021360+0000","last_clean":"2024-02-10T07:44:53.021360+0000","last_became_active":"2024-02-10T07:31:57.311427+0000","last_became_peered":"2024-02-10T07:31:57.311427+0000","last_unstale":"2024-02-10T07:44:53.021360+0000","last_undegraded":"2024-02-10T07:44:53.021360+0000","last_fullsized":"2024-02-10T07:44:53.021360+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:59.300487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:43:59.300487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:16.206253+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":506,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.955822+0000","last_change":"2024-02-10T07:44:03.427446+0000","last_active":"2024-02-10T07:44:43.955822+0000","last_peered":"2024-02-10T07:44:43.955822+0000","last_clean":"2024-02-10T07:44:43.955822+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:44:43.955822+0000","last_undegraded":"2024-02-10T07:44:43.955822+0000","last_fullsized":"2024-02-10T07:44:43.955822+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:03.427333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:44:03.427333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:16.605118+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":532,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.967483+0000","last_change":"2024-02-10T07:43:51.058868+0000","last_active":"2024-02-10T07:44:43.967483+0000","last_peered":"2024-02-10T07:44:43.967483+0000","last_clean":"2024-02-10T07:44:43.967483+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:44:43.967483+0000","last_undegraded":"2024-02-10T07:44:43.967483+0000","last_fullsized":"2024-02-10T07:44:43.967483+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:51.058792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:44.067302+0000","last_clean_scrub_stamp":"2024-02-10T07:43:51.058792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:45:08.489187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"250'39","reported_seq":561,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.950239+0000","last_change":"2024-02-10T07:41:25.892072+0000","last_active":"2024-02-10T07:44:43.950239+0000","last_peered":"2024-02-10T07:44:43.950239+0000","last_clean":"2024-02-10T07:44:43.950239+0000","last_became_active":"2024-02-10T07:31:53.292277+0000","last_became_peered":"2024-02-10T07:31:53.292277+0000","last_unstale":"2024-02-10T07:44:43.950239+0000","last_undegraded":"2024-02-10T07:44:43.950239+0000","last_fullsized":"2024-02-10T07:44:43.950239+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'25","last_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_clean_scrub_stamp":"2024-02-10T07:41:25.891983+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"293'11552","reported_seq":11727,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:49.379570+0000","last_change":"2024-02-10T07:43:53.588670+0000","last_active":"2024-02-10T07:44:49.379570+0000","last_peered":"2024-02-10T07:44:49.379570+0000","last_clean":"2024-02-10T07:44:49.379570+0000","last_became_active":"2024-02-10T07:42:09.170186+0000","last_became_peered":"2024-02-10T07:42:09.170186+0000","last_unstale":"2024-02-10T07:44:49.379570+0000","last_undegraded":"2024-02-10T07:44:49.379570+0000","last_fullsized":"2024-02-10T07:44:49.379570+0000","mapping_epoch":253,"log_start":"277'6700","ondisk_log_start":"277'6700","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'8104","last_scrub_stamp":"2024-02-10T07:43:53.588443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:08.158083+0000","last_clean_scrub_stamp":"2024-02-10T07:43:53.588443+0000","objects_scrubbed":7832,"log_size":4852,"log_dups_size":0,"ondisk_log_size":4852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:44:57.545210+0000","scrub_duration":2816,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280494080,"num_objects":4280,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11552,"num_write_kb":506624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":496,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.950792+0000","last_change":"2024-02-10T07:43:10.479711+0000","last_active":"2024-02-10T07:44:43.950792+0000","last_peered":"2024-02-10T07:44:43.950792+0000","last_clean":"2024-02-10T07:44:43.950792+0000","last_became_active":"2024-02-10T07:31:57.308608+0000","last_became_peered":"2024-02-10T07:31:57.308608+0000","last_unstale":"2024-02-10T07:44:43.950792+0000","last_undegraded":"2024-02-10T07:44:43.950792+0000","last_fullsized":"2024-02-10T07:44:43.950792+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:10.479470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:10.479470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"158'10","reported_seq":687,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-02-10T07:44:53.021047+0000","last_change":"2024-02-10T07:43:07.964022+0000","last_active":"2024-02-10T07:44:53.021047+0000","last_peered":"2024-02-10T07:44:53.021047+0000","last_clean":"2024-02-10T07:44:53.021047+0000","last_became_active":"2024-02-10T07:36:33.776454+0000","last_became_peered":"2024-02-10T07:36:33.776454+0000","last_unstale":"2024-02-10T07:44:53.021047+0000","last_undegraded":"2024-02-10T07:44:53.021047+0000","last_fullsized":"2024-02-10T07:44:53.021047+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:43:07.963897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:07.963897+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":500,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.954968+0000","last_change":"2024-02-10T07:43:09.950745+0000","last_active":"2024-02-10T07:44:43.954968+0000","last_peered":"2024-02-10T07:44:43.954968+0000","last_clean":"2024-02-10T07:44:43.954968+0000","last_became_active":"2024-02-10T07:31:51.288197+0000","last_became_peered":"2024-02-10T07:31:51.288197+0000","last_unstale":"2024-02-10T07:44:43.954968+0000","last_undegraded":"2024-02-10T07:44:43.954968+0000","last_fullsized":"2024-02-10T07:44:43.954968+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_clean_scrub_stamp":"2024-02-10T07:43:09.950640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"250'20","reported_seq":534,"reported_epoch":293,"state":"active+clean","last_fresh":"2024-02-10T07:44:43.949895+0000","last_change":"2024-02-10T07:42:10.093762+0000","last_active":"2024-02-10T07:44:43.949895+0000","last_peered":"2024-02-10T07:44:43.949895+0000","last_clean":"2024-02-10T07:44:43.949895+0000","last_became_active":"2024-02-10T07:31:53.287984+0000","last_became_peered":"2024-02-10T07:31:53.287984+0000","last_unstale":"2024-02-10T07:44:43.949895+0000","last_undegraded":"2024-02-10T07:44:43.949895+0000","last_fullsized":"2024-02-10T07:44:43.949895+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:42:10.093651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:42:10.093651+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4405264408,"num_objects":67220,"num_object_clones":0,"num_object_copies":201660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67220,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189130,"num_write_kb":8203137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13215281152,"data_stored":13215268936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83149,"ondisk_log_size":83149,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":775,"num_read_kb":770,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1122,"num_read_kb":879,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":762,"ondisk_log_size":762,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037921,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2832036,"kb_used_data":2385756,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":90913116,"statfs":{"total":95995035648,"available":93095030784,"internally_reserved":0,"allocated":2443014144,"data_stored":2442709589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":456980938},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":2.1400000000000001,"15min":1.6659999999999999},"min":{"1min":0.26800000000000002,"5min":0.224,"15min":0.20699999999999999},"max":{"1min":6.008,"5min":36.024000000000001,"15min":52.591999999999999},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":2.2210000000000001,"15min":1.5609999999999999},"min":{"1min":0.30599999999999999,"5min":0.24399999999999999,"15min":0.23100000000000001},"max":{"1min":6.681,"5min":35.548000000000002,"15min":52.494},"last":0.312}]},{"osd":1,"last update":"Sat Feb 10 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":2.8450000000000002,"5min":2.5499999999999998,"15min":1.4470000000000001},"min":{"1min":0.318,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":30.228000000000002,"5min":35.459000000000003,"15min":35.459000000000003},"last":1.1479999999999999},{"interface":"front","average":{"1min":2.859,"5min":2.4510000000000001,"15min":1.458},"min":{"1min":0.27400000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":23.239000000000001,"5min":35.225000000000001,"15min":35.225000000000001},"last":0.877}]},{"osd":2,"last update":"Sat Feb 10 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.2669999999999999,"15min":1.1020000000000001},"min":{"1min":0.314,"5min":0.29199999999999998,"15min":0.27300000000000002},"max":{"1min":5.9450000000000003,"5min":18.57,"15min":19.710000000000001},"last":3.3769999999999998},{"interface":"front","average":{"1min":1.117,"5min":1.468,"15min":1.228},"min":{"1min":0.32500000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.343,"5min":18.521000000000001,"15min":19.643000000000001},"last":2.302}]},{"osd":3,"last update":"Sat Feb 10 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":4.423,"5min":2.2200000000000002,"15min":1.7210000000000001},"min":{"1min":0.35699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":43.573999999999998,"5min":43.573999999999998,"15min":64.120999999999995},"last":3.5099999999999998},{"interface":"front","average":{"1min":4.4539999999999997,"5min":2.2789999999999999,"15min":1.8700000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":43.451000000000001,"5min":43.451000000000001,"15min":64.400000000000006},"last":2.3290000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.85299999999999998,"15min":0.61399999999999999},"min":{"1min":0.28599999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.895,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.77300000000000002,"15min":0.60299999999999998},"min":{"1min":0.33700000000000002,"5min":0.26600000000000001,"15min":0.24099999999999999},"max":{"1min":1.6699999999999999,"5min":4.6980000000000004,"15min":4.6980000000000004},"last":3.492}]},{"osd":5,"last update":"Sat Feb 10 07:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.96399999999999997,"15min":0.80500000000000005},"min":{"1min":0.31,"5min":0.245,"15min":0.245},"max":{"1min":3.395,"5min":6.1050000000000004,"15min":6.1050000000000004},"last":1.105},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.0369999999999999,"15min":0.84399999999999997},"min":{"1min":0.32300000000000001,"5min":0.32200000000000001,"15min":0.29999999999999999},"max":{"1min":2.9929999999999999,"5min":4.4939999999999998,"15min":4.4939999999999998},"last":0.52400000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1,"15min":0.73199999999999998},"min":{"1min":0.36499999999999999,"5min":0.30299999999999999,"15min":0.28199999999999997},"max":{"1min":3.2269999999999999,"5min":9.7720000000000002,"15min":9.7720000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.873,"15min":0.69499999999999995},"min":{"1min":0.40100000000000002,"5min":0.29799999999999999,"15min":0.27400000000000002},"max":{"1min":1.756,"5min":6.6109999999999998,"15min":6.6109999999999998},"last":3.431}]}]},{"osd":6,"up_from":50,"seq":214748364966,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1117136,"kb_used_data":788872,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92628016,"statfs":{"total":95995035648,"available":94851088384,"internally_reserved":0,"allocated":807804928,"data_stored":807520176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":336132550},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.3959999999999999,"15min":1.3700000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.217},"max":{"1min":4.468,"5min":13.728999999999999,"15min":63.405999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.214,"15min":1.5089999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.0140000000000002,"5min":13.768000000000001,"15min":67.748000000000005},"last":0.68700000000000006}]},{"osd":1,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":2.1579999999999999,"15min":1.4239999999999999},"min":{"1min":0.25600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":3.8149999999999999,"5min":37.659999999999997,"15min":37.659999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":2.1789999999999998,"15min":1.4379999999999999},"min":{"1min":0.27000000000000002,"5min":0.253,"15min":0.24399999999999999},"max":{"1min":4.9749999999999996,"5min":37.75,"15min":37.75},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.145,"15min":1.3839999999999999},"min":{"1min":0.28899999999999998,"5min":0.253,"15min":0.253},"max":{"1min":4.524,"5min":4.556,"15min":65.278999999999996},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.127,"5min":1.0249999999999999,"15min":1.4279999999999999},"min":{"1min":0.34599999999999997,"5min":0.28399999999999997,"15min":0.25800000000000001},"max":{"1min":5.6779999999999999,"5min":5.6779999999999999,"15min":65.265000000000001},"last":0.40600000000000003}]},{"osd":3,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.899,"5min":1.8240000000000001,"15min":1.766},"min":{"1min":0.34799999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":11.103999999999999,"5min":20.603999999999999,"15min":62.835999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":2.274,"5min":1.925,"15min":1.4259999999999999},"min":{"1min":0.36399999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":11.304,"5min":20.478999999999999,"15min":63.878999999999998},"last":0.38200000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.68799999999999994,"15min":0.57999999999999996},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.502,"5min":4.7489999999999997,"15min":7.7839999999999998},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.65400000000000003,"15min":0.56799999999999995},"min":{"1min":0.30299999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80900000000000005,"5min":4.3520000000000003,"15min":4.3520000000000003},"last":0.55100000000000005}]},{"osd":5,"last update":"Sat Feb 10 07:44:48 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.78800000000000003,"15min":0.74399999999999999},"min":{"1min":0.32300000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.4809999999999999,"5min":5.899,"15min":5.899},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.84999999999999998,"15min":0.68799999999999994},"min":{"1min":0.33300000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.536,"5min":4.9299999999999997,"15min":4.9299999999999997},"last":0.58299999999999996}]},{"osd":7,"last update":"Sat Feb 10 07:44:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.876,"15min":0.68999999999999995},"min":{"1min":0.32200000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.4279999999999999,"5min":4.641,"15min":7.859},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.86499999999999999,"15min":0.69599999999999995},"min":{"1min":0.39900000000000002,"5min":0.315,"15min":0.29699999999999999},"max":{"1min":4.4630000000000001,"5min":4.4850000000000003,"15min":4.4850000000000003},"last":0.47099999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575039,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3084348,"kb_used_data":2579764,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":90660804,"statfs":{"total":95995035648,"available":92836663296,"internally_reserved":0,"allocated":2641678336,"data_stored":2641348955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":516684230},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.9570000000000001,"5min":1.7490000000000001,"15min":1.71},"min":{"1min":0.192,"5min":0.182,"15min":0.182},"max":{"1min":4.5830000000000002,"5min":7.2850000000000001,"15min":114.27},"last":2.5659999999999998},{"interface":"front","average":{"1min":2.0270000000000001,"5min":1.726,"15min":1.8799999999999999},"min":{"1min":0.42299999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":5.4640000000000004,"5min":7.8330000000000002,"15min":119.499},"last":1.952}]},{"osd":2,"last update":"Sat Feb 10 07:43:58 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.456,"15min":1.081},"min":{"1min":0.38100000000000001,"5min":0.27600000000000002,"15min":0.19600000000000001},"max":{"1min":6.782,"5min":6.782,"15min":8.9209999999999994},"last":0.20399999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.579,"15min":1.1919999999999999},"min":{"1min":0.30399999999999999,"5min":0.25600000000000001,"15min":0.23699999999999999},"max":{"1min":3.976,"5min":8.577,"15min":8.577},"last":2.0150000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.8340000000000001,"5min":2.0939999999999999,"15min":1.46},"min":{"1min":0.30299999999999999,"5min":0.222,"15min":0.21099999999999999},"max":{"1min":8.4139999999999997,"5min":11.535,"15min":11.535},"last":2.6539999999999999},{"interface":"front","average":{"1min":2.6379999999999999,"5min":1.8859999999999999,"15min":1.3979999999999999},"min":{"1min":0.59599999999999997,"5min":0.24199999999999999,"15min":0.23799999999999999},"max":{"1min":8.6289999999999996,"5min":9.516,"15min":11.698},"last":1.6419999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:43:52 2024","interfaces":[{"interface":"back","average":{"1min":1.7649999999999999,"5min":3.536,"15min":1.8440000000000001},"min":{"1min":0.39300000000000002,"5min":0.32900000000000001,"15min":0.30499999999999999},"max":{"1min":6.8410000000000002,"5min":207.45699999999999,"15min":207.45699999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.377,"5min":1.234,"15min":0.97199999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":4.3769999999999998,"5min":4.6159999999999997,"15min":7.8079999999999998},"last":0.54400000000000004}]},{"osd":5,"last update":"Sat Feb 10 07:43:52 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":2.7789999999999999,"15min":2.0099999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":8.2170000000000005,"5min":98.703000000000003,"15min":98.703000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.8779999999999999,"5min":2.661,"15min":1.913},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":8.2289999999999992,"5min":98.664000000000001,"15min":98.664000000000001},"last":1.9870000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.6950000000000001,"5min":1.286,"15min":1.0720000000000001},"min":{"1min":0.35399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":6.7869999999999999,"5min":6.7869999999999999,"15min":11.244999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":2.335,"5min":1.6399999999999999,"15min":1.2609999999999999},"min":{"1min":0.375,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":8.6020000000000003,"5min":8.6020000000000003,"15min":9.0090000000000003},"last":0.499}]},{"osd":7,"last update":"Sat Feb 10 07:43:58 2024","interfaces":[{"interface":"back","average":{"1min":1.869,"5min":1.569,"15min":1.175},"min":{"1min":0.50700000000000001,"5min":0.36399999999999999,"15min":0.36099999999999999},"max":{"1min":7.9500000000000002,"5min":7.9500000000000002,"15min":7.9500000000000002},"last":2.0390000000000001},{"interface":"front","average":{"1min":2.032,"5min":1.4770000000000001,"15min":1.1339999999999999},"min":{"1min":0.65200000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":7.7089999999999996,"5min":7.7089999999999996,"15min":11.212},"last":0.57699999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738565,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1459000,"kb_used_data":1040492,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92286152,"statfs":{"total":95995035648,"available":94501019648,"internally_reserved":0,"allocated":1065463808,"data_stored":1065151647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:44:30 2024","interfaces":[{"interface":"back","average":{"1min":3.1640000000000001,"5min":3.6339999999999999,"15min":2.0619999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":48.447000000000003,"5min":156.16200000000001,"15min":156.16200000000001},"last":2.1539999999999999},{"interface":"front","average":{"1min":3.371,"5min":3.758,"15min":2.0790000000000002},"min":{"1min":0.19,"5min":0.183,"15min":0.183},"max":{"1min":49.302,"5min":156.941,"15min":156.941},"last":2.5499999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:44:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.827,"15min":1.4099999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.6950000000000003,"5min":28.163,"15min":28.163},"last":2.036},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.375,"15min":1.208},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":4.8090000000000002,"5min":24.059999999999999,"15min":24.059999999999999},"last":2.5819999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:44:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9640000000000004,"5min":3.6070000000000002,"15min":2.2069999999999999},"min":{"1min":0.25700000000000001,"5min":0.222,"15min":0.222},"max":{"1min":97.677000000000007,"5min":97.677000000000007,"15min":97.677000000000007},"last":2.0790000000000002},{"interface":"front","average":{"1min":5.7160000000000002,"5min":2.5670000000000002,"15min":1.784},"min":{"1min":0.32400000000000001,"5min":0.246,"15min":0.246},"max":{"1min":49.344000000000001,"5min":49.344000000000001,"15min":49.344000000000001},"last":2.5110000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:44:51 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.27600000000000002},"max":{"1min":5.0700000000000003,"5min":5.0700000000000003,"15min":5.4260000000000002},"last":2.093},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.0169999999999999,"15min":1.1859999999999999},"min":{"1min":0.29199999999999998,"5min":0.28399999999999997,"15min":0.26900000000000002},"max":{"1min":4.6970000000000001,"5min":4.8220000000000001,"15min":19.219000000000001},"last":2.1339999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:44:51 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.5640000000000001,"15min":2.5550000000000002},"min":{"1min":0.34999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.1929999999999996,"5min":23.391999999999999,"15min":33.289999999999999},"last":2.0550000000000002},{"interface":"front","average":{"1min":1.395,"5min":1.484,"15min":2.984},"min":{"1min":0.495,"5min":0.308,"15min":0.308},"max":{"1min":5.0049999999999999,"5min":23.417999999999999,"15min":33.430999999999997},"last":2.1699999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:44:30 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.89500000000000002,"15min":1.117},"min":{"1min":0.41099999999999998,"5min":0.33600000000000002,"15min":0.29599999999999999},"max":{"1min":2.286,"5min":4.4109999999999996,"15min":19.16},"last":2.6150000000000002},{"interface":"front","average":{"1min":1.028,"5min":1.248,"15min":1.1579999999999999},"min":{"1min":0.38300000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":4.1059999999999999,"5min":16.431000000000001,"15min":16.431000000000001},"last":2.1230000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:44:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.4450000000000001,"15min":1.5069999999999999},"min":{"1min":0.42199999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.3519999999999999,"5min":16.282,"15min":19.178000000000001},"last":2.1070000000000002},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1120000000000001,"15min":1.1040000000000001},"min":{"1min":0.41499999999999998,"5min":0.379,"15min":0.376},"max":{"1min":3.4630000000000001,"5min":5.5659999999999998,"15min":8.5700000000000003},"last":2.649}]}]},{"osd":2,"up_from":18,"seq":77309411514,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1229400,"kb_used_data":792848,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92515752,"statfs":{"total":95995035648,"available":94736130048,"internally_reserved":0,"allocated":811876352,"data_stored":811583894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:44:04 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":2.1659999999999999,"15min":1.9350000000000001},"min":{"1min":0.252,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":3.899,"5min":65.430999999999997,"15min":90.046999999999997},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.758,"5min":1.641,"15min":1.748},"min":{"1min":0.24099999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.2009999999999996,"5min":8.3309999999999995,"15min":90.016000000000005},"last":0.24199999999999999}]},{"osd":1,"last update":"Sat Feb 10 07:44:04 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":2.4470000000000001,"15min":1.556},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.19500000000000001},"max":{"1min":6.8280000000000003,"5min":65.795000000000002,"15min":65.795000000000002},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.8360000000000001,"15min":1.343},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.19500000000000001},"max":{"1min":5.1749999999999998,"5min":8.1010000000000009,"15min":9.5630000000000006},"last":2.4409999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:44:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":2.3300000000000001,"15min":1.6859999999999999},"min":{"1min":0.32700000000000001,"5min":0.29799999999999999,"15min":0.23400000000000001},"max":{"1min":5.6230000000000002,"5min":65.869,"15min":65.869},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.635,"5min":1.6259999999999999,"15min":1.4830000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.7160000000000002,"5min":8.9710000000000001,"15min":11.739000000000001},"last":3.589}]},{"osd":4,"last update":"Sat Feb 10 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.913,"15min":1.4330000000000001},"min":{"1min":0.34599999999999997,"5min":0.309,"15min":0.29099999999999998},"max":{"1min":4.2249999999999996,"5min":65.494,"15min":65.494},"last":1.629},{"interface":"front","average":{"1min":1.325,"5min":1.264,"15min":1.28},"min":{"1min":0.34200000000000003,"5min":0.311,"15min":0.311},"max":{"1min":6.8259999999999996,"5min":6.8259999999999996,"15min":33.554000000000002},"last":2.4780000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:44:47 2024","interfaces":[{"interface":"back","average":{"1min":1.615,"5min":1.5760000000000001,"15min":1.347},"min":{"1min":0.44,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":6.915,"5min":6.9770000000000003,"15min":7.1109999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.998,"5min":2.1070000000000002,"15min":1.742},"min":{"1min":0.38,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":4.1970000000000001,"5min":65.834000000000003,"15min":65.834000000000003},"last":1.5609999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:44:29 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.9470000000000001,"15min":1.4419999999999999},"min":{"1min":0.46300000000000002,"5min":0.36599999999999999,"15min":0.33800000000000002},"max":{"1min":3.1309999999999998,"5min":65.748000000000005,"15min":65.748000000000005},"last":1.6539999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.381,"15min":1.1870000000000001},"min":{"1min":0.42999999999999999,"5min":0.39900000000000002,"15min":0.315},"max":{"1min":3.8839999999999999,"5min":7.0359999999999996,"15min":8.1899999999999995},"last":0.56299999999999994}]},{"osd":7,"last update":"Sat Feb 10 07:44:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7470000000000001,"5min":1.5,"15min":1.208},"min":{"1min":0.42399999999999999,"5min":0.38900000000000001,"15min":0.33000000000000002},"max":{"1min":5.5,"5min":7.1779999999999999,"15min":8.2189999999999994},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.2430000000000001,"5min":2.0590000000000002,"15min":1.3979999999999999},"min":{"1min":0.47599999999999998,"5min":0.44600000000000001,"15min":0.38200000000000001},"max":{"1min":5.6849999999999996,"5min":65.539000000000001,"15min":65.539000000000001},"last":1.601}]}]},{"osd":3,"up_from":26,"seq":111669149876,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2729776,"kb_used_data":2233448,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":91015376,"statfs":{"total":95995035648,"available":93199745024,"internally_reserved":0,"allocated":2287050752,"data_stored":2286733194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":508164551},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:44:31 2024","interfaces":[{"interface":"back","average":{"1min":2.056,"5min":2.0369999999999999,"15min":1.595},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.17499999999999999},"max":{"1min":7.181,"5min":15.446,"15min":25.277000000000001},"last":2.9449999999999998},{"interface":"front","average":{"1min":1.903,"5min":1.671,"15min":1.4810000000000001},"min":{"1min":0.215,"5min":0.21299999999999999,"15min":0.18099999999999999},"max":{"1min":6.4000000000000004,"5min":9.0459999999999994,"15min":28.390000000000001},"last":3.173}]},{"osd":1,"last update":"Sat Feb 10 07:44:31 2024","interfaces":[{"interface":"back","average":{"1min":2.056,"5min":2.0699999999999998,"15min":1.5129999999999999},"min":{"1min":0.36099999999999999,"5min":0.219,"15min":0.20999999999999999},"max":{"1min":6.5289999999999999,"5min":25.105,"15min":25.105},"last":7.3559999999999999},{"interface":"front","average":{"1min":2.4470000000000001,"5min":2.6230000000000002,"15min":1.823},"min":{"1min":0.29599999999999999,"5min":0.251,"15min":0.23300000000000001},"max":{"1min":6.2489999999999997,"5min":28.555,"15min":28.555},"last":2.0600000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:44:31 2024","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.2069999999999999,"15min":2.0790000000000002},"min":{"1min":0.25900000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.9560000000000004,"5min":13.209,"15min":163.441},"last":3.3069999999999999},{"interface":"front","average":{"1min":1.7589999999999999,"5min":1.865,"15min":2.044},"min":{"1min":0.42499999999999999,"5min":0.313,"15min":0.218},"max":{"1min":5.4630000000000001,"5min":9.3989999999999991,"15min":166.82400000000001},"last":1.502}]},{"osd":4,"last update":"Sat Feb 10 07:43:50 2024","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.3819999999999999,"15min":1.121},"min":{"1min":0.42699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":6.5019999999999998,"5min":9.6609999999999996,"15min":9.6609999999999996},"last":3.2829999999999999},{"interface":"front","average":{"1min":3.1819999999999999,"5min":1.927,"15min":1.3500000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":13.276,"5min":13.276,"15min":13.276},"last":3.3450000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:43:50 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":2.2149999999999999,"15min":1.4650000000000001},"min":{"1min":0.45900000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":62.817,"5min":62.817,"15min":62.817},"last":3.2549999999999999},{"interface":"front","average":{"1min":7.5099999999999998,"5min":2.8250000000000002,"15min":1.75},"min":{"1min":0.499,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":62.865000000000002,"5min":62.865000000000002,"15min":62.865000000000002},"last":3.4849999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:44:33 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":2.1120000000000001,"15min":1.5860000000000001},"min":{"1min":0.33800000000000002,"5min":0.318,"15min":0.318},"max":{"1min":5.9219999999999997,"5min":13.307,"15min":32.048999999999999},"last":3.5139999999999998},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.665,"15min":1.3660000000000001},"min":{"1min":0.437,"5min":0.38300000000000001,"15min":0.33600000000000002},"max":{"1min":5.6340000000000003,"5min":10.191000000000001,"15min":25.937000000000001},"last":3.0920000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:44:03 2024","interfaces":[{"interface":"back","average":{"1min":3.8559999999999999,"5min":1.954,"15min":1.444},"min":{"1min":0.47599999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":31.257999999999999,"5min":31.257999999999999,"15min":31.257999999999999},"last":3.2229999999999999},{"interface":"front","average":{"1min":5.609,"5min":2.5720000000000001,"15min":1.698},"min":{"1min":0.46999999999999997,"5min":0.39300000000000002,"15min":0.33600000000000002},"max":{"1min":30.5,"5min":30.5,"15min":30.5},"last":3.5379999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953648,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1179308,"kb_used_data":805220,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92565844,"statfs":{"total":95995035648,"available":94787424256,"internally_reserved":0,"allocated":824545280,"data_stored":824231930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:44:52 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.873,"15min":1.97},"min":{"1min":0.23999999999999999,"5min":0.23300000000000001,"15min":0.22600000000000001},"max":{"1min":4.2039999999999997,"5min":45.610999999999997,"15min":74.533000000000001},"last":1.079},{"interface":"front","average":{"1min":1.052,"5min":1.8100000000000001,"15min":1.873},"min":{"1min":0.27000000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.9409999999999998,"5min":45.524999999999999,"15min":74.340999999999994},"last":1.1479999999999999}]},{"osd":1,"last update":"Sat Feb 10 07:44:52 2024","interfaces":[{"interface":"back","average":{"1min":3.6640000000000001,"5min":2.8759999999999999,"15min":1.911},"min":{"1min":0.29299999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":53.472000000000001,"5min":53.472000000000001,"15min":69.852000000000004},"last":0.95799999999999996},{"interface":"front","average":{"1min":3.7850000000000001,"5min":3.0619999999999998,"15min":1.944},"min":{"1min":0.34699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":53.536000000000001,"5min":53.536000000000001,"15min":70.692999999999998},"last":1.004}]},{"osd":2,"last update":"Sat Feb 10 07:44:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.81399999999999995,"15min":1.1910000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.881,"5min":4.4470000000000001,"15min":50.162999999999997},"last":1.206},{"interface":"front","average":{"1min":0.91600000000000004,"5min":1.0640000000000001,"15min":1.1890000000000001},"min":{"1min":0.30099999999999999,"5min":0.28399999999999997,"15min":0.27700000000000002},"max":{"1min":2.766,"5min":8.5129999999999999,"15min":50.453000000000003},"last":0.80700000000000005}]},{"osd":3,"last update":"Sat Feb 10 07:44:52 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.5369999999999999,"15min":1.6519999999999999},"min":{"1min":0.311,"5min":0.30499999999999999,"15min":0.28499999999999998},"max":{"1min":3.714,"5min":40.710000000000001,"15min":69.813000000000002},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.6160000000000001,"15min":1.6930000000000001},"min":{"1min":0.33400000000000002,"5min":0.30099999999999999,"15min":0.27900000000000003},"max":{"1min":4.609,"5min":40.981000000000002,"15min":69.760999999999996},"last":0.98799999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:44:52 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.81399999999999995,"15min":0.59899999999999998},"min":{"1min":0.25,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.8380000000000001,"5min":7.4969999999999999,"15min":7.4969999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.76700000000000002,"15min":0.60699999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.294,"5min":4.242,"15min":7.101},"last":0.48399999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:44:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.70299999999999996,"15min":0.56299999999999994},"min":{"1min":0.28899999999999998,"5min":0.28399999999999997,"15min":0.25900000000000001},"max":{"1min":2.8690000000000002,"5min":3.347,"15min":3.347},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.68999999999999995,"15min":0.57799999999999996},"min":{"1min":0.30299999999999999,"5min":0.26600000000000001,"15min":0.24399999999999999},"max":{"1min":1.744,"5min":3.0219999999999998,"15min":3.7109999999999999},"last":1.0960000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:43:57 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.69299999999999995,"15min":0.58799999999999997},"min":{"1min":0.38400000000000001,"5min":0.32700000000000001,"15min":0.29099999999999998},"max":{"1min":2.8180000000000001,"5min":2.9740000000000002,"15min":2.9740000000000002},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.80700000000000005,"15min":0.65500000000000003},"min":{"1min":0.42899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.8460000000000001,"5min":4.1710000000000003,"15min":7.0359999999999996},"last":0.92100000000000004}]}]},{"osd":5,"up_from":156,"seq":670014898277,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2801052,"kb_used_data":2437652,"kb_used_omap":4,"kb_used_meta":363387,"kb_avail":90944100,"statfs":{"total":95995035648,"available":93126758400,"internally_reserved":0,"allocated":2496155648,"data_stored":2495859300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":372108962},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:44:46 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":2.2250000000000001,"15min":1.778},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.20499999999999999},"max":{"1min":3.7989999999999999,"5min":52.938000000000002,"15min":52.938000000000002},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.752,"5min":2.6499999999999999,"15min":2.0139999999999998},"min":{"1min":0.255,"5min":0.246,"15min":0.246},"max":{"1min":17.82,"5min":52.805,"15min":52.805},"last":4.6239999999999997}]},{"osd":1,"last update":"Sat Feb 10 07:44:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":2.6509999999999998,"15min":2.952},"min":{"1min":0.316,"5min":0.26500000000000001,"15min":0.245},"max":{"1min":6.5629999999999997,"5min":54.298000000000002,"15min":54.298000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.3149999999999999,"5min":2.5739999999999998,"15min":2.9860000000000002},"min":{"1min":0.27800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.3029999999999999,"5min":52.969999999999999,"15min":52.969999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Sat Feb 10 07:44:46 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.077,"15min":1.5189999999999999},"min":{"1min":0.29399999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.718,"5min":5.5519999999999996,"15min":11.601000000000001},"last":0.432},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0580000000000001,"15min":1.5760000000000001},"min":{"1min":0.38400000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.629,"5min":4.6619999999999999,"15min":11.618},"last":0.50600000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:44:46 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":2.7999999999999998,"15min":1.9350000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":10,"5min":53.366999999999997,"15min":53.366999999999997},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.669,"5min":2.9550000000000001,"15min":2.1880000000000002},"min":{"1min":0.34999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":6.4039999999999999,"5min":52.731000000000002,"15min":52.731000000000002},"last":0.55000000000000004}]},{"osd":4,"last update":"Sat Feb 10 07:44:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.82599999999999996,"15min":0.62},"min":{"1min":0.309,"5min":0.246,"15min":0.246},"max":{"1min":1.661,"5min":5.4050000000000002,"15min":5.4050000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.84699999999999998,"15min":0.59999999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":5.5670000000000002,"5min":5.5670000000000002,"15min":5.5670000000000002},"last":0.47999999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:44:46 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.85499999999999998,"15min":0.64800000000000002},"min":{"1min":0.438,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.6619999999999999,"5min":4.3860000000000001,"15min":4.3860000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.80500000000000005,"15min":0.72899999999999998},"min":{"1min":0.40100000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.2310000000000001,"5min":2.839,"15min":4.2839999999999998},"last":0.53200000000000003}]},{"osd":7,"last update":"Sat Feb 10 07:44:46 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0880000000000001,"15min":0.74199999999999999},"min":{"1min":0.378,"5min":0.312,"15min":0.312},"max":{"1min":4.7430000000000003,"5min":8.8059999999999992,"15min":8.8059999999999992},"last":0.497},{"interface":"front","average":{"1min":0.995,"5min":0.92800000000000005,"15min":0.71499999999999997},"min":{"1min":0.36299999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":3.9009999999999998,"5min":4.3600000000000003,"15min":4.3600000000000003},"last":0.59699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1051787264,"data_stored":1051787264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2612006912,"data_stored":2612002840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":798228480,"data_stored":798228480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2282553344,"data_stored":2282553344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":811073536,"data_stored":811073536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2453344256,"data_stored":2453340184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":794296320,"data_stored":794296320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2411991040,"data_stored":2411986968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:44:58.965 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-10T07:44:58.965 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:44:58.965 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3, 4, 6, 7, 5] 2024-02-10T07:44:58.965 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-10T07:44:58.965 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.3 2024-02-10T07:44:59.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:59.003 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:59.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:44:59.049 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:44:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:44:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:44:59.105 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:59.143 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:59.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:59.244 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:59.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:59.284 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:59.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:59.297 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:44:59.321 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:44:59 smithi008 systemd[1]: Stopping Ceph osd.3 for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:44:59.321 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:59 smithi008 ceph-mon[29816]: pgmap v588: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-02-10T07:44:59.321 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:59 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:44:59.321 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:59 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:44:59.321 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:59 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:44:59.321 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:44:59 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2693406952' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:44:59.321 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:59 smithi008 ceph-mon[40743]: pgmap v588: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-02-10T07:44:59.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:59 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:44:59.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:59 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:44:59.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:59 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:44:59.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:44:59 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2693406952' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:44:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:59 smithi170 ceph-mon[33394]: pgmap v588: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-02-10T07:44:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:59 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:44:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:59 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:44:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:59 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:44:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:44:59 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2693406952' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:44:59.384 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:44:59.434 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:59.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:44:59.536 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:44:59.574 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:44:59.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:59.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:44:59.608 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:44:59.608 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:44:59.677 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:44:59.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:59.714 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:59.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:44:59.815 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:44:59.848 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:59.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:44:59.950 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:44:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:44:59.985 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:44:59.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:00.086 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:00.128 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:00.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:00.230 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:00.265 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:00.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:00.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[29816]: 5.17 scrub starts 2024-02-10T07:45:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[29816]: 5.17 scrub ok 2024-02-10T07:45:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:45:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:45:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:45:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:45:00.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2693406952' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:45:00.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[29816]: osdmap e295: 8 total, 8 up, 8 in 2024-02-10T07:45:00.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[40743]: 5.17 scrub starts 2024-02-10T07:45:00.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[40743]: 5.17 scrub ok 2024-02-10T07:45:00.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:45:00.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:45:00.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:45:00.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:45:00.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2693406952' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:45:00.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:00 smithi008 ceph-mon[40743]: osdmap e295: 8 total, 8 up, 8 in 2024-02-10T07:45:00.367 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:00 smithi170 ceph-mon[33394]: 5.17 scrub starts 2024-02-10T07:45:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:00 smithi170 ceph-mon[33394]: 5.17 scrub ok 2024-02-10T07:45:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:00 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:45:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:00 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:45:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:00 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:45:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:00 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:45:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:00 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2693406952' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:45:00.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:00 smithi170 ceph-mon[33394]: osdmap e295: 8 total, 8 up, 8 in 2024-02-10T07:45:00.402 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:00.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:00.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:00.503 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:00.535 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:00.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:00.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:00.637 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:00.672 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:00.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:00.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:00.772 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:45:00.774 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:00.806 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:00.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:00.907 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:00.938 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:00.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:01.040 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:01.074 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:01.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:01.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:01.176 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:01.211 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:01.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:01.269 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-10T07:45:01.312 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:01.353 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:01.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:01.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:01.455 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:01.502 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:01.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:01.511 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[29816]: pgmap v590: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-02-10T07:45:01.511 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[29816]: 4.c scrub starts 2024-02-10T07:45:01.511 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[29816]: 4.c scrub ok 2024-02-10T07:45:01.511 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[29816]: 4.3 scrub starts 2024-02-10T07:45:01.511 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[29816]: 4.3 scrub ok 2024-02-10T07:45:01.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[29816]: 4.7 scrub starts 2024-02-10T07:45:01.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[29816]: 4.7 scrub ok 2024-02-10T07:45:01.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[29816]: 1.0 scrub starts 2024-02-10T07:45:01.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[29816]: 1.0 scrub ok 2024-02-10T07:45:01.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[29816]: 4.19 scrub starts 2024-02-10T07:45:01.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[29816]: 4.19 scrub ok 2024-02-10T07:45:01.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[29816]: 5.6 scrub starts 2024-02-10T07:45:01.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[29816]: 5.6 scrub ok 2024-02-10T07:45:01.512 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[40743]: pgmap v590: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-02-10T07:45:01.512 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[40743]: 4.c scrub starts 2024-02-10T07:45:01.512 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[40743]: 4.c scrub ok 2024-02-10T07:45:01.512 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[40743]: 4.3 scrub starts 2024-02-10T07:45:01.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[40743]: 4.3 scrub ok 2024-02-10T07:45:01.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[40743]: 4.7 scrub starts 2024-02-10T07:45:01.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[40743]: 4.7 scrub ok 2024-02-10T07:45:01.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[40743]: 1.0 scrub starts 2024-02-10T07:45:01.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[40743]: 1.0 scrub ok 2024-02-10T07:45:01.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[40743]: 4.19 scrub starts 2024-02-10T07:45:01.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[40743]: 4.19 scrub ok 2024-02-10T07:45:01.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[40743]: 5.6 scrub starts 2024-02-10T07:45:01.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-mon[40743]: 5.6 scrub ok 2024-02-10T07:45:01.604 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:01 smithi170 ceph-mon[33394]: pgmap v590: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-02-10T07:45:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:01 smithi170 ceph-mon[33394]: 4.c scrub starts 2024-02-10T07:45:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:01 smithi170 ceph-mon[33394]: 4.c scrub ok 2024-02-10T07:45:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:01 smithi170 ceph-mon[33394]: 4.3 scrub starts 2024-02-10T07:45:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:01 smithi170 ceph-mon[33394]: 4.3 scrub ok 2024-02-10T07:45:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:01 smithi170 ceph-mon[33394]: 4.7 scrub starts 2024-02-10T07:45:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:01 smithi170 ceph-mon[33394]: 4.7 scrub ok 2024-02-10T07:45:01.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:01 smithi170 ceph-mon[33394]: 1.0 scrub starts 2024-02-10T07:45:01.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:01 smithi170 ceph-mon[33394]: 1.0 scrub ok 2024-02-10T07:45:01.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:01 smithi170 ceph-mon[33394]: 4.19 scrub starts 2024-02-10T07:45:01.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:01 smithi170 ceph-mon[33394]: 4.19 scrub ok 2024-02-10T07:45:01.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:01 smithi170 ceph-mon[33394]: 5.6 scrub starts 2024-02-10T07:45:01.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:01 smithi170 ceph-mon[33394]: 5.6 scrub ok 2024-02-10T07:45:01.637 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:01.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:01.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:01.739 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:01.776 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:01.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:01.832 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3[62633]: 2024-02-10T07:45:01.630+0000 7f3836a4c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:45:01.832 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3[62633]: 2024-02-10T07:45:01.630+0000 7f3836a4c700 -1 osd.3 295 *** Got signal Terminated *** 2024-02-10T07:45:01.832 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:45:01 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3[62633]: 2024-02-10T07:45:01.630+0000 7f3836a4c700 -1 osd.3 295 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:45:01.879 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:01.920 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:01.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:02.022 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:02.053 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:02.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:02.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:02.155 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:02.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:02.207 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:02.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:02.308 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:02.345 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:02.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:02.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:02.446 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:02.469 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:02.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:02.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:02.569 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:45:02.571 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:02.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:02 smithi008 ceph-mon[29816]: osd.3 marked itself down and dead 2024-02-10T07:45:02.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:02 smithi008 ceph-mon[40743]: osd.3 marked itself down and dead 2024-02-10T07:45:02.595 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:02.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:02.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:02.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:02 smithi170 ceph-mon[33394]: osd.3 marked itself down and dead 2024-02-10T07:45:02.697 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:02.722 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:02.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:02.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:02.824 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:02.849 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:02.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:02.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:02.951 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:02.977 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:02.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:03.079 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:03.102 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:03.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:03.204 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:03.228 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:03.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:03.295 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:45:02 smithi008 podman[151181]: 2024-02-10 07:45:02.946650676 +0000 UTC m=+1.685422994 container died 23044a16e9545fe2cb943f8a0ee5b1c6cc218214e10732f54b4291470466a981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-10T07:45:03.330 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:03.363 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:03.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:03.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:03.465 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:03.502 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:03.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:03.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[29816]: pgmap v591: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 569 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-10T07:45:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[29816]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:45:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[29816]: osdmap e296: 8 total, 7 up, 8 in 2024-02-10T07:45:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[29816]: 6.15 scrub starts 2024-02-10T07:45:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[29816]: 9.a scrub starts 2024-02-10T07:45:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[29816]: 6.15 scrub ok 2024-02-10T07:45:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[29816]: 9.a scrub ok 2024-02-10T07:45:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:45:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:45:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[40743]: pgmap v591: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 569 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-10T07:45:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[40743]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:45:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[40743]: osdmap e296: 8 total, 7 up, 8 in 2024-02-10T07:45:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[40743]: 6.15 scrub starts 2024-02-10T07:45:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[40743]: 9.a scrub starts 2024-02-10T07:45:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[40743]: 6.15 scrub ok 2024-02-10T07:45:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[40743]: 9.a scrub ok 2024-02-10T07:45:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:45:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:45:03.604 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:03.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:45:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:45:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:03 smithi170 ceph-mon[33394]: pgmap v591: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 569 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-10T07:45:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:03 smithi170 ceph-mon[33394]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-10T07:45:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:03 smithi170 ceph-mon[33394]: osdmap e296: 8 total, 7 up, 8 in 2024-02-10T07:45:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:03 smithi170 ceph-mon[33394]: 6.15 scrub starts 2024-02-10T07:45:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:03 smithi170 ceph-mon[33394]: 9.a scrub starts 2024-02-10T07:45:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:03 smithi170 ceph-mon[33394]: 6.15 scrub ok 2024-02-10T07:45:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:03 smithi170 ceph-mon[33394]: 9.a scrub ok 2024-02-10T07:45:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:45:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:45:03.640 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:03.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:03.741 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:03.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:03.776 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:03.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:03.877 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:03.911 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:03.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:03.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:04.013 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:04.045 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:04.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:04.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:04.147 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:04.182 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:04.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:04.284 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:04.311 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:04.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:04.413 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:04.445 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:04.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:04.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:04.547 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:04.582 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:04.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:04 smithi170 ceph-mon[33394]: 3.6 deep-scrub starts 2024-02-10T07:45:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:04 smithi170 ceph-mon[33394]: 3.6 deep-scrub ok 2024-02-10T07:45:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:04 smithi170 ceph-mon[33394]: osdmap e297: 8 total, 7 up, 8 in 2024-02-10T07:45:04.683 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:04.723 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:04 smithi008 ceph-mon[29816]: 3.6 deep-scrub starts 2024-02-10T07:45:04.723 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:04 smithi008 ceph-mon[29816]: 3.6 deep-scrub ok 2024-02-10T07:45:04.723 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:04 smithi008 ceph-mon[29816]: osdmap e297: 8 total, 7 up, 8 in 2024-02-10T07:45:04.723 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:04 smithi008 ceph-mon[40743]: 3.6 deep-scrub starts 2024-02-10T07:45:04.723 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:04 smithi008 ceph-mon[40743]: 3.6 deep-scrub ok 2024-02-10T07:45:04.723 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:04 smithi008 ceph-mon[40743]: osdmap e297: 8 total, 7 up, 8 in 2024-02-10T07:45:04.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:04.725 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:04.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:04.826 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:04.857 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:04.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:04.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:04.959 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:04.992 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:04.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:04.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:05.094 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:05.135 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:05.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:05.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:05.237 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:05.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:05.286 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:05.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:05.332 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:45:05 smithi008 podman[151181]: 2024-02-10 07:45:05.021450982 +0000 UTC m=+3.760223317 container cleanup 23044a16e9545fe2cb943f8a0ee5b1c6cc218214e10732f54b4291470466a981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-10T07:45:05.332 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:45:05 smithi008 bash[151181]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3 2024-02-10T07:45:05.387 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:05.421 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:05.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:05.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:05.523 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:05.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:05.561 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:05.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:05 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:05 smithi170 ceph-mon[33394]: pgmap v594: 148 pgs: 5 active+undersized+degraded, 12 active+undersized, 24 peering, 7 stale+active+clean, 100 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 170 B/s rd, 0 B/s wr, 1.24k op/s; 3376/165099 objects degraded (2.045%) 2024-02-10T07:45:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:05 smithi170 ceph-mon[33394]: 3.3 deep-scrub starts 2024-02-10T07:45:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:05 smithi170 ceph-mon[33394]: 3.3 deep-scrub ok 2024-02-10T07:45:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:05 smithi170 ceph-mon[33394]: Health check failed: Reduced data availability: 7 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-02-10T07:45:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:05 smithi170 ceph-mon[33394]: Health check failed: Degraded data redundancy: 3376/165099 objects degraded (2.045%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:05 smithi170 ceph-mon[33394]: 6.1 scrub starts 2024-02-10T07:45:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:05 smithi170 ceph-mon[33394]: 5.15 scrub starts 2024-02-10T07:45:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:05 smithi170 ceph-mon[33394]: 5.15 scrub ok 2024-02-10T07:45:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:05 smithi170 ceph-mon[33394]: 6.1 scrub ok 2024-02-10T07:45:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:05 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:45:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:05 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:45:05.662 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:05.697 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:05.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:05.799 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:05.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:05.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[29816]: pgmap v594: 148 pgs: 5 active+undersized+degraded, 12 active+undersized, 24 peering, 7 stale+active+clean, 100 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 170 B/s rd, 0 B/s wr, 1.24k op/s; 3376/165099 objects degraded (2.045%) 2024-02-10T07:45:05.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[29816]: 3.3 deep-scrub starts 2024-02-10T07:45:05.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[29816]: 3.3 deep-scrub ok 2024-02-10T07:45:05.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[29816]: Health check failed: Reduced data availability: 7 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-02-10T07:45:05.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[29816]: Health check failed: Degraded data redundancy: 3376/165099 objects degraded (2.045%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:05.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[29816]: 6.1 scrub starts 2024-02-10T07:45:05.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[29816]: 5.15 scrub starts 2024-02-10T07:45:05.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[29816]: 5.15 scrub ok 2024-02-10T07:45:05.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[29816]: 6.1 scrub ok 2024-02-10T07:45:05.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:45:05.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:45:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[40743]: pgmap v594: 148 pgs: 5 active+undersized+degraded, 12 active+undersized, 24 peering, 7 stale+active+clean, 100 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 170 B/s rd, 0 B/s wr, 1.24k op/s; 3376/165099 objects degraded (2.045%) 2024-02-10T07:45:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[40743]: 3.3 deep-scrub starts 2024-02-10T07:45:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[40743]: 3.3 deep-scrub ok 2024-02-10T07:45:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[40743]: Health check failed: Reduced data availability: 7 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-02-10T07:45:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[40743]: Health check failed: Degraded data redundancy: 3376/165099 objects degraded (2.045%), 5 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[40743]: 6.1 scrub starts 2024-02-10T07:45:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[40743]: 5.15 scrub starts 2024-02-10T07:45:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[40743]: 5.15 scrub ok 2024-02-10T07:45:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[40743]: 6.1 scrub ok 2024-02-10T07:45:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:45:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:05 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:45:05.836 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:05.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:05.938 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:05.975 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:05.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:05.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:06.076 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:06.109 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:06.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:06.211 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:06.237 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:06.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:06.339 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:06.371 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:06.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:06.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:06.473 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:06.512 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:06.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:06.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:06.614 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:06 smithi170 ceph-mon[33394]: 4.e scrub starts 2024-02-10T07:45:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:06 smithi170 ceph-mon[33394]: 4.e scrub ok 2024-02-10T07:45:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:06 smithi170 ceph-mon[33394]: osdmap e298: 8 total, 7 up, 8 in 2024-02-10T07:45:06.651 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:06.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:06.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:06.738 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:06 smithi008 ceph-mon[29816]: 4.e scrub starts 2024-02-10T07:45:06.738 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:06 smithi008 ceph-mon[29816]: 4.e scrub ok 2024-02-10T07:45:06.738 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:06 smithi008 ceph-mon[29816]: osdmap e298: 8 total, 7 up, 8 in 2024-02-10T07:45:06.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:06 smithi008 ceph-mon[40743]: 4.e scrub starts 2024-02-10T07:45:06.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:06 smithi008 ceph-mon[40743]: 4.e scrub ok 2024-02-10T07:45:06.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:06 smithi008 ceph-mon[40743]: osdmap e298: 8 total, 7 up, 8 in 2024-02-10T07:45:06.746 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:45:06.750 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:45:06.753 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:06.804 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:06.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:06.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:06.906 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:06.941 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:06.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:06.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:07.020 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:45:06 smithi008 podman[151399]: 2024-02-10 07:45:06.738242239 +0000 UTC m=+3.893776847 container remove 23044a16e9545fe2cb943f8a0ee5b1c6cc218214e10732f54b4291470466a981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-290212a, ceph=True) 2024-02-10T07:45:07.020 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:45:06 smithi008 bash[151637]: Error: no container with ID 23044a16e9545fe2cb943f8a0ee5b1c6cc218214e10732f54b4291470466a981 found in database: no such container 2024-02-10T07:45:07.042 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:07.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:07.089 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:07.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:07.189 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:07.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:07.226 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:07.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:07.327 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:07.363 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:07.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:07.464 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:07.496 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:07.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:07.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[29816]: pgmap v596: 148 pgs: 13 active+undersized+degraded, 16 active+undersized, 24 peering, 1 stale+active+clean, 94 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 10087/162297 objects degraded (6.215%); 0 B/s, 0 objects/s recovering 2024-02-10T07:45:07.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[29816]: 5.1e deep-scrub starts 2024-02-10T07:45:07.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[29816]: 5.1e deep-scrub ok 2024-02-10T07:45:07.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[29816]: 5.11 scrub starts 2024-02-10T07:45:07.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[29816]: 5.11 scrub ok 2024-02-10T07:45:07.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[29816]: 5.e scrub starts 2024-02-10T07:45:07.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[29816]: 5.e scrub ok 2024-02-10T07:45:07.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[29816]: 2.0 scrub starts 2024-02-10T07:45:07.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[29816]: 2.0 scrub ok 2024-02-10T07:45:07.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[29816]: 3.16 scrub starts 2024-02-10T07:45:07.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[29816]: 3.16 scrub ok 2024-02-10T07:45:07.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:45:07.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:45:07.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[40743]: pgmap v596: 148 pgs: 13 active+undersized+degraded, 16 active+undersized, 24 peering, 1 stale+active+clean, 94 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 10087/162297 objects degraded (6.215%); 0 B/s, 0 objects/s recovering 2024-02-10T07:45:07.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[40743]: 5.1e deep-scrub starts 2024-02-10T07:45:07.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[40743]: 5.1e deep-scrub ok 2024-02-10T07:45:07.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[40743]: 5.11 scrub starts 2024-02-10T07:45:07.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[40743]: 5.11 scrub ok 2024-02-10T07:45:07.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[40743]: 5.e scrub starts 2024-02-10T07:45:07.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[40743]: 5.e scrub ok 2024-02-10T07:45:07.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[40743]: 2.0 scrub starts 2024-02-10T07:45:07.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[40743]: 2.0 scrub ok 2024-02-10T07:45:07.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[40743]: 3.16 scrub starts 2024-02-10T07:45:07.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[40743]: 3.16 scrub ok 2024-02-10T07:45:07.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:45:07.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:07 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:45:07.598 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:07 smithi170 ceph-mon[33394]: pgmap v596: 148 pgs: 13 active+undersized+degraded, 16 active+undersized, 24 peering, 1 stale+active+clean, 94 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 10087/162297 objects degraded (6.215%); 0 B/s, 0 objects/s recovering 2024-02-10T07:45:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:07 smithi170 ceph-mon[33394]: 5.1e deep-scrub starts 2024-02-10T07:45:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:07 smithi170 ceph-mon[33394]: 5.1e deep-scrub ok 2024-02-10T07:45:07.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:07 smithi170 ceph-mon[33394]: 5.11 scrub starts 2024-02-10T07:45:07.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:07 smithi170 ceph-mon[33394]: 5.11 scrub ok 2024-02-10T07:45:07.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:07 smithi170 ceph-mon[33394]: 5.e scrub starts 2024-02-10T07:45:07.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:07 smithi170 ceph-mon[33394]: 5.e scrub ok 2024-02-10T07:45:07.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:07 smithi170 ceph-mon[33394]: 2.0 scrub starts 2024-02-10T07:45:07.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:07 smithi170 ceph-mon[33394]: 2.0 scrub ok 2024-02-10T07:45:07.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:07 smithi170 ceph-mon[33394]: 3.16 scrub starts 2024-02-10T07:45:07.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:07 smithi170 ceph-mon[33394]: 3.16 scrub ok 2024-02-10T07:45:07.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:07 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:45:07.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:07 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:45:07.635 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:07.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:07.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:07.737 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:07.773 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:07.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:07.875 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:07.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:07.970 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:07.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:07.971 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:45:07.971 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:45:08.002 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:45:08.042 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:45:08.072 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:08.078 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:45:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:08.104 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:08.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:08.106 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:45:08.126 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:45:08.127 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:45:08.127 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:45:08.127 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:45:08.127 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-10T07:45:08.127 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:45:08.127 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:45:08.127 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:45:08.128 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:45:08.205 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:08.239 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:08.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:08.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:08.341 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:08.375 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:08.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:08.477 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:08.517 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:08.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:08.619 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:08 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4239828128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:08 smithi170 ceph-mon[33394]: osdmap e299: 8 total, 7 up, 8 in 2024-02-10T07:45:08.657 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:08.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:08.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:08.753 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:08 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4239828128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:08.753 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:08 smithi008 ceph-mon[29816]: osdmap e299: 8 total, 7 up, 8 in 2024-02-10T07:45:08.753 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:08 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4239828128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:08.753 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:08 smithi008 ceph-mon[40743]: osdmap e299: 8 total, 7 up, 8 in 2024-02-10T07:45:08.759 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:08.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:08.796 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:08.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:08.897 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:08.929 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:08.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:08.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:09.030 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:09.063 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:09.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:09.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:45:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:45:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:45:09.165 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:09.199 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:09.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:09.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:09.301 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:09.336 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:09.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:09.438 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:09.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[29816]: pgmap v598: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 719 B/s rd, 0 B/s wr, 1.21k op/s; 25088/152334 objects degraded (16.469%); 0 B/s, 0 objects/s recovering 2024-02-10T07:45:09.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[29816]: 4.5 scrub starts 2024-02-10T07:45:09.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[29816]: 4.5 scrub ok 2024-02-10T07:45:09.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[29816]: 5.7 scrub starts 2024-02-10T07:45:09.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[29816]: 5.7 scrub ok 2024-02-10T07:45:09.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[29816]: 3.d scrub starts 2024-02-10T07:45:09.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[29816]: 3.d scrub ok 2024-02-10T07:45:09.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[29816]: 5.b scrub starts 2024-02-10T07:45:09.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[29816]: 5.b scrub ok 2024-02-10T07:45:09.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[29816]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 11 pgs peering) 2024-02-10T07:45:09.469 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[40743]: pgmap v598: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 719 B/s rd, 0 B/s wr, 1.21k op/s; 25088/152334 objects degraded (16.469%); 0 B/s, 0 objects/s recovering 2024-02-10T07:45:09.469 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[40743]: 4.5 scrub starts 2024-02-10T07:45:09.469 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[40743]: 4.5 scrub ok 2024-02-10T07:45:09.469 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[40743]: 5.7 scrub starts 2024-02-10T07:45:09.469 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[40743]: 5.7 scrub ok 2024-02-10T07:45:09.469 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[40743]: 3.d scrub starts 2024-02-10T07:45:09.469 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[40743]: 3.d scrub ok 2024-02-10T07:45:09.469 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[40743]: 5.b scrub starts 2024-02-10T07:45:09.469 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[40743]: 5.b scrub ok 2024-02-10T07:45:09.469 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:09 smithi008 ceph-mon[40743]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 11 pgs peering) 2024-02-10T07:45:09.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:09.470 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:09.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:09.571 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:09.606 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:09.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:09 smithi170 ceph-mon[33394]: pgmap v598: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 719 B/s rd, 0 B/s wr, 1.21k op/s; 25088/152334 objects degraded (16.469%); 0 B/s, 0 objects/s recovering 2024-02-10T07:45:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:09 smithi170 ceph-mon[33394]: 4.5 scrub starts 2024-02-10T07:45:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:09 smithi170 ceph-mon[33394]: 4.5 scrub ok 2024-02-10T07:45:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:09 smithi170 ceph-mon[33394]: 5.7 scrub starts 2024-02-10T07:45:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:09 smithi170 ceph-mon[33394]: 5.7 scrub ok 2024-02-10T07:45:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:09 smithi170 ceph-mon[33394]: 3.d scrub starts 2024-02-10T07:45:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:09 smithi170 ceph-mon[33394]: 3.d scrub ok 2024-02-10T07:45:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:09 smithi170 ceph-mon[33394]: 5.b scrub starts 2024-02-10T07:45:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:09 smithi170 ceph-mon[33394]: 5.b scrub ok 2024-02-10T07:45:09.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:09 smithi170 ceph-mon[33394]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 11 pgs peering) 2024-02-10T07:45:09.707 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:09.738 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:09.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:09.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:09.840 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:09.873 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:09.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:09.975 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:10.008 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:10.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:10.109 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:10.148 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:10.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:10.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:10.249 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:10.279 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:10.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:10.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:10.381 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:10.399 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:45:10.399 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:45:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:10.415 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:10.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:10.516 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:10.552 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:10.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:10 smithi170 ceph-mon[33394]: 3.1d scrub starts 2024-02-10T07:45:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:10 smithi170 ceph-mon[33394]: 3.1d scrub ok 2024-02-10T07:45:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:10 smithi170 ceph-mon[33394]: 3.11 scrub starts 2024-02-10T07:45:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:10 smithi170 ceph-mon[33394]: 3.11 scrub ok 2024-02-10T07:45:10.654 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:10.693 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:10.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:10.795 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:10.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:10 smithi008 ceph-mon[29816]: 3.1d scrub starts 2024-02-10T07:45:10.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:10 smithi008 ceph-mon[29816]: 3.1d scrub ok 2024-02-10T07:45:10.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:10 smithi008 ceph-mon[29816]: 3.11 scrub starts 2024-02-10T07:45:10.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:10 smithi008 ceph-mon[29816]: 3.11 scrub ok 2024-02-10T07:45:10.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:10 smithi008 ceph-mon[40743]: 3.1d scrub starts 2024-02-10T07:45:10.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:10 smithi008 ceph-mon[40743]: 3.1d scrub ok 2024-02-10T07:45:10.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:10 smithi008 ceph-mon[40743]: 3.11 scrub starts 2024-02-10T07:45:10.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:10 smithi008 ceph-mon[40743]: 3.11 scrub ok 2024-02-10T07:45:10.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:10.833 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:10.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:10.934 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:10.967 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:10.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:10.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:11.069 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:11.105 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:11.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:11.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:11.206 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:11.243 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:11.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:11.344 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:11.380 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:11.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:11.420 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:11 smithi008 ceph-mon[29816]: pgmap v599: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 307 B/s rd, 0 B/s wr, 1.28k op/s; 23762/143121 objects degraded (16.603%); 0 B/s, 0 objects/s recovering 2024-02-10T07:45:11.420 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:11 smithi008 ceph-mon[29816]: 4.1 scrub starts 2024-02-10T07:45:11.420 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:11 smithi008 ceph-mon[29816]: 4.1 scrub ok 2024-02-10T07:45:11.420 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:11 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 23762/143121 objects degraded (16.603%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:11.420 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:11 smithi008 ceph-mon[29816]: 5.1a scrub starts 2024-02-10T07:45:11.420 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:11 smithi008 ceph-mon[29816]: 5.1a scrub ok 2024-02-10T07:45:11.421 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:11 smithi008 ceph-mon[40743]: pgmap v599: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 307 B/s rd, 0 B/s wr, 1.28k op/s; 23762/143121 objects degraded (16.603%); 0 B/s, 0 objects/s recovering 2024-02-10T07:45:11.421 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:11 smithi008 ceph-mon[40743]: 4.1 scrub starts 2024-02-10T07:45:11.421 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:11 smithi008 ceph-mon[40743]: 4.1 scrub ok 2024-02-10T07:45:11.421 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:11 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 23762/143121 objects degraded (16.603%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:11.421 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:11 smithi008 ceph-mon[40743]: 5.1a scrub starts 2024-02-10T07:45:11.421 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:11 smithi008 ceph-mon[40743]: 5.1a scrub ok 2024-02-10T07:45:11.483 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:11.516 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:11.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:11.619 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:11 smithi170 ceph-mon[33394]: pgmap v599: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 307 B/s rd, 0 B/s wr, 1.28k op/s; 23762/143121 objects degraded (16.603%); 0 B/s, 0 objects/s recovering 2024-02-10T07:45:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:11 smithi170 ceph-mon[33394]: 4.1 scrub starts 2024-02-10T07:45:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:11 smithi170 ceph-mon[33394]: 4.1 scrub ok 2024-02-10T07:45:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:11 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 23762/143121 objects degraded (16.603%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:11 smithi170 ceph-mon[33394]: 5.1a scrub starts 2024-02-10T07:45:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:11 smithi170 ceph-mon[33394]: 5.1a scrub ok 2024-02-10T07:45:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:11.657 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:11.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:11.758 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:11.788 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:11.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:11.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:11.793 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:45:11.793 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:45:11.793 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.794 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25283.0:213871 9.0 e299/253 9:009b6931:::benchmark_data_smithi008_134920_object85694:head v 299'18659, mlcod=299'18655)", 2024-02-10T07:45:11.794 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:45:11.786328+0000", 2024-02-10T07:45:11.794 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0060350400000000002, 2024-02-10T07:45:11.794 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0060672829999999997, 2024-02-10T07:45:11.794 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:45:11.794 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:45:11.794 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:45:11.794 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.794 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:45:11.794 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.786328+0000", 2024-02-10T07:45:11.794 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.786328+0000", 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.786329+0000", 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.47e-07 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.786337+0000", 2024-02-10T07:45:11.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.4290000000000004e-06 2024-02-10T07:45:11.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:45:11.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.786353+0000", 2024-02-10T07:45:11.796 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6045999999999998e-05 2024-02-10T07:45:11.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:45:11.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.786361+0000", 2024-02-10T07:45:11.796 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5890000000000003e-06 2024-02-10T07:45:11.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.791560+0000", 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.005198673 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.791616+0000", 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6032000000000002e-05 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25283.0:213884 9.6 e299/253 9:6023e7bb:::benchmark_data_smithi008_134920_object85707:head v 299'13602, mlcod=299'13600)", 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:45:11.790109+0000", 2024-02-10T07:45:11.797 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0022543910000000001, 2024-02-10T07:45:11.798 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002364353, 2024-02-10T07:45:11.798 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:45:11.798 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:45:11.798 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:45:11.798 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.798 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:45:11.798 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.790109+0000", 2024-02-10T07:45:11.798 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:45:11.798 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.798 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.798 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:45:11.798 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.790109+0000", 2024-02-10T07:45:11.798 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:45:11.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:45:11.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.790110+0000", 2024-02-10T07:45:11.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0109999999999999e-06 2024-02-10T07:45:11.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:45:11.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.790118+0000", 2024-02-10T07:45:11.799 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.354e-06 2024-02-10T07:45:11.799 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.799 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.799 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:45:11.799 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.790135+0000", 2024-02-10T07:45:11.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6909e-05 2024-02-10T07:45:11.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.800 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:45:11.800 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.790143+0000", 2024-02-10T07:45:11.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8609999999999998e-06 2024-02-10T07:45:11.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.800 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:45:11.800 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.790299+0000", 2024-02-10T07:45:11.800 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015643699999999999 2024-02-10T07:45:11.800 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.800 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.800 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.790356+0000", 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6946e-05 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25283.0:213882 9.2 e299/296)", 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:45:11.791397+0000", 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.00096666199999999997, 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0011438710000000001, 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:45:11.801 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.791397+0000", 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.791397+0000", 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.791397+0000", 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.9400000000000004e-07 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:45:11.802 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.791404+0000", 2024-02-10T07:45:11.803 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1350000000000003e-06 2024-02-10T07:45:11.803 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.803 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:45:11.803 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.791419+0000", 2024-02-10T07:45:11.803 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4228e-05 2024-02-10T07:45:11.803 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:45:11.803 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:45:11.803 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:45:11.803 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:45:11.791427+0000", 2024-02-10T07:45:11.803 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8059999999999995e-06 2024-02-10T07:45:11.803 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:45:11.803 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:45:11.804 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:45:11.804 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:45:11.804 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:45:11.804 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 3 2024-02-10T07:45:11.804 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:45:11.889 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:11.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:11.922 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:11.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:12.003 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:45:11 smithi008 podman[152211]: 2024-02-10 07:45:11.7997881 +0000 UTC m=+0.034811680 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:45:12.023 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:12.058 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:12.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:12.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:12.160 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:12.204 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:12.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:12.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:12.287 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:45:12 smithi008 podman[152211]: 2024-02-10 07:45:12.146247889 +0000 UTC m=+0.381271445 container create c8897816ea8c4830c88e58805d55bb8aa06e826d8155e93874b73de636cc4e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a) 2024-02-10T07:45:12.307 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:12.341 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:12.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:12.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:12.400 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:45:12.444 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:12.527 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:12.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:12.550 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:12 smithi008 ceph-mon[29816]: 4.b scrub starts 2024-02-10T07:45:12.550 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:12 smithi008 ceph-mon[29816]: 4.b scrub ok 2024-02-10T07:45:12.550 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:12 smithi008 ceph-mon[29816]: 5.4 scrub starts 2024-02-10T07:45:12.550 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:12 smithi008 ceph-mon[29816]: 5.4 scrub ok 2024-02-10T07:45:12.551 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:12 smithi008 ceph-mon[40743]: 4.b scrub starts 2024-02-10T07:45:12.551 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:12 smithi008 ceph-mon[40743]: 4.b scrub ok 2024-02-10T07:45:12.551 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:12 smithi008 ceph-mon[40743]: 5.4 scrub starts 2024-02-10T07:45:12.551 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:12 smithi008 ceph-mon[40743]: 5.4 scrub ok 2024-02-10T07:45:12.551 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:45:12 smithi008 podman[152211]: 2024-02-10 07:45:12.433148663 +0000 UTC m=+0.668172229 container init c8897816ea8c4830c88e58805d55bb8aa06e826d8155e93874b73de636cc4e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:45:12.551 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:45:12 smithi008 podman[152211]: 2024-02-10 07:45:12.439660315 +0000 UTC m=+0.674683878 container start c8897816ea8c4830c88e58805d55bb8aa06e826d8155e93874b73de636cc4e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-10T07:45:12.551 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:45:12 smithi008 podman[152211]: 2024-02-10 07:45:12.550026011 +0000 UTC m=+0.785049565 container attach c8897816ea8c4830c88e58805d55bb8aa06e826d8155e93874b73de636cc4e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True, io.buildah.version=1.33.2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-10T07:45:12.629 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:12 smithi170 ceph-mon[33394]: 4.b scrub starts 2024-02-10T07:45:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:12 smithi170 ceph-mon[33394]: 4.b scrub ok 2024-02-10T07:45:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:12 smithi170 ceph-mon[33394]: 5.4 scrub starts 2024-02-10T07:45:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:12 smithi170 ceph-mon[33394]: 5.4 scrub ok 2024-02-10T07:45:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:12.675 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:12.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:12.776 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:12.817 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:12.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:12.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:12.919 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:12.960 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:12.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:13.062 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:13.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:13.095 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:13.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:13.196 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:13.227 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:13.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:13.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:13.328 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:13.359 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:13.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:13.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:13.413 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:45:13.458 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-10T07:45:13.461 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:13.494 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:13.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:13.595 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:13.633 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:13.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:13.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:13.735 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:13.758 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:13.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:13.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:13.831 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:45:13 smithi008 podman[152211]: 2024-02-10 07:45:13.457704735 +0000 UTC m=+1.692728296 container died c8897816ea8c4830c88e58805d55bb8aa06e826d8155e93874b73de636cc4e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2) 2024-02-10T07:45:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[29816]: pgmap v600: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 383 B/s rd, 0 B/s wr, 915 op/s; 23762/143121 objects degraded (16.603%); 0 B/s, 0 objects/s recovering 2024-02-10T07:45:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[29816]: 6.9 scrub starts 2024-02-10T07:45:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[29816]: 6.9 scrub ok 2024-02-10T07:45:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[29816]: 5.1f scrub starts 2024-02-10T07:45:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[29816]: 5.1f scrub ok 2024-02-10T07:45:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[29816]: 3.1c scrub starts 2024-02-10T07:45:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[29816]: 3.1c scrub ok 2024-02-10T07:45:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3393319923' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:45:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:45:13.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[40743]: pgmap v600: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 383 B/s rd, 0 B/s wr, 915 op/s; 23762/143121 objects degraded (16.603%); 0 B/s, 0 objects/s recovering 2024-02-10T07:45:13.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[40743]: 6.9 scrub starts 2024-02-10T07:45:13.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[40743]: 6.9 scrub ok 2024-02-10T07:45:13.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[40743]: 5.1f scrub starts 2024-02-10T07:45:13.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[40743]: 5.1f scrub ok 2024-02-10T07:45:13.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[40743]: 3.1c scrub starts 2024-02-10T07:45:13.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[40743]: 3.1c scrub ok 2024-02-10T07:45:13.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3393319923' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:45:13.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:13 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:45:13.861 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:13.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:45:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:45:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:13 smithi170 ceph-mon[33394]: pgmap v600: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 2.9 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 383 B/s rd, 0 B/s wr, 915 op/s; 23762/143121 objects degraded (16.603%); 0 B/s, 0 objects/s recovering 2024-02-10T07:45:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:13 smithi170 ceph-mon[33394]: 6.9 scrub starts 2024-02-10T07:45:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:13 smithi170 ceph-mon[33394]: 6.9 scrub ok 2024-02-10T07:45:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:13 smithi170 ceph-mon[33394]: 5.1f scrub starts 2024-02-10T07:45:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:13 smithi170 ceph-mon[33394]: 5.1f scrub ok 2024-02-10T07:45:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:13 smithi170 ceph-mon[33394]: 3.1c scrub starts 2024-02-10T07:45:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:13 smithi170 ceph-mon[33394]: 3.1c scrub ok 2024-02-10T07:45:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:13 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3393319923' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:45:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:13 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:45:13.892 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:13.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:13.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:13.994 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:14.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:14.031 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:14.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:14.132 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:14.170 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:14.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:14.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:14.271 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:14.305 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:14.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:14.407 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:14.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:14.435 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:14.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:14.536 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:14.563 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:14.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:14.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:14.664 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:14.690 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:14.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:14.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:14.791 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:14.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:14.823 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:14.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:14.831 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:14 smithi008 ceph-mon[40743]: 9.5 deep-scrub starts 2024-02-10T07:45:14.831 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:14 smithi008 ceph-mon[40743]: 6.0 scrub starts 2024-02-10T07:45:14.831 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:14 smithi008 ceph-mon[40743]: 6.0 scrub ok 2024-02-10T07:45:14.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:14 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:45:14.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:14 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:45:14.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:14 smithi008 ceph-mon[40743]: osdmap e300: 8 total, 7 up, 8 in 2024-02-10T07:45:14.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:14 smithi008 ceph-mon[29816]: 9.5 deep-scrub starts 2024-02-10T07:45:14.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:14 smithi008 ceph-mon[29816]: 6.0 scrub starts 2024-02-10T07:45:14.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:14 smithi008 ceph-mon[29816]: 6.0 scrub ok 2024-02-10T07:45:14.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:14 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:45:14.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:14 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:45:14.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:14 smithi008 ceph-mon[29816]: osdmap e300: 8 total, 7 up, 8 in 2024-02-10T07:45:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:14 smithi170 ceph-mon[33394]: 9.5 deep-scrub starts 2024-02-10T07:45:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:14 smithi170 ceph-mon[33394]: 6.0 scrub starts 2024-02-10T07:45:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:14 smithi170 ceph-mon[33394]: 6.0 scrub ok 2024-02-10T07:45:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:14 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:45:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:14 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:45:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:14 smithi170 ceph-mon[33394]: osdmap e300: 8 total, 7 up, 8 in 2024-02-10T07:45:14.924 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:14.948 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:14.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:15.049 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:15.084 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:15.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:15.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:15.185 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:15.220 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:15.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:15.321 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:15.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:15.358 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:15.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:15.459 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:15.493 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:15.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:15.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:15.595 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:15.634 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:15.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:15.736 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:15.774 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:15.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:15.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:15.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:15 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:15.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:15 smithi008 ceph-mon[29816]: pgmap v602: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 21286/126918 objects degraded (16.771%) 2024-02-10T07:45:15.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:15 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:15.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:15 smithi008 ceph-mon[40743]: pgmap v602: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 21286/126918 objects degraded (16.771%) 2024-02-10T07:45:15.876 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:15 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:15 smithi170 ceph-mon[33394]: pgmap v602: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 21286/126918 objects degraded (16.771%) 2024-02-10T07:45:15.896 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.3.service' 2024-02-10T07:45:15.910 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:15.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:15.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:15.967 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:45:15.968 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-10T07:45:15.968 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-02-10T07:45:15.968 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T07:45:15.968 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -n osd.3 --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-02-10T07:45:16.012 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:16.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:16.047 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:16.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:16.148 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:16.213 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:16.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:16.314 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:16.350 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:16.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:16.453 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:16.488 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:16.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:16.589 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:16.622 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:16.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:16.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:16.646 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:45:16.668 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/osd.3/config 2024-02-10T07:45:16.702 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:45:16.723 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:16.755 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:16.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:16.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:16.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:16 smithi008 ceph-mon[29816]: pgmap v603: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 20485/123321 objects degraded (16.611%) 2024-02-10T07:45:16.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:16 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 20485/123321 objects degraded (16.611%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:16.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:16 smithi008 ceph-mon[40743]: pgmap v603: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 20485/123321 objects degraded (16.611%) 2024-02-10T07:45:16.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:16 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 20485/123321 objects degraded (16.611%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:16.859 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:16.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:16 smithi170 ceph-mon[33394]: pgmap v603: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 20485/123321 objects degraded (16.611%) 2024-02-10T07:45:16.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:16 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 20485/123321 objects degraded (16.611%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:16.889 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:16.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:16.991 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:17.025 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:17.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:17.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:17.126 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:17.159 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:17.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:17.260 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:17.293 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:17.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:17.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:17.394 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:17.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:17.432 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:17.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:17.533 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:17.569 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:17.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:17.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:17.671 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:17.714 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:17.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:17.817 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:17.855 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:17.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:17.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:17.957 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:17.989 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:17.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:17.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:18.042 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:17 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:45:18.042 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:17 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:45:18.042 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:17 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:45:18.042 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:17 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:45:18.091 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:18.122 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:18.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:18.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:18.224 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:18.251 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:18.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:18.352 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:18.383 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:18.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:17 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:45:18.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:17 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:45:18.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:18.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:18.485 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:18.515 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:18.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:18.602 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:45:18.617 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:18.648 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:18.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:18.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:18.750 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:18.780 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:18.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:18.882 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:18.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:18.917 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:18.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:18.918 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:45:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:45:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:45:19.019 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:19.047 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:19.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:19.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:19.149 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:19.185 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:19.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:19.287 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:19.315 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:19.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:19.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:19.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:18 smithi008 ceph-mon[29816]: osdmap e301: 8 total, 7 up, 8 in 2024-02-10T07:45:19.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:45:19.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:45:19.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:18 smithi008 ceph-mon[29816]: pgmap v605: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 18835/114156 objects degraded (16.499%) 2024-02-10T07:45:19.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:18 smithi008 ceph-mon[40743]: osdmap e301: 8 total, 7 up, 8 in 2024-02-10T07:45:19.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:45:19.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:45:19.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:18 smithi008 ceph-mon[40743]: pgmap v605: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 18835/114156 objects degraded (16.499%) 2024-02-10T07:45:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:18 smithi170 ceph-mon[33394]: osdmap e301: 8 total, 7 up, 8 in 2024-02-10T07:45:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:45:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:45:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:18 smithi170 ceph-mon[33394]: pgmap v605: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 18835/114156 objects degraded (16.499%) 2024-02-10T07:45:19.417 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:19.445 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:19.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:19.547 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:19.580 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:19.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:19.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:19.682 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:19.713 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:19.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:19.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:19.814 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:19.850 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:19.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:19.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:19.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:19.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:19.990 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:19.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:20.057 INFO:teuthology.orchestra.run.smithi008.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-10T07:45:20.090 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:20.126 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:20.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:20.228 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:20.265 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:20.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:20.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:20.367 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:20.400 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:20.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:20.502 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:20.541 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:20.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:20.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:20.643 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:20.673 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:20.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:20.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:20.775 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:20.808 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:20.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:20.909 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:20.947 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:20.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:21.049 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:21.077 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:21.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:21.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:21.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:21 smithi008 ceph-mon[29816]: pgmap v606: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 17828/107064 objects degraded (16.652%) 2024-02-10T07:45:21.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:21 smithi008 ceph-mon[40743]: pgmap v606: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 17828/107064 objects degraded (16.652%) 2024-02-10T07:45:21.168 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:45:21.180 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:21.200 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:45:21.217 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:21.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:21.233 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:45:21.265 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:45:21.299 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:45:21.320 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:21.328 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:45:21.353 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:21.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:21.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:21.363 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:45:21.363 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:45:21.363 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:45:21.363 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:45:21.363 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-10T07:45:21.363 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:45:21.363 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:45:21.363 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:45:21.363 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:45:21.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:21 smithi170 ceph-mon[33394]: pgmap v606: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s; 17828/107064 objects degraded (16.652%) 2024-02-10T07:45:21.454 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:21.485 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:21.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:21.586 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:21.621 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:21.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:21.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:21.723 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:21.758 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:21.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:21.859 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:21.894 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:21.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:21.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:21.996 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:22.028 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:22.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:22.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:22.130 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:22.164 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:22.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:22.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:22.173 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:45:22.174 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:45:22.266 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:22.306 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:22.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:22.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:22 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2434337431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:22.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:22 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 17828/107064 objects degraded (16.652%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:22.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:22 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2434337431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:22.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:22 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 17828/107064 objects degraded (16.652%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:22 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2434337431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:22 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 17828/107064 objects degraded (16.652%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:22.408 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:22.443 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:22.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:22.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:22.545 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:22.578 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:22.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:22.680 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:22.714 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:22.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:22.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:22.816 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:22.852 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:22.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:22.954 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:22.987 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:22.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:22.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:23.089 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:23.126 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:23.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:23.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:23.201 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:23 smithi008 ceph-mon[29816]: pgmap v607: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 594 B/s rd, 0 B/s wr, 956 op/s; 17828/107064 objects degraded (16.652%) 2024-02-10T07:45:23.202 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:23 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:45:23.202 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:23 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:45:23.202 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:23 smithi008 ceph-mon[40743]: pgmap v607: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 594 B/s rd, 0 B/s wr, 956 op/s; 17828/107064 objects degraded (16.652%) 2024-02-10T07:45:23.202 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:23 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:45:23.202 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:23 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:45:23.214 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:45:23.227 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:23.273 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:23.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:23.301 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:45:23.301 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:45:23.373 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:23 smithi170 ceph-mon[33394]: pgmap v607: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 594 B/s rd, 0 B/s wr, 956 op/s; 17828/107064 objects degraded (16.652%) 2024-02-10T07:45:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:23 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:45:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:23 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:45:23.410 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:23.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:23.512 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:23.547 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:23.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:23.648 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:23.682 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:23.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:23.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:23.784 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:23.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:23.823 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:23.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:23.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:45:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:45:23.924 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:23.959 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:23.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:23.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:24.061 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:24.099 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:24.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:24.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:24.175 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:45:24.200 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:24.248 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:24.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:24.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:24.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:24 smithi008 ceph-mon[29816]: osdmap e302: 8 total, 7 up, 8 in 2024-02-10T07:45:24.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:24 smithi008 ceph-mon[40743]: osdmap e302: 8 total, 7 up, 8 in 2024-02-10T07:45:24.350 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:24.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:24 smithi170 ceph-mon[33394]: osdmap e302: 8 total, 7 up, 8 in 2024-02-10T07:45:24.406 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:24.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:24.508 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:24.552 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:24.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:24.654 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:24.687 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:24.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:24.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:24.789 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:24.828 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:24.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:24.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:24.968 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:24.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:25.070 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:25.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:25.107 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:25.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:25.208 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:25.246 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:25.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:25.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:25 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:25.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:25 smithi008 ceph-mon[40743]: pgmap v609: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 15808/93486 objects degraded (16.909%) 2024-02-10T07:45:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:25 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:25 smithi008 ceph-mon[29816]: pgmap v609: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 15808/93486 objects degraded (16.909%) 2024-02-10T07:45:25.348 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:25.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:25.382 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:25.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:25 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:25 smithi170 ceph-mon[33394]: pgmap v609: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 15808/93486 objects degraded (16.909%) 2024-02-10T07:45:25.483 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:25.521 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:25.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:25.606 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-10T07:45:25.622 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:25.684 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:25.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:25.788 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:25.827 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:25.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:25.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:25.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:25.962 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:25.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:26.063 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:26.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:26.139 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:26.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:26.180 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:45:26.223 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:45:26.240 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:26.273 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:26.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:26.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:26.327 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:26 smithi008 ceph-mon[29816]: 9.5 deep-scrub ok 2024-02-10T07:45:26.327 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:26 smithi008 ceph-mon[40743]: 9.5 deep-scrub ok 2024-02-10T07:45:26.375 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:26.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:26 smithi170 ceph-mon[33394]: 9.5 deep-scrub ok 2024-02-10T07:45:26.407 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:26.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:26.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:26.509 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:26.546 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:26.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:26.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:26.648 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:26.725 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:26.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:26.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:26.827 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:26.859 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:26.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:26.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:26.960 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:26.997 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:26.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:27.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:27.112 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:45:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:27.137 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:27.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:27.238 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:27.292 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:27.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:27.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:27.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:27 smithi008 ceph-mon[29816]: pgmap v610: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 1.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 15105/90360 objects degraded (16.716%) 2024-02-10T07:45:27.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:27 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3336597728' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:45:27.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:27 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:45:27.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:27 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 15105/90360 objects degraded (16.716%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:27 smithi008 ceph-mon[40743]: pgmap v610: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 1.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 15105/90360 objects degraded (16.716%) 2024-02-10T07:45:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:27 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3336597728' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:45:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:27 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:45:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:27 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 15105/90360 objects degraded (16.716%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:27 smithi170 ceph-mon[33394]: pgmap v610: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 1.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 15105/90360 objects degraded (16.716%) 2024-02-10T07:45:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:27 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3336597728' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:45:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:27 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:45:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:27 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 15105/90360 objects degraded (16.716%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:27.394 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:27.424 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:27.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:27.526 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:27.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:27.561 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:27.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:27.662 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:27.700 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:27.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:27.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:27.802 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:27.843 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:27.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:27.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:27.945 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:27.977 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:27.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:28.078 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:28.114 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:28.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:28.216 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:28.250 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:28.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:28.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:28.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:28 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:45:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:28 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:45:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:28 smithi008 ceph-mon[29816]: osdmap e303: 8 total, 7 up, 8 in 2024-02-10T07:45:28.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:28 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:45:28.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:28 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:45:28.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:28 smithi008 ceph-mon[40743]: osdmap e303: 8 total, 7 up, 8 in 2024-02-10T07:45:28.355 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:28 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:45:28.406 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:28 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:45:28.407 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:28 smithi170 ceph-mon[33394]: osdmap e303: 8 total, 7 up, 8 in 2024-02-10T07:45:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:28.410 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:28.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:28.511 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:28.546 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:28.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:28.647 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:28.680 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:28.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:28.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:28.782 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:28.819 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:28.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:28.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:28.920 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:28.953 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:28.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:28.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:29.030 INFO:teuthology.orchestra.run.smithi008.stdout:9.4 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:9.2 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:9.1 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:9.f 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:9.c 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:6.8 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:6.6 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:6.c 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:6.4 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:6.f 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:6.2 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:6.1f 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:6.16 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:6.7 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:6.11 2024-02-10T07:45:29.031 INFO:teuthology.orchestra.run.smithi008.stdout:5.a 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:5.1c 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:4.10 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:6.19 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:4.0 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:6.13 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:3.5 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:3.8 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:6.b 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:3.2 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:4.17 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:3.1e 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:6.5 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:4.16 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:4.1e 2024-02-10T07:45:29.032 INFO:teuthology.orchestra.run.smithi008.stdout:9.8 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:4.4 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:3.b 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:5.14 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:2.1 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:3.15 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:5.1 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:3.18 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:3.17 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:4.c 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:4.11 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:3.12 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:4.9 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:5.16 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:5.13 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:3.1 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:4.19 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:6.1b 2024-02-10T07:45:29.033 INFO:teuthology.orchestra.run.smithi008.stdout:3.7 2024-02-10T07:45:29.034 INFO:teuthology.orchestra.run.smithi008.stdout:3.c 2024-02-10T07:45:29.034 INFO:teuthology.orchestra.run.smithi008.stdout:3.19 2024-02-10T07:45:29.034 INFO:teuthology.orchestra.run.smithi008.stdout:4.1a 2024-02-10T07:45:29.034 INFO:teuthology.orchestra.run.smithi008.stdout:4.15 2024-02-10T07:45:29.034 INFO:teuthology.orchestra.run.smithi008.stdout:4.1c 2024-02-10T07:45:29.034 INFO:teuthology.orchestra.run.smithi008.stdout:9.3 2024-02-10T07:45:29.034 INFO:teuthology.orchestra.run.smithi008.stdout:4.1d 2024-02-10T07:45:29.034 INFO:teuthology.orchestra.run.smithi008.stdout:4.f 2024-02-10T07:45:29.034 INFO:teuthology.orchestra.run.smithi008.stdout:5.0 2024-02-10T07:45:29.034 INFO:teuthology.orchestra.run.smithi008.stdout:9.7 2024-02-10T07:45:29.034 INFO:teuthology.orchestra.run.smithi008.stdout:4.1b 2024-02-10T07:45:29.034 INFO:teuthology.orchestra.run.smithi008.stdout:5.17 2024-02-10T07:45:29.034 INFO:teuthology.orchestra.run.smithi008.stdout:5.12 2024-02-10T07:45:29.055 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:29.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:45:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:45:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:45:29.092 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:29.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:29.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:29.194 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:29.225 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:29.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:29.327 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:29.359 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:29.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:29.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:29 smithi170 ceph-mon[33394]: pgmap v612: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 13251/80250 objects degraded (16.512%) 2024-02-10T07:45:29.460 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:29.496 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:29.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:29.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:29.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:29 smithi008 ceph-mon[29816]: pgmap v612: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 13251/80250 objects degraded (16.512%) 2024-02-10T07:45:29.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:29 smithi008 ceph-mon[40743]: pgmap v612: 148 pgs: 1 active+clean+scrubbing+deep, 28 active+undersized+degraded, 34 active+undersized, 85 active+clean; 1.6 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 13251/80250 objects degraded (16.512%) 2024-02-10T07:45:29.598 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:29.635 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:29.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:29.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:29.737 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:29.768 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:29.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:29.870 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:29.900 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:29.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:29.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:30.003 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:30.042 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:30.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:30.143 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:30.178 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:30.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:30.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:30.279 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:30.313 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:30.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:30.415 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:30.452 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:30.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:30.554 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:30.592 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:30.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:30.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:30.694 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:30.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:30.733 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:30.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:30.834 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:30.871 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:30.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:30.973 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:31.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:31.012 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:31.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:31.113 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:31.147 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:31.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:31.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:31.248 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:31.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:31.287 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:31.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:31.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:31 smithi008 ceph-mon[29816]: pgmap v613: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 1.5 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 12043/72093 objects degraded (16.705%) 2024-02-10T07:45:31.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:31 smithi008 ceph-mon[40743]: pgmap v613: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 1.5 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 12043/72093 objects degraded (16.705%) 2024-02-10T07:45:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:31 smithi170 ceph-mon[33394]: pgmap v613: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 1.5 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 12043/72093 objects degraded (16.705%) 2024-02-10T07:45:31.388 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:31.427 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:31.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:31.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:31.529 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:31.560 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:31.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:31.662 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:31.693 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:31.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:31.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:31.795 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:31.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:31.836 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:31.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:31.937 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:31.972 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:31.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:32.074 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:32.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:32.117 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:32.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:32.218 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:32.249 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:32.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:32.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:32.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:32 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 12043/72093 objects degraded (16.705%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:32 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 12043/72093 objects degraded (16.705%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:32.351 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:32 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 12043/72093 objects degraded (16.705%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:32.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:32.387 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:32.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:32.406 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:45:32.489 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:32.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:32.523 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:32.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:32.624 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:32.658 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:32.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:32.714 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -n osd.3 --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 6.7 --file /var/log/ceph/exp.6.7.3 2024-02-10T07:45:32.761 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:32.797 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:32.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:32.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:32.899 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:32.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:32.942 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:32.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:32.984 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/osd.3/config 2024-02-10T07:45:33.043 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:33.076 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:33.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:33.178 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:33.211 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:33.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:33.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:33.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:33 smithi008 ceph-mon[29816]: pgmap v614: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 1.5 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 572 B/s rd, 0 B/s wr, 960 op/s; 12043/72093 objects degraded (16.705%) 2024-02-10T07:45:33.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:45:33.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:33 smithi008 ceph-mon[40743]: pgmap v614: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 1.5 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 572 B/s rd, 0 B/s wr, 960 op/s; 12043/72093 objects degraded (16.705%) 2024-02-10T07:45:33.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:45:33.313 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:33.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:33.357 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:33.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:33.363 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:45:33.363 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:45:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:33 smithi170 ceph-mon[33394]: pgmap v614: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 1.5 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 572 B/s rd, 0 B/s wr, 960 op/s; 12043/72093 objects degraded (16.705%) 2024-02-10T07:45:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:45:33.458 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:33.491 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:33.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:33.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:33.593 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:33.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:33.628 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:33.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:33.637 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:45:33.676 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:45:33.711 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:45:33.729 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:33.747 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:45:33.767 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:33.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:33.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:33.785 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:45:33.803 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:45:33.824 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:45:33.862 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:45:33.862 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:45:33.863 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:45:33.863 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:45:33.863 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-10T07:45:33.863 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:45:33.863 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:45:33.863 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:45:33.863 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:45:33.868 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:33.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:45:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:45:33.909 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:33.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:33.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:34.011 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:34.048 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:34.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:34.150 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:34.181 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:34.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:34.283 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:34.317 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:34.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:34.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:34 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1237467715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:34.419 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:34.454 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:34.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:34.460 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:34 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1237467715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:34.460 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:34 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1237467715' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:34.556 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:34.588 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:34.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:34.690 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:34.721 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:34.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:34.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:34.802 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:45:34.803 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:45:34.822 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:34.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:34.862 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:34.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:34.963 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:34.999 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:34.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:35.100 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:35.128 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:35.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:35.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:35.230 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:35.267 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:35.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:35.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:35.297 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:35 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:35.297 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:35 smithi008 ceph-mon[29816]: pgmap v615: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 1.2 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s; 9921/60609 objects degraded (16.369%) 2024-02-10T07:45:35.298 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:35 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:35.298 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:35 smithi008 ceph-mon[40743]: pgmap v615: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 1.2 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s; 9921/60609 objects degraded (16.369%) 2024-02-10T07:45:35.369 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:35 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:35 smithi170 ceph-mon[33394]: pgmap v615: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 1.2 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s; 9921/60609 objects degraded (16.369%) 2024-02-10T07:45:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:35.406 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:35.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:35.507 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:35.536 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:35.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:35.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:35.638 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:35.670 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:35.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:35.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:35.771 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:35.804 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:35.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:35.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:35.906 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:35.943 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:35.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:35.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:36.045 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:36.084 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:36.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:36.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:36.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:36.220 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:36.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:36.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:36.322 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:36.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:36.357 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:36.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:36.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:36.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:36.494 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:36.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:36.595 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:36.628 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-10T07:45:36.640 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:36.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:36.743 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:36.782 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:36.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:36.804 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:45:36.883 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:36.930 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:36.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:37.016 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:45:37.030 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:37.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:37.082 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:37.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:37.183 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:37.224 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:37.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:37.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:37.326 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:37.365 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:37.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:37.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:37.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:37 smithi170 ceph-mon[33394]: pgmap v616: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 1.0 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s; 8610/51531 objects degraded (16.708%) 2024-02-10T07:45:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:37 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8610/51531 objects degraded (16.708%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:37.467 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:37.496 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:37.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:37.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:37 smithi008 ceph-mon[29816]: pgmap v616: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 1.0 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s; 8610/51531 objects degraded (16.708%) 2024-02-10T07:45:37.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:37 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8610/51531 objects degraded (16.708%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:37 smithi008 ceph-mon[40743]: pgmap v616: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 1.0 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s; 8610/51531 objects degraded (16.708%) 2024-02-10T07:45:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:37 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8610/51531 objects degraded (16.708%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:37.598 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:37.628 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:37.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:37.730 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:37.761 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:37.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:37.862 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:37.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:37.898 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:37.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:37.999 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:38.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:38.035 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:38.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:38.136 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:38.164 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:38.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:38.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:38.266 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:38.304 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:38.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:38.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:38.406 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:38.433 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:38.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:38.535 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:38.572 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:38.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:38.674 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:38.704 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:38.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:38.806 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:38.838 INFO:teuthology.orchestra.run.smithi008.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-10T07:45:38.839 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:38.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:38.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:38.941 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:38.974 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:38.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:38.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:39.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:39.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:45:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:45:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:45:39.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:39.128 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:39.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:39.229 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:39.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:39.265 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:39.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:39.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:39 smithi170 ceph-mon[33394]: pgmap v617: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 823 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.54k op/s; 6504/40059 objects degraded (16.236%) 2024-02-10T07:45:39.367 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:39.402 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:39 smithi008 ceph-mon[29816]: pgmap v617: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 823 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.54k op/s; 6504/40059 objects degraded (16.236%) 2024-02-10T07:45:39.402 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:39 smithi008 ceph-mon[40743]: pgmap v617: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 823 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.54k op/s; 6504/40059 objects degraded (16.236%) 2024-02-10T07:45:39.414 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:39.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:39.515 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:39.556 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:39.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:39.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:39.658 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:39.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:39.691 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:39.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:39.792 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:39.830 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:39.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:39.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:39.911 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:45:39.912 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:45:39.912 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.912 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.25283.0:248002 9.4 e303/296 9:240cae2c:::benchmark_data_smithi008_134920_object119825:head v 303'15549, mlcod=303'15544)", 2024-02-10T07:45:39.912 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:45:39.839433+0000", 2024-02-10T07:45:39.912 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.0059884250000000003, 2024-02-10T07:45:39.912 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0060182389999999999, 2024-02-10T07:45:39.912 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.839433+0000", 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.839433+0000", 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.839434+0000", 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8899999999999998e-07 2024-02-10T07:45:39.913 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.839441+0000", 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8549999999999996e-06 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.839459+0000", 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8876e-05 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.839468+0000", 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3270000000000001e-06 2024-02-10T07:45:39.914 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.840202+0000", 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00073420400000000002 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.840298+0000", 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6149000000000004e-05 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:248008 9.c 9:310ec88e:::benchmark_data_smithi008_134920_object119831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-02-10T07:45:39.915 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:45:39.844976+0000", 2024-02-10T07:45:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.00044558999999999999, 2024-02-10T07:45:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00055271699999999999, 2024-02-10T07:45:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:45:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:45:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:45:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 248008 2024-02-10T07:45:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.844976+0000", 2024-02-10T07:45:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.844976+0000", 2024-02-10T07:45:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.844977+0000", 2024-02-10T07:45:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3489999999999999e-06 2024-02-10T07:45:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:39.917 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.844983+0000", 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7540000000000003e-06 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.845003+0000", 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0027e-05 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.845017+0000", 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4623999999999999e-05 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:39.918 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:39.845372+0000", 2024-02-10T07:45:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00035412300000000002 2024-02-10T07:45:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2024-02-10T07:45:39.919 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 2 2024-02-10T07:45:39.919 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:45:39.932 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:39.964 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:39.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:39.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:40.065 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:40.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:40.101 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:40.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:40.202 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:40.240 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:40.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:40.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:40.342 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:40.382 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:40.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:40.484 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:40.514 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:40.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:40.616 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:40.661 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:40.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:40.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:40.690 INFO:teuthology.orchestra.run.smithi008.stderr:Exporting 6.7 info 6.7( v 94'2 (0'0,94'2] local-lis/les=156/157 n=1 ec=72/72 lis/c=156/156 les/c/f=157/157/0 sis=156) 2024-02-10T07:45:40.727 INFO:teuthology.orchestra.run.smithi008.stderr:Read #6:e4fda9dc:users.keys::A1GPE05QVJT728K71ZP4:head# 2024-02-10T07:45:40.728 INFO:teuthology.orchestra.run.smithi008.stderr:Export successful 2024-02-10T07:45:40.749 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:45:40.763 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:40.798 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:40.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:40.899 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:40.940 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:40.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:41.042 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:41.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:41.077 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:41.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:41.103 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-10T07:45:41.159 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:45:41.178 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:41.209 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:41.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:41.311 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:41.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:41 smithi008 ceph-mon[29816]: pgmap v618: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 648 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.35k op/s; 5298/31662 objects degraded (16.733%) 2024-02-10T07:45:41.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:41 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:45:41.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:41 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:45:41.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:41 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3995991911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:45:41.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:41 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:45:41.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:41 smithi008 ceph-mon[40743]: pgmap v618: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 648 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.35k op/s; 5298/31662 objects degraded (16.733%) 2024-02-10T07:45:41.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:41 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:45:41.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:41 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:45:41.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:41 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3995991911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:45:41.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:41 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:45:41.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:41.360 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:41.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:41.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:41 smithi170 ceph-mon[33394]: pgmap v618: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 648 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.35k op/s; 5298/31662 objects degraded (16.733%) 2024-02-10T07:45:41.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:41 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:45:41.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:41 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:45:41.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:41 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3995991911' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:45:41.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:41 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:45:41.461 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:41.489 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:41.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:41.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:41.582 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:45:41.591 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:41.618 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:41.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:41.720 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:41.749 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:41.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:41.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:41.851 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:41.882 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:41.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:41.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:41.984 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:42.013 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:42.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:42.114 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:42.153 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:42.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:42.241 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -n osd.3 --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 6.7 2024-02-10T07:45:42.254 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:42.319 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:42.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:42.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:42 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:45:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:42 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:45:42.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:42 smithi170 ceph-mon[33394]: osdmap e304: 8 total, 7 up, 8 in 2024-02-10T07:45:42.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:42 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 5298/31662 objects degraded (16.733%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:42.421 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:42.460 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:42.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:42.552 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:42 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:45:42.552 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:42 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:45:42.552 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:42 smithi008 ceph-mon[29816]: osdmap e304: 8 total, 7 up, 8 in 2024-02-10T07:45:42.552 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:42 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 5298/31662 objects degraded (16.733%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:42.553 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:42 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:45:42.553 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:42 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:45:42.553 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:42 smithi008 ceph-mon[40743]: osdmap e304: 8 total, 7 up, 8 in 2024-02-10T07:45:42.553 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:42 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 5298/31662 objects degraded (16.733%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:42.559 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/osd.3/config 2024-02-10T07:45:42.561 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:42.606 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:42.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:42.707 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:42.741 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:42.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:42.843 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:42.880 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:42.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:42.982 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:43.040 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:43.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:43.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:43.142 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:43.184 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:43.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:43.287 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:43.320 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:43.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:43.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:43.357 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:43 smithi170 ceph-mon[33394]: 5.1b scrub starts 2024-02-10T07:45:43.357 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:43 smithi170 ceph-mon[33394]: 5.1b scrub ok 2024-02-10T07:45:43.357 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:43 smithi170 ceph-mon[33394]: 6.e scrub starts 2024-02-10T07:45:43.357 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:43 smithi170 ceph-mon[33394]: 6.e scrub ok 2024-02-10T07:45:43.357 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:43 smithi170 ceph-mon[33394]: pgmap v620: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 648 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 5298/31662 objects degraded (16.733%) 2024-02-10T07:45:43.392 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:43 smithi008 ceph-mon[29816]: 5.1b scrub starts 2024-02-10T07:45:43.392 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:43 smithi008 ceph-mon[29816]: 5.1b scrub ok 2024-02-10T07:45:43.392 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:43 smithi008 ceph-mon[29816]: 6.e scrub starts 2024-02-10T07:45:43.392 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:43 smithi008 ceph-mon[29816]: 6.e scrub ok 2024-02-10T07:45:43.392 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:43 smithi008 ceph-mon[29816]: pgmap v620: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 648 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 5298/31662 objects degraded (16.733%) 2024-02-10T07:45:43.392 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:43 smithi008 ceph-mon[40743]: 5.1b scrub starts 2024-02-10T07:45:43.392 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:43 smithi008 ceph-mon[40743]: 5.1b scrub ok 2024-02-10T07:45:43.392 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:43 smithi008 ceph-mon[40743]: 6.e scrub starts 2024-02-10T07:45:43.392 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:43 smithi008 ceph-mon[40743]: 6.e scrub ok 2024-02-10T07:45:43.393 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:43 smithi008 ceph-mon[40743]: pgmap v620: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 648 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 5298/31662 objects degraded (16.733%) 2024-02-10T07:45:43.422 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:43.458 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:43.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:43.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:43.560 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:43.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:43.591 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:43.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:43.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:45:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:45:43.692 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:43.727 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:43.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:43.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:43.829 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:43.871 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:43.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:43.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:43.973 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:44.003 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:44.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:44.009 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:45:44.009 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2024-02-10T07:45:44.009 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2024-02-10T07:45:44.009 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2024-02-10T07:45:44.009 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:45:44.105 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:44.136 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:44.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:44.237 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:44.264 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:44.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:44.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:44.366 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:44.394 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:44.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:44.415 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:44 smithi008 ceph-mon[29816]: 3.e scrub starts 2024-02-10T07:45:44.415 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:44 smithi008 ceph-mon[29816]: 3.e scrub ok 2024-02-10T07:45:44.415 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:44 smithi008 ceph-mon[29816]: 6.1a scrub starts 2024-02-10T07:45:44.415 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:44 smithi008 ceph-mon[29816]: 6.1a scrub ok 2024-02-10T07:45:44.415 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:44 smithi008 ceph-mon[40743]: 3.e scrub starts 2024-02-10T07:45:44.415 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:44 smithi008 ceph-mon[40743]: 3.e scrub ok 2024-02-10T07:45:44.415 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:44 smithi008 ceph-mon[40743]: 6.1a scrub starts 2024-02-10T07:45:44.415 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:44 smithi008 ceph-mon[40743]: 6.1a scrub ok 2024-02-10T07:45:44.496 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:44.534 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:44.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:44.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:44 smithi170 ceph-mon[33394]: 3.e scrub starts 2024-02-10T07:45:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:44 smithi170 ceph-mon[33394]: 3.e scrub ok 2024-02-10T07:45:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:44 smithi170 ceph-mon[33394]: 6.1a scrub starts 2024-02-10T07:45:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:44 smithi170 ceph-mon[33394]: 6.1a scrub ok 2024-02-10T07:45:44.635 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:44.665 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:44.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:44.712 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-10T07:45:44.768 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:44.801 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:44.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:44.889 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:45:44.902 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:44.927 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:45:44.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:44.936 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:44.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:44.957 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:45:45.002 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:45:45.037 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:45.039 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:45:45.076 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:45:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:45.096 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:45.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:45.113 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:45:45.114 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:45:45.114 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:45:45.114 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:45:45.114 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-10T07:45:45.114 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:45:45.114 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:45:45.114 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:45:45.114 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:45:45.155 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:45:45.197 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:45.243 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:45.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:45.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[29816]: 3.4 scrub starts 2024-02-10T07:45:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[29816]: 3.4 scrub ok 2024-02-10T07:45:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[29816]: 9.e scrub starts 2024-02-10T07:45:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[29816]: 9.e scrub ok 2024-02-10T07:45:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[29816]: pgmap v621: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 429 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s; 3372/21165 objects degraded (15.932%) 2024-02-10T07:45:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[29816]: 5.8 scrub starts 2024-02-10T07:45:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[29816]: 5.8 scrub ok 2024-02-10T07:45:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/685540157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:45.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[40743]: 3.4 scrub starts 2024-02-10T07:45:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[40743]: 3.4 scrub ok 2024-02-10T07:45:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[40743]: 9.e scrub starts 2024-02-10T07:45:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[40743]: 9.e scrub ok 2024-02-10T07:45:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[40743]: pgmap v621: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 429 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s; 3372/21165 objects degraded (15.932%) 2024-02-10T07:45:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[40743]: 5.8 scrub starts 2024-02-10T07:45:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[40743]: 5.8 scrub ok 2024-02-10T07:45:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:45 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/685540157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:45.344 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:45.383 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:45.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:45.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:45.433 INFO:teuthology.orchestra.run.smithi008.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-10T07:45:45.451 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:45 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:45.451 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:45 smithi170 ceph-mon[33394]: 3.4 scrub starts 2024-02-10T07:45:45.451 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:45 smithi170 ceph-mon[33394]: 3.4 scrub ok 2024-02-10T07:45:45.452 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:45 smithi170 ceph-mon[33394]: 9.e scrub starts 2024-02-10T07:45:45.452 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:45 smithi170 ceph-mon[33394]: 9.e scrub ok 2024-02-10T07:45:45.452 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:45 smithi170 ceph-mon[33394]: pgmap v621: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 429 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s; 3372/21165 objects degraded (15.932%) 2024-02-10T07:45:45.452 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:45 smithi170 ceph-mon[33394]: 5.8 scrub starts 2024-02-10T07:45:45.452 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:45 smithi170 ceph-mon[33394]: 5.8 scrub ok 2024-02-10T07:45:45.452 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:45 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/685540157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:45.485 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:45.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:45.525 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:45.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:45.626 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:45.655 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:45.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:45.757 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:45.785 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:45.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:45.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:45.859 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:45:45.887 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:45.924 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:45.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:46.026 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:46.059 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:46.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:46.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:46.161 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:46.186 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:45:46.187 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:45:46.193 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:46.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:46.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:46.295 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:46.333 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:46.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:46.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:46.435 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:46.468 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:46.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:46.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:46 smithi008 ceph-mon[29816]: 2.2 scrub starts 2024-02-10T07:45:46.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:46 smithi008 ceph-mon[29816]: 2.2 scrub ok 2024-02-10T07:45:46.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:46 smithi008 ceph-mon[29816]: 3.1a scrub starts 2024-02-10T07:45:46.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:46 smithi008 ceph-mon[29816]: 3.1a scrub ok 2024-02-10T07:45:46.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:46 smithi008 ceph-mon[29816]: 9.9 scrub starts 2024-02-10T07:45:46.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:46 smithi008 ceph-mon[29816]: 9.9 scrub ok 2024-02-10T07:45:46.470 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:46 smithi008 ceph-mon[40743]: 2.2 scrub starts 2024-02-10T07:45:46.470 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:46 smithi008 ceph-mon[40743]: 2.2 scrub ok 2024-02-10T07:45:46.470 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:46 smithi008 ceph-mon[40743]: 3.1a scrub starts 2024-02-10T07:45:46.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:46 smithi008 ceph-mon[40743]: 3.1a scrub ok 2024-02-10T07:45:46.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:46 smithi008 ceph-mon[40743]: 9.9 scrub starts 2024-02-10T07:45:46.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:46 smithi008 ceph-mon[40743]: 9.9 scrub ok 2024-02-10T07:45:46.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:46 smithi170 ceph-mon[33394]: 2.2 scrub starts 2024-02-10T07:45:46.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:46 smithi170 ceph-mon[33394]: 2.2 scrub ok 2024-02-10T07:45:46.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:46 smithi170 ceph-mon[33394]: 3.1a scrub starts 2024-02-10T07:45:46.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:46 smithi170 ceph-mon[33394]: 3.1a scrub ok 2024-02-10T07:45:46.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:46 smithi170 ceph-mon[33394]: 9.9 scrub starts 2024-02-10T07:45:46.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:46 smithi170 ceph-mon[33394]: 9.9 scrub ok 2024-02-10T07:45:46.554 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Removed 128174 objects 2024-02-10T07:45:46.554 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Clean up completed and total clean up time :115.754 2024-02-10T07:45:46.570 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:46.593 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:46.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:46.695 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:46.718 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:46.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:46.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:46.739 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:45:46.820 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:46.843 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:46.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:46.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:46.934 INFO:teuthology.orchestra.run.smithi008.stdout: marking collection for removal 2024-02-10T07:45:46.935 INFO:teuthology.orchestra.run.smithi008.stdout:setting '_remove' omap key 2024-02-10T07:45:46.935 INFO:teuthology.orchestra.run.smithi008.stdout:finish_remove_pgs 6.7_head removing 6.7 2024-02-10T07:45:46.945 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:46.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:46.973 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:46.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:46.973 INFO:teuthology.orchestra.run.smithi008.stdout:Remove successful 2024-02-10T07:45:47.074 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:47.099 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:47.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:47.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:47.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:47.227 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:47.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:47.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:47.329 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:47.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:47 smithi008 ceph-mon[29816]: 6.d scrub starts 2024-02-10T07:45:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:47 smithi008 ceph-mon[29816]: 6.d scrub ok 2024-02-10T07:45:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:47 smithi008 ceph-mon[29816]: 6.3 scrub starts 2024-02-10T07:45:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:47 smithi008 ceph-mon[29816]: 6.3 scrub ok 2024-02-10T07:45:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:47 smithi008 ceph-mon[29816]: pgmap v622: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 262 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 2230/13146 objects degraded (16.963%) 2024-02-10T07:45:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:47 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 2230/13146 objects degraded (16.963%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:47.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:47 smithi008 ceph-mon[40743]: 6.d scrub starts 2024-02-10T07:45:47.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:47 smithi008 ceph-mon[40743]: 6.d scrub ok 2024-02-10T07:45:47.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:47 smithi008 ceph-mon[40743]: 6.3 scrub starts 2024-02-10T07:45:47.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:47 smithi008 ceph-mon[40743]: 6.3 scrub ok 2024-02-10T07:45:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:47 smithi008 ceph-mon[40743]: pgmap v622: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 262 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 2230/13146 objects degraded (16.963%) 2024-02-10T07:45:47.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:47 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 2230/13146 objects degraded (16.963%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:47.357 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:47.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:47.459 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:47.486 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:47.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:47.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:47.587 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:47.611 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:47.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:47.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:47 smithi170 ceph-mon[33394]: 6.d scrub starts 2024-02-10T07:45:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:47 smithi170 ceph-mon[33394]: 6.d scrub ok 2024-02-10T07:45:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:47 smithi170 ceph-mon[33394]: 6.3 scrub starts 2024-02-10T07:45:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:47 smithi170 ceph-mon[33394]: 6.3 scrub ok 2024-02-10T07:45:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:47 smithi170 ceph-mon[33394]: pgmap v622: 148 pgs: 28 active+undersized+degraded, 34 active+undersized, 86 active+clean; 262 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 2230/13146 objects degraded (16.963%) 2024-02-10T07:45:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:47 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 2230/13146 objects degraded (16.963%), 28 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:47.713 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:47.737 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:47.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:47.839 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:47.866 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:47.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:47.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:47.968 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:47.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:47.997 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:47.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:48.098 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:48.122 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:48.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:48.154 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-02-10T07:45:48.154 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-02-10T07:45:48.188 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:45:48.224 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:48.266 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:48.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:48.368 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:48.393 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:48.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:48.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:48.422 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-mon[29816]: 6.a scrub starts 2024-02-10T07:45:48.422 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-mon[29816]: 6.a scrub ok 2024-02-10T07:45:48.422 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-mon[29816]: 5.10 scrub starts 2024-02-10T07:45:48.422 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-mon[29816]: 5.10 scrub ok 2024-02-10T07:45:48.422 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-mon[29816]: 5.5 scrub starts 2024-02-10T07:45:48.422 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-mon[29816]: 5.5 scrub ok 2024-02-10T07:45:48.422 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:45:48.422 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-mon[40743]: 6.a scrub starts 2024-02-10T07:45:48.422 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-mon[40743]: 6.a scrub ok 2024-02-10T07:45:48.422 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-mon[40743]: 5.10 scrub starts 2024-02-10T07:45:48.422 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-mon[40743]: 5.10 scrub ok 2024-02-10T07:45:48.422 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-mon[40743]: 5.5 scrub starts 2024-02-10T07:45:48.423 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-mon[40743]: 5.5 scrub ok 2024-02-10T07:45:48.423 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:45:48.430 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:48 smithi170 ceph-mon[33394]: 6.a scrub starts 2024-02-10T07:45:48.430 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:48 smithi170 ceph-mon[33394]: 6.a scrub ok 2024-02-10T07:45:48.430 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:48 smithi170 ceph-mon[33394]: 5.10 scrub starts 2024-02-10T07:45:48.430 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:48 smithi170 ceph-mon[33394]: 5.10 scrub ok 2024-02-10T07:45:48.430 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:48 smithi170 ceph-mon[33394]: 5.5 scrub starts 2024-02-10T07:45:48.430 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:48 smithi170 ceph-mon[33394]: 5.5 scrub ok 2024-02-10T07:45:48.430 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:48 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:45:48.495 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:48.519 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:48.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:48.620 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:48.651 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:48.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:48.752 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:48.776 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:48.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:48.833 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:45:48.833 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:45:48.878 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:48.903 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:48.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:49.004 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:49.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:49.029 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:49.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:49.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:45:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:45:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:45:49.130 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:49.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:49.155 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:49.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:49.180 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:45:49.255 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:49.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:49.281 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:49.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:49.382 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:49.406 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:49.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:49.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:49.433 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:49 smithi170 ceph-mon[33394]: 5.f scrub starts 2024-02-10T07:45:49.433 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:49 smithi170 ceph-mon[33394]: 5.f scrub ok 2024-02-10T07:45:49.433 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:49 smithi170 ceph-mon[33394]: pgmap v623: 148 pgs: 26 active+undersized+degraded, 36 active+undersized, 86 active+clean; 86 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 668/4680 objects degraded (14.274%) 2024-02-10T07:45:49.433 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:49 smithi170 ceph-mon[33394]: 6.17 scrub starts 2024-02-10T07:45:49.433 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:49 smithi170 ceph-mon[33394]: 6.17 scrub ok 2024-02-10T07:45:49.433 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:49 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1516526029' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:45:49.433 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:49 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:45:49.481 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-10T07:45:49.507 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:49.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:49.548 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:49.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:49.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:49 smithi008 ceph-mon[29816]: 5.f scrub starts 2024-02-10T07:45:49.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:49 smithi008 ceph-mon[29816]: 5.f scrub ok 2024-02-10T07:45:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:49 smithi008 ceph-mon[29816]: pgmap v623: 148 pgs: 26 active+undersized+degraded, 36 active+undersized, 86 active+clean; 86 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 668/4680 objects degraded (14.274%) 2024-02-10T07:45:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:49 smithi008 ceph-mon[29816]: 6.17 scrub starts 2024-02-10T07:45:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:49 smithi008 ceph-mon[29816]: 6.17 scrub ok 2024-02-10T07:45:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:49 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1516526029' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:45:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:49 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:45:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:49 smithi008 ceph-mon[40743]: 5.f scrub starts 2024-02-10T07:45:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:49 smithi008 ceph-mon[40743]: 5.f scrub ok 2024-02-10T07:45:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:49 smithi008 ceph-mon[40743]: pgmap v623: 148 pgs: 26 active+undersized+degraded, 36 active+undersized, 86 active+clean; 86 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s; 668/4680 objects degraded (14.274%) 2024-02-10T07:45:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:49 smithi008 ceph-mon[40743]: 6.17 scrub starts 2024-02-10T07:45:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:49 smithi008 ceph-mon[40743]: 6.17 scrub ok 2024-02-10T07:45:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:49 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1516526029' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:45:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:49 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:45:49.649 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:49.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:49.674 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:49.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:49.721 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:45:49.775 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:49.800 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:49.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:49.901 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:49.926 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:49.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:50.027 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:50.029 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -n osd.3 --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.6.7.3 2024-02-10T07:45:50.051 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:50.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:50.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:50.153 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:50.178 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:50.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:50.279 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:50.303 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:50.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:50.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:50.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:50 smithi008 ceph-mon[29816]: 6.12 scrub starts 2024-02-10T07:45:50.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:50 smithi008 ceph-mon[29816]: 6.12 scrub ok 2024-02-10T07:45:50.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:50 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:45:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:50 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:45:50.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:50 smithi008 ceph-mon[29816]: osdmap e305: 8 total, 7 up, 8 in 2024-02-10T07:45:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:50 smithi008 ceph-mon[40743]: 6.12 scrub starts 2024-02-10T07:45:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:50 smithi008 ceph-mon[40743]: 6.12 scrub ok 2024-02-10T07:45:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:50 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:45:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:50 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:45:50.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:50 smithi008 ceph-mon[40743]: osdmap e305: 8 total, 7 up, 8 in 2024-02-10T07:45:50.406 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:50.431 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:50.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:50.532 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:50.557 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:50.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:50 smithi170 ceph-mon[33394]: 6.12 scrub starts 2024-02-10T07:45:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:50 smithi170 ceph-mon[33394]: 6.12 scrub ok 2024-02-10T07:45:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:50 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:45:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:50 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:45:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:50 smithi170 ceph-mon[33394]: osdmap e305: 8 total, 7 up, 8 in 2024-02-10T07:45:50.658 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:50.683 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:50.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:50.786 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:50.812 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:50.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:50.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:50.871 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/osd.3/config 2024-02-10T07:45:50.888 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:45:50.913 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:50.939 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:50.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:51.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:51.067 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:51.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:51.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:51.169 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:51.194 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:51.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:51.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:51.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[29816]: 4.8 scrub starts 2024-02-10T07:45:51.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[29816]: 4.8 scrub ok 2024-02-10T07:45:51.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[29816]: 9.d scrub starts 2024-02-10T07:45:51.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[29816]: 9.d scrub ok 2024-02-10T07:45:51.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[29816]: pgmap v625: 148 pgs: 24 active+undersized+degraded, 38 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:51.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[29816]: 5.3 scrub starts 2024-02-10T07:45:51.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[29816]: 5.3 scrub ok 2024-02-10T07:45:51.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[29816]: 5.c scrub starts 2024-02-10T07:45:51.252 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[29816]: 5.c scrub ok 2024-02-10T07:45:51.252 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[40743]: 4.8 scrub starts 2024-02-10T07:45:51.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[40743]: 4.8 scrub ok 2024-02-10T07:45:51.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[40743]: 9.d scrub starts 2024-02-10T07:45:51.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[40743]: 9.d scrub ok 2024-02-10T07:45:51.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[40743]: pgmap v625: 148 pgs: 24 active+undersized+degraded, 38 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:51.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[40743]: 5.3 scrub starts 2024-02-10T07:45:51.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[40743]: 5.3 scrub ok 2024-02-10T07:45:51.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[40743]: 5.c scrub starts 2024-02-10T07:45:51.253 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:51 smithi008 ceph-mon[40743]: 5.c scrub ok 2024-02-10T07:45:51.295 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:51.322 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:51.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:51.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:51.424 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:51.427 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:51 smithi170 ceph-mon[33394]: 4.8 scrub starts 2024-02-10T07:45:51.427 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:51 smithi170 ceph-mon[33394]: 4.8 scrub ok 2024-02-10T07:45:51.427 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:51 smithi170 ceph-mon[33394]: 9.d scrub starts 2024-02-10T07:45:51.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:51 smithi170 ceph-mon[33394]: 9.d scrub ok 2024-02-10T07:45:51.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:51 smithi170 ceph-mon[33394]: pgmap v625: 148 pgs: 24 active+undersized+degraded, 38 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:51.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:51 smithi170 ceph-mon[33394]: 5.3 scrub starts 2024-02-10T07:45:51.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:51 smithi170 ceph-mon[33394]: 5.3 scrub ok 2024-02-10T07:45:51.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:51 smithi170 ceph-mon[33394]: 5.c scrub starts 2024-02-10T07:45:51.428 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:51 smithi170 ceph-mon[33394]: 5.c scrub ok 2024-02-10T07:45:51.448 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:51.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:51.549 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:51.575 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:51.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:51.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:51.676 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:51.702 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:51.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:51.804 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:51.829 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:51.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:51.863 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:45:51.863 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:45:51.931 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:51.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:51.958 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:51.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:52.059 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:52.085 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:52.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:52.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:52.186 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:52.212 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:52.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:52.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:52 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 87/594 objects degraded (14.646%), 24 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:52.309 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:52 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 87/594 objects degraded (14.646%), 24 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:52.314 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:52.337 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:52.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:52.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:52.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:52 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 87/594 objects degraded (14.646%), 24 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:52.439 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:52.464 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:52.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:52.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:52.504 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-10T07:45:52.566 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:52.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:52.591 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:52.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:52.656 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:45:52.674 INFO:teuthology.orchestra.run.smithi008.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-10T07:45:52.686 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:45:52.692 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:52.715 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:52.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:52.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:52.744 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:45:52.817 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:52.840 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:52.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:52.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:52.943 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:52.966 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:52.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:52.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:53.068 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:53.092 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:53.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:53.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:53.194 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:53.220 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:53.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:53.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:53.322 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:53.349 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:53.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:53.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:53.439 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:53 smithi008 ceph-mon[29816]: pgmap v626: 148 pgs: 24 active+undersized+degraded, 38 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:53.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:53 smithi008 ceph-mon[40743]: pgmap v626: 148 pgs: 24 active+undersized+degraded, 38 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:53.450 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:53.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:53.476 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:53.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:53.504 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:53 smithi170 ceph-mon[33394]: pgmap v626: 148 pgs: 24 active+undersized+degraded, 38 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:53.577 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:53.601 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:53.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:53.703 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:53.726 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:53.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:53.828 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:53.852 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:53.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:53.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:53.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:45:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:45:53.953 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:53.977 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:53.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:53.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:54.078 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:54.104 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:54.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:54.205 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:54.229 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:54.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:54.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:54.331 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:54.355 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:54.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:54.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:54.358 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:54 smithi170 ceph-mon[33394]: 4.14 scrub starts 2024-02-10T07:45:54.358 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:54 smithi170 ceph-mon[33394]: 4.14 scrub ok 2024-02-10T07:45:54.358 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:54 smithi170 ceph-mon[33394]: 5.19 scrub starts 2024-02-10T07:45:54.358 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:54 smithi170 ceph-mon[33394]: 5.19 scrub ok 2024-02-10T07:45:54.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:54.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:54.482 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:54.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:54.559 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:54 smithi008 ceph-mon[29816]: 4.14 scrub starts 2024-02-10T07:45:54.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:54 smithi008 ceph-mon[29816]: 4.14 scrub ok 2024-02-10T07:45:54.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:54 smithi008 ceph-mon[29816]: 5.19 scrub starts 2024-02-10T07:45:54.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:54 smithi008 ceph-mon[29816]: 5.19 scrub ok 2024-02-10T07:45:54.560 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:54 smithi008 ceph-mon[40743]: 4.14 scrub starts 2024-02-10T07:45:54.560 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:54 smithi008 ceph-mon[40743]: 4.14 scrub ok 2024-02-10T07:45:54.560 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:54 smithi008 ceph-mon[40743]: 5.19 scrub starts 2024-02-10T07:45:54.560 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:54 smithi008 ceph-mon[40743]: 5.19 scrub ok 2024-02-10T07:45:54.583 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:54.607 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:54.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:54.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:54.697 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:45:54.697 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:45:54.709 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:54.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:54.734 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:54.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:54.834 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:54.858 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:54.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:54.959 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:54.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:54.986 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:54.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:55.088 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:55.111 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:55.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:55.212 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:55.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:55.237 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:55.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:55.338 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:55.351 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:55 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:55.351 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:55 smithi170 ceph-mon[33394]: 9.6 scrub starts 2024-02-10T07:45:55.351 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:55 smithi170 ceph-mon[33394]: 9.6 scrub ok 2024-02-10T07:45:55.351 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:55 smithi170 ceph-mon[33394]: pgmap v627: 148 pgs: 21 active+undersized+degraded, 41 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 686 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:55.364 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:55.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:55.377 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-10T07:45:55.395 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:45:55.448 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:45:55.448 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:55 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:55.448 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:55 smithi008 ceph-mon[29816]: 9.6 scrub starts 2024-02-10T07:45:55.448 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:55 smithi008 ceph-mon[29816]: 9.6 scrub ok 2024-02-10T07:45:55.448 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:55 smithi008 ceph-mon[29816]: pgmap v627: 148 pgs: 21 active+undersized+degraded, 41 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 686 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:55.449 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:55 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:45:55.449 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:55 smithi008 ceph-mon[40743]: 9.6 scrub starts 2024-02-10T07:45:55.449 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:55 smithi008 ceph-mon[40743]: 9.6 scrub ok 2024-02-10T07:45:55.449 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:55 smithi008 ceph-mon[40743]: pgmap v627: 148 pgs: 21 active+undersized+degraded, 41 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 686 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:55.450 INFO:teuthology.orchestra.run.smithi008.stdout:get_pg_num_history pg_num_history pg_num_history(e295 pg_nums {1={19=1},2={65=3},3={66=32},4={68=32},5={70=32},6={72=32},8={200=16},9={253=16}} deleted_pools 251,8) 2024-02-10T07:45:55.450 INFO:teuthology.orchestra.run.smithi008.stdout:Importing pgid 6.7 2024-02-10T07:45:55.450 INFO:teuthology.orchestra.run.smithi008.stdout:Write #6:e4fda9dc:users.keys::A1GPE05QVJT728K71ZP4:head# 2024-02-10T07:45:55.450 INFO:teuthology.orchestra.run.smithi008.stdout:snapset 0=[]:{} 2024-02-10T07:45:55.450 INFO:teuthology.orchestra.run.smithi008.stdout:write_pg epoch 256 info 6.7( v 94'2 (0'0,94'2] local-lis/les=156/157 n=1 ec=72/72 lis/c=156/156 les/c/f=157/157/0 sis=156) 2024-02-10T07:45:55.450 INFO:teuthology.orchestra.run.smithi008.stdout:Import successful 2024-02-10T07:45:55.466 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:55.469 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:45:55.489 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:45:55.491 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:55.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:55.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:55.510 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:45:55.530 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:45:55.552 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:45:55.572 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:45:55.572 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:45:55.573 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:45:55.573 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:45:55.573 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-10T07:45:55.573 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:45:55.573 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:45:55.573 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:45:55.573 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:45:55.593 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:55.615 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:45:55.618 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:55.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:55.720 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:55.743 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:55.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:55.845 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:55.870 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:55.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:55.972 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:55.997 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:55.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:55.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:56.098 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:56.122 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:56.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:56.224 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:56.248 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:56.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:56.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:56 smithi008 ceph-mon[29816]: 3.14 scrub starts 2024-02-10T07:45:56.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:56 smithi008 ceph-mon[29816]: 3.14 scrub ok 2024-02-10T07:45:56.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:56 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2956059596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:56 smithi008 ceph-mon[40743]: 3.14 scrub starts 2024-02-10T07:45:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:56 smithi008 ceph-mon[40743]: 3.14 scrub ok 2024-02-10T07:45:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:56 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2956059596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:56.351 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:56.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:56.378 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:56.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:56.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:56 smithi170 ceph-mon[33394]: 3.14 scrub starts 2024-02-10T07:45:56.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:56 smithi170 ceph-mon[33394]: 3.14 scrub ok 2024-02-10T07:45:56.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:56 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2956059596' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:45:56.479 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:56.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:56.505 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:56.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:56.606 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:56.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:56.635 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:56.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:56.736 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:56.763 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:56.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:56.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:56.865 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:56.892 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:56.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:56.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:56.994 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:57.018 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:57.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:57.120 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:57.143 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:57.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:57.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:57.245 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:57.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:57.273 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:57.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:57.368 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:45:57.369 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:45:57.374 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:57.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:57.399 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:57.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:57.448 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:57 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:45:57.448 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:57 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:45:57.448 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:57 smithi170 ceph-mon[33394]: 5.18 scrub starts 2024-02-10T07:45:57.448 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:57 smithi170 ceph-mon[33394]: 5.18 scrub ok 2024-02-10T07:45:57.448 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:57 smithi170 ceph-mon[33394]: pgmap v628: 148 pgs: 20 active+undersized+degraded, 42 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 419 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:57.448 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:57 smithi170 ceph-mon[33394]: 3.1b scrub starts 2024-02-10T07:45:57.448 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:57 smithi170 ceph-mon[33394]: 3.1b scrub ok 2024-02-10T07:45:57.449 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:57 smithi170 ceph-mon[33394]: osdmap e306: 8 total, 7 up, 8 in 2024-02-10T07:45:57.449 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:57 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 87/594 objects degraded (14.646%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:57.449 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:45:57.449 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2024-02-10T07:45:57.449 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2024-02-10T07:45:57.450 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:45:57.450 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.450 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210702 7.12 7:49611789:::benchmark_data_smithi008_82046_object95564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:45:57.450 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.058811+0000", 2024-02-10T07:45:57.450 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 502.38796497999999, 2024-02-10T07:45:57.450 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0023876279999999, 2024-02-10T07:45:57.450 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.450 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.450 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.450 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.450 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.450 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210702 2024-02-10T07:45:57.450 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.450 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058811+0000", 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058811+0000", 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058812+0000", 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6799999999999999e-07 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.451 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.452 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.452 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058815+0000", 2024-02-10T07:45:57.452 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.881e-06 2024-02-10T07:45:57.452 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.452 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.452 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.452 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058824+0000", 2024-02-10T07:45:57.452 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.2650000000000002e-06 2024-02-10T07:45:57.452 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.452 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.452 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.452 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058830+0000", 2024-02-10T07:45:57.452 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.242e-06 2024-02-10T07:45:57.452 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058883+0000", 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3631999999999997e-05 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.058893+0000", 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0258000000000001e-05 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.042033+0000", 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.98313994000000005 2024-02-10T07:45:57.453 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.454 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.454 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.454 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.042128+0000", 2024-02-10T07:45:57.454 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4141000000000005e-05 2024-02-10T07:45:57.454 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.454 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.454 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:45:57.454 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.042439+0000", 2024-02-10T07:45:57.454 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00031155199999999999 2024-02-10T07:45:57.454 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.454 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.454 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.454 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.045767+0000", 2024-02-10T07:45:57.454 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033280839999999998 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.048185+0000", 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0024177080000000002 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061099+0000", 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012914583 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061157+0000", 2024-02-10T07:45:57.455 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7429000000000002e-05 2024-02-10T07:45:57.456 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.456 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.456 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.456 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061199+0000", 2024-02-10T07:45:57.456 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2045000000000001e-05 2024-02-10T07:45:57.456 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.456 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.456 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.456 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.456 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.456 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210713 7.12 7:48995235:::benchmark_data_smithi008_82046_object95575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:45:57.456 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.069890+0000", 2024-02-10T07:45:57.456 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 502.37688648, 2024-02-10T07:45:57.456 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.99147127099999999, 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210713 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069890+0000", 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.457 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.458 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069890+0000", 2024-02-10T07:45:57.458 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.458 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.458 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.458 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.458 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069891+0000", 2024-02-10T07:45:57.458 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5880000000000001e-06 2024-02-10T07:45:57.458 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.458 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.458 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.458 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069896+0000", 2024-02-10T07:45:57.458 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9250000000000003e-06 2024-02-10T07:45:57.458 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069909+0000", 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2471e-05 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.069917+0000", 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1850000000000004e-06 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.070014+0000", 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6675999999999994e-05 2024-02-10T07:45:57.459 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.460 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.460 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.460 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.070022+0000", 2024-02-10T07:45:57.460 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7499999999999992e-06 2024-02-10T07:45:57.460 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.460 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.460 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.460 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.044238+0000", 2024-02-10T07:45:57.460 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.97421551399999995 2024-02-10T07:45:57.460 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.460 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.460 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.460 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.044356+0000", 2024-02-10T07:45:57.460 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000117617 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.044601+0000", 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024531800000000001 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049272+0000", 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0046710420000000002 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050384+0000", 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001111721 2024-02-10T07:45:57.461 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061261+0000", 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010877045 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061319+0000", 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8415999999999998e-05 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061361+0000", 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2002999999999997e-05 2024-02-10T07:45:57.462 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.463 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.463 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.463 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.463 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.463 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210719 7.12 7:4a954bf6:::benchmark_data_smithi008_82046_object95581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:45:57.463 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.075516+0000", 2024-02-10T07:45:57.463 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 502.37126024200001, 2024-02-10T07:45:57.463 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.98756840400000001, 2024-02-10T07:45:57.463 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.463 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.463 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.463 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.463 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210719 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075516+0000", 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075516+0000", 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.464 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075517+0000", 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4219999999999999e-06 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075522+0000", 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7779999999999998e-06 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075534+0000", 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.173e-05 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.465 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075544+0000", 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0105000000000001e-05 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075592+0000", 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8173000000000001e-05 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.075601+0000", 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5890000000000003e-06 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.466 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.045802+0000", 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.97020084299999998 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.045871+0000", 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9725000000000001e-05 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.046021+0000", 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014953899999999999 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049278+0000", 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0032569140000000001 2024-02-10T07:45:57.467 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.468 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.468 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.468 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050430+0000", 2024-02-10T07:45:57.468 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0011520949999999999 2024-02-10T07:45:57.468 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.468 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.468 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.468 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.062999+0000", 2024-02-10T07:45:57.468 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012568823999999999 2024-02-10T07:45:57.468 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.468 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.468 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.468 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063050+0000", 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0788999999999998e-05 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063084+0000", 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4878e-05 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210730 7.12 7:4fc3a9f7:::benchmark_data_smithi008_82046_object95592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.081362+0000", 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 502.36541396400003, 2024-02-10T07:45:57.469 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.98183658399999996, 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210730 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081362+0000", 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.470 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081362+0000", 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081364+0000", 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4219999999999999e-06 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081368+0000", 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5770000000000004e-06 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.471 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081381+0000", 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3118e-05 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081389+0000", 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.379e-06 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081443+0000", 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4475000000000003e-05 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.472 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.081452+0000", 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.32e-06 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.046907+0000", 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.96545532300000003 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.047023+0000", 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000115832 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.047228+0000", 2024-02-10T07:45:57.473 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020531600000000001 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050394+0000", 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0031658239999999998 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055109+0000", 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0047147710000000004 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063136+0000", 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0080273949999999997 2024-02-10T07:45:57.474 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.475 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.475 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.475 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063176+0000", 2024-02-10T07:45:57.475 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0142999999999998e-05 2024-02-10T07:45:57.475 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.476 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:45:57.476 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:45:57.476 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[29816]: 5.18 scrub starts 2024-02-10T07:45:57.476 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[29816]: 5.18 scrub ok 2024-02-10T07:45:57.476 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[29816]: pgmap v628: 148 pgs: 20 active+undersized+degraded, 42 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 419 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:57.476 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[29816]: 3.1b scrub starts 2024-02-10T07:45:57.476 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[29816]: 3.1b scrub ok 2024-02-10T07:45:57.476 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[29816]: osdmap e306: 8 total, 7 up, 8 in 2024-02-10T07:45:57.476 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 87/594 objects degraded (14.646%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:57.477 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:45:57.477 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:45:57.477 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[40743]: 5.18 scrub starts 2024-02-10T07:45:57.477 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[40743]: 5.18 scrub ok 2024-02-10T07:45:57.477 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[40743]: pgmap v628: 148 pgs: 20 active+undersized+degraded, 42 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 419 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:57.477 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[40743]: 3.1b scrub starts 2024-02-10T07:45:57.477 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[40743]: 3.1b scrub ok 2024-02-10T07:45:57.477 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[40743]: osdmap e306: 8 total, 7 up, 8 in 2024-02-10T07:45:57.477 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:57 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 87/594 objects degraded (14.646%), 20 pgs degraded (PG_DEGRADED) 2024-02-10T07:45:57.478 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.478 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.478 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063199+0000", 2024-02-10T07:45:57.478 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2688999999999999e-05 2024-02-10T07:45:57.478 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.478 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.478 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.478 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.478 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.478 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210739 7.12 7:4d4f1063:::benchmark_data_smithi008_82046_object95601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:45:57.478 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.090931+0000", 2024-02-10T07:45:57.478 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 502.35584506399999, 2024-02-10T07:45:57.479 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.97239903000000005, 2024-02-10T07:45:57.479 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.479 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.479 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.479 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.479 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.479 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210739 2024-02-10T07:45:57.479 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.479 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.479 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.479 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.479 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090931+0000", 2024-02-10T07:45:57.479 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.479 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090931+0000", 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090932+0000", 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2309999999999999e-06 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090937+0000", 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5650000000000003e-06 2024-02-10T07:45:57.480 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090954+0000", 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6515e-05 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.090962+0000", 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0600000000000008e-06 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.091039+0000", 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7632999999999993e-05 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.481 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.091050+0000", 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0491e-05 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.048220+0000", 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.95717007499999995 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.048294+0000", 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4536000000000006e-05 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:45:57.482 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.048444+0000", 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014942699999999999 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.051558+0000", 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0031144889999999998 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.058790+0000", 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0072319430000000002 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.483 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063249+0000", 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0044584890000000004 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063296+0000", 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7327999999999998e-05 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063330+0000", 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4248000000000001e-05 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.484 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.485 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.485 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210750 7.12 7:4d3f77f1:::benchmark_data_smithi008_82046_object95612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:45:57.485 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.097423+0000", 2024-02-10T07:45:57.485 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 502.34935287799999, 2024-02-10T07:45:57.485 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.96606917000000003, 2024-02-10T07:45:57.485 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.485 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.485 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.485 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.485 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.485 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210750 2024-02-10T07:45:57.485 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.485 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.485 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097423+0000", 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097423+0000", 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097425+0000", 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9080000000000002e-06 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.486 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.487 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097430+0000", 2024-02-10T07:45:57.487 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2390000000000004e-06 2024-02-10T07:45:57.487 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.487 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.487 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.487 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097441+0000", 2024-02-10T07:45:57.487 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1257000000000001e-05 2024-02-10T07:45:57.487 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.487 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.487 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.487 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097449+0000", 2024-02-10T07:45:57.487 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7430000000000002e-06 2024-02-10T07:45:57.487 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.487 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097502+0000", 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3361e-05 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.097510+0000", 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.4449999999999996e-06 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049307+0000", 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.95179659999999999 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.488 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049373+0000", 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5612999999999997e-05 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.049521+0000", 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014814799999999999 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.053835+0000", 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0043143499999999998 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.489 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.490 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.490 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061401+0000", 2024-02-10T07:45:57.490 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0075663919999999999 2024-02-10T07:45:57.490 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.490 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.490 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.490 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063378+0000", 2024-02-10T07:45:57.490 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0019766499999999999 2024-02-10T07:45:57.490 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.490 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.490 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.490 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063453+0000", 2024-02-10T07:45:57.490 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4935e-05 2024-02-10T07:45:57.490 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063493+0000", 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9529e-05 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210807 7.12 7:4f675a45:::benchmark_data_smithi008_82046_object95669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.148770+0000", 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 502.29800624199999, 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.91486412100000003, 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.491 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.492 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.492 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.492 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.492 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210807 2024-02-10T07:45:57.492 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.492 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.492 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.492 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.492 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148770+0000", 2024-02-10T07:45:57.492 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.492 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.492 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.492 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.492 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148770+0000", 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148771+0000", 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1710000000000001e-06 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148775+0000", 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1420000000000003e-06 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.493 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148788+0000", 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2897e-05 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148797+0000", 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2689999999999999e-06 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148849+0000", 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2095000000000002e-05 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.148857+0000", 2024-02-10T07:45:57.494 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9389999999999997e-06 2024-02-10T07:45:57.495 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.495 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.495 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.495 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050456+0000", 2024-02-10T07:45:57.495 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.90159990199999995 2024-02-10T07:45:57.495 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.495 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.495 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.495 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050527+0000", 2024-02-10T07:45:57.495 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0321000000000007e-05 2024-02-10T07:45:57.495 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.495 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.495 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:45:57.495 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.050675+0000", 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000147725 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055123+0000", 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0044484379999999999 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061450+0000", 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0063272709999999998 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.496 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063551+0000", 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0021005830000000001 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063603+0000", 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2441000000000002e-05 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063634+0000", 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0926999999999997e-05 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.497 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210838 7.12 7:4d53824f:::benchmark_data_smithi008_82046_object95700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.176589+0000", 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 502.27018753700003, 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.88719636599999996, 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210838 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176589+0000", 2024-02-10T07:45:57.498 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176589+0000", 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176590+0000", 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.198e-06 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176594+0000", 2024-02-10T07:45:57.499 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8840000000000004e-06 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176607+0000", 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2855e-05 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176616+0000", 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1339999999999993e-06 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176667+0000", 2024-02-10T07:45:57.500 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0887999999999998e-05 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.176676+0000", 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1009999999999998e-06 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.051582+0000", 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.87490583200000005 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.051647+0000", 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5252999999999993e-05 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.501 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.051803+0000", 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000156336 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055129+0000", 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033258559999999999 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.062510+0000", 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0073805119999999997 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.502 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063680+0000", 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001170508 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063745+0000", 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5273999999999999e-05 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.063785+0000", 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9734999999999998e-05 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.503 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210862 7.12 7:48875dbe:::benchmark_data_smithi008_82046_object95724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.256071+0000", 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 502.19070487499999, 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.81128573100000001, 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210862 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.504 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256071+0000", 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256071+0000", 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256072+0000", 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5199999999999995e-07 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.505 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256076+0000", 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7910000000000002e-06 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256084+0000", 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0860000000000002e-06 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256089+0000", 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0549999999999999e-06 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256126+0000", 2024-02-10T07:45:57.506 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6480000000000003e-05 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.256131+0000", 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7259999999999999e-06 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.052653+0000", 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.79652150600000005 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.052764+0000", 2024-02-10T07:45:57.507 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000111022 2024-02-10T07:45:57.508 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.508 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.508 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:45:57.508 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.052966+0000", 2024-02-10T07:45:57.508 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000202442 2024-02-10T07:45:57.508 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.508 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.508 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.508 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057531+0000", 2024-02-10T07:45:57.508 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0045644800000000001 2024-02-10T07:45:57.508 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.508 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.508 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.508 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.062542+0000", 2024-02-10T07:45:57.509 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0050112000000000004 2024-02-10T07:45:57.509 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.509 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.509 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.509 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.067300+0000", 2024-02-10T07:45:57.509 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0047581560000000004 2024-02-10T07:45:57.509 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.509 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.509 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.509 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.067333+0000", 2024-02-10T07:45:57.509 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2437999999999997e-05 2024-02-10T07:45:57.509 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.509 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.509 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.067357+0000", 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4496999999999999e-05 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210891 7.12 7:4e0bd40b:::benchmark_data_smithi008_82046_object95753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.296428+0000", 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 502.150348415, 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.77191008299999997, 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.510 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210891 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296428+0000", 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296428+0000", 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.511 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296430+0000", 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9659999999999999e-06 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296435+0000", 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8509999999999998e-06 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296452+0000", 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6888000000000002e-05 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.512 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296461+0000", 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4429999999999992e-06 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296628+0000", 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000166675 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.296636+0000", 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6770000000000001e-06 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.513 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.053881+0000", 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.75724445799999995 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.053969+0000", 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7972000000000004e-05 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.054161+0000", 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019222400000000001 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.514 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057538+0000", 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033767060000000002 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.067399+0000", 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0098609119999999995 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.068281+0000", 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000881857 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.068312+0000", 2024-02-10T07:45:57.515 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.1659999999999998e-05 2024-02-10T07:45:57.516 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.516 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.516 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.516 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.068338+0000", 2024-02-10T07:45:57.516 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5794000000000001e-05 2024-02-10T07:45:57.516 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.516 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.516 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.516 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.516 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.516 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210902 7.12 7:4cdd2df1:::benchmark_data_smithi008_82046_object95764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:45:57.516 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.308067+0000", 2024-02-10T07:45:57.516 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 502.13870915500002, 2024-02-10T07:45:57.516 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.76995323400000004, 2024-02-10T07:45:57.517 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.517 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.517 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.517 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.517 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.517 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210902 2024-02-10T07:45:57.517 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.517 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.517 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.517 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.517 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308067+0000", 2024-02-10T07:45:57.517 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.517 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.517 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308067+0000", 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308068+0000", 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.077e-06 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308072+0000", 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8020000000000002e-06 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.518 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308083+0000", 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0743e-05 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308091+0000", 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.824e-06 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308147+0000", 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5998000000000002e-05 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.519 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.308157+0000", 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9250000000000007e-06 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055146+0000", 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.74698950799999997 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055221+0000", 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4597000000000004e-05 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:45:57.520 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.055375+0000", 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015408300000000001 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.058799+0000", 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0034239380000000001 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.068376+0000", 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0095769779999999999 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.077940+0000", 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0095647360000000008 2024-02-10T07:45:57.521 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.077981+0000", 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0688000000000001e-05 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078020+0000", 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9337e-05 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210911 7.12 7:4f38fa58:::benchmark_data_smithi008_82046_object95773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:45:57.522 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.318835+0000", 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 502.12794115899999, 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.75930091600000005, 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210911 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318835+0000", 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.523 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318835+0000", 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318836+0000", 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1510000000000001e-06 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318840+0000", 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1309999999999999e-06 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.524 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318850+0000", 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6690000000000001e-06 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318855+0000", 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3410000000000002e-06 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318904+0000", 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8712999999999999e-05 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.318913+0000", 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.8650000000000007e-06 2024-02-10T07:45:57.525 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.056341+0000", 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.73742814899999998 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.056452+0000", 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000110511 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.056655+0000", 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00020361199999999999 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.526 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.527 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061057+0000", 2024-02-10T07:45:57.527 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.004401861 2024-02-10T07:45:57.527 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.527 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.527 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.527 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.076677+0000", 2024-02-10T07:45:57.527 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015619687 2024-02-10T07:45:57.527 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.527 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.527 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.527 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078061+0000", 2024-02-10T07:45:57.527 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0013845089999999999 2024-02-10T07:45:57.527 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.527 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.528 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.528 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078102+0000", 2024-02-10T07:45:57.528 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.108e-05 2024-02-10T07:45:57.528 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.528 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.528 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.528 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078136+0000", 2024-02-10T07:45:57.528 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3637000000000003e-05 2024-02-10T07:45:57.528 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.528 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.528 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.528 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.528 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.528 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210950 7.12 7:4bea578c:::benchmark_data_smithi008_82046_object95812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.363086+0000", 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 502.08369018799999, 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.71519846200000003, 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210950 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363086+0000", 2024-02-10T07:45:57.529 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363086+0000", 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363087+0000", 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.232e-06 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363091+0000", 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7780000000000001e-06 2024-02-10T07:45:57.530 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363101+0000", 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9580000000000002e-06 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363108+0000", 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6239999999999996e-06 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363163+0000", 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4987000000000001e-05 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.531 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.363173+0000", 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0338e-05 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057579+0000", 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.69440612899999998 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057654+0000", 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4476999999999999e-05 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.532 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.057814+0000", 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000160216 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061207+0000", 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0033935850000000002 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.076729+0000", 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015521517 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.533 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078191+0000", 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0014615749999999999 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078243+0000", 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2627000000000003e-05 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078285+0000", 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1418999999999998e-05 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.534 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:210966 7.12 7:4dc81df4:::benchmark_data_smithi008_82046_object95828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:35.386841+0000", 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 502.059935129, 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.69155063900000002, 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 210966 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.535 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386841+0000", 2024-02-10T07:45:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386841+0000", 2024-02-10T07:45:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386842+0000", 2024-02-10T07:45:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3499999999999996e-07 2024-02-10T07:45:57.536 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386844+0000", 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3709999999999998e-06 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386851+0000", 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2490000000000004e-06 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386856+0000", 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5600000000000004e-06 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.537 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386907+0000", 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0946000000000003e-05 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:35.386916+0000", 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0450000000000006e-06 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.058828+0000", 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.67191242600000001 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.538 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.058892+0000", 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3441999999999993e-05 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,2", 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.059060+0000", 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000167808 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.061212+0000", 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0021523620000000001 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.539 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.540 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.076780+0000", 2024-02-10T07:45:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015567713 2024-02-10T07:45:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078326+0000", 2024-02-10T07:45:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0015468789999999999 2024-02-10T07:45:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.541 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078357+0000", 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0905000000000003e-05 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.078392+0000", 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4397999999999998e-05 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211133 7.19 7:9c7b01de:::benchmark_data_smithi008_82046_object95995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.029346+0000", 2024-02-10T07:45:57.542 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 501.41742988599998, 2024-02-10T07:45:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.75181978400000005, 2024-02-10T07:45:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211133 2024-02-10T07:45:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029346+0000", 2024-02-10T07:45:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.543 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029346+0000", 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029348+0000", 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.347e-06 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029352+0000", 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4689999999999999e-06 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.544 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029364+0000", 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1712999999999999e-05 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029372+0000", 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8869999999999992e-06 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029414+0000", 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2472000000000003e-05 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.545 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.029421+0000", 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5880000000000003e-06 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.774874+0000", 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.74545264499999997 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.774967+0000", 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3955000000000004e-05 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.546 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.775347+0000", 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00037925300000000002 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.780874+0000", 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0055271779999999998 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.780892+0000", 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8397999999999999e-05 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.547 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.548 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781061+0000", 2024-02-10T07:45:57.548 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016846299999999999 2024-02-10T07:45:57.548 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.548 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.548 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.548 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781118+0000", 2024-02-10T07:45:57.548 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7451000000000003e-05 2024-02-10T07:45:57.548 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.548 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.548 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.548 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781166+0000", 2024-02-10T07:45:57.548 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7964999999999999e-05 2024-02-10T07:45:57.548 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.548 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.549 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.549 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.549 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.549 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211195 7.19 7:98ba502a:::benchmark_data_smithi008_82046_object96057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:45:57.549 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.123282+0000", 2024-02-10T07:45:57.549 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 501.32349425699999, 2024-02-10T07:45:57.549 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.65998720099999997, 2024-02-10T07:45:57.549 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.549 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.549 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.549 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.549 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.549 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211195 2024-02-10T07:45:57.549 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123282+0000", 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123282+0000", 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123283+0000", 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9099999999999991e-07 2024-02-10T07:45:57.550 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123287+0000", 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6839999999999998e-06 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123297+0000", 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0237e-05 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123302+0000", 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4909999999999996e-06 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.551 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123356+0000", 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3748000000000001e-05 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.123366+0000", 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0139000000000001e-05 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.777248+0000", 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.65388193900000002 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.552 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.777361+0000", 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000112723 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.777593+0000", 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00023236400000000001 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781245+0000", 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0036513639999999998 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.553 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.554 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781258+0000", 2024-02-10T07:45:57.554 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3458999999999999e-05 2024-02-10T07:45:57.554 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.554 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.554 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.554 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783159+0000", 2024-02-10T07:45:57.554 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001900484 2024-02-10T07:45:57.554 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.554 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.554 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.554 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783222+0000", 2024-02-10T07:45:57.554 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3149000000000001e-05 2024-02-10T07:45:57.554 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.554 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783269+0000", 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7429000000000003e-05 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211237 7.19 7:9803a4c9:::benchmark_data_smithi008_82046_object96099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.157249+0000", 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 501.28952686600002, 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.62624425500000003, 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.555 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211237 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157249+0000", 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157249+0000", 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.556 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.557 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.557 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157250+0000", 2024-02-10T07:45:57.557 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.47e-07 2024-02-10T07:45:57.557 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.557 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.557 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.557 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157255+0000", 2024-02-10T07:45:57.557 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2889999999999999e-06 2024-02-10T07:45:57.557 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.557 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.557 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.557 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157265+0000", 2024-02-10T07:45:57.557 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0158e-05 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157272+0000", 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3010000000000001e-06 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157324+0000", 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1997999999999999e-05 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.157332+0000", 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4839999999999998e-06 2024-02-10T07:45:57.558 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.559 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.559 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.559 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.778870+0000", 2024-02-10T07:45:57.559 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.62153816500000003 2024-02-10T07:45:57.559 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.559 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.559 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.559 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.778991+0000", 2024-02-10T07:45:57.559 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000121272 2024-02-10T07:45:57.559 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.559 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.559 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:45:57.559 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.779232+0000", 2024-02-10T07:45:57.559 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024073399999999999 2024-02-10T07:45:57.560 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.560 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.560 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.560 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781264+0000", 2024-02-10T07:45:57.560 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020323899999999998 2024-02-10T07:45:57.560 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.560 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.560 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.560 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781322+0000", 2024-02-10T07:45:57.560 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8e-05 2024-02-10T07:45:57.560 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.560 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.560 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.560 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783335+0000", 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020125339999999999 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783454+0000", 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011938300000000001 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783494+0000", 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.96e-05 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.561 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211239 7.19 7:9b3bb0be:::benchmark_data_smithi008_82046_object96101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.158720+0000", 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 501.288056391, 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.62931068000000001, 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211239 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.562 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158720+0000", 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158720+0000", 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158721+0000", 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1289999999999999e-06 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158726+0000", 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7010000000000004e-06 2024-02-10T07:45:57.563 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158747+0000", 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1362e-05 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158755+0000", 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7680000000000008e-06 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158807+0000", 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2522000000000003e-05 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.564 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.158817+0000", 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3870000000000001e-06 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781362+0000", 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.62254494699999996 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781463+0000", 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000101592 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.565 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.566 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:45:57.566 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.781718+0000", 2024-02-10T07:45:57.566 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00025504199999999997 2024-02-10T07:45:57.566 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.566 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.566 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.566 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.785814+0000", 2024-02-10T07:45:57.566 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0040961110000000004 2024-02-10T07:45:57.566 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.566 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.566 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.566 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787830+0000", 2024-02-10T07:45:57.566 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020153760000000001 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787925+0000", 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5309000000000005e-05 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787989+0000", 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3578999999999996e-05 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.788031+0000", 2024-02-10T07:45:57.567 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1854999999999998e-05 2024-02-10T07:45:57.568 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.568 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.568 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.568 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.568 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.568 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211270 7.19 7:9a9ecfcf:::benchmark_data_smithi008_82046_object96132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:45:57.568 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.185809+0000", 2024-02-10T07:45:57.568 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 501.26096734999999, 2024-02-10T07:45:57.568 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.60674339600000005, 2024-02-10T07:45:57.568 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.568 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.568 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.568 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.569 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.569 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211270 2024-02-10T07:45:57.569 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.569 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.569 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.569 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.569 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185809+0000", 2024-02-10T07:45:57.569 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.569 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.569 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.569 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.569 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185809+0000", 2024-02-10T07:45:57.569 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.570 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.570 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.570 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.570 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185810+0000", 2024-02-10T07:45:57.570 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.215e-06 2024-02-10T07:45:57.570 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.570 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.570 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.570 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185814+0000", 2024-02-10T07:45:57.570 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3259999999999997e-06 2024-02-10T07:45:57.570 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.570 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.570 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.570 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185827+0000", 2024-02-10T07:45:57.571 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2669e-05 2024-02-10T07:45:57.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.571 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.571 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.571 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.185835+0000", 2024-02-10T07:45:57.571 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4769999999999997e-06 2024-02-10T07:45:57.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.571 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.571 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.571 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.186136+0000", 2024-02-10T07:45:57.571 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000301155 2024-02-10T07:45:57.571 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.186151+0000", 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5316e-05 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783531+0000", 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.59737997799999998 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783650+0000", 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000118968 2024-02-10T07:45:57.572 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.783954+0000", 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00030440799999999999 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787866+0000", 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0039120379999999996 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.787875+0000", 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1249999999999993e-06 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.573 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792479+0000", 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0046047429999999997 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792525+0000", 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5262000000000001e-05 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792552+0000", 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7716e-05 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.574 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:211303 7.19 7:9bbbc725:::benchmark_data_smithi008_82046_object96165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:37:36.210320+0000", 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 501.23645674599999, 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.58484913400000005, 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 211303 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:45:57.575 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.576 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:45:57.576 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210320+0000", 2024-02-10T07:45:57.576 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.576 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.576 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.576 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:45:57.576 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210320+0000", 2024-02-10T07:45:57.576 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:45:57.576 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.576 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.576 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:45:57.576 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210320+0000", 2024-02-10T07:45:57.576 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1200000000000002e-07 2024-02-10T07:45:57.576 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210324+0000", 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2459999999999999e-06 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210344+0000", 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0020000000000001e-05 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210352+0000", 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0430000000000008e-06 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.577 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210431+0000", 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9482999999999998e-05 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.210439+0000", 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0909999999999993e-06 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.785850+0000", 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.57541028900000002 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.578 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:45:57.579 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.785977+0000", 2024-02-10T07:45:57.579 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012783599999999999 2024-02-10T07:45:57.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.579 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.579 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:45:57.579 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.786232+0000", 2024-02-10T07:45:57.579 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00025462599999999998 2024-02-10T07:45:57.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.579 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.579 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:45:57.579 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792371+0000", 2024-02-10T07:45:57.579 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0061389529999999999 2024-02-10T07:45:57.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.579 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.792433+0000", 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2088999999999995e-05 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.795068+0000", 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0026348130000000002 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.795124+0000", 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6365e-05 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:45:57.580 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:45:57.581 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:45:57.581 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:37:36.795169+0000", 2024-02-10T07:45:57.581 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4468000000000002e-05 2024-02-10T07:45:57.581 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.581 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.581 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.581 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:45:57.581 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:45:57.581 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:45:57.605 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:57.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:57.707 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:57.731 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:57.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:57.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:57.832 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:57.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:57.858 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:57.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:57.959 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:57.983 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:57.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:57.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:58.085 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:58.108 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:58.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:58.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:58.210 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:45:58.234 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:58.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:58.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:45:58.259 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-10T07:45:58.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:58 smithi170 ceph-mon[33394]: 9.b scrub starts 2024-02-10T07:45:58.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:58 smithi170 ceph-mon[33394]: 9.b scrub ok 2024-02-10T07:45:58.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:58 smithi170 ceph-mon[33394]: 3.a scrub starts 2024-02-10T07:45:58.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:58 smithi170 ceph-mon[33394]: 3.a scrub ok 2024-02-10T07:45:58.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:58 smithi170 ceph-mon[33394]: 6.1d scrub starts 2024-02-10T07:45:58.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:58 smithi170 ceph-mon[33394]: 6.1d scrub ok 2024-02-10T07:45:58.336 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:58.359 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:58.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:58.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:58.461 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:45:58.485 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:58.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:58.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:45:58.502 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:58 smithi008 ceph-mon[29816]: 9.b scrub starts 2024-02-10T07:45:58.502 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:58 smithi008 ceph-mon[29816]: 9.b scrub ok 2024-02-10T07:45:58.503 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:58 smithi008 ceph-mon[29816]: 3.a scrub starts 2024-02-10T07:45:58.503 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:58 smithi008 ceph-mon[29816]: 3.a scrub ok 2024-02-10T07:45:58.503 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:58 smithi008 ceph-mon[29816]: 6.1d scrub starts 2024-02-10T07:45:58.503 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:58 smithi008 ceph-mon[29816]: 6.1d scrub ok 2024-02-10T07:45:58.503 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:58 smithi008 ceph-mon[40743]: 9.b scrub starts 2024-02-10T07:45:58.503 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:58 smithi008 ceph-mon[40743]: 9.b scrub ok 2024-02-10T07:45:58.503 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:58 smithi008 ceph-mon[40743]: 3.a scrub starts 2024-02-10T07:45:58.503 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:58 smithi008 ceph-mon[40743]: 3.a scrub ok 2024-02-10T07:45:58.503 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:58 smithi008 ceph-mon[40743]: 6.1d scrub starts 2024-02-10T07:45:58.503 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:58 smithi008 ceph-mon[40743]: 6.1d scrub ok 2024-02-10T07:45:58.546 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:45:58.587 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:58.610 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:58.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:58.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:58.642 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/exp.6.7.3 2024-02-10T07:45:58.712 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:58.750 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:58.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:58.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:58.852 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:58.876 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:58.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:58.978 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:59.002 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:59.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:59.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:59.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:45:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:45:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:45:59.104 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:59.128 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:59.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:59.207 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'unique_pool_2' removed 2024-02-10T07:45:59.229 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:59.254 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:59.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:59.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:59.355 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:45:59.370 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:45:59.379 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:59.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:45:59.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:59 smithi170 ceph-mon[33394]: pgmap v630: 148 pgs: 20 active+undersized+degraded, 42 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:59 smithi170 ceph-mon[33394]: 4.6 scrub starts 2024-02-10T07:45:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:59 smithi170 ceph-mon[33394]: 4.6 scrub ok 2024-02-10T07:45:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:59 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3100411134' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:45:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:59 smithi170 ceph-mon[33394]: 3.10 scrub starts 2024-02-10T07:45:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:59 smithi170 ceph-mon[33394]: 3.10 scrub ok 2024-02-10T07:45:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:59 smithi170 ceph-mon[33394]: 4.2 scrub starts 2024-02-10T07:45:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:59 smithi170 ceph-mon[33394]: 4.2 scrub ok 2024-02-10T07:45:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:45:59 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:45:59.482 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:45:59.509 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:59.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:45:59.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[29816]: pgmap v630: 148 pgs: 20 active+undersized+degraded, 42 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:59.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[29816]: 4.6 scrub starts 2024-02-10T07:45:59.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[29816]: 4.6 scrub ok 2024-02-10T07:45:59.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3100411134' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:45:59.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[29816]: 3.10 scrub starts 2024-02-10T07:45:59.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[29816]: 3.10 scrub ok 2024-02-10T07:45:59.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[29816]: 4.2 scrub starts 2024-02-10T07:45:59.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[29816]: 4.2 scrub ok 2024-02-10T07:45:59.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:45:59.564 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[40743]: pgmap v630: 148 pgs: 20 active+undersized+degraded, 42 active+undersized, 86 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:45:59.564 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[40743]: 4.6 scrub starts 2024-02-10T07:45:59.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[40743]: 4.6 scrub ok 2024-02-10T07:45:59.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3100411134' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:45:59.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[40743]: 3.10 scrub starts 2024-02-10T07:45:59.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[40743]: 3.10 scrub ok 2024-02-10T07:45:59.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[40743]: 4.2 scrub starts 2024-02-10T07:45:59.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[40743]: 4.2 scrub ok 2024-02-10T07:45:59.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:45:59 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:45:59.611 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:45:59.635 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:59.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:45:59.737 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:45:59.763 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:45:59.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:59.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:45:59.865 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:45:59.888 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:45:59.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:45:59.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:45:59.990 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:00.013 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:00.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:00.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:00.115 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:00.138 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:00.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:00.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:00.240 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:00.263 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:00.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:00.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:00.365 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:00.388 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:00.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:00.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:00.475 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:00 smithi170 ceph-mon[33394]: 6.10 deep-scrub starts 2024-02-10T07:46:00.476 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:00 smithi170 ceph-mon[33394]: 6.10 deep-scrub ok 2024-02-10T07:46:00.476 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:00 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3100411134' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:46:00.476 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:00 smithi170 ceph-mon[33394]: osdmap e307: 8 total, 7 up, 8 in 2024-02-10T07:46:00.476 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:00 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:46:00.476 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:00 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:46:00.476 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:00 smithi170 ceph-mon[33394]: 4.12 scrub starts 2024-02-10T07:46:00.476 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:00 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:46:00.476 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:00 smithi170 ceph-mon[33394]: 4.12 scrub ok 2024-02-10T07:46:00.490 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:00.513 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:00.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:00.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[29816]: 6.10 deep-scrub starts 2024-02-10T07:46:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[29816]: 6.10 deep-scrub ok 2024-02-10T07:46:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3100411134' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:46:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[29816]: osdmap e307: 8 total, 7 up, 8 in 2024-02-10T07:46:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:46:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:46:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[29816]: 4.12 scrub starts 2024-02-10T07:46:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:46:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[29816]: 4.12 scrub ok 2024-02-10T07:46:00.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[40743]: 6.10 deep-scrub starts 2024-02-10T07:46:00.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[40743]: 6.10 deep-scrub ok 2024-02-10T07:46:00.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3100411134' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:46:00.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[40743]: osdmap e307: 8 total, 7 up, 8 in 2024-02-10T07:46:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:46:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:46:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[40743]: 4.12 scrub starts 2024-02-10T07:46:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:46:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:00 smithi008 ceph-mon[40743]: 4.12 scrub ok 2024-02-10T07:46:00.614 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:00.639 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:00.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:00.712 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-10T07:46:00.713 INFO:tasks.radosbench:Beginning radosbench... 2024-02-10T07:46:00.713 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-02-10T07:46:00.713 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-02-10T07:46:00.741 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:00.748 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:46:00.748 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2024-02-10T07:46:00.748 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2024-02-10T07:46:00.748 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:46:00.776 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:00.780 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:00.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:00.881 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:00.906 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:00.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:00.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:01.007 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:01.031 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:01.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:01.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:01.133 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:01.156 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:01.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:01.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:01.258 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:01.282 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:01.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:01.383 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:01.407 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:01.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:01.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:01.427 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:46:01.508 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:01.532 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:01.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:01.558 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-10T07:46:01.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:01 smithi170 ceph-mon[33394]: pgmap v632: 132 pgs: 20 active+undersized+degraded, 34 active+undersized, 78 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:01.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:01 smithi170 ceph-mon[33394]: 5.2 deep-scrub starts 2024-02-10T07:46:01.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:01 smithi170 ceph-mon[33394]: 5.2 deep-scrub ok 2024-02-10T07:46:01.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:01 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/302827180' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:46:01.633 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:01.657 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:01.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:01.759 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:01.784 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:01.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:01.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:01 smithi008 ceph-mon[29816]: pgmap v632: 132 pgs: 20 active+undersized+degraded, 34 active+undersized, 78 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:01.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:01 smithi008 ceph-mon[29816]: 5.2 deep-scrub starts 2024-02-10T07:46:01.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:01 smithi008 ceph-mon[29816]: 5.2 deep-scrub ok 2024-02-10T07:46:01.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:01 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/302827180' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:46:01.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:01 smithi008 ceph-mon[40743]: pgmap v632: 132 pgs: 20 active+undersized+degraded, 34 active+undersized, 78 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:01.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:01 smithi008 ceph-mon[40743]: 5.2 deep-scrub starts 2024-02-10T07:46:01.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:01 smithi008 ceph-mon[40743]: 5.2 deep-scrub ok 2024-02-10T07:46:01.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:01 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/302827180' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:46:01.842 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:46:01.886 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:01.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:01.910 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:01.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:02.011 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:02.036 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:02.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:02.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:02.137 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:02.160 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:02.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:02.262 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:02.285 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:02.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:02.387 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:02.411 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:02.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:02.513 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:02.536 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:02.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:02.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:02 smithi008 ceph-mon[29816]: 5.9 scrub starts 2024-02-10T07:46:02.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:02 smithi008 ceph-mon[29816]: 5.9 scrub ok 2024-02-10T07:46:02.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:02 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:46:02.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:02 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/302827180' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:46:02.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:02 smithi008 ceph-mon[29816]: osdmap e308: 8 total, 7 up, 8 in 2024-02-10T07:46:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:02 smithi008 ceph-mon[40743]: 5.9 scrub starts 2024-02-10T07:46:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:02 smithi008 ceph-mon[40743]: 5.9 scrub ok 2024-02-10T07:46:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:02 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:46:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:02 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/302827180' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:46:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:02 smithi008 ceph-mon[40743]: osdmap e308: 8 total, 7 up, 8 in 2024-02-10T07:46:02.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:02 smithi170 ceph-mon[33394]: 5.9 scrub starts 2024-02-10T07:46:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:02 smithi170 ceph-mon[33394]: 5.9 scrub ok 2024-02-10T07:46:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:02 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:46:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:02 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/302827180' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:46:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:02 smithi170 ceph-mon[33394]: osdmap e308: 8 total, 7 up, 8 in 2024-02-10T07:46:02.638 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:02.661 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:02.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:02.763 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:02.787 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:02.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:02.808 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:02.888 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:02.911 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:02.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:03.013 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:03.036 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:03.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:03.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:03.138 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:03.161 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:03.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:03.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:03.263 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:03.286 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:03.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:03.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:03.387 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:03.410 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:03.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:03.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:03.512 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:03.536 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:03.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:03.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:03.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:46:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:46:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:03 smithi170 ceph-mon[33394]: pgmap v634: 132 pgs: 20 active+undersized+degraded, 34 active+undersized, 78 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:46:03.635 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:46:03.636 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2024-02-10T07:46:03.636 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2024-02-10T07:46:03.636 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2024-02-10T07:46:03.636 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:46:03.638 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:03.662 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:03.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:03.687 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 6, 7, 5] 2024-02-10T07:46:03.687 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:46:03.688 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-10T07:46:03.688 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd out 2 2024-02-10T07:46:03.709 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:03 smithi008 ceph-mon[29816]: pgmap v634: 132 pgs: 20 active+undersized+degraded, 34 active+undersized, 78 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:03.709 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:46:03.710 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:03 smithi008 ceph-mon[40743]: pgmap v634: 132 pgs: 20 active+undersized+degraded, 34 active+undersized, 78 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:03.710 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:46:03.764 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:03.788 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:03.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:03.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:03.889 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:03.914 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:03.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:03.998 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:46:04.016 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:04.017 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:46:04.037 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:46:04.040 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:04.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:04.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:04.058 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:46:04.077 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:46:04.097 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:46:04.117 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:46:04.117 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:46:04.117 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:46:04.117 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:46:04.117 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-10T07:46:04.117 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:46:04.117 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:46:04.118 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:46:04.118 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:46:04.141 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:04.164 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:04.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:04.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:04.179 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:46:04.206 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-10T07:46:04.266 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:04.289 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:04.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:04.391 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:04.414 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:04.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:04.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:04.448 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:46:04.516 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:04.540 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:04.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:04.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:04.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:04 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3388218601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:04.641 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:04.665 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:04.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:04.748 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:04 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3388218601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:04.748 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:04 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3388218601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:04.768 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:04.792 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:04.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:04.893 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:04.917 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:04.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:05.018 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:05.042 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:05.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:05.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:05.144 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:05.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:05.169 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:05.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:05.269 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:05.293 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:05.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:05.394 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:05.418 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:05.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:05.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:05.519 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:05.543 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:05.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:05.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:05.644 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:05.668 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:05.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:05.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:05.669 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:46:05.670 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:46:05.690 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'unique_pool_3' created 2024-02-10T07:46:05.769 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:05.793 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:05.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:05.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:05 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:05 smithi170 ceph-mon[33394]: pgmap v635: 132 pgs: 20 active+undersized+degraded, 34 active+undersized, 78 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:05 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 87/594 objects degraded (14.646%), 20 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:05.896 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:05.919 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:05.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:05.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:06.021 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:06.032 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:05 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:06.032 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:05 smithi008 ceph-mon[40743]: pgmap v635: 132 pgs: 20 active+undersized+degraded, 34 active+undersized, 78 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:06.032 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:05 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 87/594 objects degraded (14.646%), 20 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:06.032 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:05 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:06.033 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:05 smithi008 ceph-mon[29816]: pgmap v635: 132 pgs: 20 active+undersized+degraded, 34 active+undersized, 78 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:06.033 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:05 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 87/594 objects degraded (14.646%), 20 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:06.039 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:46:06.039 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:46:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:06.046 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:06.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:06.147 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:06.171 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:06.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:06.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:06.272 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:06.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:06.297 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:06.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:06.378 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-02-10T07:46:06.398 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:06.421 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:06.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:06.427 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:06.432 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:06.523 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:06.546 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:06.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:06.648 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:06.671 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:06.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:06.773 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:06.796 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:06.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:06.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:06.898 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:06.921 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:06 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3803100441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-10T07:46:06.921 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:06 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-10T07:46:06.921 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:06 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-10T07:46:06.922 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:06 smithi170 ceph-mon[33394]: osdmap e309: 8 total, 7 up, 8 in 2024-02-10T07:46:06.922 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:06 smithi170 ceph-mon[33394]: pgmap v637: 148 pgs: 16 unknown, 20 active+undersized+degraded, 34 active+undersized, 78 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 750 B/s rd, 0 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:06.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:06.922 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:06.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:06.946 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:06 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3803100441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-10T07:46:06.946 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:06 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-10T07:46:06.946 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:06 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-10T07:46:06.946 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:06 smithi008 ceph-mon[29816]: osdmap e309: 8 total, 7 up, 8 in 2024-02-10T07:46:06.946 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:06 smithi008 ceph-mon[29816]: pgmap v637: 148 pgs: 16 unknown, 20 active+undersized+degraded, 34 active+undersized, 78 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 750 B/s rd, 0 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:06.946 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:06 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3803100441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-10T07:46:06.946 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:06 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-10T07:46:06.946 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:06 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-10T07:46:06.946 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:06 smithi008 ceph-mon[40743]: osdmap e309: 8 total, 7 up, 8 in 2024-02-10T07:46:06.946 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:06 smithi008 ceph-mon[40743]: pgmap v637: 148 pgs: 16 unknown, 20 active+undersized+degraded, 34 active+undersized, 78 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 750 B/s rd, 0 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:06.948 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T07:46:07.023 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:07.046 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:07.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:07.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:07.148 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:07.171 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:07.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:07.187 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:46:07.273 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:07.296 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:07.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:07.397 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:07.421 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:07.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:07.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:07.522 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:07.546 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:07.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:07.647 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:07.670 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:46:07.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:07.672 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:07.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:07.773 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:07.796 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:07.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:07.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:07.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:07 smithi008 ceph-mon[29816]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:46:07.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:07 smithi008 ceph-mon[29816]: osdmap e310: 8 total, 7 up, 8 in 2024-02-10T07:46:07.831 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:07 smithi008 ceph-mon[40743]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:46:07.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:07 smithi008 ceph-mon[40743]: osdmap e310: 8 total, 7 up, 8 in 2024-02-10T07:46:07.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:07 smithi170 ceph-mon[33394]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:46:07.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:07 smithi170 ceph-mon[33394]: osdmap e310: 8 total, 7 up, 8 in 2024-02-10T07:46:07.898 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:07.923 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:07.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:07.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:08.024 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:08.048 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:08.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:08.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:08.149 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:08.174 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:08.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:08.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:08.275 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:08.299 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:08.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:08.401 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:08.425 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:08.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:08.526 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:08.550 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:08.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:08.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:08.652 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:08.675 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:08.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:08.777 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:08.800 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:08.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:08.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:08.902 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:08.926 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:08.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:08.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:08.966 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:46:08.967 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:46:08.967 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:08 smithi170 ceph-mon[33394]: osdmap e311: 8 total, 7 up, 8 in 2024-02-10T07:46:08.967 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:08 smithi170 ceph-mon[33394]: pgmap v640: 148 pgs: 5 unknown, 20 active+undersized+degraded, 36 active+undersized, 87 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:09.028 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:09.051 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:09.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:09.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:09.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:08 smithi008 ceph-mon[40743]: osdmap e311: 8 total, 7 up, 8 in 2024-02-10T07:46:09.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:08 smithi008 ceph-mon[40743]: pgmap v640: 148 pgs: 5 unknown, 20 active+undersized+degraded, 36 active+undersized, 87 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:09.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:08 smithi008 ceph-mon[29816]: osdmap e311: 8 total, 7 up, 8 in 2024-02-10T07:46:09.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:08 smithi008 ceph-mon[29816]: pgmap v640: 148 pgs: 5 unknown, 20 active+undersized+degraded, 36 active+undersized, 87 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:09.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:46:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:46:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:46:09.153 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:09.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:09.178 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:09.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:09.279 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:09.302 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:09.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:09.404 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:09.428 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:09.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:09.529 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:09.536 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:09.537 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:09.555 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:09.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:09.655 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-10T07:46:09.657 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:09.680 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:09.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:09.714 INFO:teuthology.orchestra.run.smithi008.stderr:marked out osd.2. 2024-02-10T07:46:09.781 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:09.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:09 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a[29812]: 2024-02-10T07:46:09.505+0000 7ff409c1e700 -1 mon.a@0(leader).osd e311 definitely_dead 0 2024-02-10T07:46:09.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:09 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2921842929' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-10T07:46:09.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:09 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-10T07:46:09.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:09 smithi008 ceph-mon[29816]: Client client.admin marked osd.2 out, while it was still marked up 2024-02-10T07:46:09.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:09 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2921842929' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-10T07:46:09.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:09 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-10T07:46:09.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:09 smithi008 ceph-mon[40743]: Client client.admin marked osd.2 out, while it was still marked up 2024-02-10T07:46:09.806 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:09.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:09.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:09.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:09 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2921842929' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-10T07:46:09.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:09 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-10T07:46:09.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:09 smithi170 ceph-mon[33394]: Client client.admin marked osd.2 out, while it was still marked up 2024-02-10T07:46:09.908 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:09.909 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:46:09.933 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:09.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:10.035 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:10.058 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:10.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:10.159 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:10.183 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:10.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:10.285 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:10.308 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:10.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:10.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:10.409 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:10.432 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:10.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:10.535 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:10.558 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:10.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:10.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:10.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:10.683 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:10.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:10.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:10.785 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:10.809 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:10.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:10.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:10.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:10 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-10T07:46:10.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:10 smithi008 ceph-mon[29816]: osdmap e312: 8 total, 7 up, 7 in 2024-02-10T07:46:10.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:10 smithi008 ceph-mon[29816]: pgmap v642: 148 pgs: 20 active+undersized+degraded, 39 active+undersized, 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:10.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:10 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-10T07:46:10.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:10 smithi008 ceph-mon[40743]: osdmap e312: 8 total, 7 up, 7 in 2024-02-10T07:46:10.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:10 smithi008 ceph-mon[40743]: pgmap v642: 148 pgs: 20 active+undersized+degraded, 39 active+undersized, 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:10 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-10T07:46:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:10 smithi170 ceph-mon[33394]: osdmap e312: 8 total, 7 up, 7 in 2024-02-10T07:46:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:10 smithi170 ceph-mon[33394]: pgmap v642: 148 pgs: 20 active+undersized+degraded, 39 active+undersized, 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:10.911 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:10.934 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:10.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:10.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:11.036 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:11.059 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:11.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:11.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:11.160 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:11.184 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:11.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:11.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:11.285 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:11.308 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:11.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:11.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:11.410 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:11.433 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:11.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:11.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:11.535 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:11.560 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:11.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:11.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:11.662 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:11.686 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:11.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:11.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:11.788 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:11.811 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:11.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:11.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:11.824 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:11 smithi008 ceph-mon[29816]: osdmap e313: 8 total, 7 up, 7 in 2024-02-10T07:46:11.824 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:11 smithi008 ceph-mon[40743]: osdmap e313: 8 total, 7 up, 7 in 2024-02-10T07:46:11.829 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:46:11.829 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:46:11.829 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:11 smithi170 ceph-mon[33394]: osdmap e313: 8 total, 7 up, 7 in 2024-02-10T07:46:11.914 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:11.940 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:11.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:12.043 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:12.067 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:12.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:12.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:12.169 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:12.192 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:12.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:12.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:12.294 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:12.317 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:12.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:12.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:12.419 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:12.442 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:12.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:12.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:12.486 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:46:12.506 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:46:12.525 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:46:12.543 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:12.547 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:46:12.567 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:46:12.568 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:12.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:12.573 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-10T07:46:12.588 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:46:12.608 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:46:12.608 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:46:12.608 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:46:12.608 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:46:12.608 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-10T07:46:12.608 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:46:12.608 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:46:12.608 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:46:12.608 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:46:12.669 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:12.692 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:12.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:12.794 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:12.818 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:12.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:12.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:12.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:12 smithi008 ceph-mon[29816]: osdmap e314: 8 total, 7 up, 7 in 2024-02-10T07:46:12.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:12 smithi008 ceph-mon[29816]: pgmap v645: 148 pgs: 20 active+undersized+degraded, 39 active+undersized, 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:12.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:12 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3779479179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:12.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:12 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1480797465' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:46:12.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:12 smithi008 ceph-mon[40743]: osdmap e314: 8 total, 7 up, 7 in 2024-02-10T07:46:12.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:12 smithi008 ceph-mon[40743]: pgmap v645: 148 pgs: 20 active+undersized+degraded, 39 active+undersized, 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:12.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:12 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3779479179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:12.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:12 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1480797465' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:46:12.863 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:46:12.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:12 smithi170 ceph-mon[33394]: osdmap e314: 8 total, 7 up, 7 in 2024-02-10T07:46:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:12 smithi170 ceph-mon[33394]: pgmap v645: 148 pgs: 20 active+undersized+degraded, 39 active+undersized, 89 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/594 objects degraded (14.646%) 2024-02-10T07:46:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:12 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3779479179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:12 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1480797465' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:46:12.920 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:12.931 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:46:12.946 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:12.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:12.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:13.048 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:13.072 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:13.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:13.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:13.174 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:13.197 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:13.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:13.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:13.299 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:13.325 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:13.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:13.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:13.368 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:46:13.368 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:46:13.427 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:13.450 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:13.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:13.552 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:13.575 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:13.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:13.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:13.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:46:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:46:13.677 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:13.702 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:13.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:13.803 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:13.827 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:13.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:13.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:13.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:13.952 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:13.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:13.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:14.022 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:13 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:46:14.022 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:13 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:46:14.022 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:13 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:46:14.022 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:13 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1480797465' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:46:14.022 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:13 smithi170 ceph-mon[33394]: osdmap e315: 8 total, 7 up, 7 in 2024-02-10T07:46:14.054 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:14.077 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:14.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:14.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:14.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:13 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:46:14.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:13 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:46:14.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:13 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:46:14.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:13 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1480797465' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:46:14.081 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:13 smithi008 ceph-mon[40743]: osdmap e315: 8 total, 7 up, 7 in 2024-02-10T07:46:14.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:13 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:46:14.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:13 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:46:14.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:13 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:46:14.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:13 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1480797465' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:46:14.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:13 smithi008 ceph-mon[29816]: osdmap e315: 8 total, 7 up, 7 in 2024-02-10T07:46:14.179 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:14.203 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:14.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:14.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:14.308 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:14.336 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:14.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:14.437 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:14.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:14.462 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:14.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:14.564 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:14.587 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:14.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:14.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:14.620 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:46:14.620 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2024-02-10T07:46:14.620 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2024-02-10T07:46:14.620 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:46:14.620 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.620 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24740.0:154826 7.0 7.577e9d00 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:36:32.885749+0000", 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 581.73280658199997, 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.86189586600000001, 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24740", 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 154826 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885749+0000", 2024-02-10T07:46:14.621 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885749+0000", 2024-02-10T07:46:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885751+0000", 2024-02-10T07:46:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2109999999999999e-06 2024-02-10T07:46:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.622 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885755+0000", 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7960000000000003e-06 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885765+0000", 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0101999999999999e-05 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885775+0000", 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9359999999999994e-06 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.623 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885836+0000", 2024-02-10T07:46:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0615999999999997e-05 2024-02-10T07:46:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:46:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:32.885856+0000", 2024-02-10T07:46:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0418999999999999e-05 2024-02-10T07:46:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:33.747631+0000", 2024-02-10T07:46:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.86177504800000004 2024-02-10T07:46:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.624 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:36:33.747645+0000", 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3738e-05 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:168459 8.d 8:b99d57f7:::benchmark_data_smithi008_109566_object43372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:34.586096+0000", 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.03245972000002, 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.21918052700000001, 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.625 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 168459 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.586096+0000", 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.586096+0000", 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.626 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.586098+0000", 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.291e-06 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.586101+0000", 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6459999999999999e-06 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.586336+0000", 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00023513200000000001 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.627 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.628 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.586346+0000", 2024-02-10T07:46:14.628 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5750000000000007e-06 2024-02-10T07:46:14.628 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.628 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.628 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.628 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.588181+0000", 2024-02-10T07:46:14.628 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001835218 2024-02-10T07:46:14.628 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.628 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.628 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.628 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.588297+0000", 2024-02-10T07:46:14.628 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000115851 2024-02-10T07:46:14.628 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.628 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.588414+0000", 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011712199999999999 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.597335+0000", 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0089213580000000008 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.599845+0000", 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0025098989999999999 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.629 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805192+0000", 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.20534719500000001 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805231+0000", 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8674000000000003e-05 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805277+0000", 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5565999999999997e-05 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.630 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:168468 8.4 8:2f095741:::benchmark_data_smithi008_109566_object43381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:34.587949+0000", 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.03060711299997, 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.21746917700000001, 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 168468 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.631 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.587949+0000", 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.587949+0000", 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.587950+0000", 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.8199999999999998e-07 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.632 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.633 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.633 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.587954+0000", 2024-02-10T07:46:14.633 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5689999999999998e-06 2024-02-10T07:46:14.633 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.633 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.633 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.633 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.587967+0000", 2024-02-10T07:46:14.634 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2918999999999999e-05 2024-02-10T07:46:14.634 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.634 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.634 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.634 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.587975+0000", 2024-02-10T07:46:14.634 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2179999999999999e-06 2024-02-10T07:46:14.634 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.634 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.634 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.634 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.588053+0000", 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7847000000000002e-05 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.588185+0000", 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000132123 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.588303+0000", 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000117546 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.635 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.597168+0000", 2024-02-10T07:46:14.636 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0088654700000000003 2024-02-10T07:46:14.636 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.636 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.636 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.636 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.598139+0000", 2024-02-10T07:46:14.636 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00097050100000000002 2024-02-10T07:46:14.636 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.636 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.636 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.636 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805343+0000", 2024-02-10T07:46:14.636 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.207204164 2024-02-10T07:46:14.636 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.636 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.636 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.637 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805384+0000", 2024-02-10T07:46:14.637 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1090000000000001e-05 2024-02-10T07:46:14.637 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.637 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.637 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.637 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805418+0000", 2024-02-10T07:46:14.637 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3847999999999998e-05 2024-02-10T07:46:14.637 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.637 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.637 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.637 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.637 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.637 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:168471 8.d 8:b73a9cff:::benchmark_data_smithi008_109566_object43384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-10T07:46:14.637 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:34.591099+0000", 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.02745741400003, 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.21437766799999999, 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 168471 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591099+0000", 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.638 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591099+0000", 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591100+0000", 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1340000000000001e-06 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591103+0000", 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.129e-06 2024-02-10T07:46:14.639 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591111+0000", 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2810000000000008e-06 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591117+0000", 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4870000000000002e-06 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591394+0000", 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00027699099999999998 2024-02-10T07:46:14.640 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591520+0000", 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012602199999999999 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.591642+0000", 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012226899999999999 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.597340+0000", 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0056981710000000001 2024-02-10T07:46:14.641 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.599885+0000", 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002544883 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805391+0000", 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.20550612700000001 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805440+0000", 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8715999999999999e-05 2024-02-10T07:46:14.642 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.805476+0000", 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6458000000000002e-05 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:168493 8.4 8:2f97658c:::benchmark_data_smithi008_109566_object43406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:34.612764+0000", 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 340.00579207099997, 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.19627367900000001, 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.643 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 168493 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612764+0000", 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612764+0000", 2024-02-10T07:46:14.644 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612765+0000", 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.483e-06 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612771+0000", 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6169999999999998e-06 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612783+0000", 2024-02-10T07:46:14.645 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2041999999999999e-05 2024-02-10T07:46:14.646 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.646 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.646 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.646 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612791+0000", 2024-02-10T07:46:14.646 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6839999999999996e-06 2024-02-10T07:46:14.646 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.646 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.646 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.646 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612849+0000", 2024-02-10T07:46:14.646 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8146000000000002e-05 2024-02-10T07:46:14.646 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.646 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.646 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.646 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.612960+0000", 2024-02-10T07:46:14.647 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000111319 2024-02-10T07:46:14.647 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.647 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.647 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:46:14.647 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.613089+0000", 2024-02-10T07:46:14.647 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012928899999999999 2024-02-10T07:46:14.647 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.647 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.647 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.647 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.616007+0000", 2024-02-10T07:46:14.647 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0029177740000000001 2024-02-10T07:46:14.647 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.647 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.647 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.619638+0000", 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0036306519999999998 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.808936+0000", 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.189297875 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.808996+0000", 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9920999999999998e-05 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.648 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.809038+0000", 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1876999999999999e-05 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:172622 8.d 8:b9980fd8:::benchmark_data_smithi008_109566_object47535:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:38.753034+0000", 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 335.86552220300001, 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.14670670899999999, 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.649 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 172622 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753034+0000", 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753034+0000", 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.650 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753035+0000", 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1459999999999999e-06 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753039+0000", 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4100000000000001e-06 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753049+0000", 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9240000000000002e-06 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.651 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753055+0000", 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4960000000000004e-06 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753104+0000", 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9364999999999999e-05 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.753182+0000", 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8361000000000004e-05 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.652 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.881819+0000", 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12863662100000001 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.891604+0000", 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0097851339999999992 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.891619+0000", 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4630000000000001e-05 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.653 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.899625+0000", 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0080057779999999999 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.899695+0000", 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0203000000000005e-05 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:38.899740+0000", 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5640999999999999e-05 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.654 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:149438 9.f 9:fdc268d9:::benchmark_data_smithi008_134920_object21261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:44:13.201771+0000", 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 121.41678464500001, 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.160330584, 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 149438 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.655 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.201771+0000", 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.201771+0000", 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.201773+0000", 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.305e-06 2024-02-10T07:46:14.656 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.201776+0000", 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6040000000000001e-06 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.201787+0000", 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.098e-05 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.201793+0000", 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3160000000000002e-06 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.657 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.658 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.658 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.230611+0000", 2024-02-10T07:46:14.658 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.028817097999999999 2024-02-10T07:46:14.658 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.658 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.658 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.658 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.230689+0000", 2024-02-10T07:46:14.658 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8620000000000003e-05 2024-02-10T07:46:14.658 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.658 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.658 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:46:14.658 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.230770+0000", 2024-02-10T07:46:14.658 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1044000000000005e-05 2024-02-10T07:46:14.658 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.235547+0000", 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0047767219999999997 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.302035+0000", 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.066488363999999994 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.362004+0000", 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.059968650999999998 2024-02-10T07:46:14.659 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.660 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.660 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.660 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.362042+0000", 2024-02-10T07:46:14.660 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7895000000000002e-05 2024-02-10T07:46:14.660 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.660 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.660 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.660 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.362102+0000", 2024-02-10T07:46:14.660 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9985000000000003e-05 2024-02-10T07:46:14.660 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.660 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.660 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.660 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.660 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:149446 9.7 9:efb5045b:::benchmark_data_smithi008_134920_object21269:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:44:13.210657+0000", 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 121.40789849399999, 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.15139239700000001, 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 149446 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.661 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.210657+0000", 2024-02-10T07:46:14.662 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.662 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.662 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.662 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.662 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.210657+0000", 2024-02-10T07:46:14.662 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.662 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.662 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.662 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.662 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.210658+0000", 2024-02-10T07:46:14.662 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7199999999999999e-07 2024-02-10T07:46:14.662 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.662 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.662 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.663 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.210660+0000", 2024-02-10T07:46:14.663 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1940000000000001e-06 2024-02-10T07:46:14.663 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.663 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.663 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.663 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.210667+0000", 2024-02-10T07:46:14.663 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3900000000000004e-06 2024-02-10T07:46:14.663 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.663 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.663 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.663 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.210671+0000", 2024-02-10T07:46:14.663 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.76e-06 2024-02-10T07:46:14.663 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.232726+0000", 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.022054397 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.232820+0000", 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4481999999999994e-05 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.232888+0000", 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8017999999999998e-05 2024-02-10T07:46:14.664 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.235559+0000", 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0026711590000000002 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.235907+0000", 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00034743799999999999 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.361989+0000", 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12608243299999999 2024-02-10T07:46:14.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.362024+0000", 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5052999999999998e-05 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.362050+0000", 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5601000000000001e-05 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.666 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:149448 9.f 9:f7ffa3ab:::benchmark_data_smithi008_134920_object21271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:44:13.214173+0000", 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 121.404383083, 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.14819568, 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 149448 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.214173+0000", 2024-02-10T07:46:14.667 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.668 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.668 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.668 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.214173+0000", 2024-02-10T07:46:14.668 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.668 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.668 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.668 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.214174+0000", 2024-02-10T07:46:14.668 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0400000000000005e-07 2024-02-10T07:46:14.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.668 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.668 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.668 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.214176+0000", 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.6400000000000001e-06 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.214184+0000", 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7300000000000005e-06 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.214189+0000", 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5480000000000003e-06 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.669 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.236014+0000", 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.021825441000000001 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.236134+0000", 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0001198 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.236260+0000", 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012575299999999999 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.238994+0000", 2024-02-10T07:46:14.670 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0027348429999999998 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.302064+0000", 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.063069691999999997 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.362289+0000", 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.060224999000000001 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.362324+0000", 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4579999999999998e-05 2024-02-10T07:46:14.671 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.672 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.672 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.672 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:13.362369+0000", 2024-02-10T07:46:14.672 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4849999999999999e-05 2024-02-10T07:46:14.672 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.672 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.672 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.672 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.672 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.672 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:150433 9.f 9:fbebd46b:::benchmark_data_smithi008_134920_object22256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-02-10T07:46:14.673 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:44:14.544803+0000", 2024-02-10T07:46:14.673 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 120.073752501, 2024-02-10T07:46:14.673 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.18665638200000001, 2024-02-10T07:46:14.673 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.673 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 150433 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.544803+0000", 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.544803+0000", 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.674 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.544804+0000", 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4299999999999999e-07 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.544805+0000", 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.486e-06 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.544809+0000", 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6940000000000001e-06 2024-02-10T07:46:14.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.676 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.676 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.676 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.544812+0000", 2024-02-10T07:46:14.676 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8499999999999998e-06 2024-02-10T07:46:14.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.676 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.676 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.676 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.591412+0000", 2024-02-10T07:46:14.676 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.046599979999999999 2024-02-10T07:46:14.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.676 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.676 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.676 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.591495+0000", 2024-02-10T07:46:14.676 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2795999999999998e-05 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.591563+0000", 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8701000000000005e-05 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.595045+0000", 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0034816109999999999 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.645263+0000", 2024-02-10T07:46:14.677 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.050218332999999997 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.731392+0000", 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.086129057999999994 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.731426+0000", 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4150000000000003e-05 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.731460+0000", 2024-02-10T07:46:14.678 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3380000000000002e-05 2024-02-10T07:46:14.679 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.679 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.679 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.679 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:150444 9.f 9:fb7378f2:::benchmark_data_smithi008_134920_object22267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-02-10T07:46:14.679 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:44:14.560148+0000", 2024-02-10T07:46:14.679 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 120.058407519, 2024-02-10T07:46:14.679 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.17144129699999999, 2024-02-10T07:46:14.679 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.679 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.679 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.679 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:46:14.679 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 150444 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.560148+0000", 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.560148+0000", 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.680 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.560150+0000", 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.13e-06 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.560153+0000", 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8600000000000003e-06 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.560163+0000", 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.8079999999999999e-06 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.681 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.560169+0000", 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8479999999999997e-06 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.592725+0000", 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.032555728999999999 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.592841+0000", 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011598499999999999 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.592947+0000", 2024-02-10T07:46:14.682 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000106257 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.595049+0000", 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002102128 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.645297+0000", 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.050248156000000002 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.731512+0000", 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.086215172000000007 2024-02-10T07:46:14.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.731545+0000", 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2369999999999997e-05 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:14.731590+0000", 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4854000000000001e-05 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.684 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:150783 9.f 9:fbcaa5a0:::benchmark_data_smithi008_134920_object22606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:44:15.015187+0000", 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 119.603369255, 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.161046983, 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 150783 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:15.015187+0000", 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:15.015187+0000", 2024-02-10T07:46:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:15.015187+0000", 2024-02-10T07:46:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5700000000000002e-07 2024-02-10T07:46:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:15.015191+0000", 2024-02-10T07:46:14.686 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.726e-06 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:15.015200+0000", 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5320000000000006e-06 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:15.015206+0000", 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0560000000000001e-06 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:15.055254+0000", 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.040048325000000003 2024-02-10T07:46:14.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:15.055368+0000", 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011359299999999999 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:15.055506+0000", 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013842300000000001 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:15.058140+0000", 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0026336770000000001 2024-02-10T07:46:14.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:15.060871+0000", 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0027308530000000001 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:15.176122+0000", 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.115251806 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:15.176176+0000", 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3384000000000003e-05 2024-02-10T07:46:14.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:15.176234+0000", 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7850999999999999e-05 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:152613 9.f 9:faf0a7d4:::benchmark_data_smithi008_134920_object24436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:44:17.322755+0000", 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 117.29580074099999, 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.192546096, 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.690 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 152613 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.322755+0000", 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.322755+0000", 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.322756+0000", 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6899999999999997e-07 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.322759+0000", 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7930000000000002e-06 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.322765+0000", 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.708e-06 2024-02-10T07:46:14.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.322770+0000", 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6580000000000001e-06 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.341208+0000", 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.018438217 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.341338+0000", 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013020299999999999 2024-02-10T07:46:14.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.341461+0000", 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012303699999999999 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.344278+0000", 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0028165410000000001 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.372416+0000", 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.028137954 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.515227+0000", 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.14281085399999999 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.515264+0000", 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6832000000000002e-05 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.515301+0000", 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.773e-05 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.695 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:152630 9.f 9:ffb186e7:::benchmark_data_smithi008_134920_object24453:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:44:17.360309+0000", 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 117.25824707699999, 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.15532327100000001, 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 152630 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.696 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.360309+0000", 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.360309+0000", 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.360310+0000", 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.265e-06 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.360316+0000", 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3850000000000001e-06 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.360327+0000", 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1229999999999999e-05 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.360334+0000", 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5549999999999997e-06 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.360391+0000", 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7119e-05 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.360529+0000", 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000137995 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.360660+0000", 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000130357 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.699 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.700 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.363874+0000", 2024-02-10T07:46:14.700 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.003214312 2024-02-10T07:46:14.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.700 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.700 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.372457+0000", 2024-02-10T07:46:14.700 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0085829930000000006 2024-02-10T07:46:14.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.700 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.700 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.515543+0000", 2024-02-10T07:46:14.700 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.14308605099999999 2024-02-10T07:46:14.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.515600+0000", 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6957999999999999e-05 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:17.515632+0000", 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2051000000000002e-05 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:156362 9.f 9:f6f7384a:::benchmark_data_smithi008_134920_object28185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:44:22.257538+0000", 2024-02-10T07:46:14.701 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 112.361018046, 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.14190599400000001, 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 156362 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:22.257538+0000", 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:22.257538+0000", 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:22.257539+0000", 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1999999999999999e-07 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:22.257542+0000", 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.0900000000000001e-06 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.703 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:22.257554+0000", 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1926999999999999e-05 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:22.257561+0000", 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.1969999999999999e-06 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:22.269968+0000", 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012406858 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:22.270110+0000", 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014247399999999999 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:22.270233+0000", 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000122731 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:22.273937+0000", 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0037038359999999998 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:22.336188+0000", 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.062251872999999999 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:22.399364+0000", 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.063175908000000003 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:22.399403+0000", 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9029000000000002e-05 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:22.399444+0000", 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0451000000000003e-05 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:157665 9.f 9:f1515bbe:::benchmark_data_smithi008_134920_object29488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:44:23.932279+0000", 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 110.686277291, 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.16612461100000001, 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.707 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 157665 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.932279+0000", 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.932279+0000", 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.708 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.932280+0000", 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6080000000000001e-06 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.932285+0000", 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4329999999999997e-06 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.932296+0000", 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.135e-05 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.932304+0000", 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0520000000000002e-06 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.996472+0000", 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.064167759000000005 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.996590+0000", 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000118541 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.996720+0000", 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012942800000000001 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.999942+0000", 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0032223099999999999 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.039551+0000", 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.039609241000000003 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.098273+0000", 2024-02-10T07:46:14.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.058721964000000001 2024-02-10T07:46:14.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.098338+0000", 2024-02-10T07:46:14.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4307000000000006e-05 2024-02-10T07:46:14.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.098403+0000", 2024-02-10T07:46:14.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.5617999999999994e-05 2024-02-10T07:46:14.712 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:157680 9.7 9:e398eb4c:::benchmark_data_smithi008_134920_object29503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:44:23.948119+0000", 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 110.670437394, 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.15120054399999999, 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 157680 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.713 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.948119+0000", 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.948119+0000", 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.948119+0000", 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.2699999999999998e-07 2024-02-10T07:46:14.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.948124+0000", 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0899999999999998e-06 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.948137+0000", 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3597e-05 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.948145+0000", 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2560000000000002e-06 2024-02-10T07:46:14.715 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.716 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.716 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.716 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.002865+0000", 2024-02-10T07:46:14.716 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.054719352999999998 2024-02-10T07:46:14.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.716 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.716 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.716 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.002970+0000", 2024-02-10T07:46:14.716 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010569999999999999 2024-02-10T07:46:14.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.716 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.716 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:46:14.716 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.003080+0000", 2024-02-10T07:46:14.716 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000109399 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.008839+0000", 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.005759152 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.008850+0000", 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1365e-05 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.099216+0000", 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.090365307000000006 2024-02-10T07:46:14.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.099268+0000", 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2333000000000002e-05 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.099319+0000", 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1065e-05 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.718 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:157685 9.f 9:f5a4dc5e:::benchmark_data_smithi008_134920_object29508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:44:23.951117+0000", 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 110.667439278, 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.147449628, 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 157685 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.951117+0000", 2024-02-10T07:46:14.719 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.951117+0000", 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.951117+0000", 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.7700000000000004e-07 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.951120+0000", 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.745e-06 2024-02-10T07:46:14.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.951128+0000", 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6539999999999999e-06 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.951133+0000", 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6300000000000003e-06 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.002673+0000", 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.051539309999999998 2024-02-10T07:46:14.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.002786+0000", 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000113651 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.002904+0000", 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000117995 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.011076+0000", 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0081712250000000007 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.039587+0000", 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.028511686000000001 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.098450+0000", 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.058862914000000002 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.098513+0000", 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3131000000000003e-05 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.098566+0000", 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3010000000000002e-05 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:157686 9.7 9:e63b27fa:::benchmark_data_smithi008_134920_object29509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:44:23.951143+0000", 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 110.667412802, 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.14837521200000001, 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.724 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 157686 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.951143+0000", 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.951143+0000", 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.951144+0000", 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9400000000000005e-07 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.951146+0000", 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2369999999999999e-06 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.951152+0000", 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3910000000000003e-06 2024-02-10T07:46:14.726 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:23.951158+0000", 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2689999999999999e-06 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.005475+0000", 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.054317828999999998 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.005573+0000", 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.8048999999999996e-05 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:46:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.005676+0000", 2024-02-10T07:46:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010277600000000001 2024-02-10T07:46:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.011101+0000", 2024-02-10T07:46:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.005424936 2024-02-10T07:46:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.011159+0000", 2024-02-10T07:46:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7516999999999998e-05 2024-02-10T07:46:14.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.099419+0000", 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.088260036 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.099480+0000", 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0949999999999998e-05 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:24.099518+0000", 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8627999999999997e-05 2024-02-10T07:46:14.729 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:158911 9.f 9:fa76f5e5:::benchmark_data_smithi008_134920_object30734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:44:25.497828+0000", 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 109.120727659, 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.14688874299999999, 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 158911 2024-02-10T07:46:14.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:25.497828+0000", 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:25.497828+0000", 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:25.497829+0000", 2024-02-10T07:46:14.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1489999999999999e-06 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:25.497834+0000", 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3919999999999996e-06 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:25.497847+0000", 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3538e-05 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:25.497856+0000", 2024-02-10T07:46:14.732 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.4889999999999995e-06 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:25.528716+0000", 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.030859865 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:25.528836+0000", 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012026700000000001 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3", 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:25.528951+0000", 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000114881 2024-02-10T07:46:14.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:25.532468+0000", 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0035173510000000002 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:25.584323+0000", 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.051854513999999997 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:25.644594+0000", 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.060271164000000002 2024-02-10T07:46:14.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:25.644654+0000", 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.012e-05 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:44:25.644717+0000", 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3013000000000001e-05 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:46:14.735 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:46:14.736 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:14.759 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:14.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:14.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:14.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:14 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:14.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:14 smithi008 ceph-mon[29816]: pgmap v647: 148 pgs: 13 active+clean+remapped, 18 active+undersized+degraded, 30 active+undersized, 87 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 81/594 objects degraded (13.636%); 7/594 objects misplaced (1.178%); 127 B/s, 5 objects/s recovering 2024-02-10T07:46:14.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:14 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:14.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:14 smithi008 ceph-mon[40743]: pgmap v647: 148 pgs: 13 active+clean+remapped, 18 active+undersized+degraded, 30 active+undersized, 87 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 81/594 objects degraded (13.636%); 7/594 objects misplaced (1.178%); 127 B/s, 5 objects/s recovering 2024-02-10T07:46:14.861 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:14 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:14 smithi170 ceph-mon[33394]: pgmap v647: 148 pgs: 13 active+clean+remapped, 18 active+undersized+degraded, 30 active+undersized, 87 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 81/594 objects degraded (13.636%); 7/594 objects misplaced (1.178%); 127 B/s, 5 objects/s recovering 2024-02-10T07:46:14.884 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:14.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:14.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:14.986 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:15.010 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:15.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:15.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:15.111 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:15.135 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:15.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:15.237 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:15.260 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:15.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:15.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:15.362 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:15.369 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:46:15.385 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:15.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:15.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:15.487 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:15.502 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-10T07:46:15.511 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:15.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:15.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:15.548 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [3] live_osds: [0, 1, 2, 4, 6, 7, 5] 2024-02-10T07:46:15.548 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:46:15.548 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-10T07:46:15.548 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.433461 2024-02-10T07:46:15.548 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 3 0.4334606235648304 2024-02-10T07:46:15.612 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:15.636 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:15.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:15.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:15.651 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:15.738 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:15.762 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:15.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:15.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:15.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:15 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 81/594 objects degraded (13.636%), 18 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:15.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:15 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 81/594 objects degraded (13.636%), 18 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:15.863 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:15 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 81/594 objects degraded (13.636%), 18 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:15.887 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:15.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:15.989 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:16.013 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:16.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:16.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:16.115 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:16.138 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:16.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:16.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:16.240 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:16.266 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:16.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:16.368 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:16.393 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:16.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:16.495 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:16.518 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:16.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:16.527 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:46:16.620 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:16.648 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:16.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:16.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:16.728 INFO:teuthology.orchestra.run.smithi008.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-02-10T07:46:16.749 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:16.776 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:16.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:16.831 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:16 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3015966581' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:46:16.831 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:16 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:46:16.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:16 smithi008 ceph-mon[40743]: pgmap v648: 148 pgs: 19 active+clean+remapped, 15 active+undersized+degraded, 28 active+undersized, 86 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 57/594 objects degraded (9.596%); 31/594 objects misplaced (5.219%); 351 B/s, 9 objects/s recovering 2024-02-10T07:46:16.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:16 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3015966581' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:46:16.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:16 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:46:16.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:16 smithi008 ceph-mon[29816]: pgmap v648: 148 pgs: 19 active+clean+remapped, 15 active+undersized+degraded, 28 active+undersized, 86 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 57/594 objects degraded (9.596%); 31/594 objects misplaced (5.219%); 351 B/s, 9 objects/s recovering 2024-02-10T07:46:16.878 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:16.906 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:16.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:16.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:17.008 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:17.032 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:17.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:17.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:16 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3015966581' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:46:17.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:16 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:46:17.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:16 smithi170 ceph-mon[33394]: pgmap v648: 148 pgs: 19 active+clean+remapped, 15 active+undersized+degraded, 28 active+undersized, 86 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 57/594 objects degraded (9.596%); 31/594 objects misplaced (5.219%); 351 B/s, 9 objects/s recovering 2024-02-10T07:46:17.133 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:17.160 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:17.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:17.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:17.262 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:17.285 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:17.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:17.387 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:17.415 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:17.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:17.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:17.516 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:17.544 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:17.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:17.643 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:17.646 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:17.674 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:17.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:17.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:17.690 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:17.707 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:46:17.730 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:46:17.750 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:46:17.770 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:46:17.775 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:17.790 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:46:17.801 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:17.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:17.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:17.811 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:46:17.822 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:17 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:46:17.822 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:17 smithi008 ceph-mon[29816]: osdmap e316: 8 total, 7 up, 7 in 2024-02-10T07:46:17.822 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:17 smithi008 ceph-mon[29816]: osdmap e317: 8 total, 7 up, 7 in 2024-02-10T07:46:17.822 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:17 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1693992035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:17.822 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:17 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:46:17.822 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:17 smithi008 ceph-mon[40743]: osdmap e316: 8 total, 7 up, 7 in 2024-02-10T07:46:17.822 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:17 smithi008 ceph-mon[40743]: osdmap e317: 8 total, 7 up, 7 in 2024-02-10T07:46:17.822 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:17 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1693992035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:17.832 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:46:17.832 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:46:17.832 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:46:17.832 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:46:17.832 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-10T07:46:17.832 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:46:17.832 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:46:17.832 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:46:17.832 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:46:17.903 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:17.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:17.928 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:17.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:18.029 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:18.056 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:18.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:18.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:17 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:46:18.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:17 smithi170 ceph-mon[33394]: osdmap e316: 8 total, 7 up, 7 in 2024-02-10T07:46:18.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:17 smithi170 ceph-mon[33394]: osdmap e317: 8 total, 7 up, 7 in 2024-02-10T07:46:18.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:17 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1693992035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:18.158 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:18.183 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:18.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:18.284 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:18.310 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:18.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:18.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:18.411 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:18.434 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:18.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:18.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:18.536 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:18.564 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:18.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:18.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:18.666 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:18.693 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:18.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:18.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:18.712 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:46:18.712 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:46:18.795 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:18.819 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:18.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:18.821 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:46:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:46:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:46:18.920 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:18.949 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:18.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:18.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:19.051 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:19.077 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:19.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:19.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:19.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:46:19.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:46:19.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:18 smithi008 ceph-mon[29816]: pgmap v651: 148 pgs: 19 active+clean+remapped, 15 active+undersized+degraded, 28 active+undersized, 86 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 57/594 objects degraded (9.596%); 31/594 objects misplaced (5.219%); 351 B/s, 9 objects/s recovering 2024-02-10T07:46:19.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:46:19.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:46:19.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:18 smithi008 ceph-mon[40743]: pgmap v651: 148 pgs: 19 active+clean+remapped, 15 active+undersized+degraded, 28 active+undersized, 86 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 57/594 objects degraded (9.596%); 31/594 objects misplaced (5.219%); 351 B/s, 9 objects/s recovering 2024-02-10T07:46:19.179 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:19.206 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:19.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:19.308 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:19.336 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:19.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:46:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:46:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:18 smithi170 ceph-mon[33394]: pgmap v651: 148 pgs: 19 active+clean+remapped, 15 active+undersized+degraded, 28 active+undersized, 86 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 57/594 objects degraded (9.596%); 31/594 objects misplaced (5.219%); 351 B/s, 9 objects/s recovering 2024-02-10T07:46:19.437 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:19.462 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:19.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:19.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:19.563 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:19.588 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:19.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:19.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:19.658 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:19.690 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:19.718 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:19.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:19.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:19.820 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:19.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:19.849 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:19.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:19.950 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:19.978 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:19.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:19.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:20.079 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:20.104 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:20.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:20.205 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:20.233 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:20.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:20.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:20.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:19 smithi008 ceph-mon[29816]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:46:20.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:19 smithi008 ceph-mon[40743]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:46:20.334 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:20.358 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:20.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:20.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:20.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:19 smithi170 ceph-mon[33394]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:46:20.460 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:20.487 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:20.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:20.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:20.589 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:20.614 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:20.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:20.689 DEBUG:teuthology.orchestra.run.smithi008:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-02-10T07:46:20.692 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-10T07:46:20.692 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-10T07:46:20.713 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:46:20.716 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:20.779 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:20.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:20.881 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:20.909 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:20.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:20.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:20.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:hints = 1 2024-02-10T07:46:20.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-10T07:46:20.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Object prefix: benchmark_data_smithi008_161452 2024-02-10T07:46:20.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:46:20.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 0 0 0 0 0 0 - 0 2024-02-10T07:46:21.010 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:21.050 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:21.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:21.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:21.152 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:21.189 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:21.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:21.277 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:20 smithi008 ceph-mon[29816]: pgmap v652: 148 pgs: 19 active+clean+remapped, 15 active+undersized+degraded, 28 active+undersized, 86 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 57/594 objects degraded (9.596%); 31/594 objects misplaced (5.219%); 297 B/s, 8 objects/s recovering 2024-02-10T07:46:21.277 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:20 smithi008 ceph-mon[40743]: pgmap v652: 148 pgs: 19 active+clean+remapped, 15 active+undersized+degraded, 28 active+undersized, 86 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 57/594 objects degraded (9.596%); 31/594 objects misplaced (5.219%); 297 B/s, 8 objects/s recovering 2024-02-10T07:46:21.291 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:21.330 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:21.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:21.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:21.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:20 smithi170 ceph-mon[33394]: pgmap v652: 148 pgs: 19 active+clean+remapped, 15 active+undersized+degraded, 28 active+undersized, 86 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 57/594 objects degraded (9.596%); 31/594 objects misplaced (5.219%); 297 B/s, 8 objects/s recovering 2024-02-10T07:46:21.432 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:21.470 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:21.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:21.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:21.572 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:21.607 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:21.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:21.709 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:21.747 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:21.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:21.849 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:21.886 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:21.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:21.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 1 16 3483 3467 216.674 216.688 0.00503143 0.00459366 2024-02-10T07:46:21.987 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:22.027 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:22.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:22.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:22.129 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:22.164 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:22.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:22.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:22.240 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:21 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 57/594 objects degraded (9.596%), 15 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:22.240 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:21 smithi008 ceph-mon[40743]: osdmap e318: 8 total, 7 up, 7 in 2024-02-10T07:46:22.240 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:21 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1310439874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.4334606235648304}]: dispatch 2024-02-10T07:46:22.240 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:21 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.4334606235648304}]: dispatch 2024-02-10T07:46:22.240 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:21 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 57/594 objects degraded (9.596%), 15 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:22.240 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:21 smithi008 ceph-mon[29816]: osdmap e318: 8 total, 7 up, 7 in 2024-02-10T07:46:22.240 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:21 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1310439874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.4334606235648304}]: dispatch 2024-02-10T07:46:22.240 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:21 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.4334606235648304}]: dispatch 2024-02-10T07:46:22.266 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:22.305 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:22.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:22.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:21 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 57/594 objects degraded (9.596%), 15 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:21 smithi170 ceph-mon[33394]: osdmap e318: 8 total, 7 up, 7 in 2024-02-10T07:46:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:21 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1310439874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.4334606235648304}]: dispatch 2024-02-10T07:46:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:21 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.4334606235648304}]: dispatch 2024-02-10T07:46:22.406 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:22.438 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:22.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:22.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:22.540 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:22.578 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:22.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:22.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:22.680 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:22.719 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:22.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:22.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:22.821 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:22.862 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:22.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:22.943 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.3 primary-affinity to 0.433461 (8284072) 2024-02-10T07:46:22.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 2 15 6792 6777 211.763 206.875 0.00753969 0.00470445 2024-02-10T07:46:22.963 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:23.000 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:23.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:23.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:23.102 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:22 smithi170 ceph-mon[33394]: pgmap v654: 148 pgs: 19 active+clean+remapped, 15 active+undersized+degraded, 28 active+undersized, 86 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 57/594 objects degraded (9.596%); 31/594 objects misplaced (5.219%) 2024-02-10T07:46:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:22 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.4334606235648304}]': finished 2024-02-10T07:46:23.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:22 smithi170 ceph-mon[33394]: osdmap e319: 8 total, 7 up, 7 in 2024-02-10T07:46:23.140 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:23.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:23.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:23.154 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:46:23.154 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:46:23.154 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:22 smithi008 ceph-mon[29816]: pgmap v654: 148 pgs: 19 active+clean+remapped, 15 active+undersized+degraded, 28 active+undersized, 86 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 57/594 objects degraded (9.596%); 31/594 objects misplaced (5.219%) 2024-02-10T07:46:23.154 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:22 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.4334606235648304}]': finished 2024-02-10T07:46:23.154 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:22 smithi008 ceph-mon[29816]: osdmap e319: 8 total, 7 up, 7 in 2024-02-10T07:46:23.155 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:22 smithi008 ceph-mon[40743]: pgmap v654: 148 pgs: 19 active+clean+remapped, 15 active+undersized+degraded, 28 active+undersized, 86 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 57/594 objects degraded (9.596%); 31/594 objects misplaced (5.219%) 2024-02-10T07:46:23.155 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:22 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.4334606235648304}]': finished 2024-02-10T07:46:23.155 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:22 smithi008 ceph-mon[40743]: osdmap e319: 8 total, 7 up, 7 in 2024-02-10T07:46:23.242 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:23.280 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:23.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:23.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:23.382 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:23.418 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:23.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:23.519 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:23.553 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:23.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:23.655 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:23.692 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:23.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:23.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:23.794 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:23.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:23.833 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:23.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:23.845 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-10T07:46:23.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:46:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:46:23.934 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:23.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 3 16 9876 9860 205.4 192.688 0.00447559 0.0048586 2024-02-10T07:46:23.971 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:23.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:23.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:24.073 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:24.111 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:24.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:24.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:24.213 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:24.252 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:24.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:24.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:24.328 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:24.354 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:24.392 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:24.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:24.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:24.494 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:24.530 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:24.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:24.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:24.632 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:24.672 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:24.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:24.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:24.774 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:24.926 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:24.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:24.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 4 16 12785 12769 199.499 181.812 0.00837109 0.00500397 2024-02-10T07:46:25.027 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:25.060 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:25.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:25.162 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:25.198 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:25.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:25.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:25.300 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:46:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:46:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:25.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:46:25.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:46:25.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[29816]: osdmap e320: 8 total, 7 up, 7 in 2024-02-10T07:46:25.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[29816]: pgmap v657: 148 pgs: 19 active+clean+remapped, 16 active+undersized+degraded, 27 active+undersized, 86 active+clean; 293 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 49 MiB/s wr, 780 op/s; 499/14640 objects degraded (3.408%); 746/14640 objects misplaced (5.096%) 2024-02-10T07:46:25.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:46:25.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:46:25.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:46:25.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:46:25.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:25.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:46:25.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:46:25.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[40743]: osdmap e320: 8 total, 7 up, 7 in 2024-02-10T07:46:25.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[40743]: pgmap v657: 148 pgs: 19 active+clean+remapped, 16 active+undersized+degraded, 27 active+undersized, 86 active+clean; 293 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 49 MiB/s wr, 780 op/s; 499/14640 objects degraded (3.408%); 746/14640 objects misplaced (5.096%) 2024-02-10T07:46:25.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:46:25.335 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:25 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:46:25.337 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:25.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:25 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:46:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:25 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:46:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:25 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:25 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:46:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:25 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:46:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:25 smithi170 ceph-mon[33394]: osdmap e320: 8 total, 7 up, 7 in 2024-02-10T07:46:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:25 smithi170 ceph-mon[33394]: pgmap v657: 148 pgs: 19 active+clean+remapped, 16 active+undersized+degraded, 27 active+undersized, 86 active+clean; 293 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 49 MiB/s wr, 780 op/s; 499/14640 objects degraded (3.408%); 746/14640 objects misplaced (5.096%) 2024-02-10T07:46:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:25 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:46:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:25 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:46:25.440 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:25.478 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:25.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:25.580 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:25.617 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:25.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:25.718 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:25.756 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:25.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:25.858 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:25.895 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:25.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:25.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:25.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 5 16 15524 15508 193.834 171.188 0.00500062 0.00515112 2024-02-10T07:46:25.997 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:26.029 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:26.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:26.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:26.131 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:26.168 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:26.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:26.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:26.270 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:26.305 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:26.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:26.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:26 smithi008 ceph-mon[29816]: osdmap e321: 8 total, 7 up, 7 in 2024-02-10T07:46:26.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:26 smithi008 ceph-mon[40743]: osdmap e321: 8 total, 7 up, 7 in 2024-02-10T07:46:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:26 smithi170 ceph-mon[33394]: osdmap e321: 8 total, 7 up, 7 in 2024-02-10T07:46:26.407 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:26.444 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:26.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:26.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:26.473 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:26.546 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:26.583 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:26.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:26.685 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:26.721 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:26.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:26.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:26.823 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:26.860 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:26.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:26.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:26.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 6 16 18019 18003 187.516 155.938 0.00902108 0.00532549 2024-02-10T07:46:26.961 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:26.998 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:26.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:27.027 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:46:27.099 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:27.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:27.211 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:27.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:27.312 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:27.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:27 smithi008 ceph-mon[29816]: pgmap v659: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 454 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s; 1121/22335 objects degraded (5.019%); 1369/22335 objects misplaced (6.129%) 2024-02-10T07:46:27.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:27 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3703107298' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:46:27.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:27 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:46:27.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:27 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 499/14640 objects degraded (3.408%), 16 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:27 smithi008 ceph-mon[40743]: pgmap v659: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 454 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s; 1121/22335 objects degraded (5.019%); 1369/22335 objects misplaced (6.129%) 2024-02-10T07:46:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:27 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3703107298' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:46:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:27 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:46:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:27 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 499/14640 objects degraded (3.408%), 16 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:27.349 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:27.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:27.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:27 smithi170 ceph-mon[33394]: pgmap v659: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 454 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s; 1121/22335 objects degraded (5.019%); 1369/22335 objects misplaced (6.129%) 2024-02-10T07:46:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:27 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3703107298' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:46:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:27 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:46:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:27 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 499/14640 objects degraded (3.408%), 16 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:27.451 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:27.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:27.491 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:27.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:27.592 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:27.630 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:27.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:27.731 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:27.764 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:27.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:27.865 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:27.904 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:27.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:27.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:27.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 7 16 20415 20399 182.119 149.75 0.00990269 0.00548527 2024-02-10T07:46:28.007 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:28.044 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:28.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:28.147 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:28.185 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:28.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:28.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:28.190 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:28 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:46:28.190 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:28 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:46:28.190 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:28 smithi008 ceph-mon[29816]: osdmap e322: 8 total, 7 up, 7 in 2024-02-10T07:46:28.190 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:28 smithi008 ceph-mon[29816]: 4.1c scrub starts 2024-02-10T07:46:28.190 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:28 smithi008 ceph-mon[29816]: 4.1c scrub ok 2024-02-10T07:46:28.191 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:28 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:46:28.191 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:28 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:46:28.191 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:28 smithi008 ceph-mon[40743]: osdmap e322: 8 total, 7 up, 7 in 2024-02-10T07:46:28.191 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:28 smithi008 ceph-mon[40743]: 4.1c scrub starts 2024-02-10T07:46:28.191 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:28 smithi008 ceph-mon[40743]: 4.1c scrub ok 2024-02-10T07:46:28.288 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:28.320 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:28.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:28 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:46:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:28 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:46:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:28 smithi170 ceph-mon[33394]: osdmap e322: 8 total, 7 up, 7 in 2024-02-10T07:46:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:28 smithi170 ceph-mon[33394]: 4.1c scrub starts 2024-02-10T07:46:28.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:28 smithi170 ceph-mon[33394]: 4.1c scrub ok 2024-02-10T07:46:28.422 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:28.455 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:28.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:28.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:28.557 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:28.594 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:28.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:28.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:28.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:28.732 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:28.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:28.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:28.834 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:28.866 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:28.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:28.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 8 16 22803 22787 178.009 149.25 0.00722099 0.00561219 2024-02-10T07:46:28.968 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:29.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:29.006 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:29.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:29.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:29 smithi008 ceph-mon[29816]: 6.18 scrub starts 2024-02-10T07:46:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:29 smithi008 ceph-mon[29816]: 6.18 scrub ok 2024-02-10T07:46:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:29 smithi008 ceph-mon[29816]: 6.2 scrub starts 2024-02-10T07:46:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:29 smithi008 ceph-mon[29816]: 6.2 scrub ok 2024-02-10T07:46:29.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:29 smithi008 ceph-mon[29816]: pgmap v661: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.1 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 2.0 KiB/s rd, 205 MiB/s wr, 3.28k op/s; 1983/53184 objects degraded (3.729%); 3064/53184 objects misplaced (5.761%) 2024-02-10T07:46:29.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:46:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:46:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:46:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:29 smithi008 ceph-mon[40743]: 6.18 scrub starts 2024-02-10T07:46:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:29 smithi008 ceph-mon[40743]: 6.18 scrub ok 2024-02-10T07:46:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:29 smithi008 ceph-mon[40743]: 6.2 scrub starts 2024-02-10T07:46:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:29 smithi008 ceph-mon[40743]: 6.2 scrub ok 2024-02-10T07:46:29.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:29 smithi008 ceph-mon[40743]: pgmap v661: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.1 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 2.0 KiB/s rd, 205 MiB/s wr, 3.28k op/s; 1983/53184 objects degraded (3.729%); 3064/53184 objects misplaced (5.761%) 2024-02-10T07:46:29.107 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:29.144 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:29.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:29.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:29.246 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:29.282 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:29.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:29.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:29.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:29 smithi170 ceph-mon[33394]: 6.18 scrub starts 2024-02-10T07:46:29.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:29 smithi170 ceph-mon[33394]: 6.18 scrub ok 2024-02-10T07:46:29.283 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:29 smithi170 ceph-mon[33394]: 6.2 scrub starts 2024-02-10T07:46:29.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:29 smithi170 ceph-mon[33394]: 6.2 scrub ok 2024-02-10T07:46:29.284 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:29 smithi170 ceph-mon[33394]: pgmap v661: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.1 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 2.0 KiB/s rd, 205 MiB/s wr, 3.28k op/s; 1983/53184 objects degraded (3.729%); 3064/53184 objects misplaced (5.761%) 2024-02-10T07:46:29.384 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:29.423 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:29.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:29.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:29.525 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:29.554 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:29.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:29.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:29.656 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:29.685 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:29.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:29.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:29.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:29.825 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:29.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:29.890 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [3] live_osds: [0, 1, 2, 4, 6, 7, 5] 2024-02-10T07:46:29.890 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:46:29.890 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.12943885521271156 2024-02-10T07:46:29.890 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 3 0.12943885521271156 2024-02-10T07:46:29.925 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:29.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 9 16 24791 24775 172.035 124.25 0.0136896 0.00580395 2024-02-10T07:46:29.963 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:29.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:30.065 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:30.102 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:30.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:30.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:30.204 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:30.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:30.244 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:30.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:30.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[29816]: 4.3 scrub starts 2024-02-10T07:46:30.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[29816]: 4.3 scrub ok 2024-02-10T07:46:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[29816]: 5.a scrub starts 2024-02-10T07:46:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[29816]: 5.a scrub ok 2024-02-10T07:46:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[29816]: 3.6 scrub starts 2024-02-10T07:46:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[29816]: 3.6 scrub ok 2024-02-10T07:46:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[29816]: 4.4 scrub starts 2024-02-10T07:46:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[29816]: 4.4 scrub ok 2024-02-10T07:46:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:46:30.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:46:30.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[40743]: 4.3 scrub starts 2024-02-10T07:46:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[40743]: 4.3 scrub ok 2024-02-10T07:46:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[40743]: 5.a scrub starts 2024-02-10T07:46:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[40743]: 5.a scrub ok 2024-02-10T07:46:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[40743]: 3.6 scrub starts 2024-02-10T07:46:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[40743]: 3.6 scrub ok 2024-02-10T07:46:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[40743]: 4.4 scrub starts 2024-02-10T07:46:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[40743]: 4.4 scrub ok 2024-02-10T07:46:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:46:30.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:30 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:46:30.345 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:30.376 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:30.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:30.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:30 smithi170 ceph-mon[33394]: 4.3 scrub starts 2024-02-10T07:46:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:30 smithi170 ceph-mon[33394]: 4.3 scrub ok 2024-02-10T07:46:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:30 smithi170 ceph-mon[33394]: 5.a scrub starts 2024-02-10T07:46:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:30 smithi170 ceph-mon[33394]: 5.a scrub ok 2024-02-10T07:46:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:30 smithi170 ceph-mon[33394]: 3.6 scrub starts 2024-02-10T07:46:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:30 smithi170 ceph-mon[33394]: 3.6 scrub ok 2024-02-10T07:46:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:30 smithi170 ceph-mon[33394]: 4.4 scrub starts 2024-02-10T07:46:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:30 smithi170 ceph-mon[33394]: 4.4 scrub ok 2024-02-10T07:46:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:30 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:46:30.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:30 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:46:30.478 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:30.509 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:30.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:30.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:30.610 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:30.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:30.646 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:30.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:30.747 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:30.784 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:30.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:30.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:30.885 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:30.922 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:30.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:30.944 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 10 16 26722 26706 166.899 120.688 0.0066036 0.00598418 2024-02-10T07:46:31.023 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:31.062 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:31.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:31.164 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:31.190 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:31.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:31.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:31.292 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:31.322 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:31.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:31.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:31.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:31 smithi008 ceph-mon[29816]: pgmap v662: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.3 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 849 B/s rd, 167 MiB/s wr, 2.67k op/s; 2802/62979 objects degraded (4.449%); 3912/62979 objects misplaced (6.212%) 2024-02-10T07:46:31.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:31 smithi008 ceph-mon[29816]: osdmap e323: 8 total, 7 up, 7 in 2024-02-10T07:46:31.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:31 smithi008 ceph-mon[29816]: 4.7 scrub starts 2024-02-10T07:46:31.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:31 smithi008 ceph-mon[29816]: 4.7 scrub ok 2024-02-10T07:46:31.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:31 smithi008 ceph-mon[40743]: pgmap v662: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.3 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 849 B/s rd, 167 MiB/s wr, 2.67k op/s; 2802/62979 objects degraded (4.449%); 3912/62979 objects misplaced (6.212%) 2024-02-10T07:46:31.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:31 smithi008 ceph-mon[40743]: osdmap e323: 8 total, 7 up, 7 in 2024-02-10T07:46:31.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:31 smithi008 ceph-mon[40743]: 4.7 scrub starts 2024-02-10T07:46:31.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:31 smithi008 ceph-mon[40743]: 4.7 scrub ok 2024-02-10T07:46:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:31 smithi170 ceph-mon[33394]: pgmap v662: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.3 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 849 B/s rd, 167 MiB/s wr, 2.67k op/s; 2802/62979 objects degraded (4.449%); 3912/62979 objects misplaced (6.212%) 2024-02-10T07:46:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:31 smithi170 ceph-mon[33394]: osdmap e323: 8 total, 7 up, 7 in 2024-02-10T07:46:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:31 smithi170 ceph-mon[33394]: 4.7 scrub starts 2024-02-10T07:46:31.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:31 smithi170 ceph-mon[33394]: 4.7 scrub ok 2024-02-10T07:46:31.424 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:31.461 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:31.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:31.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:31.563 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:31.594 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:31.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:31.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:31.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:31.734 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:31.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:31.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:31.836 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:31.872 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:31.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:31.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:31.945 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 11 16 28580 28564 162.282 116.125 0.00951876 0.00615551 2024-02-10T07:46:31.974 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:32.003 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:32.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:32.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:32.105 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:32.142 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:32.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:32.244 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:32.273 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:46:32.274 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:46:32.274 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:32 smithi008 ceph-mon[29816]: 5.4 scrub starts 2024-02-10T07:46:32.274 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:32 smithi008 ceph-mon[29816]: 5.4 scrub ok 2024-02-10T07:46:32.274 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:32 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 2802/62979 objects degraded (4.449%), 17 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:32.274 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:32 smithi008 ceph-mon[40743]: 5.4 scrub starts 2024-02-10T07:46:32.274 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:32 smithi008 ceph-mon[40743]: 5.4 scrub ok 2024-02-10T07:46:32.274 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:32 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 2802/62979 objects degraded (4.449%), 17 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:32.279 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:32.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:32.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:32.381 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:32 smithi170 ceph-mon[33394]: 5.4 scrub starts 2024-02-10T07:46:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:32 smithi170 ceph-mon[33394]: 5.4 scrub ok 2024-02-10T07:46:32.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:32 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 2802/62979 objects degraded (4.449%), 17 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:32.421 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:32.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:32.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:32.523 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:32.560 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:32.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:32.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:32.566 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:46:32.601 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:46:32.633 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:46:32.662 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:32.673 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:46:32.702 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:32.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:32.706 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:46:32.739 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:46:32.776 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:46:32.776 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:46:32.776 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:46:32.776 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:46:32.776 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-10T07:46:32.776 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:46:32.776 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:46:32.776 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:46:32.776 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:46:32.804 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:32.837 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:32.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:32.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:32.939 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:32.945 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 12 16 30555 30539 159.044 123.438 0.00465486 0.00628139 2024-02-10T07:46:32.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:32.978 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:32.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:33.079 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:33.116 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:33.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:33.217 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:33.255 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:33.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:33.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[29816]: 3.1e scrub starts 2024-02-10T07:46:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[29816]: 3.1e scrub ok 2024-02-10T07:46:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[29816]: 6.8 scrub starts 2024-02-10T07:46:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[29816]: 6.8 scrub ok 2024-02-10T07:46:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[29816]: pgmap v664: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.3 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 729 B/s rd, 121 MiB/s wr, 1.93k op/s; 2802/62979 objects degraded (4.449%); 3912/62979 objects misplaced (6.212%) 2024-02-10T07:46:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[29816]: 5.1d scrub starts 2024-02-10T07:46:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[29816]: 5.1d scrub ok 2024-02-10T07:46:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3978591322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:46:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:46:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[40743]: 3.1e scrub starts 2024-02-10T07:46:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[40743]: 3.1e scrub ok 2024-02-10T07:46:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[40743]: 6.8 scrub starts 2024-02-10T07:46:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[40743]: 6.8 scrub ok 2024-02-10T07:46:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[40743]: pgmap v664: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.3 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 729 B/s rd, 121 MiB/s wr, 1.93k op/s; 2802/62979 objects degraded (4.449%); 3912/62979 objects misplaced (6.212%) 2024-02-10T07:46:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[40743]: 5.1d scrub starts 2024-02-10T07:46:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[40743]: 5.1d scrub ok 2024-02-10T07:46:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3978591322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:46:33.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:46:33.356 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:33 smithi170 ceph-mon[33394]: 3.1e scrub starts 2024-02-10T07:46:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:33 smithi170 ceph-mon[33394]: 3.1e scrub ok 2024-02-10T07:46:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:33 smithi170 ceph-mon[33394]: 6.8 scrub starts 2024-02-10T07:46:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:33 smithi170 ceph-mon[33394]: 6.8 scrub ok 2024-02-10T07:46:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:33 smithi170 ceph-mon[33394]: pgmap v664: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.3 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 729 B/s rd, 121 MiB/s wr, 1.93k op/s; 2802/62979 objects degraded (4.449%); 3912/62979 objects misplaced (6.212%) 2024-02-10T07:46:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:33 smithi170 ceph-mon[33394]: 5.1d scrub starts 2024-02-10T07:46:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:33 smithi170 ceph-mon[33394]: 5.1d scrub ok 2024-02-10T07:46:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:33 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3978591322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:46:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:46:33.387 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:33.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:33.437 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:46:33.489 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:33.526 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:33.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:33.628 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:33.666 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:33.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:33.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:33.767 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:33.798 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:33.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:33.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:46:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:46:33.900 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:33.908 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-10T07:46:33.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:33.937 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:33.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:33.945 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 13 16 32479 32463 156.059 120.25 0.015769 0.00640212 2024-02-10T07:46:34.038 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:34.074 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:34.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:34.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:34.176 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:34.214 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:34.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:34.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:34.235 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:34 smithi008 ceph-mon[29816]: 4.a scrub starts 2024-02-10T07:46:34.235 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:34 smithi008 ceph-mon[29816]: 4.a scrub ok 2024-02-10T07:46:34.235 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:34 smithi008 ceph-mon[29816]: 5.14 scrub starts 2024-02-10T07:46:34.236 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:34 smithi008 ceph-mon[29816]: 5.14 scrub ok 2024-02-10T07:46:34.236 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:34 smithi008 ceph-mon[29816]: 3.0 scrub starts 2024-02-10T07:46:34.236 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:34 smithi008 ceph-mon[29816]: 3.0 scrub ok 2024-02-10T07:46:34.236 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:34 smithi008 ceph-mon[40743]: 4.a scrub starts 2024-02-10T07:46:34.236 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:34 smithi008 ceph-mon[40743]: 4.a scrub ok 2024-02-10T07:46:34.236 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:34 smithi008 ceph-mon[40743]: 5.14 scrub starts 2024-02-10T07:46:34.236 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:34 smithi008 ceph-mon[40743]: 5.14 scrub ok 2024-02-10T07:46:34.236 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:34 smithi008 ceph-mon[40743]: 3.0 scrub starts 2024-02-10T07:46:34.236 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:34 smithi008 ceph-mon[40743]: 3.0 scrub ok 2024-02-10T07:46:34.316 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:34.346 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:34.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:34.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:34 smithi170 ceph-mon[33394]: 4.a scrub starts 2024-02-10T07:46:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:34 smithi170 ceph-mon[33394]: 4.a scrub ok 2024-02-10T07:46:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:34 smithi170 ceph-mon[33394]: 5.14 scrub starts 2024-02-10T07:46:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:34 smithi170 ceph-mon[33394]: 5.14 scrub ok 2024-02-10T07:46:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:34 smithi170 ceph-mon[33394]: 3.0 scrub starts 2024-02-10T07:46:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:34 smithi170 ceph-mon[33394]: 3.0 scrub ok 2024-02-10T07:46:34.448 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:34.488 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:34.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:34.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:34.590 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:34.626 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:34.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:34.694 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:34.697 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:46:34.701 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:46:34.728 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:34.764 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:34.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:34.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:34.866 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:34.904 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:34.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:34.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:34.945 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 14 15 34398 34383 153.483 120 0.0339396 0.00650887 2024-02-10T07:46:35.006 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:35.029 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:35.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:35.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:35.131 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:35.165 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:35.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:35.267 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:35.307 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:35.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:35.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:35.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:35 smithi008 ceph-mon[29816]: 3.5 scrub starts 2024-02-10T07:46:35.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:35 smithi008 ceph-mon[29816]: 3.5 scrub ok 2024-02-10T07:46:35.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:35 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:35.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:35 smithi008 ceph-mon[29816]: pgmap v665: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 166 MiB/s wr, 2.65k op/s; 3459/85923 objects degraded (4.026%); 5210/85923 objects misplaced (6.064%) 2024-02-10T07:46:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:35 smithi008 ceph-mon[40743]: 3.5 scrub starts 2024-02-10T07:46:35.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:35 smithi008 ceph-mon[40743]: 3.5 scrub ok 2024-02-10T07:46:35.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:35 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:35.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:35 smithi008 ceph-mon[40743]: pgmap v665: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 166 MiB/s wr, 2.65k op/s; 3459/85923 objects degraded (4.026%); 5210/85923 objects misplaced (6.064%) 2024-02-10T07:46:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:35 smithi170 ceph-mon[33394]: 3.5 scrub starts 2024-02-10T07:46:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:35 smithi170 ceph-mon[33394]: 3.5 scrub ok 2024-02-10T07:46:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:35 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:35 smithi170 ceph-mon[33394]: pgmap v665: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.7 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 166 MiB/s wr, 2.65k op/s; 3459/85923 objects degraded (4.026%); 5210/85923 objects misplaced (6.064%) 2024-02-10T07:46:35.408 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:35.440 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:35.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:35.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:35.542 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:35.573 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:35.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:35.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:35.675 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:35.705 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:35.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:35.807 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:35.845 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:35.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:35.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:35.945 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 15 16 36175 36159 150.65 111 0.00990304 0.0066322 2024-02-10T07:46:35.947 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:35.979 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:35.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:35.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:36.081 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:36.117 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:36.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:36.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:36.218 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:36.257 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:36.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:36.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:36 smithi008 ceph-mon[29816]: 3.16 scrub starts 2024-02-10T07:46:36.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:36 smithi008 ceph-mon[29816]: 3.16 scrub ok 2024-02-10T07:46:36.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:36 smithi008 ceph-mon[29816]: 6.15 scrub starts 2024-02-10T07:46:36.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:36 smithi008 ceph-mon[29816]: 6.15 scrub ok 2024-02-10T07:46:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:36 smithi008 ceph-mon[40743]: 3.16 scrub starts 2024-02-10T07:46:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:36 smithi008 ceph-mon[40743]: 3.16 scrub ok 2024-02-10T07:46:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:36 smithi008 ceph-mon[40743]: 6.15 scrub starts 2024-02-10T07:46:36.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:36 smithi008 ceph-mon[40743]: 6.15 scrub ok 2024-02-10T07:46:36.358 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:36 smithi170 ceph-mon[33394]: 3.16 scrub starts 2024-02-10T07:46:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:36 smithi170 ceph-mon[33394]: 3.16 scrub ok 2024-02-10T07:46:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:36 smithi170 ceph-mon[33394]: 6.15 scrub starts 2024-02-10T07:46:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:36 smithi170 ceph-mon[33394]: 6.15 scrub ok 2024-02-10T07:46:36.396 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:36.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:36.499 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:36.529 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:36.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:36.631 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:36.662 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:36.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:36.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:36.702 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:46:36.765 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:36.803 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:36.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:36.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:36.904 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:36.940 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:36.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:36.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:36.945 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 16 16 38031 38015 148.484 116 0.0119872 0.00672572 2024-02-10T07:46:37.041 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:37.074 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:37.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:37.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:37.162 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:37.176 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:37.214 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:37.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:37.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:37.221 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:37.315 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:37.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:37 smithi008 ceph-mon[29816]: 3.f scrub starts 2024-02-10T07:46:37.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:37 smithi008 ceph-mon[29816]: 3.f scrub ok 2024-02-10T07:46:37.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:37 smithi008 ceph-mon[29816]: pgmap v666: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 164 MiB/s wr, 2.62k op/s; 4053/93120 objects degraded (4.352%); 5815/93120 objects misplaced (6.245%) 2024-02-10T07:46:37.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:37 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 4053/93120 objects degraded (4.352%), 17 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:37 smithi008 ceph-mon[40743]: 3.f scrub starts 2024-02-10T07:46:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:37 smithi008 ceph-mon[40743]: 3.f scrub ok 2024-02-10T07:46:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:37 smithi008 ceph-mon[40743]: pgmap v666: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 164 MiB/s wr, 2.62k op/s; 4053/93120 objects degraded (4.352%); 5815/93120 objects misplaced (6.245%) 2024-02-10T07:46:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:37 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 4053/93120 objects degraded (4.352%), 17 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:37.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:37.346 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:37.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:37 smithi170 ceph-mon[33394]: 3.f scrub starts 2024-02-10T07:46:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:37 smithi170 ceph-mon[33394]: 3.f scrub ok 2024-02-10T07:46:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:37 smithi170 ceph-mon[33394]: pgmap v666: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 1.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 164 MiB/s wr, 2.62k op/s; 4053/93120 objects degraded (4.352%); 5815/93120 objects misplaced (6.245%) 2024-02-10T07:46:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:37 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 4053/93120 objects degraded (4.352%), 17 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:37.447 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:37.477 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:37.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:37.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:37.579 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:37.615 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:37.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:37.716 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:37.756 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:37.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:37.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:37.858 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:37.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:37.892 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:37.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:37.945 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 17 16 39871 39855 146.514 115 0.00401325 0.00681994 2024-02-10T07:46:37.992 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:38.028 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:38.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:38.130 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:38.166 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:38.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:38.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:38.210 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:38 smithi008 ceph-mon[29816]: 3.13 scrub starts 2024-02-10T07:46:38.210 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:38 smithi008 ceph-mon[29816]: 3.13 scrub ok 2024-02-10T07:46:38.210 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:38 smithi008 ceph-mon[29816]: 4.18 scrub starts 2024-02-10T07:46:38.210 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:38 smithi008 ceph-mon[29816]: 4.18 scrub ok 2024-02-10T07:46:38.210 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:38 smithi008 ceph-mon[40743]: 3.13 scrub starts 2024-02-10T07:46:38.210 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:38 smithi008 ceph-mon[40743]: 3.13 scrub ok 2024-02-10T07:46:38.210 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:38 smithi008 ceph-mon[40743]: 4.18 scrub starts 2024-02-10T07:46:38.210 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:38 smithi008 ceph-mon[40743]: 4.18 scrub ok 2024-02-10T07:46:38.268 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:38.309 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:38.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:38.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:38 smithi170 ceph-mon[33394]: 3.13 scrub starts 2024-02-10T07:46:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:38 smithi170 ceph-mon[33394]: 3.13 scrub ok 2024-02-10T07:46:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:38 smithi170 ceph-mon[33394]: 4.18 scrub starts 2024-02-10T07:46:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:38 smithi170 ceph-mon[33394]: 4.18 scrub ok 2024-02-10T07:46:38.411 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:38.451 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:38.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:38.553 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:38.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:38.590 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:38.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:38.691 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:38.728 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:38.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:38.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:38.829 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:38.866 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:38.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:38.945 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 18 16 41657 41641 144.575 111.625 0.00694641 0.00691166 2024-02-10T07:46:38.969 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:39.007 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:39.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:39.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:39.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:46:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:46:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:46:39.109 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:39.110 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.3 to 0.129439 (2122) 2024-02-10T07:46:39.151 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:39.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:39.253 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:39.276 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:39.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:39.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:39.378 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:39 smithi170 ceph-mon[33394]: 4.1f scrub starts 2024-02-10T07:46:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:39 smithi170 ceph-mon[33394]: 4.1f scrub ok 2024-02-10T07:46:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:39 smithi170 ceph-mon[33394]: 3.9 scrub starts 2024-02-10T07:46:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:39 smithi170 ceph-mon[33394]: 3.9 scrub ok 2024-02-10T07:46:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:39 smithi170 ceph-mon[33394]: pgmap v667: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.04k op/s; 4634/114255 objects degraded (4.056%); 6983/114255 objects misplaced (6.112%) 2024-02-10T07:46:39.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:39 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4082689174' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.12943885521271156}]: dispatch 2024-02-10T07:46:39.401 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:39 smithi008 ceph-mon[29816]: 4.1f scrub starts 2024-02-10T07:46:39.401 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:39 smithi008 ceph-mon[29816]: 4.1f scrub ok 2024-02-10T07:46:39.401 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:39 smithi008 ceph-mon[29816]: 3.9 scrub starts 2024-02-10T07:46:39.401 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:39 smithi008 ceph-mon[29816]: 3.9 scrub ok 2024-02-10T07:46:39.401 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:39 smithi008 ceph-mon[29816]: pgmap v667: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.04k op/s; 4634/114255 objects degraded (4.056%); 6983/114255 objects misplaced (6.112%) 2024-02-10T07:46:39.401 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:39 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4082689174' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.12943885521271156}]: dispatch 2024-02-10T07:46:39.401 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:39 smithi008 ceph-mon[40743]: 4.1f scrub starts 2024-02-10T07:46:39.401 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:39 smithi008 ceph-mon[40743]: 4.1f scrub ok 2024-02-10T07:46:39.401 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:39 smithi008 ceph-mon[40743]: 3.9 scrub starts 2024-02-10T07:46:39.402 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:39 smithi008 ceph-mon[40743]: 3.9 scrub ok 2024-02-10T07:46:39.402 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:39 smithi008 ceph-mon[40743]: pgmap v667: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.04k op/s; 4634/114255 objects degraded (4.056%); 6983/114255 objects misplaced (6.112%) 2024-02-10T07:46:39.402 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:39 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4082689174' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.12943885521271156}]: dispatch 2024-02-10T07:46:39.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:39.403 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:39.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:39.504 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:39.527 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:39.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:39.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:39.629 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:39.653 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:39.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:39.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:39.755 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:39.779 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:39.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:39.881 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:39.904 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:39.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:39.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:39.945 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 19 16 41991 41975 138.064 20.875 0.0018354 0.0069328 2024-02-10T07:46:40.006 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:40.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:40.030 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:40.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:40.108 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:46:40.130 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:40.156 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:40.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:40.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:40.258 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:40.281 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:40.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:40.382 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:40.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:40 smithi170 ceph-mon[33394]: 5.d scrub starts 2024-02-10T07:46:40.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:40 smithi170 ceph-mon[33394]: 5.d scrub ok 2024-02-10T07:46:40.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:40 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4082689174' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.12943885521271156}]': finished 2024-02-10T07:46:40.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:40 smithi170 ceph-mon[33394]: osdmap e324: 8 total, 7 up, 7 in 2024-02-10T07:46:40.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:40 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3931702580' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:46:40.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:40 smithi170 ceph-mon[33394]: 6.1 scrub starts 2024-02-10T07:46:40.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:40 smithi170 ceph-mon[33394]: 6.1 scrub ok 2024-02-10T07:46:40.410 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:40.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:40.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:40 smithi008 ceph-mon[29816]: 5.d scrub starts 2024-02-10T07:46:40.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:40 smithi008 ceph-mon[29816]: 5.d scrub ok 2024-02-10T07:46:40.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:40 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4082689174' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.12943885521271156}]': finished 2024-02-10T07:46:40.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:40 smithi008 ceph-mon[29816]: osdmap e324: 8 total, 7 up, 7 in 2024-02-10T07:46:40.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:40 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3931702580' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:46:40.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:40 smithi008 ceph-mon[29816]: 6.1 scrub starts 2024-02-10T07:46:40.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:40 smithi008 ceph-mon[29816]: 6.1 scrub ok 2024-02-10T07:46:40.470 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:40 smithi008 ceph-mon[40743]: 5.d scrub starts 2024-02-10T07:46:40.470 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:40 smithi008 ceph-mon[40743]: 5.d scrub ok 2024-02-10T07:46:40.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:40 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4082689174' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.12943885521271156}]': finished 2024-02-10T07:46:40.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:40 smithi008 ceph-mon[40743]: osdmap e324: 8 total, 7 up, 7 in 2024-02-10T07:46:40.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:40 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3931702580' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:46:40.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:40 smithi008 ceph-mon[40743]: 6.1 scrub starts 2024-02-10T07:46:40.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:40 smithi008 ceph-mon[40743]: 6.1 scrub ok 2024-02-10T07:46:40.511 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:40.536 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:40.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:40.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:40.638 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:40.662 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:40.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:40.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:40.764 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:40.788 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:40.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:40.890 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:40.915 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:40.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:40.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:40.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:46:40.946451+0000 min lat: 0.00147754 max lat: 0.110166 avg lat: 0.0069328 2024-02-10T07:46:40.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:46:40.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 20 16 41991 41975 131.161 0 - 0.0069328 2024-02-10T07:46:41.017 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:41.043 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:41.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:41.145 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:41.177 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:41.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:41.279 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:41.320 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:41.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:41 smithi170 ceph-mon[33394]: 5.6 scrub starts 2024-02-10T07:46:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:41 smithi170 ceph-mon[33394]: 5.6 scrub ok 2024-02-10T07:46:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:41 smithi170 ceph-mon[33394]: pgmap v669: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 5196/121158 objects degraded (4.289%); 7551/121158 objects misplaced (6.232%) 2024-02-10T07:46:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:41 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:46:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:41 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3931702580' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:46:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:41 smithi170 ceph-mon[33394]: osdmap e325: 8 total, 7 up, 7 in 2024-02-10T07:46:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:41 smithi170 ceph-mon[33394]: 3.12 scrub starts 2024-02-10T07:46:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:41 smithi170 ceph-mon[33394]: 3.12 scrub ok 2024-02-10T07:46:41.422 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:41.444 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[29816]: 5.6 scrub starts 2024-02-10T07:46:41.444 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[29816]: 5.6 scrub ok 2024-02-10T07:46:41.444 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[29816]: pgmap v669: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 5196/121158 objects degraded (4.289%); 7551/121158 objects misplaced (6.232%) 2024-02-10T07:46:41.444 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:46:41.444 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3931702580' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:46:41.444 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[29816]: osdmap e325: 8 total, 7 up, 7 in 2024-02-10T07:46:41.444 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[29816]: 3.12 scrub starts 2024-02-10T07:46:41.444 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[29816]: 3.12 scrub ok 2024-02-10T07:46:41.445 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[40743]: 5.6 scrub starts 2024-02-10T07:46:41.445 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[40743]: 5.6 scrub ok 2024-02-10T07:46:41.445 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[40743]: pgmap v669: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 5196/121158 objects degraded (4.289%); 7551/121158 objects misplaced (6.232%) 2024-02-10T07:46:41.445 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:46:41.445 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3931702580' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:46:41.445 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[40743]: osdmap e325: 8 total, 7 up, 7 in 2024-02-10T07:46:41.445 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[40743]: 3.12 scrub starts 2024-02-10T07:46:41.445 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:41 smithi008 ceph-mon[40743]: 3.12 scrub ok 2024-02-10T07:46:41.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:41.463 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:41.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:41.564 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:41.603 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:41.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:41.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:41.740 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:41.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:41.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:41.842 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:41.876 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:41.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:41.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:41.945 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 21 16 42809 42793 127.35 25.5625 0.0144721 0.00784449 2024-02-10T07:46:41.978 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:41.990 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:46:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-02-10T07:46:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-02-10T07:46:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:46:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.24740.0:212345 7.f 7:f80fa090:::benchmark_data_smithi008_82046_object97207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-10T07:46:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.578926+0000", 2024-02-10T07:46:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 544.40789823900002, 2024-02-10T07:46:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24282195200000001, 2024-02-10T07:46:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:41.990 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.24740", 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/3827575830", 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 212345 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578926+0000", 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578926+0000", 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:41.991 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578928+0000", 2024-02-10T07:46:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.486e-06 2024-02-10T07:46:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578932+0000", 2024-02-10T07:46:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2699999999999998e-06 2024-02-10T07:46:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578945+0000", 2024-02-10T07:46:41.992 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2703e-05 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.578954+0000", 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8710000000000003e-06 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.579336+0000", 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00038255500000000003 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.579461+0000", 2024-02-10T07:46:41.993 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000124621 2024-02-10T07:46:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:46:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.579564+0000", 2024-02-10T07:46:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000103317 2024-02-10T07:46:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:46:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.582702+0000", 2024-02-10T07:46:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0031373400000000002 2024-02-10T07:46:41.994 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583326+0000", 2024-02-10T07:46:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00062478000000000002 2024-02-10T07:46:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821642+0000", 2024-02-10T07:46:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.238315844 2024-02-10T07:46:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821699+0000", 2024-02-10T07:46:41.995 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6635000000000002e-05 2024-02-10T07:46:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821748+0000", 2024-02-10T07:46:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9530000000000002e-05 2024-02-10T07:46:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212347 7.12 e187/185 7:4b6dc533:::benchmark_data_smithi008_82046_object97209:head v 187'10763, mlcod=187'10761)", 2024-02-10T07:46:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.583166+0000", 2024-02-10T07:46:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 544.40365853499998, 2024-02-10T07:46:41.996 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24110515900000001, 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583166+0000", 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583166+0000", 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.997 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583167+0000", 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1299999999999999e-07 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583172+0000", 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6740000000000002e-06 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583182+0000", 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8130000000000007e-06 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.998 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583189+0000", 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9140000000000002e-06 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583624+0000", 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00043515399999999999 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583646+0000", 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1474000000000001e-05 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:41.999 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.824236+0000", 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.24059001699999999 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.824271+0000", 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.54e-05 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212349 7.e e187/166 7:7df8d3ee:::benchmark_data_smithi008_82046_object97211:head v 187'13215, mlcod=187'13213)", 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.583663+0000", 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 544.403161264, 2024-02-10T07:46:42.000 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23797284099999999, 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583663+0000", 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583663+0000", 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.001 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583664+0000", 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3900000000000004e-07 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583670+0000", 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7790000000000001e-06 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583680+0000", 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7419999999999992e-06 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.002 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.583686+0000", 2024-02-10T07:46:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5740000000000001e-06 2024-02-10T07:46:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.584048+0000", 2024-02-10T07:46:42.003 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00036165999999999998 2024-02-10T07:46:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.584076+0000", 2024-02-10T07:46:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8121000000000001e-05 2024-02-10T07:46:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821584+0000", 2024-02-10T07:46:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.237508318 2024-02-10T07:46:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.004 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821636+0000", 2024-02-10T07:46:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1907999999999998e-05 2024-02-10T07:46:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop_reply(client.24740.0:212345 7.f e187/98 ondisk, result = 0)", 2024-02-10T07:46:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.585629+0000", 2024-02-10T07:46:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 544.40119592799999, 2024-02-10T07:46:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23612559999999999, 2024-02-10T07:46:42.005 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585629+0000", 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585629+0000", 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.006 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585630+0000", 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0999999999999997e-07 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585635+0000", 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1050000000000003e-06 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585645+0000", 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0484e-05 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.585652+0000", 2024-02-10T07:46:42.007 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2570000000000001e-06 2024-02-10T07:46:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821625+0000", 2024-02-10T07:46:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23597299399999999 2024-02-10T07:46:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821635+0000", 2024-02-10T07:46:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9240000000000002e-06 2024-02-10T07:46:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.008 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821754+0000", 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011892600000000001 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDPGPushReply(7.14 187/156 [PushReplyOp(7:2f8c8aed:::benchmark_data_smithi008_82046_object99852:head)])", 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 544.39863448599999, 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23338334399999999, 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "reached pg", 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.009 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588190+0000", 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2499999999999999e-07 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.010 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588193+0000", 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.881e-06 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588200+0000", 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.703e-06 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588206+0000", 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4869999999999999e-06 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.011 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.588865+0000", 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00065820199999999998 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821573+0000", 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.232708846 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212353 7.12 e187/185 7:4af281cd:::benchmark_data_smithi008_82046_object97215:head v 187'10764, mlcod=187'10761)", 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.589498+0000", 2024-02-10T07:46:42.012 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 544.39732675000005, 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23719131800000001, 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589498+0000", 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589498+0000", 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.013 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589499+0000", 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8500000000000001e-07 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.589505+0000", 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4060000000000002e-06 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821581+0000", 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23207629699999999 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.014 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821593+0000", 2024-02-10T07:46:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1387e-05 2024-02-10T07:46:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821991+0000", 2024-02-10T07:46:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00039868400000000001 2024-02-10T07:46:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.015 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.822013+0000", 2024-02-10T07:46:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1191000000000001e-05 2024-02-10T07:46:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.826650+0000", 2024-02-10T07:46:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0046371650000000004 2024-02-10T07:46:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.826689+0000", 2024-02-10T07:46:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9502999999999998e-05 2024-02-10T07:46:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.016 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.24740.0:212363 7.5 e187/135 7:a6287109:::benchmark_data_smithi008_82046_object97225:head v 187'10252, mlcod=187'10250)", 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:37:37.594086+0000", 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 544.39273897800001, 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.23397765000000001, 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594086+0000", 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.017 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.018 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.018 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.018 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594086+0000", 2024-02-10T07:46:42.018 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.018 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.018 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.018 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.018 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594087+0000", 2024-02-10T07:46:42.018 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.595e-06 2024-02-10T07:46:42.018 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.018 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.018 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.018 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.594097+0000", 2024-02-10T07:46:42.018 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4560000000000006e-06 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821570+0000", 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.22747376999999999 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821582+0000", 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1093e-05 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821684+0000", 2024-02-10T07:46:42.019 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010225699999999999 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.821708+0000", 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4641e-05 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.828021+0000", 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0063127080000000002 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:37:37.828063+0000", 2024-02-10T07:46:42.020 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2129999999999998e-05 2024-02-10T07:46:42.021 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.021 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.021 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.021 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.021 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.021 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15630.0:32134 10.8 10:162b9c67:::benchmark_data_smithi008_161452_object32133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-02-10T07:46:42.021 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:46:33.730210+0000", 2024-02-10T07:46:42.021 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 8.2566142609999993, 2024-02-10T07:46:42.021 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.046734584000000003, 2024-02-10T07:46:42.021 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.021 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.021 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:46:42.023 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15630", 2024-02-10T07:46:42.023 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:46:42.023 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 32134 2024-02-10T07:46:42.023 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.023 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.023 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.023 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.023 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:33.730210+0000", 2024-02-10T07:46:42.023 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.023 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.023 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.023 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.023 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:33.730210+0000", 2024-02-10T07:46:42.023 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:33.730212+0000", 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.424e-06 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:33.730264+0000", 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1929999999999999e-05 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:33.730280+0000", 2024-02-10T07:46:42.024 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6166000000000001e-05 2024-02-10T07:46:42.025 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.025 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.025 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.025 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:33.730288+0000", 2024-02-10T07:46:42.025 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.8630000000000008e-06 2024-02-10T07:46:42.025 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.025 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.026 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.026 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:33.730383+0000", 2024-02-10T07:46:42.026 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.5470000000000006e-05 2024-02-10T07:46:42.026 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.026 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.026 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.026 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:33.730530+0000", 2024-02-10T07:46:42.026 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014668299999999999 2024-02-10T07:46:42.026 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.026 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.026 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,7", 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:33.730660+0000", 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013045 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:33.732015+0000", 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001354935 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:33.732324+0000", 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00030899900000000001 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.027 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.028 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:33.776837+0000", 2024-02-10T07:46:42.028 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.044512650000000001 2024-02-10T07:46:42.028 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.028 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.028 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.028 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:33.776902+0000", 2024-02-10T07:46:42.028 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4978e-05 2024-02-10T07:46:42.028 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.028 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.028 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.028 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:33.776945+0000", 2024-02-10T07:46:42.028 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3035999999999999e-05 2024-02-10T07:46:42.028 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.028 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15630.0:34296 10.5 10:a3403e0b:::benchmark_data_smithi008_161452_object34295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:46:34.879846+0000", 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 7.1069785420000002, 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.065088347000000005, 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15630", 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 34296 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.029 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.879846+0000", 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.879846+0000", 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.879847+0000", 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.235e-06 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.030 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.879868+0000", 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0466e-05 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.879882+0000", 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3969e-05 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.879890+0000", 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3580000000000003e-06 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.031 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.032 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.879946+0000", 2024-02-10T07:46:42.032 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.5473999999999998e-05 2024-02-10T07:46:42.032 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.032 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.032 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.032 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.880086+0000", 2024-02-10T07:46:42.032 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014084899999999999 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.880249+0000", 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016260100000000001 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.881770+0000", 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015211249999999999 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.881863+0000", 2024-02-10T07:46:42.033 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3074999999999999e-05 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.944777+0000", 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.062913854000000005 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.944836+0000", 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9243000000000002e-05 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.944934+0000", 2024-02-10T07:46:42.034 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8097999999999995e-05 2024-02-10T07:46:42.035 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.035 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.035 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.035 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.035 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.035 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15630.0:34314 10.5 10:aa4d3703:::benchmark_data_smithi008_161452_object34313:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-02-10T07:46:42.035 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:46:34.891084+0000", 2024-02-10T07:46:42.035 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 7.0957407799999999, 2024-02-10T07:46:42.035 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.054069662999999997, 2024-02-10T07:46:42.035 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.035 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.035 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:46:42.035 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15630", 2024-02-10T07:46:42.035 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:46:42.036 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 34314 2024-02-10T07:46:42.036 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.036 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.036 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.036 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.036 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.891084+0000", 2024-02-10T07:46:42.036 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.036 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.036 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.037 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.037 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.891084+0000", 2024-02-10T07:46:42.037 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.037 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.037 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.037 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.037 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.891085+0000", 2024-02-10T07:46:42.037 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.232e-06 2024-02-10T07:46:42.037 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.037 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.037 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.037 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.891112+0000", 2024-02-10T07:46:42.037 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6911e-05 2024-02-10T07:46:42.037 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.891125+0000", 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2660000000000001e-05 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.891132+0000", 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3069999999999997e-06 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.891182+0000", 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0393999999999999e-05 2024-02-10T07:46:42.038 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.891298+0000", 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000115296 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.891390+0000", 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2590000000000006e-05 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.892700+0000", 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0013100939999999999 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.039 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.040 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.040 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.893141+0000", 2024-02-10T07:46:42.040 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000441042 2024-02-10T07:46:42.040 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.040 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.040 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.040 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.944995+0000", 2024-02-10T07:46:42.040 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.051853398000000002 2024-02-10T07:46:42.040 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.040 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.040 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.040 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.945055+0000", 2024-02-10T07:46:42.040 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0168999999999999e-05 2024-02-10T07:46:42.040 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.945153+0000", 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.857e-05 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15630.0:34327 10.5 10:ac5f7bfa:::benchmark_data_smithi008_161452_object34326:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:46:34.898114+0000", 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 7.0887105300000002, 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.047179470000000001, 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.041 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15630", 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 34327 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.898114+0000", 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.898114+0000", 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.042 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.898115+0000", 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4899999999999995e-07 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.898147+0000", 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1986999999999997e-05 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.898156+0000", 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7739999999999999e-06 2024-02-10T07:46:42.043 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.898164+0000", 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1790000000000002e-06 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.898216+0000", 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1958e-05 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.898342+0000", 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000126367 2024-02-10T07:46:42.044 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.045 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.045 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:46:42.045 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.898483+0000", 2024-02-10T07:46:42.045 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014107200000000001 2024-02-10T07:46:42.045 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.045 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.045 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:46:42.045 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.899836+0000", 2024-02-10T07:46:42.045 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0013531979999999999 2024-02-10T07:46:42.045 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.045 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.045 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.045 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.900202+0000", 2024-02-10T07:46:42.046 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:42.046 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00036630799999999998 2024-02-10T07:46:42.046 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.046 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.046 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.046 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.945199+0000", 2024-02-10T07:46:42.046 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.044996133000000001 2024-02-10T07:46:42.047 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.047 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.047 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.047 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.945233+0000", 2024-02-10T07:46:42.047 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3936000000000002e-05 2024-02-10T07:46:42.047 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.047 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.047 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.047 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.945294+0000", 2024-02-10T07:46:42.047 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1008999999999998e-05 2024-02-10T07:46:42.047 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.047 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.047 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.047 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15630.0:34331 10.5 10:a6b72cb6:::benchmark_data_smithi008_161452_object34330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:46:34.901225+0000", 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 7.0856000589999999, 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.044253632000000001, 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15630", 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 34331 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.048 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.901225+0000", 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.901225+0000", 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.901226+0000", 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.362e-06 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.049 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.050 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.901245+0000", 2024-02-10T07:46:42.050 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8711000000000001e-05 2024-02-10T07:46:42.050 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.050 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.050 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.050 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.901257+0000", 2024-02-10T07:46:42.050 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2473999999999999e-05 2024-02-10T07:46:42.050 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.050 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.050 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.050 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.901262+0000", 2024-02-10T07:46:42.050 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3619999999999997e-06 2024-02-10T07:46:42.050 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.050 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.901311+0000", 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8789999999999999e-05 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.901420+0000", 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000108346 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.901519+0000", 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9024999999999995e-05 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.051 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.902532+0000", 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0010130359999999999 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.903768+0000", 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0012364629999999999 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.945349+0000", 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.041580935999999999 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.052 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.945393+0000", 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4203999999999999e-05 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.945478+0000", 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4923e-05 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15630.0:34333 10.5 10:a8a591ca:::benchmark_data_smithi008_161452_object34332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-02-10T07:46:42.053 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:46:34.902545+0000", 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 7.084279853, 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.043115054999999999, 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15630", 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 34333 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.902545+0000", 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.054 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.902545+0000", 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.902546+0000", 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0299999999999998e-07 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.902566+0000", 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0319999999999999e-05 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.055 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.902578+0000", 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.22e-05 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.902585+0000", 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.34e-06 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.902638+0000", 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2939000000000003e-05 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.056 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.902779+0000", 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000140357 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.902932+0000", 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015315899999999999 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.903850+0000", 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00091827499999999997 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.057 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.904265+0000", 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00041476600000000001 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.945524+0000", 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.041258817000000003 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.945567+0000", 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3058e-05 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.058 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.059 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.059 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.945660+0000", 2024-02-10T07:46:42.059 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3021000000000003e-05 2024-02-10T07:46:42.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:42.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:42.060 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.060 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.061 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.061 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.061 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.061 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15630.0:34344 10.5 10:a687f279:::benchmark_data_smithi008_161452_object34343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-02-10T07:46:42.061 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:46:34.909014+0000", 2024-02-10T07:46:42.061 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 7.077810167, 2024-02-10T07:46:42.061 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.036783075999999998, 2024-02-10T07:46:42.061 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.061 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.061 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:46:42.061 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15630", 2024-02-10T07:46:42.061 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:46:42.061 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 34344 2024-02-10T07:46:42.061 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.909014+0000", 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.909014+0000", 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.909015+0000", 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7300000000000004e-07 2024-02-10T07:46:42.062 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.909040+0000", 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4108999999999999e-05 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.909049+0000", 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7659999999999993e-06 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.909055+0000", 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3730000000000001e-06 2024-02-10T07:46:42.063 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.064 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.064 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.064 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.909116+0000", 2024-02-10T07:46:42.064 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0949999999999998e-05 2024-02-10T07:46:42.064 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.064 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.064 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.064 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.909277+0000", 2024-02-10T07:46:42.064 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000161688 2024-02-10T07:46:42.064 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.064 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.064 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,7", 2024-02-10T07:46:42.064 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.909432+0000", 2024-02-10T07:46:42.064 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015449800000000001 2024-02-10T07:46:42.065 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.065 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.065 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:46:42.065 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.910628+0000", 2024-02-10T07:46:42.065 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001195991 2024-02-10T07:46:42.065 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.065 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.065 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.065 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.910960+0000", 2024-02-10T07:46:42.065 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000332524 2024-02-10T07:46:42.065 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.065 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.065 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.065 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.945710+0000", 2024-02-10T07:46:42.066 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.034749246999999997 2024-02-10T07:46:42.066 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.066 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.066 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.066 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.945745+0000", 2024-02-10T07:46:42.066 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.5784999999999997e-05 2024-02-10T07:46:42.066 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.066 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.066 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.066 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:34.945798+0000", 2024-02-10T07:46:42.066 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2172000000000001e-05 2024-02-10T07:46:42.066 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.066 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.066 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15630.0:39549 10.8 10:1ffed51c:::benchmark_data_smithi008_161452_object39548:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:46:37.766194+0000", 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.2206305669999997, 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.047145820999999997, 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15630", 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 39549 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.067 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.068 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.068 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.766194+0000", 2024-02-10T07:46:42.068 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.068 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.068 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.068 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.068 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.766194+0000", 2024-02-10T07:46:42.068 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.068 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.068 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.068 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.068 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.766195+0000", 2024-02-10T07:46:42.068 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7800000000000001e-07 2024-02-10T07:46:42.068 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.069 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.069 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.069 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.766214+0000", 2024-02-10T07:46:42.069 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9763e-05 2024-02-10T07:46:42.069 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.069 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.069 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.069 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.766226+0000", 2024-02-10T07:46:42.069 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1035e-05 2024-02-10T07:46:42.069 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.069 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.069 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.069 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.766232+0000", 2024-02-10T07:46:42.069 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6139999999999997e-06 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.769304+0000", 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003071402 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.769440+0000", 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013608399999999999 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,7", 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.770225+0000", 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00078583799999999997 2024-02-10T07:46:42.070 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.772771+0000", 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0025457090000000002 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.774089+0000", 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0013174020000000001 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.813225+0000", 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.039136775999999998 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.071 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.813299+0000", 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3678999999999997e-05 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.813340+0000", 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0840999999999997e-05 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15630.0:39570 10.8 10:1874ef04:::benchmark_data_smithi008_161452_object39569:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-02-10T07:46:42.072 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:46:37.779611+0000", 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.2072131349999999, 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.041100721999999999, 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15630", 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 39570 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.779611+0000", 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.073 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.779611+0000", 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.779613+0000", 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0470000000000001e-06 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.779636+0000", 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3028999999999999e-05 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.074 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.779649+0000", 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3811999999999999e-05 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.779656+0000", 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4880000000000004e-06 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.779719+0000", 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3530999999999999e-05 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.075 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.076 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.779878+0000", 2024-02-10T07:46:42.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015848500000000001 2024-02-10T07:46:42.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.076 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.076 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,7", 2024-02-10T07:46:42.076 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.780042+0000", 2024-02-10T07:46:42.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016419500000000001 2024-02-10T07:46:42.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.076 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.076 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:46:42.076 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.781502+0000", 2024-02-10T07:46:42.076 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001460184 2024-02-10T07:46:42.076 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.076 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782280+0000", 2024-02-10T07:46:42.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00077736299999999999 2024-02-10T07:46:42.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.820648+0000", 2024-02-10T07:46:42.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.038368041999999998 2024-02-10T07:46:42.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.077 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.077 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.077 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.820685+0000", 2024-02-10T07:46:42.077 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7066000000000001e-05 2024-02-10T07:46:42.077 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.820712+0000", 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7480000000000001e-05 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15630.0:39575 10.8 10:1e9f4d13:::benchmark_data_smithi008_161452_object39574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:46:37.782394+0000", 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.2044310810000001, 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.038460499000000002, 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.078 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15630", 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 39575 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782394+0000", 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782394+0000", 2024-02-10T07:46:42.079 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.080 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.080 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.080 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.080 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782395+0000", 2024-02-10T07:46:42.080 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.122e-06 2024-02-10T07:46:42.080 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.080 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.080 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.080 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782417+0000", 2024-02-10T07:46:42.080 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2670000000000001e-05 2024-02-10T07:46:42.080 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.080 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.080 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.080 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782431+0000", 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3190000000000001e-05 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782438+0000", 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.6129999999999998e-06 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782499+0000", 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0395e-05 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.081 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782599+0000", 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010051399999999999 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,7", 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782690+0000", 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1353000000000004e-05 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.785483+0000", 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027924870000000002 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.082 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.787229+0000", 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0017466249999999999 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.820759+0000", 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.033529524999999998 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.820816+0000", 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6867999999999998e-05 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.083 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.820854+0000", 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8136999999999998e-05 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15630.0:39576 10.8 10:12e3b5ae:::benchmark_data_smithi008_161452_object39575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:46:37.782473+0000", 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4.2043514719999999, 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.041037559000000001, 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15630", 2024-02-10T07:46:42.084 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 39576 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782473+0000", 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782473+0000", 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.085 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782474+0000", 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4900000000000001e-07 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782488+0000", 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4644000000000001e-05 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782494+0000", 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.083e-06 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.086 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.782498+0000", 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0249999999999996e-06 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.784038+0000", 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0015395319999999999 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.784167+0000", 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012956299999999999 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.087 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,7", 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.784308+0000", 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014105699999999999 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.785629+0000", 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001320805 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.787377+0000", 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001747869 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.088 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.089 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.823331+0000", 2024-02-10T07:46:42.089 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.035953855999999999 2024-02-10T07:46:42.089 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.089 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.089 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.089 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.823424+0000", 2024-02-10T07:46:42.089 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3115999999999994e-05 2024-02-10T07:46:42.089 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.089 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.089 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.089 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:37.823511+0000", 2024-02-10T07:46:42.089 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6559999999999998e-05 2024-02-10T07:46:42.089 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.089 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15630.0:41950 10.5 10:a7058607:::benchmark_data_smithi008_161452_object41949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:46:39.137891+0000", 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.8489334369999999, 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0860107750000001, 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15630", 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 41950 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.090 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.137891+0000", 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.137891+0000", 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.137893+0000", 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.584e-06 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.091 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.137916+0000", 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2818000000000001e-05 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.137927+0000", 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1582e-05 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.137935+0000", 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0560000000000005e-06 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.092 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.138122+0000", 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000186973 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.138135+0000", 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2627e-05 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:40.113114+0000", 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.97497898000000005 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.093 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:40.113125+0000", 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1382999999999999e-05 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.217291+0000", 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1041660310000001 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.217465+0000", 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017416600000000001 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.094 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,7", 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.217654+0000", 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00018860099999999999 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.222355+0000", 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0047009579999999999 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.222372+0000", 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7187e-05 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.095 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.096 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.096 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.222400+0000", 2024-02-10T07:46:42.096 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7698000000000001e-05 2024-02-10T07:46:42.096 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.096 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.096 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.096 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.222448+0000", 2024-02-10T07:46:42.096 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8542999999999999e-05 2024-02-10T07:46:42.096 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.096 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.096 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.096 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.223902+0000", 2024-02-10T07:46:42.096 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001453588 2024-02-10T07:46:42.096 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.15630.0:41989 10.5 10:a7e04e3e:::benchmark_data_smithi008_161452_object41988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:46:39.168216+0000", 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.8186082309999998, 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0602803550000002, 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.15630", 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 41989 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.097 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.168216+0000", 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.168216+0000", 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.168217+0000", 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8100000000000001e-07 2024-02-10T07:46:42.098 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.168238+0000", 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0543e-05 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.168250+0000", 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2544999999999999e-05 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.168258+0000", 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7179999999999996e-06 2024-02-10T07:46:42.099 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.168313+0000", 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4945000000000003e-05 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:39.168321+0000", 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7489999999999998e-06 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:40.113154+0000", 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.94483372600000004 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.100 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for peered", 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:40.113163+0000", 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.551e-06 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.220719+0000", 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1075558999999999 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.220831+0000", 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011225700000000001 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.101 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 5,7", 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.220938+0000", 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010660800000000001 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.224015+0000", 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0030768039999999998 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.224060+0000", 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5410000000000001e-05 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.102 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.226316+0000", 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022559569999999998 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.226361+0000", 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5454000000000002e-05 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:46:41.228497+0000", 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0021353069999999999 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.103 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.104 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:42.104 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:42.104 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:46:42.160 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:42.192 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:42.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:42.293 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:42.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:42 smithi008 ceph-mon[29816]: osdmap e326: 8 total, 7 up, 7 in 2024-02-10T07:46:42.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:42 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 5196/121158 objects degraded (4.289%), 17 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:42.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:42 smithi008 ceph-mon[29816]: osdmap e327: 8 total, 7 up, 7 in 2024-02-10T07:46:42.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:42 smithi008 ceph-mon[40743]: osdmap e326: 8 total, 7 up, 7 in 2024-02-10T07:46:42.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:42 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 5196/121158 objects degraded (4.289%), 17 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:42.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:42 smithi008 ceph-mon[40743]: osdmap e327: 8 total, 7 up, 7 in 2024-02-10T07:46:42.334 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:42.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:42.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:42 smithi170 ceph-mon[33394]: osdmap e326: 8 total, 7 up, 7 in 2024-02-10T07:46:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:42 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 5196/121158 objects degraded (4.289%), 17 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:42 smithi170 ceph-mon[33394]: osdmap e327: 8 total, 7 up, 7 in 2024-02-10T07:46:42.436 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:42.475 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:42.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:42.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:42.576 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:42.611 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:42.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:42.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:42.713 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:42.754 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:42.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:42.827 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:46:42.856 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:42.906 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:42.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:42.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 22 16 43979 43963 124.885 73.125 0.0113143 0.00799898 2024-02-10T07:46:43.008 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:43.048 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:43.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:43.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:43.150 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:43.189 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:43.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:43.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:43.291 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:43 smithi170 ceph-mon[33394]: pgmap v673: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 36 MiB/s wr, 575 op/s; 5196/121158 objects degraded (4.289%); 7551/121158 objects misplaced (6.232%) 2024-02-10T07:46:43.447 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:43.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:43.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:43.534 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:43 smithi008 ceph-mon[29816]: pgmap v673: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 36 MiB/s wr, 575 op/s; 5196/121158 objects degraded (4.289%); 7551/121158 objects misplaced (6.232%) 2024-02-10T07:46:43.534 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:43 smithi008 ceph-mon[40743]: pgmap v673: 148 pgs: 19 active+clean+remapped, 17 active+undersized+degraded, 26 active+undersized, 86 active+clean; 2.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 36 MiB/s wr, 575 op/s; 5196/121158 objects degraded (4.289%); 7551/121158 objects misplaced (6.232%) 2024-02-10T07:46:43.549 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:43.584 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:43.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:43.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:43.686 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:43.725 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:43.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:43.826 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:43.866 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:43.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:43.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:43.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:46:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:46:43.884 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:46:43.915 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:46:43.944 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:46:43.945 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 23 16 45174 45158 122.702 74.6875 0.00853953 0.00814327 2024-02-10T07:46:43.968 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:43.984 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:46:44.006 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:44.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:44.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:44.015 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:46:44.046 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:46:44.078 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:46:44.079 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:46:44.079 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:46:44.079 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:46:44.079 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-10T07:46:44.079 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:46:44.079 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:46:44.079 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:46:44.079 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:46:44.107 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:44.144 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:44.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:44.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:44.246 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:44.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:44.288 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:44.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:44.304 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-10T07:46:44.374 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:44 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1886801606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:44.374 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:44 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1886801606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:44 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1886801606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:44.389 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:44.427 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:44.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:44.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:44.521 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:44.528 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:44.566 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:44.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:44.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:44.668 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:44.708 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:44.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:44.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:44.810 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:44.851 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:44.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:44.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 24 16 46338 46322 120.62 72.75 0.0207072 0.00828336 2024-02-10T07:46:44.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:44.988 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:44.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:44.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:45.068 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:46:45.068 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:46:45.089 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:45.118 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:45.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:45.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:45.199 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [3] live_osds: [0, 1, 2, 4, 6, 7, 5] 2024-02-10T07:46:45.200 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:46:45.200 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-10T07:46:45.200 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-10T07:46:45.219 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:45.260 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:45.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:45.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:45.292 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:45 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:45.292 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:45 smithi008 ceph-mon[29816]: pgmap v674: 148 pgs: 6 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 2 active+recovering+undersized+remapped, 17 active+undersized, 111 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 40 MiB/s wr, 638 op/s; 5246/130605 objects degraded (4.017%); 7785/130605 objects misplaced (5.961%); 1.7 MiB/s, 34 objects/s recovering 2024-02-10T07:46:45.292 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:45 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:45.292 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:45 smithi008 ceph-mon[40743]: pgmap v674: 148 pgs: 6 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 2 active+recovering+undersized+remapped, 17 active+undersized, 111 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 40 MiB/s wr, 638 op/s; 5246/130605 objects degraded (4.017%); 7785/130605 objects misplaced (5.961%); 1.7 MiB/s, 34 objects/s recovering 2024-02-10T07:46:45.362 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:45 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:45 smithi170 ceph-mon[33394]: pgmap v674: 148 pgs: 6 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 11 active+undersized+degraded, 2 active+recovering+undersized+remapped, 17 active+undersized, 111 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 40 MiB/s wr, 638 op/s; 5246/130605 objects degraded (4.017%); 7785/130605 objects misplaced (5.961%); 1.7 MiB/s, 34 objects/s recovering 2024-02-10T07:46:45.399 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:45.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:45.500 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:45.528 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:45.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:45.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:45.630 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:45.672 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:45.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:45.774 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:45.797 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:45.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:45.899 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:45.928 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:45.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:45.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:45.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 25 16 47394 47378 118.435 66 0.0126601 0.00843667 2024-02-10T07:46:46.030 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:46.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:46.056 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:46.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:46.157 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:46.194 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:46.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:46.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:46.296 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:46.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:46.343 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:46.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:46.443 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:46.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:46.484 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:46.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:46.585 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:46.611 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:46.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:46.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:46.712 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:46.746 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:46.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:46.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:46.848 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:46.876 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:46.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:46.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:46.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 26 16 48536 48520 116.625 71.375 0.0096205 0.00856812 2024-02-10T07:46:46.978 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:47.008 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:47.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:47.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:47.068 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:46:47.109 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:47.149 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:47.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:47.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:47.177 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:47 smithi008 ceph-mon[29816]: pgmap v675: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 2.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 42 MiB/s wr, 665 op/s; 5432/133125 objects degraded (4.080%); 7611/133125 objects misplaced (5.717%); 3.0 MiB/s, 60 objects/s recovering 2024-02-10T07:46:47.177 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:47 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 5432/133125 objects degraded (4.080%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:47.178 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:47 smithi008 ceph-mon[40743]: pgmap v675: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 2.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 42 MiB/s wr, 665 op/s; 5432/133125 objects degraded (4.080%); 7611/133125 objects misplaced (5.717%); 3.0 MiB/s, 60 objects/s recovering 2024-02-10T07:46:47.178 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:47 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 5432/133125 objects degraded (4.080%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:47.251 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:47.285 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:47.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:47.321 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:47 smithi170 ceph-mon[33394]: pgmap v675: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 2.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 42 MiB/s wr, 665 op/s; 5432/133125 objects degraded (4.080%); 7611/133125 objects misplaced (5.717%); 3.0 MiB/s, 60 objects/s recovering 2024-02-10T07:46:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:47 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 5432/133125 objects degraded (4.080%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:47.385 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:47.429 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:47.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:47.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:47.531 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:47.570 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:47.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:47.671 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:47.711 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:47.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:47.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:47.812 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:47.854 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:47.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:47.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 27 16 49630 49614 114.838 68.375 0.00657409 0.00870122 2024-02-10T07:46:47.955 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:47.996 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:47.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:47.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:48.098 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:48.106 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:48.124 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:48.139 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:46:48.141 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:48.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:48.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:48.243 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:48.273 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:48.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:48.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:48.375 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:48.405 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:48.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:48.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:48.507 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:48.545 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:48.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:48.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:48.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:48 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/822353201' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:46:48.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:48 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:46:48.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:48 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:46:48.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:48 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:46:48.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:48 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/822353201' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:46:48.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:48 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:46:48.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:48 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:46:48.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:48 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:46:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:48 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/822353201' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:46:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:48 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:46:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:48 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:46:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:48 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:46:48.647 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:48.686 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:48.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:48.788 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:48.821 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:48.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:48.922 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:48.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 28 16 50648 50632 113.009 63.625 0.0242082 0.00884188 2024-02-10T07:46:48.965 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:48.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:49.067 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:49.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:46:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:46:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:46:49.103 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:49.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:49.205 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:49.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:49.234 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:49.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:49.335 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:49.365 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:49.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:49.467 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:49.496 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:49.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:49.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:49.549 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:49 smithi008 ceph-mon[29816]: pgmap v676: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 49 MiB/s wr, 778 op/s; 5432/139674 objects degraded (3.889%); 7403/139674 objects misplaced (5.300%); 3.9 MiB/s, 72 objects/s recovering 2024-02-10T07:46:49.549 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:49 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:46:49.549 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:49 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:46:49.549 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:49 smithi008 ceph-mon[29816]: osdmap e328: 8 total, 7 up, 7 in 2024-02-10T07:46:49.549 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:49 smithi008 ceph-mon[40743]: pgmap v676: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 49 MiB/s wr, 778 op/s; 5432/139674 objects degraded (3.889%); 7403/139674 objects misplaced (5.300%); 3.9 MiB/s, 72 objects/s recovering 2024-02-10T07:46:49.549 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:49 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:46:49.549 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:49 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:46:49.549 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:49 smithi008 ceph-mon[40743]: osdmap e328: 8 total, 7 up, 7 in 2024-02-10T07:46:49.598 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:49 smithi170 ceph-mon[33394]: pgmap v676: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 2.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 49 MiB/s wr, 778 op/s; 5432/139674 objects degraded (3.889%); 7403/139674 objects misplaced (5.300%); 3.9 MiB/s, 72 objects/s recovering 2024-02-10T07:46:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:49 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:46:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:49 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:46:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:49 smithi170 ceph-mon[33394]: osdmap e328: 8 total, 7 up, 7 in 2024-02-10T07:46:49.638 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:49.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:49.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:49.740 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:49.770 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:49.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:49.873 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:49.911 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:49.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:49.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:49.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 29 15 51735 51720 111.456 68 0.0390273 0.00896607 2024-02-10T07:46:50.014 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:50.049 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:50.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:50.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:50.151 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:50.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:50.195 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:50.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:50.296 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:50.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:50.338 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:50.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:50.398 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:50 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:46:50.398 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:50 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:46:50.398 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:50 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:46:50.399 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:50 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:46:50.439 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:50.472 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:50.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:50.574 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:50.604 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:50.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:50 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:46:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:50 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:46:50.705 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:50.736 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:50.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:50.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:50.838 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:50.883 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:50.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:50.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:50.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 30 16 52825 52809 110.01 68.0625 0.00960854 0.00908437 2024-02-10T07:46:50.985 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:51.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:51.025 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:51.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:51.126 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:51.157 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:51.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:51.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:51.259 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:51.290 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:51.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:51.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:51 smithi008 ceph-mon[29816]: pgmap v678: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 5605/150024 objects degraded (3.736%); 6964/150024 objects misplaced (4.642%); 8.3 MiB/s, 142 objects/s recovering 2024-02-10T07:46:51.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:51 smithi008 ceph-mon[29816]: osdmap e329: 8 total, 7 up, 7 in 2024-02-10T07:46:51.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:51 smithi008 ceph-mon[40743]: pgmap v678: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 5605/150024 objects degraded (3.736%); 6964/150024 objects misplaced (4.642%); 8.3 MiB/s, 142 objects/s recovering 2024-02-10T07:46:51.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:51 smithi008 ceph-mon[40743]: osdmap e329: 8 total, 7 up, 7 in 2024-02-10T07:46:51.392 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:51.433 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:51.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:51.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:51.535 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:51.563 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:51.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:51 smithi170 ceph-mon[33394]: pgmap v678: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 5605/150024 objects degraded (3.736%); 6964/150024 objects misplaced (4.642%); 8.3 MiB/s, 142 objects/s recovering 2024-02-10T07:46:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:51 smithi170 ceph-mon[33394]: osdmap e329: 8 total, 7 up, 7 in 2024-02-10T07:46:51.665 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:51.695 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:51.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:51.797 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:51.804 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:46:51.804 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:46:51.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:51.834 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:51.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:51.935 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:51.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 31 16 53904 53888 108.636 67.4375 0.0249348 0.0091986 2024-02-10T07:46:51.984 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:51.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:51.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:52.086 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:52.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:52.110 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:52.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:52.211 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:52.241 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:52.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:52.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:52.343 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:52.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:52.382 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:52.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:52.483 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:52.525 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:52.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:52.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:52.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:52 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 5605/150024 objects degraded (3.736%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:52.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:52 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 5605/150024 objects degraded (3.736%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:52.627 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:52 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 5605/150024 objects degraded (3.736%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:52.667 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:52.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:52.769 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:52.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:52.808 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:52.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:52.909 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:52.937 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:52.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:52.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:52.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 32 16 54824 54808 107.038 57.5 0.00753762 0.00933678 2024-02-10T07:46:53.039 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:53.083 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:53.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:53.184 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:53.221 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:53.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:53.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:53.323 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:53.361 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:53.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:53.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:53.406 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:46:53.434 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:53 smithi008 ceph-mon[29816]: pgmap v680: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 809 op/s; 5605/150024 objects degraded (3.736%); 6964/150024 objects misplaced (4.642%); 7.4 MiB/s, 123 objects/s recovering 2024-02-10T07:46:53.435 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:53 smithi008 ceph-mon[40743]: pgmap v680: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 809 op/s; 5605/150024 objects degraded (3.736%); 6964/150024 objects misplaced (4.642%); 7.4 MiB/s, 123 objects/s recovering 2024-02-10T07:46:53.463 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:53.495 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:53.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:53.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:53.565 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:53 smithi170 ceph-mon[33394]: pgmap v680: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 51 MiB/s wr, 809 op/s; 5605/150024 objects degraded (3.736%); 6964/150024 objects misplaced (4.642%); 7.4 MiB/s, 123 objects/s recovering 2024-02-10T07:46:53.597 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:53.637 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:53.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:53.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:53.739 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:53.767 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:53.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:53.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:53.869 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:53.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:46:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:46:53.897 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:53.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:53.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:53.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 33 16 55790 55774 105.624 60.375 0.00798197 0.00946182 2024-02-10T07:46:53.999 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:54.042 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:54.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:54.144 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:54.172 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:54.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:54.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:54.274 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:54.311 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:54.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:54.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:54.412 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:54.439 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:54.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:54.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:54.541 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:54.570 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:54.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:54.672 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:54.704 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:54.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:54.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:54.806 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:54.835 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:54.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:54.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:54.940 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:54.946 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 34 16 56754 56738 104.289 60.25 0.0247695 0.00958269 2024-02-10T07:46:54.968 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:54.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:54.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:55.070 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:55.099 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:55.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:55.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:55.201 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:55.229 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:55.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:55.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:55.332 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:55.369 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:55.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:55.471 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:55.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:55 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:55.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:55 smithi008 ceph-mon[29816]: pgmap v681: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 5458/161880 objects degraded (3.372%); 6585/161880 objects misplaced (4.068%); 10 MiB/s, 164 objects/s recovering 2024-02-10T07:46:55.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:55 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:55.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:55 smithi008 ceph-mon[40743]: pgmap v681: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 5458/161880 objects degraded (3.372%); 6585/161880 objects misplaced (4.068%); 10 MiB/s, 164 objects/s recovering 2024-02-10T07:46:55.584 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:55.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:55.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:55 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:46:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:55 smithi170 ceph-mon[33394]: pgmap v681: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 5458/161880 objects degraded (3.372%); 6585/161880 objects misplaced (4.068%); 10 MiB/s, 164 objects/s recovering 2024-02-10T07:46:55.686 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:55.723 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:55.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:55.824 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:55.852 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:55.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:55.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:55.947 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 35 16 57720 57704 103.034 60.375 0.04713 0.00969878 2024-02-10T07:46:55.954 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:55.983 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:55.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:55.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:56.085 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:56.113 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:56.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:56.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:56.215 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:56.358 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:56.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:56.460 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:56.500 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:56.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:56.602 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:56.640 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:56.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:56.742 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:56.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:56.782 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:56.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:56.883 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:56.926 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:56.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:56.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:56.947 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 36 16 58696 58680 101.867 61 0.00785379 0.00981099 2024-02-10T07:46:57.028 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:57.072 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:57.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:57.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:57.173 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:57.213 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:57.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:57.315 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:57.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:57 smithi008 ceph-mon[40743]: pgmap v682: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s; 5785/165762 objects degraded (3.490%); 6306/165762 objects misplaced (3.804%); 11 MiB/s, 173 objects/s recovering 2024-02-10T07:46:57.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:57 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 5785/165762 objects degraded (3.490%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:57 smithi008 ceph-mon[29816]: pgmap v682: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s; 5785/165762 objects degraded (3.490%); 6306/165762 objects misplaced (3.804%); 11 MiB/s, 173 objects/s recovering 2024-02-10T07:46:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:57 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 5785/165762 objects degraded (3.490%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:57.346 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:57.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:57.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:57.447 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:57.486 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:57.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:57.503 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-10T07:46:57.588 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:57.619 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:57.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:57.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:57 smithi170 ceph-mon[33394]: pgmap v682: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s; 5785/165762 objects degraded (3.490%); 6306/165762 objects misplaced (3.804%); 11 MiB/s, 173 objects/s recovering 2024-02-10T07:46:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:57 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 5785/165762 objects degraded (3.490%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:46:57.721 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:57.757 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:57.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:57.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:57.859 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:57.899 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:57.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:57.947 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 37 16 59731 59715 100.862 64.6875 0.00846469 0.00990797 2024-02-10T07:46:58.000 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:58.035 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:58.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:58.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:58.137 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:58.174 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:58.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:58.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:58.276 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:58.303 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:58.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:58.405 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:58.437 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:58.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:58.539 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:58.580 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:58.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:58.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:46:58.682 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:46:58.716 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:58.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:58.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:46:58.818 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:46:58.831 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:46:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:46:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:46:58.947 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 38 16 60657 60641 99.7302 57.875 0.0205171 0.0100215 2024-02-10T07:46:59.054 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:59.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:59.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:46:59.101 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:59.119 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:46:59.148 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:46:59.156 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:59.173 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:46:59.173 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:46:59.173 INFO:teuthology.orchestra.run.smithi008.stdout: "epoch": 329, 2024-02-10T07:46:59.173 INFO:teuthology.orchestra.run.smithi008.stdout: "fsid": "8c8244e6-c7e5-11ee-95b8-87774f69a715", 2024-02-10T07:46:59.173 INFO:teuthology.orchestra.run.smithi008.stdout: "created": "2024-02-10T07:26:21.316239+0000", 2024-02-10T07:46:59.173 INFO:teuthology.orchestra.run.smithi008.stdout: "modified": "2024-02-10T07:46:50.140309+0000", 2024-02-10T07:46:59.173 INFO:teuthology.orchestra.run.smithi008.stdout: "last_up_change": "2024-02-10T07:45:02.292328+0000", 2024-02-10T07:46:59.173 INFO:teuthology.orchestra.run.smithi008.stdout: "last_in_change": "2024-02-10T07:46:09.693935+0000", 2024-02-10T07:46:59.173 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-10T07:46:59.173 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_num": 5801984, 2024-02-10T07:46:59.173 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_set": [ 2024-02-10T07:46:59.173 INFO:teuthology.orchestra.run.smithi008.stdout: "noscrub", 2024-02-10T07:46:59.174 INFO:teuthology.orchestra.run.smithi008.stdout: "pglog_hardlimit", 2024-02-10T07:46:59.174 INFO:teuthology.orchestra.run.smithi008.stdout: "purged_snapdirs", 2024-02-10T07:46:59.174 INFO:teuthology.orchestra.run.smithi008.stdout: "recovery_deletes", 2024-02-10T07:46:59.174 INFO:teuthology.orchestra.run.smithi008.stdout: "sortbitwise" 2024-02-10T07:46:59.174 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:46:59.174 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_version": 19, 2024-02-10T07:46:59.174 INFO:teuthology.orchestra.run.smithi008.stdout: "full_ratio": 0.94999998807907104, 2024-02-10T07:46:59.174 INFO:teuthology.orchestra.run.smithi008.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-10T07:46:59.174 INFO:teuthology.orchestra.run.smithi008.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-10T07:46:59.174 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_snapshot": "", 2024-02-10T07:46:59.174 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_max": 10, 2024-02-10T07:46:59.174 INFO:teuthology.orchestra.run.smithi008.stdout: "max_osd": 8, 2024-02-10T07:46:59.174 INFO:teuthology.orchestra.run.smithi008.stdout: "require_min_compat_client": "luminous", 2024-02-10T07:46:59.174 INFO:teuthology.orchestra.run.smithi008.stdout: "min_compat_client": "luminous", 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "require_osd_release": "squid", 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "allow_crimson": false, 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "pools": [ 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "pool": 1, 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_name": ".mgr", 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "create_time": "2024-02-10T07:29:32.704872+0000", 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": 1, 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_names": "hashpspool", 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "type": 1, 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 3, 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "min_size": 2, 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_rule": 0, 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_count": 0, 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_target": 0, 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T07:46:59.175 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T07:46:59.176 INFO:teuthology.orchestra.run.smithi008.stdout: "object_hash": 2, 2024-02-10T07:46:59.176 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_mode": "off", 2024-02-10T07:46:59.176 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num": 1, 2024-02-10T07:46:59.176 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num": 1, 2024-02-10T07:46:59.176 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num_target": 1, 2024-02-10T07:46:59.176 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_target": 1, 2024-02-10T07:46:59.176 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_pending": 1, 2024-02-10T07:46:59.176 INFO:teuthology.orchestra.run.smithi008.stdout: "last_pg_merge_meta": { 2024-02-10T07:46:59.176 INFO:teuthology.orchestra.run.smithi008.stdout: "source_pgid": "0.0", 2024-02-10T07:46:59.176 INFO:teuthology.orchestra.run.smithi008.stdout: "ready_epoch": 0, 2024-02-10T07:46:59.176 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_started": 0, 2024-02-10T07:46:59.176 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_clean": 0, 2024-02-10T07:46:59.176 INFO:teuthology.orchestra.run.smithi008.stdout: "source_version": "0'0", 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: "target_version": "0'0" 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: "last_change": "21", 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend": "0", 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: "auid": 0, 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_mode": "selfmanaged", 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_seq": 0, 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_epoch": 0, 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_snaps": [], 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps": "[]", 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_bytes": 0, 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_objects": 0, 2024-02-10T07:46:59.177 INFO:teuthology.orchestra.run.smithi008.stdout: "tiers": [], 2024-02-10T07:46:59.178 INFO:teuthology.orchestra.run.smithi008.stdout: "tier_of": -1, 2024-02-10T07:46:59.178 INFO:teuthology.orchestra.run.smithi008.stdout: "read_tier": -1, 2024-02-10T07:46:59.178 INFO:teuthology.orchestra.run.smithi008.stdout: "write_tier": -1, 2024-02-10T07:46:59.178 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_mode": "none", 2024-02-10T07:46:59.178 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_bytes": 0, 2024-02-10T07:46:59.178 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_objects": 0, 2024-02-10T07:46:59.178 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T07:46:59.178 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T07:46:59.178 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T07:46:59.178 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_flush_age": 0, 2024-02-10T07:46:59.178 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_evict_age": 0, 2024-02-10T07:46:59.178 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profile": "", 2024-02-10T07:46:59.178 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_params": { 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "none" 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_period": 0, 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_count": 0, 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "use_gmt_hitset": true, 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "min_read_recency_for_promote": 0, 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "min_write_recency_for_promote": 0, 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_search_last_n": 0, 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "grade_table": [], 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "stripe_width": 0, 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "expected_num_objects": 0, 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "fast_read": false, 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "options": { 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_max": 32, 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_min": 1 2024-02-10T07:46:59.179 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.180 INFO:teuthology.orchestra.run.smithi008.stdout: "application_metadata": { 2024-02-10T07:46:59.180 INFO:teuthology.orchestra.run.smithi008.stdout: "mgr": {} 2024-02-10T07:46:59.180 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.180 INFO:teuthology.orchestra.run.smithi008.stdout: "read_balance": { 2024-02-10T07:46:59.180 INFO:teuthology.orchestra.run.smithi008.stdout: "score_type": "Fair distribution", 2024-02-10T07:46:59.180 INFO:teuthology.orchestra.run.smithi008.stdout: "score_acting": 7.8899998664855957, 2024-02-10T07:46:59.180 INFO:teuthology.orchestra.run.smithi008.stdout: "score_stable": 7.8899998664855957, 2024-02-10T07:46:59.180 INFO:teuthology.orchestra.run.smithi008.stdout: "optimal_score": 0.37999999523162842, 2024-02-10T07:46:59.180 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_acting": 3, 2024-02-10T07:46:59.180 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_stable": 3, 2024-02-10T07:46:59.180 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity_weighted": 0.92918205261230469, 2024-02-10T07:46:59.180 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity": 0.93000000715255737, 2024-02-10T07:46:59.180 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: "pool": 2, 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_name": "datapool", 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: "create_time": "2024-02-10T07:31:49.150741+0000", 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": 8193, 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: "type": 1, 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 3, 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: "min_size": 2, 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_rule": 0, 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_count": 0, 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_target": 0, 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T07:46:59.181 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "object_hash": 2, 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_mode": "off", 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num": 3, 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num": 3, 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num_target": 3, 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_target": 3, 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_pending": 3, 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "last_pg_merge_meta": { 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "source_pgid": "0.0", 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "ready_epoch": 0, 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_started": 0, 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_clean": 0, 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "source_version": "0'0", 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "target_version": "0'0" 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.182 INFO:teuthology.orchestra.run.smithi008.stdout: "last_change": "70", 2024-02-10T07:46:59.183 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend": "0", 2024-02-10T07:46:59.183 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T07:46:59.183 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T07:46:59.183 INFO:teuthology.orchestra.run.smithi008.stdout: "auid": 0, 2024-02-10T07:46:59.183 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_mode": "selfmanaged", 2024-02-10T07:46:59.183 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_seq": 2, 2024-02-10T07:46:59.183 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_epoch": 70, 2024-02-10T07:46:59.183 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_snaps": [], 2024-02-10T07:46:59.183 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps": "[]", 2024-02-10T07:46:59.183 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_bytes": 0, 2024-02-10T07:46:59.183 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_objects": 0, 2024-02-10T07:46:59.183 INFO:teuthology.orchestra.run.smithi008.stdout: "tiers": [], 2024-02-10T07:46:59.183 INFO:teuthology.orchestra.run.smithi008.stdout: "tier_of": -1, 2024-02-10T07:46:59.183 INFO:teuthology.orchestra.run.smithi008.stdout: "read_tier": -1, 2024-02-10T07:46:59.184 INFO:teuthology.orchestra.run.smithi008.stdout: "write_tier": -1, 2024-02-10T07:46:59.184 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_mode": "none", 2024-02-10T07:46:59.184 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_bytes": 0, 2024-02-10T07:46:59.184 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_objects": 0, 2024-02-10T07:46:59.184 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T07:46:59.184 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T07:46:59.184 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T07:46:59.184 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_flush_age": 0, 2024-02-10T07:46:59.184 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_evict_age": 0, 2024-02-10T07:46:59.184 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profile": "", 2024-02-10T07:46:59.184 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_params": { 2024-02-10T07:46:59.184 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "none" 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_period": 0, 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_count": 0, 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: "use_gmt_hitset": true, 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: "min_read_recency_for_promote": 0, 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: "min_write_recency_for_promote": 0, 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_search_last_n": 0, 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: "grade_table": [], 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: "stripe_width": 0, 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: "expected_num_objects": 0, 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: "fast_read": false, 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: "options": {}, 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: "application_metadata": { 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: "rbd": {} 2024-02-10T07:46:59.185 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: "read_balance": { 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: "score_type": "Fair distribution", 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: "score_acting": 2.6500000953674316, 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: "score_stable": 2.6500000953674316, 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: "optimal_score": 0.62999999523162842, 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_acting": 1.6699999570846558, 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_stable": 1.6699999570846558, 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity_weighted": 0.92918205261230469, 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity": 0.93000000715255737, 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: "pool": 3, 2024-02-10T07:46:59.186 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_name": ".rgw.root", 2024-02-10T07:46:59.187 INFO:teuthology.orchestra.run.smithi008.stdout: "create_time": "2024-02-10T07:31:49.831683+0000", 2024-02-10T07:46:59.187 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": 1, 2024-02-10T07:46:59.187 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_names": "hashpspool", 2024-02-10T07:46:59.187 INFO:teuthology.orchestra.run.smithi008.stdout: "type": 1, 2024-02-10T07:46:59.187 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 3, 2024-02-10T07:46:59.187 INFO:teuthology.orchestra.run.smithi008.stdout: "min_size": 2, 2024-02-10T07:46:59.187 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_rule": 0, 2024-02-10T07:46:59.187 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_count": 0, 2024-02-10T07:46:59.187 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_target": 0, 2024-02-10T07:46:59.187 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T07:46:59.187 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T07:46:59.187 INFO:teuthology.orchestra.run.smithi008.stdout: "object_hash": 2, 2024-02-10T07:46:59.187 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_mode": "off", 2024-02-10T07:46:59.187 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num": 32, 2024-02-10T07:46:59.188 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num": 32, 2024-02-10T07:46:59.188 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num_target": 32, 2024-02-10T07:46:59.188 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_target": 32, 2024-02-10T07:46:59.188 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_pending": 32, 2024-02-10T07:46:59.188 INFO:teuthology.orchestra.run.smithi008.stdout: "last_pg_merge_meta": { 2024-02-10T07:46:59.188 INFO:teuthology.orchestra.run.smithi008.stdout: "source_pgid": "0.0", 2024-02-10T07:46:59.188 INFO:teuthology.orchestra.run.smithi008.stdout: "ready_epoch": 0, 2024-02-10T07:46:59.188 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_started": 0, 2024-02-10T07:46:59.188 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_clean": 0, 2024-02-10T07:46:59.188 INFO:teuthology.orchestra.run.smithi008.stdout: "source_version": "0'0", 2024-02-10T07:46:59.188 INFO:teuthology.orchestra.run.smithi008.stdout: "target_version": "0'0" 2024-02-10T07:46:59.188 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.188 INFO:teuthology.orchestra.run.smithi008.stdout: "last_change": "68", 2024-02-10T07:46:59.188 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend": "0", 2024-02-10T07:46:59.189 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T07:46:59.189 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T07:46:59.189 INFO:teuthology.orchestra.run.smithi008.stdout: "auid": 0, 2024-02-10T07:46:59.189 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_mode": "selfmanaged", 2024-02-10T07:46:59.189 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_seq": 0, 2024-02-10T07:46:59.189 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_epoch": 0, 2024-02-10T07:46:59.189 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_snaps": [], 2024-02-10T07:46:59.189 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps": "[]", 2024-02-10T07:46:59.189 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_bytes": 0, 2024-02-10T07:46:59.189 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_objects": 0, 2024-02-10T07:46:59.189 INFO:teuthology.orchestra.run.smithi008.stdout: "tiers": [], 2024-02-10T07:46:59.189 INFO:teuthology.orchestra.run.smithi008.stdout: "tier_of": -1, 2024-02-10T07:46:59.189 INFO:teuthology.orchestra.run.smithi008.stdout: "read_tier": -1, 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: "write_tier": -1, 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_mode": "none", 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_bytes": 0, 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_objects": 0, 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_flush_age": 0, 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_evict_age": 0, 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profile": "", 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_params": { 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "none" 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_period": 0, 2024-02-10T07:46:59.190 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_count": 0, 2024-02-10T07:46:59.191 INFO:teuthology.orchestra.run.smithi008.stdout: "use_gmt_hitset": true, 2024-02-10T07:46:59.191 INFO:teuthology.orchestra.run.smithi008.stdout: "min_read_recency_for_promote": 0, 2024-02-10T07:46:59.191 INFO:teuthology.orchestra.run.smithi008.stdout: "min_write_recency_for_promote": 0, 2024-02-10T07:46:59.191 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T07:46:59.191 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_search_last_n": 0, 2024-02-10T07:46:59.191 INFO:teuthology.orchestra.run.smithi008.stdout: "grade_table": [], 2024-02-10T07:46:59.191 INFO:teuthology.orchestra.run.smithi008.stdout: "stripe_width": 0, 2024-02-10T07:46:59.191 INFO:teuthology.orchestra.run.smithi008.stdout: "expected_num_objects": 0, 2024-02-10T07:46:59.191 INFO:teuthology.orchestra.run.smithi008.stdout: "fast_read": false, 2024-02-10T07:46:59.191 INFO:teuthology.orchestra.run.smithi008.stdout: "options": {}, 2024-02-10T07:46:59.191 INFO:teuthology.orchestra.run.smithi008.stdout: "application_metadata": { 2024-02-10T07:46:59.191 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw": {} 2024-02-10T07:46:59.192 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.192 INFO:teuthology.orchestra.run.smithi008.stdout: "read_balance": { 2024-02-10T07:46:59.192 INFO:teuthology.orchestra.run.smithi008.stdout: "score_type": "Fair distribution", 2024-02-10T07:46:59.192 INFO:teuthology.orchestra.run.smithi008.stdout: "score_acting": 2, 2024-02-10T07:46:59.192 INFO:teuthology.orchestra.run.smithi008.stdout: "score_stable": 2, 2024-02-10T07:46:59.192 INFO:teuthology.orchestra.run.smithi008.stdout: "optimal_score": 0.75, 2024-02-10T07:46:59.192 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_acting": 1.5, 2024-02-10T07:46:59.192 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_stable": 1.5, 2024-02-10T07:46:59.192 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity_weighted": 0.92918205261230469, 2024-02-10T07:46:59.192 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity": 0.93000000715255737, 2024-02-10T07:46:59.192 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-02-10T07:46:59.192 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.193 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.193 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.193 INFO:teuthology.orchestra.run.smithi008.stdout: "pool": 4, 2024-02-10T07:46:59.193 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_name": "default.rgw.log", 2024-02-10T07:46:59.193 INFO:teuthology.orchestra.run.smithi008.stdout: "create_time": "2024-02-10T07:31:51.622544+0000", 2024-02-10T07:46:59.193 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": 1, 2024-02-10T07:46:59.193 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_names": "hashpspool", 2024-02-10T07:46:59.193 INFO:teuthology.orchestra.run.smithi008.stdout: "type": 1, 2024-02-10T07:46:59.193 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 3, 2024-02-10T07:46:59.193 INFO:teuthology.orchestra.run.smithi008.stdout: "min_size": 2, 2024-02-10T07:46:59.193 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_rule": 0, 2024-02-10T07:46:59.193 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_count": 0, 2024-02-10T07:46:59.193 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_target": 0, 2024-02-10T07:46:59.193 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T07:46:59.194 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T07:46:59.194 INFO:teuthology.orchestra.run.smithi008.stdout: "object_hash": 2, 2024-02-10T07:46:59.194 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_mode": "off", 2024-02-10T07:46:59.194 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num": 32, 2024-02-10T07:46:59.194 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num": 32, 2024-02-10T07:46:59.194 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num_target": 32, 2024-02-10T07:46:59.194 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_target": 32, 2024-02-10T07:46:59.194 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_pending": 32, 2024-02-10T07:46:59.194 INFO:teuthology.orchestra.run.smithi008.stdout: "last_pg_merge_meta": { 2024-02-10T07:46:59.194 INFO:teuthology.orchestra.run.smithi008.stdout: "source_pgid": "0.0", 2024-02-10T07:46:59.194 INFO:teuthology.orchestra.run.smithi008.stdout: "ready_epoch": 0, 2024-02-10T07:46:59.194 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_started": 0, 2024-02-10T07:46:59.194 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_clean": 0, 2024-02-10T07:46:59.194 INFO:teuthology.orchestra.run.smithi008.stdout: "source_version": "0'0", 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: "target_version": "0'0" 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: "last_change": "70", 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend": "0", 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: "auid": 0, 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_mode": "selfmanaged", 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_seq": 0, 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_epoch": 0, 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_snaps": [], 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps": "[]", 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_bytes": 0, 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_objects": 0, 2024-02-10T07:46:59.195 INFO:teuthology.orchestra.run.smithi008.stdout: "tiers": [], 2024-02-10T07:46:59.196 INFO:teuthology.orchestra.run.smithi008.stdout: "tier_of": -1, 2024-02-10T07:46:59.196 INFO:teuthology.orchestra.run.smithi008.stdout: "read_tier": -1, 2024-02-10T07:46:59.196 INFO:teuthology.orchestra.run.smithi008.stdout: "write_tier": -1, 2024-02-10T07:46:59.196 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_mode": "none", 2024-02-10T07:46:59.196 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_bytes": 0, 2024-02-10T07:46:59.196 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_objects": 0, 2024-02-10T07:46:59.196 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T07:46:59.196 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T07:46:59.196 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T07:46:59.196 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_flush_age": 0, 2024-02-10T07:46:59.196 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_evict_age": 0, 2024-02-10T07:46:59.196 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profile": "", 2024-02-10T07:46:59.197 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_params": { 2024-02-10T07:46:59.197 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "none" 2024-02-10T07:46:59.197 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.197 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_period": 0, 2024-02-10T07:46:59.197 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_count": 0, 2024-02-10T07:46:59.197 INFO:teuthology.orchestra.run.smithi008.stdout: "use_gmt_hitset": true, 2024-02-10T07:46:59.197 INFO:teuthology.orchestra.run.smithi008.stdout: "min_read_recency_for_promote": 0, 2024-02-10T07:46:59.197 INFO:teuthology.orchestra.run.smithi008.stdout: "min_write_recency_for_promote": 0, 2024-02-10T07:46:59.197 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T07:46:59.197 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_search_last_n": 0, 2024-02-10T07:46:59.197 INFO:teuthology.orchestra.run.smithi008.stdout: "grade_table": [], 2024-02-10T07:46:59.197 INFO:teuthology.orchestra.run.smithi008.stdout: "stripe_width": 0, 2024-02-10T07:46:59.197 INFO:teuthology.orchestra.run.smithi008.stdout: "expected_num_objects": 0, 2024-02-10T07:46:59.197 INFO:teuthology.orchestra.run.smithi008.stdout: "fast_read": false, 2024-02-10T07:46:59.198 INFO:teuthology.orchestra.run.smithi008.stdout: "options": {}, 2024-02-10T07:46:59.198 INFO:teuthology.orchestra.run.smithi008.stdout: "application_metadata": { 2024-02-10T07:46:59.198 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw": {} 2024-02-10T07:46:59.198 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.198 INFO:teuthology.orchestra.run.smithi008.stdout: "read_balance": { 2024-02-10T07:46:59.198 INFO:teuthology.orchestra.run.smithi008.stdout: "score_type": "Fair distribution", 2024-02-10T07:46:59.198 INFO:teuthology.orchestra.run.smithi008.stdout: "score_acting": 2.25, 2024-02-10T07:46:59.198 INFO:teuthology.orchestra.run.smithi008.stdout: "score_stable": 2.25, 2024-02-10T07:46:59.198 INFO:teuthology.orchestra.run.smithi008.stdout: "optimal_score": 0.75, 2024-02-10T07:46:59.198 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_acting": 1.690000057220459, 2024-02-10T07:46:59.198 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_stable": 1.690000057220459, 2024-02-10T07:46:59.198 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity_weighted": 0.92918205261230469, 2024-02-10T07:46:59.199 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity": 0.93000000715255737, 2024-02-10T07:46:59.199 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-02-10T07:46:59.199 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.199 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.199 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.199 INFO:teuthology.orchestra.run.smithi008.stdout: "pool": 5, 2024-02-10T07:46:59.199 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_name": "default.rgw.control", 2024-02-10T07:46:59.199 INFO:teuthology.orchestra.run.smithi008.stdout: "create_time": "2024-02-10T07:31:53.550827+0000", 2024-02-10T07:46:59.199 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": 1, 2024-02-10T07:46:59.199 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_names": "hashpspool", 2024-02-10T07:46:59.199 INFO:teuthology.orchestra.run.smithi008.stdout: "type": 1, 2024-02-10T07:46:59.199 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 3, 2024-02-10T07:46:59.199 INFO:teuthology.orchestra.run.smithi008.stdout: "min_size": 2, 2024-02-10T07:46:59.200 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_rule": 0, 2024-02-10T07:46:59.200 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_count": 0, 2024-02-10T07:46:59.200 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_target": 0, 2024-02-10T07:46:59.200 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T07:46:59.200 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T07:46:59.200 INFO:teuthology.orchestra.run.smithi008.stdout: "object_hash": 2, 2024-02-10T07:46:59.200 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_mode": "off", 2024-02-10T07:46:59.200 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num": 32, 2024-02-10T07:46:59.200 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num": 32, 2024-02-10T07:46:59.200 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num_target": 32, 2024-02-10T07:46:59.200 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_target": 32, 2024-02-10T07:46:59.200 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_pending": 32, 2024-02-10T07:46:59.200 INFO:teuthology.orchestra.run.smithi008.stdout: "last_pg_merge_meta": { 2024-02-10T07:46:59.200 INFO:teuthology.orchestra.run.smithi008.stdout: "source_pgid": "0.0", 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "ready_epoch": 0, 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_started": 0, 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_clean": 0, 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "source_version": "0'0", 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "target_version": "0'0" 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "last_change": "72", 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend": "0", 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "auid": 0, 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_mode": "selfmanaged", 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_seq": 0, 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_epoch": 0, 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_snaps": [], 2024-02-10T07:46:59.201 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps": "[]", 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_bytes": 0, 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_objects": 0, 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "tiers": [], 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "tier_of": -1, 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "read_tier": -1, 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "write_tier": -1, 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_mode": "none", 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_bytes": 0, 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_objects": 0, 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_flush_age": 0, 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_evict_age": 0, 2024-02-10T07:46:59.202 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profile": "", 2024-02-10T07:46:59.203 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_params": { 2024-02-10T07:46:59.203 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "none" 2024-02-10T07:46:59.203 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.203 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_period": 0, 2024-02-10T07:46:59.203 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_count": 0, 2024-02-10T07:46:59.203 INFO:teuthology.orchestra.run.smithi008.stdout: "use_gmt_hitset": true, 2024-02-10T07:46:59.203 INFO:teuthology.orchestra.run.smithi008.stdout: "min_read_recency_for_promote": 0, 2024-02-10T07:46:59.203 INFO:teuthology.orchestra.run.smithi008.stdout: "min_write_recency_for_promote": 0, 2024-02-10T07:46:59.203 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T07:46:59.203 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_search_last_n": 0, 2024-02-10T07:46:59.203 INFO:teuthology.orchestra.run.smithi008.stdout: "grade_table": [], 2024-02-10T07:46:59.203 INFO:teuthology.orchestra.run.smithi008.stdout: "stripe_width": 0, 2024-02-10T07:46:59.203 INFO:teuthology.orchestra.run.smithi008.stdout: "expected_num_objects": 0, 2024-02-10T07:46:59.203 INFO:teuthology.orchestra.run.smithi008.stdout: "fast_read": false, 2024-02-10T07:46:59.204 INFO:teuthology.orchestra.run.smithi008.stdout: "options": {}, 2024-02-10T07:46:59.204 INFO:teuthology.orchestra.run.smithi008.stdout: "application_metadata": { 2024-02-10T07:46:59.204 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw": {} 2024-02-10T07:46:59.204 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.204 INFO:teuthology.orchestra.run.smithi008.stdout: "read_balance": { 2024-02-10T07:46:59.204 INFO:teuthology.orchestra.run.smithi008.stdout: "score_type": "Fair distribution", 2024-02-10T07:46:59.204 INFO:teuthology.orchestra.run.smithi008.stdout: "score_acting": 1.5099999904632568, 2024-02-10T07:46:59.204 INFO:teuthology.orchestra.run.smithi008.stdout: "score_stable": 1.5099999904632568, 2024-02-10T07:46:59.204 INFO:teuthology.orchestra.run.smithi008.stdout: "optimal_score": 0.75, 2024-02-10T07:46:59.204 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_acting": 1.1299999952316284, 2024-02-10T07:46:59.204 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_stable": 1.1299999952316284, 2024-02-10T07:46:59.204 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity_weighted": 0.92918205261230469, 2024-02-10T07:46:59.205 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity": 0.93000000715255737, 2024-02-10T07:46:59.205 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-02-10T07:46:59.205 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.205 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.205 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.205 INFO:teuthology.orchestra.run.smithi008.stdout: "pool": 6, 2024-02-10T07:46:59.206 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_name": "default.rgw.meta", 2024-02-10T07:46:59.206 INFO:teuthology.orchestra.run.smithi008.stdout: "create_time": "2024-02-10T07:31:55.600117+0000", 2024-02-10T07:46:59.206 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": 1, 2024-02-10T07:46:59.206 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_names": "hashpspool", 2024-02-10T07:46:59.206 INFO:teuthology.orchestra.run.smithi008.stdout: "type": 1, 2024-02-10T07:46:59.206 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 3, 2024-02-10T07:46:59.206 INFO:teuthology.orchestra.run.smithi008.stdout: "min_size": 2, 2024-02-10T07:46:59.206 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_rule": 0, 2024-02-10T07:46:59.206 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_count": 0, 2024-02-10T07:46:59.206 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_target": 0, 2024-02-10T07:46:59.206 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T07:46:59.206 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T07:46:59.206 INFO:teuthology.orchestra.run.smithi008.stdout: "object_hash": 2, 2024-02-10T07:46:59.206 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_mode": "off", 2024-02-10T07:46:59.207 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num": 32, 2024-02-10T07:46:59.207 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num": 32, 2024-02-10T07:46:59.207 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num_target": 32, 2024-02-10T07:46:59.207 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_target": 32, 2024-02-10T07:46:59.207 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_pending": 32, 2024-02-10T07:46:59.207 INFO:teuthology.orchestra.run.smithi008.stdout: "last_pg_merge_meta": { 2024-02-10T07:46:59.207 INFO:teuthology.orchestra.run.smithi008.stdout: "source_pgid": "0.0", 2024-02-10T07:46:59.207 INFO:teuthology.orchestra.run.smithi008.stdout: "ready_epoch": 0, 2024-02-10T07:46:59.207 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_started": 0, 2024-02-10T07:46:59.207 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_clean": 0, 2024-02-10T07:46:59.207 INFO:teuthology.orchestra.run.smithi008.stdout: "source_version": "0'0", 2024-02-10T07:46:59.207 INFO:teuthology.orchestra.run.smithi008.stdout: "target_version": "0'0" 2024-02-10T07:46:59.207 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.207 INFO:teuthology.orchestra.run.smithi008.stdout: "last_change": "74", 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend": "0", 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "auid": 0, 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_mode": "selfmanaged", 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_seq": 0, 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_epoch": 0, 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_snaps": [], 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps": "[]", 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_bytes": 0, 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_objects": 0, 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "tiers": [], 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "tier_of": -1, 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "read_tier": -1, 2024-02-10T07:46:59.208 INFO:teuthology.orchestra.run.smithi008.stdout: "write_tier": -1, 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_mode": "none", 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_bytes": 0, 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_objects": 0, 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_flush_age": 0, 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_evict_age": 0, 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profile": "", 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_params": { 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "none" 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_period": 0, 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_count": 0, 2024-02-10T07:46:59.209 INFO:teuthology.orchestra.run.smithi008.stdout: "use_gmt_hitset": true, 2024-02-10T07:46:59.210 INFO:teuthology.orchestra.run.smithi008.stdout: "min_read_recency_for_promote": 0, 2024-02-10T07:46:59.210 INFO:teuthology.orchestra.run.smithi008.stdout: "min_write_recency_for_promote": 0, 2024-02-10T07:46:59.210 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T07:46:59.210 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_search_last_n": 0, 2024-02-10T07:46:59.210 INFO:teuthology.orchestra.run.smithi008.stdout: "grade_table": [], 2024-02-10T07:46:59.210 INFO:teuthology.orchestra.run.smithi008.stdout: "stripe_width": 0, 2024-02-10T07:46:59.210 INFO:teuthology.orchestra.run.smithi008.stdout: "expected_num_objects": 0, 2024-02-10T07:46:59.210 INFO:teuthology.orchestra.run.smithi008.stdout: "fast_read": false, 2024-02-10T07:46:59.210 INFO:teuthology.orchestra.run.smithi008.stdout: "options": { 2024-02-10T07:46:59.210 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_bias": 4 2024-02-10T07:46:59.210 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.210 INFO:teuthology.orchestra.run.smithi008.stdout: "application_metadata": { 2024-02-10T07:46:59.211 INFO:teuthology.orchestra.run.smithi008.stdout: "rgw": {} 2024-02-10T07:46:59.211 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.211 INFO:teuthology.orchestra.run.smithi008.stdout: "read_balance": { 2024-02-10T07:46:59.211 INFO:teuthology.orchestra.run.smithi008.stdout: "score_type": "Fair distribution", 2024-02-10T07:46:59.211 INFO:teuthology.orchestra.run.smithi008.stdout: "score_acting": 2, 2024-02-10T07:46:59.211 INFO:teuthology.orchestra.run.smithi008.stdout: "score_stable": 2, 2024-02-10T07:46:59.211 INFO:teuthology.orchestra.run.smithi008.stdout: "optimal_score": 0.75, 2024-02-10T07:46:59.211 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_acting": 1.5, 2024-02-10T07:46:59.211 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_stable": 1.5, 2024-02-10T07:46:59.211 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity_weighted": 0.92918205261230469, 2024-02-10T07:46:59.211 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity": 0.93000000715255737, 2024-02-10T07:46:59.211 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-02-10T07:46:59.211 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.211 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.212 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.212 INFO:teuthology.orchestra.run.smithi008.stdout: "pool": 10, 2024-02-10T07:46:59.212 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_name": "unique_pool_3", 2024-02-10T07:46:59.212 INFO:teuthology.orchestra.run.smithi008.stdout: "create_time": "2024-02-10T07:46:05.631764+0000", 2024-02-10T07:46:59.212 INFO:teuthology.orchestra.run.smithi008.stdout: "flags": 1, 2024-02-10T07:46:59.212 INFO:teuthology.orchestra.run.smithi008.stdout: "flags_names": "hashpspool", 2024-02-10T07:46:59.212 INFO:teuthology.orchestra.run.smithi008.stdout: "type": 1, 2024-02-10T07:46:59.212 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 3, 2024-02-10T07:46:59.212 INFO:teuthology.orchestra.run.smithi008.stdout: "min_size": 2, 2024-02-10T07:46:59.212 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_rule": 0, 2024-02-10T07:46:59.212 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_count": 0, 2024-02-10T07:46:59.212 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_target": 0, 2024-02-10T07:46:59.213 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_barrier": 0, 2024-02-10T07:46:59.213 INFO:teuthology.orchestra.run.smithi008.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-10T07:46:59.213 INFO:teuthology.orchestra.run.smithi008.stdout: "object_hash": 2, 2024-02-10T07:46:59.213 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_autoscale_mode": "off", 2024-02-10T07:46:59.213 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num": 16, 2024-02-10T07:46:59.213 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num": 16, 2024-02-10T07:46:59.213 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_placement_num_target": 16, 2024-02-10T07:46:59.213 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_target": 16, 2024-02-10T07:46:59.213 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_num_pending": 16, 2024-02-10T07:46:59.213 INFO:teuthology.orchestra.run.smithi008.stdout: "last_pg_merge_meta": { 2024-02-10T07:46:59.213 INFO:teuthology.orchestra.run.smithi008.stdout: "source_pgid": "0.0", 2024-02-10T07:46:59.213 INFO:teuthology.orchestra.run.smithi008.stdout: "ready_epoch": 0, 2024-02-10T07:46:59.213 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_started": 0, 2024-02-10T07:46:59.213 INFO:teuthology.orchestra.run.smithi008.stdout: "last_epoch_clean": 0, 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: "source_version": "0'0", 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: "target_version": "0'0" 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: "last_change": "316", 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend": "0", 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: "last_force_op_resend_preluminous": "0", 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: "auid": 0, 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_mode": "selfmanaged", 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_seq": 0, 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: "snap_epoch": 0, 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: "pool_snaps": [], 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps": "[]", 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_bytes": 0, 2024-02-10T07:46:59.214 INFO:teuthology.orchestra.run.smithi008.stdout: "quota_max_objects": 0, 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "tiers": [], 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "tier_of": -1, 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "read_tier": -1, 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "write_tier": -1, 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_mode": "none", 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_bytes": 0, 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "target_max_objects": 0, 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_flush_age": 0, 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "cache_min_evict_age": 0, 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profile": "", 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_params": { 2024-02-10T07:46:59.215 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "none" 2024-02-10T07:46:59.216 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.216 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_period": 0, 2024-02-10T07:46:59.216 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_count": 0, 2024-02-10T07:46:59.216 INFO:teuthology.orchestra.run.smithi008.stdout: "use_gmt_hitset": true, 2024-02-10T07:46:59.216 INFO:teuthology.orchestra.run.smithi008.stdout: "min_read_recency_for_promote": 0, 2024-02-10T07:46:59.216 INFO:teuthology.orchestra.run.smithi008.stdout: "min_write_recency_for_promote": 0, 2024-02-10T07:46:59.216 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_grade_decay_rate": 0, 2024-02-10T07:46:59.216 INFO:teuthology.orchestra.run.smithi008.stdout: "hit_set_search_last_n": 0, 2024-02-10T07:46:59.216 INFO:teuthology.orchestra.run.smithi008.stdout: "grade_table": [], 2024-02-10T07:46:59.216 INFO:teuthology.orchestra.run.smithi008.stdout: "stripe_width": 0, 2024-02-10T07:46:59.216 INFO:teuthology.orchestra.run.smithi008.stdout: "expected_num_objects": 0, 2024-02-10T07:46:59.216 INFO:teuthology.orchestra.run.smithi008.stdout: "fast_read": false, 2024-02-10T07:46:59.216 INFO:teuthology.orchestra.run.smithi008.stdout: "options": {}, 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: "application_metadata": { 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: "rados": {} 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: "read_balance": { 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: "score_type": "Fair distribution", 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: "score_acting": 3, 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: "score_stable": 3, 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: "optimal_score": 0.75, 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_acting": 2.25, 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: "raw_score_stable": 2.25, 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity_weighted": 0.92918205261230469, 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity": 0.93000000715255737, 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: "average_primary_affinity_weighted": 0.93000000715255737 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.217 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: "osds": [ 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 0, 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "63abc4f1-1c83-4e03-8592-959c51d10847", 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 1, 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 8, 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 325, 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 0, 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:46:59.218 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6802", 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6803", 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.219 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.220 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6804", 2024-02-10T07:46:59.220 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:46:59.220 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.220 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.220 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.220 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6805", 2024-02-10T07:46:59.220 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:46:59.220 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.220 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.220 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.220 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:46:59.220 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.220 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.220 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.221 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6808", 2024-02-10T07:46:59.221 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:46:59.221 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.221 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.221 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.221 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6809", 2024-02-10T07:46:59.221 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:46:59.221 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.221 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.221 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.221 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:46:59.221 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.221 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.222 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:59 smithi008 ceph-mon[29816]: pgmap v683: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 845 op/s; 5785/171183 objects degraded (3.379%); 6092/171183 objects misplaced (3.559%); 8.2 MiB/s, 131 objects/s recovering 2024-02-10T07:46:59.222 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:46:59 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3484983068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:59.222 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:59 smithi008 ceph-mon[40743]: pgmap v683: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 845 op/s; 5785/171183 objects degraded (3.379%); 6092/171183 objects misplaced (3.559%); 8.2 MiB/s, 131 objects/s recovering 2024-02-10T07:46:59.222 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:46:59 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3484983068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:59.223 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:46:59.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:59.223 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6806", 2024-02-10T07:46:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:46:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6807", 2024-02-10T07:46:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1442795455 2024-02-10T07:46:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.8:6803/1442795455", 2024-02-10T07:46:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.8:6805/1442795455", 2024-02-10T07:46:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.8:6809/1442795455", 2024-02-10T07:46:59.224 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.8:6807/1442795455", 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 1, 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "88307389-bfa8-402e-92b9-9fa5094262c4", 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 1, 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 13, 2024-02-10T07:46:59.225 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 325, 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 0, 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6810", 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6811", 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.226 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6812", 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6813", 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.227 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:46:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6816", 2024-02-10T07:46:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:46:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6817", 2024-02-10T07:46:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:46:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.228 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:46:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6814", 2024-02-10T07:46:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:46:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6815", 2024-02-10T07:46:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3278758893 2024-02-10T07:46:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.229 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.8:6811/3278758893", 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.8:6813/3278758893", 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.8:6817/3278758893", 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.8:6815/3278758893", 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 2, 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "c6120003-9ab5-4cf5-87ac-6351ed2c6226", 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 0, 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 0, 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:46:59.230 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 18, 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 309, 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 0, 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6818", 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.231 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6819", 2024-02-10T07:46:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:46:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:46:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6820", 2024-02-10T07:46:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:46:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.232 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6821", 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6824", 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6825", 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.233 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6822", 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6823", 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 481314802 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.234 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.8:6819/481314802", 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.8:6821/481314802", 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.8:6825/481314802", 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.8:6823/481314802", 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 3, 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "6ca8f6f3-013a-48cb-9083-be37c1767489", 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 0, 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 0.129425048828125, 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 0.4334564208984375, 2024-02-10T07:46:59.235 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:46:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:46:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 26, 2024-02-10T07:46:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 253, 2024-02-10T07:46:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 296, 2024-02-10T07:46:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:46:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:46:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6826", 2024-02-10T07:46:59.236 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:46:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6827", 2024-02-10T07:46:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:46:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:46:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6828", 2024-02-10T07:46:59.237 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:46:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6829", 2024-02-10T07:46:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:46:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:46:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.238 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6832", 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6833", 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6830", 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:46:59.239 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:6831", 2024-02-10T07:46:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3692331651 2024-02-10T07:46:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.8:6827/3692331651", 2024-02-10T07:46:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.8:6829/3692331651", 2024-02-10T07:46:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.8:6833/3692331651", 2024-02-10T07:46:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.8:6831/3692331651", 2024-02-10T07:46:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:46:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: "exists" 2024-02-10T07:46:59.240 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 4, 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "aa667766-e57f-447b-9a31-e3f3e61ac346", 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 1, 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 32, 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 325, 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 0, 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:46:59.241 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:46:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6800", 2024-02-10T07:46:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:46:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6801", 2024-02-10T07:46:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:46:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:46:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6802", 2024-02-10T07:46:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:46:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6803", 2024-02-10T07:46:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:46:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.243 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6806", 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6807", 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:46:59.244 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6804", 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6805", 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3997132504 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.170:6801/3997132504", 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.170:6803/3997132504", 2024-02-10T07:46:59.245 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.170:6807/3997132504", 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.170:6805/3997132504", 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 5, 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "2e9ab14d-1aff-4638-9f41-311f6b5f6770", 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 1, 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 40, 2024-02-10T07:46:59.246 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 139, 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 156, 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 325, 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 140, 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6808", 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3225334381 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6809", 2024-02-10T07:46:59.247 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3225334381 2024-02-10T07:46:59.248 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.248 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.248 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:46:59.248 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.248 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.248 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6810", 2024-02-10T07:46:59.248 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3225334381 2024-02-10T07:46:59.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.248 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.248 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6811", 2024-02-10T07:46:59.248 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3225334381 2024-02-10T07:46:59.249 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.249 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.249 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:46:59.249 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.249 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.249 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6814", 2024-02-10T07:46:59.249 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3225334381 2024-02-10T07:46:59.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.249 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.249 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6815", 2024-02-10T07:46:59.249 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3225334381 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6812", 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3225334381 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6813", 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3225334381 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.250 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.170:6809/3225334381", 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.170:6811/3225334381", 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.170:6815/3225334381", 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.170:6813/3225334381", 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 6, 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "61970c81-4b79-4743-a379-50d5d102b373", 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:46:59.251 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 1, 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 50, 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 325, 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 0, 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6816", 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.252 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.253 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.253 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6817", 2024-02-10T07:46:59.253 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:46:59.253 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.253 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.253 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:46:59.253 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.253 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.253 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6818", 2024-02-10T07:46:59.253 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:46:59.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.254 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.254 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6819", 2024-02-10T07:46:59.254 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:46:59.254 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.254 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.254 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:46:59.254 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.254 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.254 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6822", 2024-02-10T07:46:59.254 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:46:59.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.255 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.255 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6823", 2024-02-10T07:46:59.255 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:46:59.255 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.255 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.255 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:46:59.255 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.255 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.255 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.255 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6820", 2024-02-10T07:46:59.255 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:46:59.256 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.256 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.256 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.256 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6821", 2024-02-10T07:46:59.256 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 1764436984 2024-02-10T07:46:59.256 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.256 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.256 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.256 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.170:6817/1764436984", 2024-02-10T07:46:59.256 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.170:6819/1764436984", 2024-02-10T07:46:59.256 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.170:6823/1764436984", 2024-02-10T07:46:59.256 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.170:6821/1764436984", 2024-02-10T07:46:59.256 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:46:59.256 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:46:59.257 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:46:59.257 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.257 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.257 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 7, 2024-02-10T07:46:59.257 INFO:teuthology.orchestra.run.smithi008.stdout: "uuid": "2e303441-b454-4750-8c5a-96c5348129a8", 2024-02-10T07:46:59.257 INFO:teuthology.orchestra.run.smithi008.stdout: "up": 1, 2024-02-10T07:46:59.257 INFO:teuthology.orchestra.run.smithi008.stdout: "in": 1, 2024-02-10T07:46:59.257 INFO:teuthology.orchestra.run.smithi008.stdout: "weight": 1, 2024-02-10T07:46:59.257 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_affinity": 1, 2024-02-10T07:46:59.257 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_begin": 0, 2024-02-10T07:46:59.257 INFO:teuthology.orchestra.run.smithi008.stdout: "last_clean_end": 0, 2024-02-10T07:46:59.257 INFO:teuthology.orchestra.run.smithi008.stdout: "up_from": 60, 2024-02-10T07:46:59.257 INFO:teuthology.orchestra.run.smithi008.stdout: "up_thru": 325, 2024-02-10T07:46:59.258 INFO:teuthology.orchestra.run.smithi008.stdout: "down_at": 0, 2024-02-10T07:46:59.258 INFO:teuthology.orchestra.run.smithi008.stdout: "lost_at": 0, 2024-02-10T07:46:59.258 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addrs": { 2024-02-10T07:46:59.258 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.258 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:46:59.258 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:59.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:59.259 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.259 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.259 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6824", 2024-02-10T07:46:59.259 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:46:59.259 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.259 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.259 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.259 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6825", 2024-02-10T07:46:59.260 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:46:59.260 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.260 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.260 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.260 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addrs": { 2024-02-10T07:46:59.260 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.261 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.261 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6826", 2024-02-10T07:46:59.261 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:46:59.261 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.261 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.261 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6827", 2024-02-10T07:46:59.261 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:46:59.261 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.261 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.261 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.261 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addrs": { 2024-02-10T07:46:59.261 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.262 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.262 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.262 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6830", 2024-02-10T07:46:59.262 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:46:59.262 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.262 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.262 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.262 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6831", 2024-02-10T07:46:59.262 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:46:59.262 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.262 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.262 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.262 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addrs": { 2024-02-10T07:46:59.262 INFO:teuthology.orchestra.run.smithi008.stdout: "addrvec": [ 2024-02-10T07:46:59.263 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.263 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v2", 2024-02-10T07:46:59.263 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6828", 2024-02-10T07:46:59.263 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:46:59.263 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.263 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.263 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2024-02-10T07:46:59.263 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.170:6829", 2024-02-10T07:46:59.263 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 4048577717 2024-02-10T07:46:59.263 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.263 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.263 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.263 INFO:teuthology.orchestra.run.smithi008.stdout: "public_addr": "172.21.15.170:6825/4048577717", 2024-02-10T07:46:59.263 INFO:teuthology.orchestra.run.smithi008.stdout: "cluster_addr": "172.21.15.170:6827/4048577717", 2024-02-10T07:46:59.264 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_back_addr": "172.21.15.170:6831/4048577717", 2024-02-10T07:46:59.264 INFO:teuthology.orchestra.run.smithi008.stdout: "heartbeat_front_addr": "172.21.15.170:6829/4048577717", 2024-02-10T07:46:59.264 INFO:teuthology.orchestra.run.smithi008.stdout: "state": [ 2024-02-10T07:46:59.264 INFO:teuthology.orchestra.run.smithi008.stdout: "exists", 2024-02-10T07:46:59.264 INFO:teuthology.orchestra.run.smithi008.stdout: "up" 2024-02-10T07:46:59.264 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.264 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.264 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:46:59.264 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_xinfo": [ 2024-02-10T07:46:59.264 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.264 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 0, 2024-02-10T07:46:59.264 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:46:59.264 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:46:59.264 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:46:59.265 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:46:59.265 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:46:59.265 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:46:23.255441+0000", 2024-02-10T07:46:59.265 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 0 2024-02-10T07:46:59.265 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.265 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.265 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 1, 2024-02-10T07:46:59.265 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:46:59.265 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:46:59.265 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:46:59.265 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:46:59.265 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:46:59.265 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:45:55.514238+0000", 2024-02-10T07:46:59.265 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 0 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 2, 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 65536, 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:46:29.520399+0000", 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 0 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 3, 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:46:59.266 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:46:59.267 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:46:59.267 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:46:59.267 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:46:59.267 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:44:07.925841+0000", 2024-02-10T07:46:59.267 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 295 2024-02-10T07:46:59.267 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.267 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.267 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 4, 2024-02-10T07:46:59.267 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:46:59.267 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:46:59.267 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:46:59.267 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:46:59.267 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:46:59.267 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:46:49.461663+0000", 2024-02-10T07:46:59.268 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 0 2024-02-10T07:46:59.268 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.268 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.268 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 5, 2024-02-10T07:46:59.268 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:46:59.268 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:46:59.268 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:46:59.268 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:46:59.268 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:46:59.268 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:46:23.825003+0000", 2024-02-10T07:46:59.268 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 139 2024-02-10T07:46:59.268 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.268 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 6, 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:46:12.306030+0000", 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 0 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: "osd": 7, 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: "down_stamp": "0.000000", 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_probability": 0, 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: "laggy_interval": 0, 2024-02-10T07:46:59.269 INFO:teuthology.orchestra.run.smithi008.stdout: "features": 4540701547738038271, 2024-02-10T07:46:59.270 INFO:teuthology.orchestra.run.smithi008.stdout: "old_weight": 0, 2024-02-10T07:46:59.270 INFO:teuthology.orchestra.run.smithi008.stdout: "last_purged_snaps_scrub": "2024-02-10T07:46:24.515183+0000", 2024-02-10T07:46:59.270 INFO:teuthology.orchestra.run.smithi008.stdout: "dead_epoch": 0 2024-02-10T07:46:59.270 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.270 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:46:59.270 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_upmap": [ 2024-02-10T07:46:59.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.270 INFO:teuthology.orchestra.run.smithi008.stdout: "pgid": "6.0", 2024-02-10T07:46:59.270 INFO:teuthology.orchestra.run.smithi008.stdout: "osds": [ 2024-02-10T07:46:59.270 INFO:teuthology.orchestra.run.smithi008.stdout: 6, 2024-02-10T07:46:59.270 INFO:teuthology.orchestra.run.smithi008.stdout: 0, 2024-02-10T07:46:59.270 INFO:teuthology.orchestra.run.smithi008.stdout: 4 2024-02-10T07:46:59.270 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.270 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_upmap_items": [ 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: "pgid": "4.14", 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: "mappings": [ 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: "from": 3, 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: "to": 5 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_upmap_primaries": [], 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: "pg_temp": [ 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.271 INFO:teuthology.orchestra.run.smithi008.stdout: "pgid": "10.3", 2024-02-10T07:46:59.272 INFO:teuthology.orchestra.run.smithi008.stdout: "osds": [ 2024-02-10T07:46:59.272 INFO:teuthology.orchestra.run.smithi008.stdout: 5, 2024-02-10T07:46:59.272 INFO:teuthology.orchestra.run.smithi008.stdout: 4 2024-02-10T07:46:59.272 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.272 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.272 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.272 INFO:teuthology.orchestra.run.smithi008.stdout: "pgid": "10.5", 2024-02-10T07:46:59.272 INFO:teuthology.orchestra.run.smithi008.stdout: "osds": [ 2024-02-10T07:46:59.272 INFO:teuthology.orchestra.run.smithi008.stdout: 0, 2024-02-10T07:46:59.272 INFO:teuthology.orchestra.run.smithi008.stdout: 7 2024-02-10T07:46:59.272 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: "pgid": "10.7", 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: "osds": [ 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: 6, 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: 5 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: "pgid": "10.f", 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: "osds": [ 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: 6, 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: 4 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:46:59.273 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "primary_temp": [], 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "blocklist": { 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/2210906095": "2024-02-11T07:32:31.651879+0000", 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:6800/1412156979": "2024-02-11T07:27:16.371197+0000", 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/2515892714": "2024-02-11T07:26:49.500037+0000", 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:6801/4001702773": "2024-02-11T07:26:49.500037+0000", 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:6801/1412156979": "2024-02-11T07:27:16.371197+0000", 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/1462056527": "2024-02-11T07:27:16.371197+0000", 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:6800/4001702773": "2024-02-11T07:26:49.500037+0000", 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/1905866501": "2024-02-11T07:26:49.500037+0000", 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/2381650878": "2024-02-11T07:27:16.371197+0000", 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/802075678": "2024-02-11T07:26:49.500037+0000", 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:6801/2677792951": "2024-02-11T07:32:31.651879+0000", 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/3905205183": "2024-02-11T07:32:31.651879+0000", 2024-02-10T07:46:59.274 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/3314827874": "2024-02-11T07:32:31.651879+0000", 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/1624888680": "2024-02-11T07:32:31.651879+0000", 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/2789652913": "2024-02-11T07:27:16.371197+0000", 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:0/4074987889": "2024-02-11T07:32:31.651879+0000", 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: "172.21.15.8:6800/2677792951": "2024-02-11T07:32:31.651879+0000" 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: "range_blocklist": {}, 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: "erasure_code_profiles": { 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: "default": { 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: "crush-failure-domain": "osd", 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: "k": "2", 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: "m": "1", 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: "plugin": "jerasure", 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: "technique": "reed_sol_van" 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: "removed_snaps_queue": [], 2024-02-10T07:46:59.275 INFO:teuthology.orchestra.run.smithi008.stdout: "new_removed_snaps": [], 2024-02-10T07:46:59.276 INFO:teuthology.orchestra.run.smithi008.stdout: "new_purged_snaps": [], 2024-02-10T07:46:59.276 INFO:teuthology.orchestra.run.smithi008.stdout: "crush_node_flags": {}, 2024-02-10T07:46:59.276 INFO:teuthology.orchestra.run.smithi008.stdout: "device_class_flags": {}, 2024-02-10T07:46:59.276 INFO:teuthology.orchestra.run.smithi008.stdout: "stretch_mode": { 2024-02-10T07:46:59.276 INFO:teuthology.orchestra.run.smithi008.stdout: "stretch_mode_enabled": false, 2024-02-10T07:46:59.276 INFO:teuthology.orchestra.run.smithi008.stdout: "stretch_bucket_count": 0, 2024-02-10T07:46:59.276 INFO:teuthology.orchestra.run.smithi008.stdout: "degraded_stretch_mode": 0, 2024-02-10T07:46:59.276 INFO:teuthology.orchestra.run.smithi008.stdout: "recovering_stretch_mode": 0, 2024-02-10T07:46:59.276 INFO:teuthology.orchestra.run.smithi008.stdout: "stretch_mode_bucket": 0 2024-02-10T07:46:59.276 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:46:59.276 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:46:59.277 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:46:59.350 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:46:59.359 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:46:59.389 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:46:59.396 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:59.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:46:59.422 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:46:59.422 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:46:59.422 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:46:59.422 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:46:59.422 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-10T07:46:59.422 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:46:59.422 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:46:59.422 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:46:59.422 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:46:59.434 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:59 smithi170 ceph-mon[33394]: pgmap v683: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 845 op/s; 5785/171183 objects degraded (3.379%); 6092/171183 objects misplaced (3.559%); 8.2 MiB/s, 131 objects/s recovering 2024-02-10T07:46:59.434 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:46:59 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3484983068' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:46:59.498 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:46:59.540 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:59.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:59.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:46:59.643 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:46:59.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:59.677 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:59.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:46:59.779 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:46:59.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:59.814 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:46:59.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:46:59.915 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:46:59.947 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 39 16 61621 61605 98.7177 60.25 0.0219381 0.0101241 2024-02-10T07:46:59.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:46:59.995 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:46:59.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:00.096 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:00.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:00.141 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:00.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:00.242 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:00.272 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:00.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:00.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:00.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:00 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/238287623' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T07:47:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:00 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:47:00.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:00 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/238287623' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T07:47:00.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:00 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:47:00.374 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:00.410 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:00.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:00.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:00.512 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:00.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:00.539 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:00.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:00 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/238287623' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-10T07:47:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:00 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:47:00.640 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:00.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:00.681 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:00.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:00.783 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:00.812 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:00.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:00.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:00.913 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:00.947 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:47:00.948154+0000 min lat: 0.00147754 max lat: 2.14964 avg lat: 0.010238 2024-02-10T07:47:00.947 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:47:00.947 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 40 16 62501 62485 97.6247 55 0.0174126 0.010238 2024-02-10T07:47:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:00.953 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:00.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:01.054 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:01.084 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:01.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:01.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:01.186 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:01.217 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:01.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:01.319 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:01.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:01 smithi008 ceph-mon[29816]: pgmap v684: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s; 5925/180435 objects degraded (3.284%); 5611/180435 objects misplaced (3.110%); 11 MiB/s, 168 objects/s recovering 2024-02-10T07:47:01.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:01 smithi008 ceph-mon[40743]: pgmap v684: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s; 5925/180435 objects degraded (3.284%); 5611/180435 objects misplaced (3.110%); 11 MiB/s, 168 objects/s recovering 2024-02-10T07:47:01.349 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:01.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:01.451 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:01.491 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:01.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:01.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:01.593 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:01 smithi170 ceph-mon[33394]: pgmap v684: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 63 MiB/s wr, 1.01k op/s; 5925/180435 objects degraded (3.284%); 5611/180435 objects misplaced (3.110%); 11 MiB/s, 168 objects/s recovering 2024-02-10T07:47:01.634 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:01.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:01.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:01.736 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:01.775 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:01.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:01.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:01.877 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:01.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:01.939 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:01.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:01.947 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 41 16 63411 63395 96.6307 56.875 0.0126903 0.0103429 2024-02-10T07:47:02.039 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:02.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:02.093 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:02.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:02.194 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:02.221 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:02.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:02.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:02.323 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:02.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:02 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 5925/180435 objects degraded (3.284%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:02 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 5925/180435 objects degraded (3.284%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:02.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:02 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 5925/180435 objects degraded (3.284%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:02.850 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:02.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:02.947 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 42 15 64285 64270 95.6319 54.6875 0.0138828 0.0104508 2024-02-10T07:47:02.951 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:02.979 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:02.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:03.081 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:03.121 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:03.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:03.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:03.222 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:03.263 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:03.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:03.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:03.365 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:03.422 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:03.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:03.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:03.478 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:03 smithi008 ceph-mon[29816]: pgmap v685: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 860 B/s rd, 53 MiB/s wr, 853 op/s; 5925/180435 objects degraded (3.284%); 5611/180435 objects misplaced (3.110%); 8.8 MiB/s, 141 objects/s recovering 2024-02-10T07:47:03.478 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:03.478 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:47:03.478 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:03 smithi008 ceph-mon[40743]: pgmap v685: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 860 B/s rd, 53 MiB/s wr, 853 op/s; 5925/180435 objects degraded (3.284%); 5611/180435 objects misplaced (3.110%); 8.8 MiB/s, 141 objects/s recovering 2024-02-10T07:47:03.478 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:03.478 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:47:03.518 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:47:03.519 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:47:03.523 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:03.565 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:03.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:03.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:03 smithi170 ceph-mon[33394]: pgmap v685: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.7 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 860 B/s rd, 53 MiB/s wr, 853 op/s; 5925/180435 objects degraded (3.284%); 5611/180435 objects misplaced (3.110%); 8.8 MiB/s, 141 objects/s recovering 2024-02-10T07:47:03.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:03.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:47:03.668 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:03.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:03.725 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:03.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:03.826 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:03.856 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:03.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:03.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:03.885 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:47:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:47:03.947 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 43 16 65181 65165 94.7087 55.9375 0.00978288 0.0105519 2024-02-10T07:47:03.958 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:04.013 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:04.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:04.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:04.115 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:04.189 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:04.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:04.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:04.291 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:04.334 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:04.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:04.435 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:04.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:04.484 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:04.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:04.585 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:04.634 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:04.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:04.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:04.735 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:04.806 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:04.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:04.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:04.908 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:04.938 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:04.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:04.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:04.947 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 44 16 66089 66073 93.8459 56.75 0.0111025 0.0106503 2024-02-10T07:47:05.040 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:05.056 INFO:tasks.thrashosds.thrasher:j is {'epoch': 329, 'fsid': '8c8244e6-c7e5-11ee-95b8-87774f69a715', 'created': '2024-02-10T07:26:21.316239+0000', 'modified': '2024-02-10T07:46:50.140309+0000', 'last_up_change': '2024-02-10T07:45:02.292328+0000', 'last_in_change': '2024-02-10T07:46:09.693935+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-10T07:29:32.704872+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9291820526123047, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.9300000071525574}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-10T07:31:49.150741+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.9291820526123047, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.9300000071525574}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-10T07:31:49.831683+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9291820526123047, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.9300000071525574}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-10T07:31:51.622544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 0.75, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9291820526123047, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.9300000071525574}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-10T07:31:53.550827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5099999904632568, 'score_stable': 1.5099999904632568, 'optimal_score': 0.75, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 0.9291820526123047, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.9300000071525574}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-10T07:31:55.600117+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9291820526123047, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.9300000071525574}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-02-10T07:46:05.631764+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '316', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9291820526123047, 'average_primary_affinity': 0.9300000071525574, 'average_primary_affinity_weighted': 0.9300000071525574}}], 'osds': [{'osd': 0, 'uuid': '63abc4f1-1c83-4e03-8592-959c51d10847', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 325, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6802', 'nonce': 1442795455}, {'type': 'v1', 'addr': '172.21.15.8:6803', 'nonce': 1442795455}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6804', 'nonce': 1442795455}, {'type': 'v1', 'addr': '172.21.15.8:6805', 'nonce': 1442795455}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6808', 'nonce': 1442795455}, {'type': 'v1', 'addr': '172.21.15.8:6809', 'nonce': 1442795455}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6806', 'nonce': 1442795455}, {'type': 'v1', 'addr': '172.21.15.8:6807', 'nonce': 1442795455}]}, 'public_addr': '172.21.15.8:6803/1442795455', 'cluster_addr': '172.21.15.8:6805/1442795455', 'heartbeat_back_addr': '172.21.15.8:6809/1442795455', 'heartbeat_front_addr': '172.21.15.8:6807/1442795455', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '88307389-bfa8-402e-92b9-9fa5094262c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 325, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6810', 'nonce': 3278758893}, {'type': 'v1', 'addr': '172.21.15.8:6811', 'nonce': 3278758893}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6812', 'nonce': 3278758893}, {'type': 'v1', 'addr': '172.21.15.8:6813', 'nonce': 3278758893}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6816', 'nonce': 3278758893}, {'type': 'v1', 'addr': '172.21.15.8:6817', 'nonce': 3278758893}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6814', 'nonce': 3278758893}, {'type': 'v1', 'addr': '172.21.15.8:6815', 'nonce': 3278758893}]}, 'public_addr': '172.21.15.8:6811/3278758893', 'cluster_addr': '172.21.15.8:6813/3278758893', 'heartbeat_back_addr': '172.21.15.8:6817/3278758893', 'heartbeat_front_addr': '172.21.15.8:6815/3278758893', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c6120003-9ab5-4cf5-87ac-6351ed2c6226', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 309, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6818', 'nonce': 481314802}, {'type': 'v1', 'addr': '172.21.15.8:6819', 'nonce': 481314802}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6820', 'nonce': 481314802}, {'type': 'v1', 'addr': '172.21.15.8:6821', 'nonce': 481314802}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6824', 'nonce': 481314802}, {'type': 'v1', 'addr': '172.21.15.8:6825', 'nonce': 481314802}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6822', 'nonce': 481314802}, {'type': 'v1', 'addr': '172.21.15.8:6823', 'nonce': 481314802}]}, 'public_addr': '172.21.15.8:6819/481314802', 'cluster_addr': '172.21.15.8:6821/481314802', 'heartbeat_back_addr': '172.21.15.8:6825/481314802', 'heartbeat_front_addr': '172.21.15.8:6823/481314802', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6ca8f6f3-013a-48cb-9083-be37c1767489', 'up': 0, 'in': 1, 'weight': 0.129425048828125, 'primary_affinity': 0.4334564208984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 253, 'down_at': 296, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6826', 'nonce': 3692331651}, {'type': 'v1', 'addr': '172.21.15.8:6827', 'nonce': 3692331651}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6828', 'nonce': 3692331651}, {'type': 'v1', 'addr': '172.21.15.8:6829', 'nonce': 3692331651}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6832', 'nonce': 3692331651}, {'type': 'v1', 'addr': '172.21.15.8:6833', 'nonce': 3692331651}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.8:6830', 'nonce': 3692331651}, {'type': 'v1', 'addr': '172.21.15.8:6831', 'nonce': 3692331651}]}, 'public_addr': '172.21.15.8:6827/3692331651', 'cluster_addr': '172.21.15.8:6829/3692331651', 'heartbeat_back_addr': '172.21.15.8:6833/3692331651', 'heartbeat_front_addr': '172.21.15.8:6831/3692331651', 'state': ['exists']}, {'osd': 4, 'uuid': 'aa667766-e57f-447b-9a31-e3f3e61ac346', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 325, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6800', 'nonce': 3997132504}, {'type': 'v1', 'addr': '172.21.15.170:6801', 'nonce': 3997132504}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6802', 'nonce': 3997132504}, {'type': 'v1', 'addr': '172.21.15.170:6803', 'nonce': 3997132504}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6806', 'nonce': 3997132504}, {'type': 'v1', 'addr': '172.21.15.170:6807', 'nonce': 3997132504}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6804', 'nonce': 3997132504}, {'type': 'v1', 'addr': '172.21.15.170:6805', 'nonce': 3997132504}]}, 'public_addr': '172.21.15.170:6801/3997132504', 'cluster_addr': '172.21.15.170:6803/3997132504', 'heartbeat_back_addr': '172.21.15.170:6807/3997132504', 'heartbeat_front_addr': '172.21.15.170:6805/3997132504', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '2e9ab14d-1aff-4638-9f41-311f6b5f6770', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 40, 'last_clean_end': 139, 'up_from': 156, 'up_thru': 325, 'down_at': 140, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6808', 'nonce': 3225334381}, {'type': 'v1', 'addr': '172.21.15.170:6809', 'nonce': 3225334381}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6810', 'nonce': 3225334381}, {'type': 'v1', 'addr': '172.21.15.170:6811', 'nonce': 3225334381}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6814', 'nonce': 3225334381}, {'type': 'v1', 'addr': '172.21.15.170:6815', 'nonce': 3225334381}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6812', 'nonce': 3225334381}, {'type': 'v1', 'addr': '172.21.15.170:6813', 'nonce': 3225334381}]}, 'public_addr': '172.21.15.170:6809/3225334381', 'cluster_addr': '172.21.15.170:6811/3225334381', 'heartbeat_back_addr': '172.21.15.170:6815/3225334381', 'heartbeat_front_addr': '172.21.15.170:6813/3225334381', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '61970c81-4b79-4743-a379-50d5d102b373', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 325, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6816', 'nonce': 1764436984}, {'type': 'v1', 'addr': '172.21.15.170:6817', 'nonce': 1764436984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6818', 'nonce': 1764436984}, {'type': 'v1', 'addr': '172.21.15.170:6819', 'nonce': 1764436984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6822', 'nonce': 1764436984}, {'type': 'v1', 'addr': '172.21.15.170:6823', 'nonce': 1764436984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6820', 'nonce': 1764436984}, {'type': 'v1', 'addr': '172.21.15.170:6821', 'nonce': 1764436984}]}, 'public_addr': '172.21.15.170:6817/1764436984', 'cluster_addr': '172.21.15.170:6819/1764436984', 'heartbeat_back_addr': '172.21.15.170:6823/1764436984', 'heartbeat_front_addr': '172.21.15.170:6821/1764436984', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '2e303441-b454-4750-8c5a-96c5348129a8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 325, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6824', 'nonce': 4048577717}, {'type': 'v1', 'addr': '172.21.15.170:6825', 'nonce': 4048577717}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6826', 'nonce': 4048577717}, {'type': 'v1', 'addr': '172.21.15.170:6827', 'nonce': 4048577717}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6830', 'nonce': 4048577717}, {'type': 'v1', 'addr': '172.21.15.170:6831', 'nonce': 4048577717}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6828', 'nonce': 4048577717}, {'type': 'v1', 'addr': '172.21.15.170:6829', 'nonce': 4048577717}]}, 'public_addr': '172.21.15.170:6825/4048577717', 'cluster_addr': '172.21.15.170:6827/4048577717', 'heartbeat_back_addr': '172.21.15.170:6831/4048577717', 'heartbeat_front_addr': '172.21.15.170:6829/4048577717', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:46:23.255441+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:45:55.514238+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-10T07:46:29.520399+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:44:07.925841+0000', 'dead_epoch': 295}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:46:49.461663+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:46:23.825003+0000', 'dead_epoch': 139}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:46:12.306030+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-10T07:46:24.515183+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '6.0', 'osds': [6, 0, 4]}], 'pg_upmap_items': [{'pgid': '4.14', 'mappings': [{'from': 3, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '10.3', 'osds': [5, 4]}, {'pgid': '10.5', 'osds': [0, 7]}, {'pgid': '10.7', 'osds': [6, 5]}, {'pgid': '10.f', 'osds': [6, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.8:0/2210906095': '2024-02-11T07:32:31.651879+0000', '172.21.15.8:6800/1412156979': '2024-02-11T07:27:16.371197+0000', '172.21.15.8:0/2515892714': '2024-02-11T07:26:49.500037+0000', '172.21.15.8:6801/4001702773': '2024-02-11T07:26:49.500037+0000', '172.21.15.8:6801/1412156979': '2024-02-11T07:27:16.371197+0000', '172.21.15.8:0/1462056527': '2024-02-11T07:27:16.371197+0000', '172.21.15.8:6800/4001702773': '2024-02-11T07:26:49.500037+0000', '172.21.15.8:0/1905866501': '2024-02-11T07:26:49.500037+0000', '172.21.15.8:0/2381650878': '2024-02-11T07:27:16.371197+0000', '172.21.15.8:0/802075678': '2024-02-11T07:26:49.500037+0000', '172.21.15.8:6801/2677792951': '2024-02-11T07:32:31.651879+0000', '172.21.15.8:0/3905205183': '2024-02-11T07:32:31.651879+0000', '172.21.15.8:0/3314827874': '2024-02-11T07:32:31.651879+0000', '172.21.15.8:0/1624888680': '2024-02-11T07:32:31.651879+0000', '172.21.15.8:0/2789652913': '2024-02-11T07:27:16.371197+0000', '172.21.15.8:0/4074987889': '2024-02-11T07:32:31.651879+0000', '172.21.15.8:6800/2677792951': '2024-02-11T07:32:31.651879+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-10T07:47:05.056 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:47:05.084 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:05.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:05.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:05.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:05.244 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:05.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:05.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:05 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:05.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:05 smithi008 ceph-mon[29816]: pgmap v686: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.13k op/s; 5759/190686 objects degraded (3.020%); 5181/190686 objects misplaced (2.717%); 12 MiB/s, 189 objects/s recovering 2024-02-10T07:47:05.329 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:05 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:05.329 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:05 smithi008 ceph-mon[40743]: pgmap v686: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.13k op/s; 5759/190686 objects degraded (3.020%); 5181/190686 objects misplaced (2.717%); 12 MiB/s, 189 objects/s recovering 2024-02-10T07:47:05.345 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:05.398 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:05.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:05.499 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:05.520 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:47:05.535 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:05.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:05.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:05 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:05.586 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:05 smithi170 ceph-mon[33394]: pgmap v686: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 71 MiB/s wr, 1.13k op/s; 5759/190686 objects degraded (3.020%); 5181/190686 objects misplaced (2.717%); 12 MiB/s, 189 objects/s recovering 2024-02-10T07:47:05.637 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:05.671 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:47:05.671 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:47:05.671 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-02-10T07:47:05.671 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-02-10T07:47:05.671 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:47:05.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:05.676 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:05.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:05.777 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:05.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:05.822 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:05.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:05.923 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:05.948 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 45 16 67036 67020 93.0756 59.1875 0.00818544 0.0107375 2024-02-10T07:47:05.953 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:05.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:06.055 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:06.097 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:06.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:06.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:06.199 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:06.265 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:06.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:06.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:06.367 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:06.436 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:06.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:06.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:06.538 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:06.579 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:06.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:06.680 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:06.718 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:06.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:06.819 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:06.851 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:06.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:06.926 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-10T07:47:06.933 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:06.948 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 46 16 67962 67946 92.3093 57.875 0.0279988 0.0108265 2024-02-10T07:47:06.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:06.982 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:06.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:06.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:07.084 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:07.111 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:07.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:07.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:07.195 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:47:07.198 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:07 smithi170 ceph-mon[33394]: pgmap v687: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 896 op/s; 6041/194133 objects degraded (3.112%); 4895/194133 objects misplaced (2.521%); 11 MiB/s, 169 objects/s recovering 2024-02-10T07:47:07.198 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:07 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1392428700' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:07.199 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:07 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:07.199 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:07 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 6041/194133 objects degraded (3.112%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:07.213 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:07.265 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:07.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:07.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:07.366 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:07.411 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:07.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:07.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:07.512 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:07.530 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:07 smithi008 ceph-mon[29816]: pgmap v687: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 896 op/s; 6041/194133 objects degraded (3.112%); 4895/194133 objects misplaced (2.521%); 11 MiB/s, 169 objects/s recovering 2024-02-10T07:47:07.530 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:07 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1392428700' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:07.530 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:07 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:07.530 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:07 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 6041/194133 objects degraded (3.112%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:07.531 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:07 smithi008 ceph-mon[40743]: pgmap v687: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 896 op/s; 6041/194133 objects degraded (3.112%); 4895/194133 objects misplaced (2.521%); 11 MiB/s, 169 objects/s recovering 2024-02-10T07:47:07.531 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:07 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1392428700' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:07.531 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:07 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:07.531 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:07 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 6041/194133 objects degraded (3.112%), 7 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:07.552 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:07.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:07.653 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:07.683 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:07.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:07.785 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:07.812 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:07.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:07.914 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:07.938 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:07.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:07.948 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 47 16 68714 68698 91.3452 47 0.00445876 0.010924 2024-02-10T07:47:08.040 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:08.084 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:08.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:08.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:08.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:08.226 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:08.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:08.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:08.328 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:08.368 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:08.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:08.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:08.470 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:08.496 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:08.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:08.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:08.569 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:08 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:47:08.569 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:08 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:47:08.569 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:08 smithi008 ceph-mon[29816]: osdmap e330: 8 total, 7 up, 7 in 2024-02-10T07:47:08.570 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:08 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:47:08.570 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:08 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:47:08.570 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:08 smithi008 ceph-mon[40743]: osdmap e330: 8 total, 7 up, 7 in 2024-02-10T07:47:08.598 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:08.636 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:08.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:08.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:08.638 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:08 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:47:08.638 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:08 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:47:08.638 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:08 smithi170 ceph-mon[33394]: osdmap e330: 8 total, 7 up, 7 in 2024-02-10T07:47:08.737 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:08.767 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:08.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:08.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:08.831 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:47:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:47:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:47:08.869 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:08.915 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:08.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:08.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:08.948 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 48 16 69562 69546 90.5463 53 0.0154798 0.0110373 2024-02-10T07:47:09.017 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:09.079 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:09.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:09.182 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:09.211 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:09.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:09.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:09.239 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:09 smithi008 ceph-mon[29816]: pgmap v689: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 511 B/s rd, 56 MiB/s wr, 898 op/s; 6041/198132 objects degraded (3.049%); 4638/198132 objects misplaced (2.341%); 11 MiB/s, 180 objects/s recovering 2024-02-10T07:47:09.239 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:09 smithi008 ceph-mon[40743]: pgmap v689: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 511 B/s rd, 56 MiB/s wr, 898 op/s; 6041/198132 objects degraded (3.049%); 4638/198132 objects misplaced (2.341%); 11 MiB/s, 180 objects/s recovering 2024-02-10T07:47:09.313 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:09.368 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:09.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:09.469 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:09.513 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:09.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:09.614 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:09 smithi170 ceph-mon[33394]: pgmap v689: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 511 B/s rd, 56 MiB/s wr, 898 op/s; 6041/198132 objects degraded (3.049%); 4638/198132 objects misplaced (2.341%); 11 MiB/s, 180 objects/s recovering 2024-02-10T07:47:09.643 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:09.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:09.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:09.744 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:09.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:09.792 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:09.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:09.893 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:09.932 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:09.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:09.948 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 49 16 70494 70478 89.8871 58.25 0.00941786 0.01112 2024-02-10T07:47:10.034 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:10.065 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:10.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:10.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:10.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:10.232 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:10.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:10.290 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:10.291 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:10.333 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:10.375 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:10.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:10.477 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:10.525 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:47:10.528 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:47:10.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:10.529 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:10.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:10.630 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:10.660 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:10.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:10.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:10.728 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:10 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:10.728 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:10 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:10.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:10 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:10.728 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:10 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:10.762 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:10.789 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:10.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:10 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:10 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:10.890 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:10.930 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:10.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:10.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:10.948 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 50 16 71381 71365 89.198 55.4375 0.0107085 0.0112043 2024-02-10T07:47:11.026 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:47:11.032 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:11.102 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:11.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:11.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:11.204 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:11.249 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:11.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:11.350 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:11.389 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:11.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:11.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:11.491 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:11.528 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:11.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:11.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:11.632 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:11.660 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:11.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:11.762 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:11.806 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:11.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:11.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:11.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:11 smithi008 ceph-mon[29816]: pgmap v690: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 56 MiB/s wr, 896 op/s; 6184/207309 objects degraded (2.983%); 4101/207309 objects misplaced (1.978%); 11 MiB/s, 180 objects/s recovering 2024-02-10T07:47:11.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:11 smithi008 ceph-mon[40743]: pgmap v690: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 56 MiB/s wr, 896 op/s; 6184/207309 objects degraded (2.983%); 4101/207309 objects misplaced (1.978%); 11 MiB/s, 180 objects/s recovering 2024-02-10T07:47:11.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:11 smithi170 ceph-mon[33394]: pgmap v690: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 56 MiB/s wr, 896 op/s; 6184/207309 objects degraded (2.983%); 4101/207309 objects misplaced (1.978%); 11 MiB/s, 180 objects/s recovering 2024-02-10T07:47:11.909 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:11.940 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:11.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:11.948 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 51 16 72374 72358 88.6658 62.0625 0.0124643 0.0112731 2024-02-10T07:47:12.042 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:12.081 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:12.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:12.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:12.183 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:12.215 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:12.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:12.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:12.317 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:12.346 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:12.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:12.448 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:12.491 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:12.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:12.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:12.593 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:12.635 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:12.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:12.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:12.737 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:12.766 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:12.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:12.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:12 smithi008 ceph-mon[29816]: from='client.25297 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:47:12.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:12 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 6184/207309 objects degraded (2.983%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:12.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:12 smithi008 ceph-mon[29816]: osdmap e331: 8 total, 7 up, 7 in 2024-02-10T07:47:12.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:12 smithi008 ceph-mon[40743]: from='client.25297 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:47:12.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:12 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 6184/207309 objects degraded (2.983%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:12.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:12 smithi008 ceph-mon[40743]: osdmap e331: 8 total, 7 up, 7 in 2024-02-10T07:47:12.868 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:12 smithi170 ceph-mon[33394]: from='client.25297 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:47:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:12 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 6184/207309 objects degraded (2.983%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:12 smithi170 ceph-mon[33394]: osdmap e331: 8 total, 7 up, 7 in 2024-02-10T07:47:12.897 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:12.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:12.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:12.949 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 52 16 73357 73341 88.1421 61.4375 0.00939612 0.0113392 2024-02-10T07:47:12.999 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:13.030 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:13.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:13.132 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:13.151 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":690,"stamp":"2024-02-10T07:47:10.045212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4516354396,"num_objects":69103,"num_object_clones":0,"num_object_copies":207309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6184,"num_objects_misplaced":4101,"num_objects_unfound":0,"num_objects_dirty":69103,"num_whiteouts":0,"num_read":2261,"num_read_kb":2107,"num_write":69806,"num_write_kb":4412381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4745,"num_bytes_recovered":300882557,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67601,"ondisk_log_size":67601,"up":434,"acting":430,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":496,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":18045952,"kb_used_data":14414724,"kb_used_omap":13,"kb_used_meta":3631026,"kb_avail":638170112,"statfs":{"total":671965249536,"available":653486194688,"internally_reserved":0,"allocated":14760677376,"data_stored":14757502766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14007,"internal_metadata":3718170953},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4,2,2,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":587071488,"num_objects":8958,"num_object_clones":0,"num_object_copies":26874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":259,"num_objects_misplaced":-1510,"num_objects_unfound":0,"num_objects_dirty":8958,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8958,"num_write_kb":573312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1801,"num_bytes_recovered":118030336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002934"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":587,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.191104+0000","last_change":"2024-02-10T07:45:43.013900+0000","last_active":"2024-02-10T07:47:07.191104+0000","last_peered":"2024-02-10T07:47:07.191104+0000","last_clean":"2024-02-10T07:47:07.191104+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:47:07.191104+0000","last_undegraded":"2024-02-10T07:47:07.191104+0000","last_fullsized":"2024-02-10T07:47:07.191104+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:43.013774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:45:43.013774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":597,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.204629+0000","last_change":"2024-02-10T07:45:53.422092+0000","last_active":"2024-02-10T07:47:07.204629+0000","last_peered":"2024-02-10T07:47:07.204629+0000","last_clean":"2024-02-10T07:47:07.204629+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:47:07.204629+0000","last_undegraded":"2024-02-10T07:47:07.204629+0000","last_fullsized":"2024-02-10T07:47:07.204629+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:53.421994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:53.421994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:10.282402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":616,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202280+0000","last_change":"2024-02-10T07:46:10.735143+0000","last_active":"2024-02-10T07:47:07.202280+0000","last_peered":"2024-02-10T07:47:07.202280+0000","last_clean":"2024-02-10T07:47:07.202280+0000","last_became_active":"2024-02-10T07:46:10.718945+0000","last_became_peered":"2024-02-10T07:46:10.718945+0000","last_unstale":"2024-02-10T07:47:07.202280+0000","last_undegraded":"2024-02-10T07:47:07.202280+0000","last_fullsized":"2024-02-10T07:47:07.202280+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:44:51.181293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:44:51.181293+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"287'19","reported_seq":655,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.190927+0000","last_change":"2024-02-10T07:46:37.008978+0000","last_active":"2024-02-10T07:47:07.190927+0000","last_peered":"2024-02-10T07:47:07.190927+0000","last_clean":"2024-02-10T07:47:07.190927+0000","last_became_active":"2024-02-10T07:46:10.717475+0000","last_became_peered":"2024-02-10T07:46:10.717475+0000","last_unstale":"2024-02-10T07:47:07.190927+0000","last_undegraded":"2024-02-10T07:47:07.190927+0000","last_fullsized":"2024-02-10T07:47:07.190927+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'19","last_scrub_stamp":"2024-02-10T07:46:37.008841+0000","last_deep_scrub":"287'19","last_deep_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.008841+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:02.207790+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":80,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.193764+0000","last_change":"2024-02-10T07:46:40.116633+0000","last_active":"2024-02-10T07:47:07.193764+0000","last_peered":"2024-02-10T07:47:07.193764+0000","last_clean":"2024-02-10T07:47:07.193764+0000","last_became_active":"2024-02-10T07:46:40.116294+0000","last_became_peered":"2024-02-10T07:46:40.116294+0000","last_unstale":"2024-02-10T07:47:07.193764+0000","last_undegraded":"2024-02-10T07:47:07.193764+0000","last_fullsized":"2024-02-10T07:47:07.193764+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:48.894116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:48.894116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":590,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.190790+0000","last_change":"2024-02-10T07:45:55.951770+0000","last_active":"2024-02-10T07:47:07.190790+0000","last_peered":"2024-02-10T07:47:07.190790+0000","last_clean":"2024-02-10T07:47:07.190790+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:47:07.190790+0000","last_undegraded":"2024-02-10T07:47:07.190790+0000","last_fullsized":"2024-02-10T07:47:07.190790+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:55.951689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:55.951689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:23.163073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":93,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202382+0000","last_change":"2024-02-10T07:46:41.128244+0000","last_active":"2024-02-10T07:47:07.202382+0000","last_peered":"2024-02-10T07:47:07.202382+0000","last_clean":"2024-02-10T07:47:07.202382+0000","last_became_active":"2024-02-10T07:46:41.127905+0000","last_became_peered":"2024-02-10T07:46:41.127905+0000","last_unstale":"2024-02-10T07:47:07.202382+0000","last_undegraded":"2024-02-10T07:47:07.202382+0000","last_fullsized":"2024-02-10T07:47:07.202382+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:31.386263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:46:31.386263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:50.322949+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"250'27","reported_seq":532,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.201549+0000","last_change":"2024-02-10T07:46:41.153628+0000","last_active":"2024-02-10T07:47:07.201549+0000","last_peered":"2024-02-10T07:47:07.201549+0000","last_clean":"2024-02-10T07:47:07.201549+0000","last_became_active":"2024-02-10T07:46:41.117335+0000","last_became_peered":"2024-02-10T07:46:41.117335+0000","last_unstale":"2024-02-10T07:47:07.201549+0000","last_undegraded":"2024-02-10T07:47:07.201549+0000","last_fullsized":"2024-02-10T07:47:07.201549+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:45:00.449705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:45:00.449705+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":586,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.201056+0000","last_change":"2024-02-10T07:46:27.392670+0000","last_active":"2024-02-10T07:47:07.201056+0000","last_peered":"2024-02-10T07:47:07.201056+0000","last_clean":"2024-02-10T07:47:07.201056+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:47:07.201056+0000","last_undegraded":"2024-02-10T07:47:07.201056+0000","last_fullsized":"2024-02-10T07:47:07.201056+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:27.392544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:46:27.392544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:55.497558+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":551,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.216825+0000","last_change":"2024-02-10T07:45:41.296239+0000","last_active":"2024-02-10T07:47:07.216825+0000","last_peered":"2024-02-10T07:47:07.216825+0000","last_clean":"2024-02-10T07:47:07.216825+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:47:07.216825+0000","last_undegraded":"2024-02-10T07:47:07.216825+0000","last_fullsized":"2024-02-10T07:47:07.216825+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:41.296129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:41.296129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":532,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.214081+0000","last_change":"2024-02-10T07:45:09.404927+0000","last_active":"2024-02-10T07:47:07.214081+0000","last_peered":"2024-02-10T07:47:07.214081+0000","last_clean":"2024-02-10T07:47:07.214081+0000","last_became_active":"2024-02-10T07:36:33.771340+0000","last_became_peered":"2024-02-10T07:36:33.771340+0000","last_unstale":"2024-02-10T07:47:07.214081+0000","last_undegraded":"2024-02-10T07:47:07.214081+0000","last_fullsized":"2024-02-10T07:47:07.214081+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:45:09.404822+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:34:49.349360+0000","last_clean_scrub_stamp":"2024-02-10T07:45:09.404822+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"250'15","reported_seq":606,"reported_epoch":330,"state":"active+undersized+degraded","last_fresh":"2024-02-10T07:47:07.191257+0000","last_change":"2024-02-10T07:45:03.340347+0000","last_active":"2024-02-10T07:47:07.191257+0000","last_peered":"2024-02-10T07:47:07.191257+0000","last_clean":"2024-02-10T07:44:59.287494+0000","last_became_active":"2024-02-10T07:45:03.340347+0000","last_became_peered":"2024-02-10T07:45:03.340347+0000","last_unstale":"2024-02-10T07:47:07.191257+0000","last_undegraded":"2024-02-10T07:45:03.334583+0000","last_fullsized":"2024-02-10T07:45:03.334398+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:42:06.021407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:42:06.021407+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":552,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.213532+0000","last_change":"2024-02-10T07:46:40.125182+0000","last_active":"2024-02-10T07:47:07.213532+0000","last_peered":"2024-02-10T07:47:07.213532+0000","last_clean":"2024-02-10T07:47:07.213532+0000","last_became_active":"2024-02-10T07:46:40.123966+0000","last_became_peered":"2024-02-10T07:46:40.123966+0000","last_unstale":"2024-02-10T07:47:07.213532+0000","last_undegraded":"2024-02-10T07:47:07.213532+0000","last_fullsized":"2024-02-10T07:47:07.213532+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:00.364505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:44:00.364505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":590,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.193889+0000","last_change":"2024-02-10T07:45:10.505657+0000","last_active":"2024-02-10T07:47:07.193889+0000","last_peered":"2024-02-10T07:47:07.193889+0000","last_clean":"2024-02-10T07:47:07.193889+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:47:07.193889+0000","last_undegraded":"2024-02-10T07:47:07.193889+0000","last_fullsized":"2024-02-10T07:47:07.193889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:10.505564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:10.505564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":555,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.213017+0000","last_change":"2024-02-10T07:45:12.464256+0000","last_active":"2024-02-10T07:47:07.213017+0000","last_peered":"2024-02-10T07:47:07.213017+0000","last_clean":"2024-02-10T07:47:07.213017+0000","last_became_active":"2024-02-10T07:36:33.767970+0000","last_became_peered":"2024-02-10T07:36:33.767970+0000","last_unstale":"2024-02-10T07:47:07.213017+0000","last_undegraded":"2024-02-10T07:47:07.213017+0000","last_fullsized":"2024-02-10T07:47:07.213017+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:12.464185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.473686+0000","last_clean_scrub_stamp":"2024-02-10T07:45:12.464185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"250'7","reported_seq":594,"reported_epoch":330,"state":"active+undersized+degraded","last_fresh":"2024-02-10T07:47:07.192375+0000","last_change":"2024-02-10T07:45:03.337712+0000","last_active":"2024-02-10T07:47:07.192375+0000","last_peered":"2024-02-10T07:47:07.192375+0000","last_clean":"2024-02-10T07:44:59.287834+0000","last_became_active":"2024-02-10T07:45:03.337712+0000","last_became_peered":"2024-02-10T07:45:03.337712+0000","last_unstale":"2024-02-10T07:47:07.192375+0000","last_undegraded":"2024-02-10T07:45:03.334114+0000","last_fullsized":"2024-02-10T07:45:03.334008+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7","last_scrub_stamp":"2024-02-10T07:43:45.444068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:45.444068+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":588,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.195512+0000","last_change":"2024-02-10T07:44:48.094231+0000","last_active":"2024-02-10T07:47:07.195512+0000","last_peered":"2024-02-10T07:47:07.195512+0000","last_clean":"2024-02-10T07:47:07.195512+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:47:07.195512+0000","last_undegraded":"2024-02-10T07:47:07.195512+0000","last_fullsized":"2024-02-10T07:47:07.195512+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:48.094138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:44:48.094138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":596,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.205681+0000","last_change":"2024-02-10T07:46:32.360289+0000","last_active":"2024-02-10T07:47:07.205681+0000","last_peered":"2024-02-10T07:47:07.205681+0000","last_clean":"2024-02-10T07:47:07.205681+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:47:07.205681+0000","last_undegraded":"2024-02-10T07:47:07.205681+0000","last_fullsized":"2024-02-10T07:47:07.205681+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:32.360208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:32.360208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:38.722065+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":598,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202355+0000","last_change":"2024-02-10T07:45:56.038392+0000","last_active":"2024-02-10T07:47:07.202355+0000","last_peered":"2024-02-10T07:47:07.202355+0000","last_clean":"2024-02-10T07:47:07.202355+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:47:07.202355+0000","last_undegraded":"2024-02-10T07:47:07.202355+0000","last_fullsized":"2024-02-10T07:47:07.202355+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:56.038313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:45:56.038313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"250'34","reported_seq":545,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.205164+0000","last_change":"2024-02-10T07:46:41.158963+0000","last_active":"2024-02-10T07:47:07.205164+0000","last_peered":"2024-02-10T07:47:07.205164+0000","last_clean":"2024-02-10T07:47:07.205164+0000","last_became_active":"2024-02-10T07:46:41.123507+0000","last_became_peered":"2024-02-10T07:46:41.123507+0000","last_unstale":"2024-02-10T07:47:07.205164+0000","last_undegraded":"2024-02-10T07:47:07.205164+0000","last_fullsized":"2024-02-10T07:47:07.205164+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:46:27.370078+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:46:27.370078+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:30.590380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":270,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.211005+0000","last_change":"2024-02-10T07:46:40.125117+0000","last_active":"2024-02-10T07:47:07.211005+0000","last_peered":"2024-02-10T07:47:07.211005+0000","last_clean":"2024-02-10T07:47:07.211005+0000","last_became_active":"2024-02-10T07:46:40.124797+0000","last_became_peered":"2024-02-10T07:46:40.124797+0000","last_unstale":"2024-02-10T07:47:07.211005+0000","last_undegraded":"2024-02-10T07:47:07.211005+0000","last_fullsized":"2024-02-10T07:47:07.211005+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:45.077266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:45.077266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":595,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.198069+0000","last_change":"2024-02-10T07:46:41.121554+0000","last_active":"2024-02-10T07:47:07.198069+0000","last_peered":"2024-02-10T07:47:07.198069+0000","last_clean":"2024-02-10T07:47:07.198069+0000","last_became_active":"2024-02-10T07:46:41.121201+0000","last_became_peered":"2024-02-10T07:46:41.121201+0000","last_unstale":"2024-02-10T07:47:07.198069+0000","last_undegraded":"2024-02-10T07:47:07.198069+0000","last_fullsized":"2024-02-10T07:47:07.198069+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:56.206287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_clean_scrub_stamp":"2024-02-10T07:44:56.206287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":604,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.191330+0000","last_change":"2024-02-10T07:46:10.717885+0000","last_active":"2024-02-10T07:47:07.191330+0000","last_peered":"2024-02-10T07:47:07.191330+0000","last_clean":"2024-02-10T07:47:07.191330+0000","last_became_active":"2024-02-10T07:46:10.717041+0000","last_became_peered":"2024-02-10T07:46:10.717041+0000","last_unstale":"2024-02-10T07:47:07.191330+0000","last_undegraded":"2024-02-10T07:47:07.191330+0000","last_fullsized":"2024-02-10T07:47:07.191330+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:44.989380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:45:44.989380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:13.129441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"250'18","reported_seq":530,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.204917+0000","last_change":"2024-02-10T07:46:40.144593+0000","last_active":"2024-02-10T07:47:07.204917+0000","last_peered":"2024-02-10T07:47:07.204917+0000","last_clean":"2024-02-10T07:47:07.204917+0000","last_became_active":"2024-02-10T07:46:40.123579+0000","last_became_peered":"2024-02-10T07:46:40.123579+0000","last_unstale":"2024-02-10T07:47:07.204917+0000","last_undegraded":"2024-02-10T07:47:07.204917+0000","last_fullsized":"2024-02-10T07:47:07.204917+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'18","last_scrub_stamp":"2024-02-10T07:43:16.944627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:16.944627+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":602,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.195268+0000","last_change":"2024-02-10T07:46:41.127935+0000","last_active":"2024-02-10T07:47:07.195268+0000","last_peered":"2024-02-10T07:47:07.195268+0000","last_clean":"2024-02-10T07:47:07.195268+0000","last_became_active":"2024-02-10T07:46:41.116241+0000","last_became_peered":"2024-02-10T07:46:41.116241+0000","last_unstale":"2024-02-10T07:47:07.195268+0000","last_undegraded":"2024-02-10T07:47:07.195268+0000","last_fullsized":"2024-02-10T07:47:07.195268+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:44:51.349776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:44:51.349776+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":801,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.208374+0000","last_change":"2024-02-10T07:45:12.363524+0000","last_active":"2024-02-10T07:47:07.208374+0000","last_peered":"2024-02-10T07:47:07.208374+0000","last_clean":"2024-02-10T07:47:07.208374+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:47:07.208374+0000","last_undegraded":"2024-02-10T07:47:07.208374+0000","last_fullsized":"2024-02-10T07:47:07.208374+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:45:12.363405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:12.363405+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":600,"reported_epoch":330,"state":"active+undersized+degraded","last_fresh":"2024-02-10T07:47:07.204615+0000","last_change":"2024-02-10T07:45:03.348472+0000","last_active":"2024-02-10T07:47:07.204615+0000","last_peered":"2024-02-10T07:47:07.204615+0000","last_clean":"2024-02-10T07:44:59.299796+0000","last_became_active":"2024-02-10T07:45:03.348472+0000","last_became_peered":"2024-02-10T07:45:03.348472+0000","last_unstale":"2024-02-10T07:47:07.204615+0000","last_undegraded":"2024-02-10T07:45:03.335366+0000","last_fullsized":"2024-02-10T07:45:03.335230+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:44:54.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:44:54.243093+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"250'25","reported_seq":651,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202659+0000","last_change":"2024-02-10T07:46:40.135390+0000","last_active":"2024-02-10T07:47:07.202659+0000","last_peered":"2024-02-10T07:47:07.202659+0000","last_clean":"2024-02-10T07:47:07.202659+0000","last_became_active":"2024-02-10T07:46:40.127716+0000","last_became_peered":"2024-02-10T07:46:40.127716+0000","last_unstale":"2024-02-10T07:47:07.202659+0000","last_undegraded":"2024-02-10T07:47:07.202659+0000","last_fullsized":"2024-02-10T07:47:07.202659+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:44:06.180439+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-02-10T07:42:17.820959+0000","last_clean_scrub_stamp":"2024-02-10T07:44:06.180439+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":593,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.205820+0000","last_change":"2024-02-10T07:45:57.490792+0000","last_active":"2024-02-10T07:47:07.205820+0000","last_peered":"2024-02-10T07:47:07.205820+0000","last_clean":"2024-02-10T07:47:07.205820+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:47:07.205820+0000","last_undegraded":"2024-02-10T07:47:07.205820+0000","last_fullsized":"2024-02-10T07:47:07.205820+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:57.490690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_clean_scrub_stamp":"2024-02-10T07:45:57.490690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:13.274950+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":589,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.203014+0000","last_change":"2024-02-10T07:46:10.724526+0000","last_active":"2024-02-10T07:47:07.203014+0000","last_peered":"2024-02-10T07:47:07.203014+0000","last_clean":"2024-02-10T07:47:07.203014+0000","last_became_active":"2024-02-10T07:46:10.724252+0000","last_became_peered":"2024-02-10T07:46:10.724252+0000","last_unstale":"2024-02-10T07:47:07.203014+0000","last_undegraded":"2024-02-10T07:47:07.203014+0000","last_fullsized":"2024-02-10T07:47:07.203014+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:06.200009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:06.200009+0000","last_clean_scrub_stamp":"2024-02-10T07:45:06.200009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":81,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.200980+0000","last_change":"2024-02-10T07:46:40.129869+0000","last_active":"2024-02-10T07:47:07.200980+0000","last_peered":"2024-02-10T07:47:07.200980+0000","last_clean":"2024-02-10T07:47:07.200980+0000","last_became_active":"2024-02-10T07:46:40.129571+0000","last_became_peered":"2024-02-10T07:46:40.129571+0000","last_unstale":"2024-02-10T07:47:07.200980+0000","last_undegraded":"2024-02-10T07:47:07.200980+0000","last_fullsized":"2024-02-10T07:47:07.200980+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:59.897726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:59.897726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"250'19","reported_seq":642,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.210795+0000","last_change":"2024-02-10T07:46:37.460129+0000","last_active":"2024-02-10T07:47:07.210795+0000","last_peered":"2024-02-10T07:47:07.210795+0000","last_clean":"2024-02-10T07:47:07.210795+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:47:07.210795+0000","last_undegraded":"2024-02-10T07:47:07.210795+0000","last_fullsized":"2024-02-10T07:47:07.210795+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:46:37.459988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.459988+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:52.034484+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":596,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.193812+0000","last_change":"2024-02-10T07:46:41.115159+0000","last_active":"2024-02-10T07:47:07.193812+0000","last_peered":"2024-02-10T07:47:07.193812+0000","last_clean":"2024-02-10T07:47:07.193812+0000","last_became_active":"2024-02-10T07:46:41.114829+0000","last_became_peered":"2024-02-10T07:46:41.114829+0000","last_unstale":"2024-02-10T07:47:07.193812+0000","last_undegraded":"2024-02-10T07:47:07.193812+0000","last_fullsized":"2024-02-10T07:47:07.193812+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:48.648483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:45:48.648483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":597,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202001+0000","last_change":"2024-02-10T07:45:06.239298+0000","last_active":"2024-02-10T07:47:07.202001+0000","last_peered":"2024-02-10T07:47:07.202001+0000","last_clean":"2024-02-10T07:47:07.202001+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:47:07.202001+0000","last_undegraded":"2024-02-10T07:47:07.202001+0000","last_fullsized":"2024-02-10T07:47:07.202001+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:06.239224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:06.239224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":616,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202585+0000","last_change":"2024-02-10T07:46:40.128287+0000","last_active":"2024-02-10T07:47:07.202585+0000","last_peered":"2024-02-10T07:47:07.202585+0000","last_clean":"2024-02-10T07:47:07.202585+0000","last_became_active":"2024-02-10T07:46:40.127951+0000","last_became_peered":"2024-02-10T07:46:40.127951+0000","last_unstale":"2024-02-10T07:47:07.202585+0000","last_undegraded":"2024-02-10T07:47:07.202585+0000","last_fullsized":"2024-02-10T07:47:07.202585+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:55.277457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:44:55.277457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"250'10","reported_seq":499,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.204984+0000","last_change":"2024-02-10T07:46:40.145373+0000","last_active":"2024-02-10T07:47:07.204984+0000","last_peered":"2024-02-10T07:47:07.204984+0000","last_clean":"2024-02-10T07:47:07.204984+0000","last_became_active":"2024-02-10T07:46:40.124444+0000","last_became_peered":"2024-02-10T07:46:40.124444+0000","last_unstale":"2024-02-10T07:47:07.204984+0000","last_undegraded":"2024-02-10T07:47:07.204984+0000","last_fullsized":"2024-02-10T07:47:07.204984+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'10","last_scrub_stamp":"2024-02-10T07:43:06.022190+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:43:06.022190+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":81,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.201883+0000","last_change":"2024-02-10T07:46:40.131712+0000","last_active":"2024-02-10T07:47:07.201883+0000","last_peered":"2024-02-10T07:47:07.201883+0000","last_clean":"2024-02-10T07:47:07.201883+0000","last_became_active":"2024-02-10T07:46:40.131402+0000","last_became_peered":"2024-02-10T07:46:40.131402+0000","last_unstale":"2024-02-10T07:47:07.201883+0000","last_undegraded":"2024-02-10T07:47:07.201883+0000","last_fullsized":"2024-02-10T07:47:07.201883+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:43.019550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:43.019550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":590,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.194300+0000","last_change":"2024-02-10T07:45:46.640690+0000","last_active":"2024-02-10T07:47:07.194300+0000","last_peered":"2024-02-10T07:47:07.194300+0000","last_clean":"2024-02-10T07:47:07.194300+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:47:07.194300+0000","last_undegraded":"2024-02-10T07:47:07.194300+0000","last_fullsized":"2024-02-10T07:47:07.194300+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:46.640569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:46.640569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":523,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.213728+0000","last_change":"2024-02-10T07:46:34.542170+0000","last_active":"2024-02-10T07:47:07.213728+0000","last_peered":"2024-02-10T07:47:07.213728+0000","last_clean":"2024-02-10T07:47:07.213728+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:47:07.213728+0000","last_undegraded":"2024-02-10T07:47:07.213728+0000","last_fullsized":"2024-02-10T07:47:07.213728+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:34.541307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:46:34.541307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:58.011213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"250'19","reported_seq":530,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.195813+0000","last_change":"2024-02-10T07:46:40.137152+0000","last_active":"2024-02-10T07:47:07.195813+0000","last_peered":"2024-02-10T07:47:07.195813+0000","last_clean":"2024-02-10T07:47:07.195813+0000","last_became_active":"2024-02-10T07:46:40.123712+0000","last_became_peered":"2024-02-10T07:46:40.123712+0000","last_unstale":"2024-02-10T07:47:07.195813+0000","last_undegraded":"2024-02-10T07:47:07.195813+0000","last_fullsized":"2024-02-10T07:47:07.195813+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:43:40.895108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:40.895108+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":587,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.200779+0000","last_change":"2024-02-10T07:45:59.179456+0000","last_active":"2024-02-10T07:47:07.200779+0000","last_peered":"2024-02-10T07:47:07.200779+0000","last_clean":"2024-02-10T07:47:07.200779+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:47:07.200779+0000","last_undegraded":"2024-02-10T07:47:07.200779+0000","last_fullsized":"2024-02-10T07:47:07.200779+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:59.179342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:59.179342+0000","last_clean_scrub_stamp":"2024-02-10T07:45:59.179342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:28.951183+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":81,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202036+0000","last_change":"2024-02-10T07:46:40.124112+0000","last_active":"2024-02-10T07:47:07.202036+0000","last_peered":"2024-02-10T07:47:07.202036+0000","last_clean":"2024-02-10T07:47:07.202036+0000","last_became_active":"2024-02-10T07:46:40.123780+0000","last_became_peered":"2024-02-10T07:46:40.123780+0000","last_unstale":"2024-02-10T07:47:07.202036+0000","last_undegraded":"2024-02-10T07:47:07.202036+0000","last_fullsized":"2024-02-10T07:47:07.202036+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:46.833613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:46.833613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":593,"reported_epoch":330,"state":"active+undersized","last_fresh":"2024-02-10T07:47:07.195537+0000","last_change":"2024-02-10T07:45:03.342854+0000","last_active":"2024-02-10T07:47:07.195537+0000","last_peered":"2024-02-10T07:47:07.195537+0000","last_clean":"2024-02-10T07:44:59.290511+0000","last_became_active":"2024-02-10T07:45:03.342854+0000","last_became_peered":"2024-02-10T07:45:03.342854+0000","last_unstale":"2024-02-10T07:47:07.195537+0000","last_undegraded":"2024-02-10T07:47:07.195537+0000","last_fullsized":"2024-02-10T07:45:03.334237+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:06.386654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:06.386654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"250'19","reported_seq":415,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.204980+0000","last_change":"2024-02-10T07:46:10.740738+0000","last_active":"2024-02-10T07:47:07.204980+0000","last_peered":"2024-02-10T07:47:07.204980+0000","last_clean":"2024-02-10T07:47:07.204980+0000","last_became_active":"2024-02-10T07:46:10.726359+0000","last_became_peered":"2024-02-10T07:46:10.726359+0000","last_unstale":"2024-02-10T07:47:07.204980+0000","last_undegraded":"2024-02-10T07:47:07.204980+0000","last_fullsized":"2024-02-10T07:47:07.204980+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:45:59.430820+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-02-10T07:39:32.934269+0000","last_clean_scrub_stamp":"2024-02-10T07:45:59.430820+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:12.081448+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":269,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.201455+0000","last_change":"2024-02-10T07:46:40.130055+0000","last_active":"2024-02-10T07:47:07.201455+0000","last_peered":"2024-02-10T07:47:07.201455+0000","last_clean":"2024-02-10T07:47:07.201455+0000","last_became_active":"2024-02-10T07:46:40.129733+0000","last_became_peered":"2024-02-10T07:46:40.129733+0000","last_unstale":"2024-02-10T07:47:07.201455+0000","last_undegraded":"2024-02-10T07:47:07.201455+0000","last_fullsized":"2024-02-10T07:47:07.201455+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:11.935253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:11.935253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":606,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.206065+0000","last_change":"2024-02-10T07:46:40.122301+0000","last_active":"2024-02-10T07:47:07.206065+0000","last_peered":"2024-02-10T07:47:07.206065+0000","last_clean":"2024-02-10T07:47:07.206065+0000","last_became_active":"2024-02-10T07:46:40.121948+0000","last_became_peered":"2024-02-10T07:46:40.121948+0000","last_unstale":"2024-02-10T07:47:07.206065+0000","last_undegraded":"2024-02-10T07:47:07.206065+0000","last_fullsized":"2024-02-10T07:47:07.206065+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:56.271925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:44:56.271925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":598,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.197790+0000","last_change":"2024-02-10T07:45:54.972676+0000","last_active":"2024-02-10T07:47:07.197790+0000","last_peered":"2024-02-10T07:47:07.197790+0000","last_clean":"2024-02-10T07:47:07.197790+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:47:07.197790+0000","last_undegraded":"2024-02-10T07:47:07.197790+0000","last_fullsized":"2024-02-10T07:47:07.197790+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:54.972600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:45:54.972600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"250'24","reported_seq":658,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.192181+0000","last_change":"2024-02-10T07:46:10.734169+0000","last_active":"2024-02-10T07:47:07.192181+0000","last_peered":"2024-02-10T07:47:07.192181+0000","last_clean":"2024-02-10T07:47:07.192181+0000","last_became_active":"2024-02-10T07:46:10.721336+0000","last_became_peered":"2024-02-10T07:46:10.721336+0000","last_unstale":"2024-02-10T07:47:07.192181+0000","last_undegraded":"2024-02-10T07:47:07.192181+0000","last_fullsized":"2024-02-10T07:47:07.192181+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:44:02.402293+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:44:02.402293+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":587,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202319+0000","last_change":"2024-02-10T07:46:40.128437+0000","last_active":"2024-02-10T07:47:07.202319+0000","last_peered":"2024-02-10T07:47:07.202319+0000","last_clean":"2024-02-10T07:47:07.202319+0000","last_became_active":"2024-02-10T07:46:40.128084+0000","last_became_peered":"2024-02-10T07:46:40.128084+0000","last_unstale":"2024-02-10T07:47:07.202319+0000","last_undegraded":"2024-02-10T07:47:07.202319+0000","last_fullsized":"2024-02-10T07:47:07.202319+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:00.233946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:44:00.233946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"158'10","reported_seq":692,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.213983+0000","last_change":"2024-02-10T07:45:04.393110+0000","last_active":"2024-02-10T07:47:07.213983+0000","last_peered":"2024-02-10T07:47:07.213983+0000","last_clean":"2024-02-10T07:47:07.213983+0000","last_became_active":"2024-02-10T07:36:33.774773+0000","last_became_peered":"2024-02-10T07:36:33.774773+0000","last_unstale":"2024-02-10T07:47:07.213983+0000","last_undegraded":"2024-02-10T07:47:07.213983+0000","last_fullsized":"2024-02-10T07:47:07.213983+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:45:04.392999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:04.392999+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":612,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.194106+0000","last_change":"2024-02-10T07:46:36.855577+0000","last_active":"2024-02-10T07:47:07.194106+0000","last_peered":"2024-02-10T07:47:07.194106+0000","last_clean":"2024-02-10T07:47:07.194106+0000","last_became_active":"2024-02-10T07:46:10.716647+0000","last_became_peered":"2024-02-10T07:46:10.716647+0000","last_unstale":"2024-02-10T07:47:07.194106+0000","last_undegraded":"2024-02-10T07:47:07.194106+0000","last_fullsized":"2024-02-10T07:47:07.194106+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:36.855461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:46:36.855461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:58.583156+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"250'25","reported_seq":440,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.212674+0000","last_change":"2024-02-10T07:45:52.483394+0000","last_active":"2024-02-10T07:47:07.212674+0000","last_peered":"2024-02-10T07:47:07.212674+0000","last_clean":"2024-02-10T07:47:07.212674+0000","last_became_active":"2024-02-10T07:36:33.752317+0000","last_became_peered":"2024-02-10T07:36:33.752317+0000","last_unstale":"2024-02-10T07:47:07.212674+0000","last_undegraded":"2024-02-10T07:47:07.212674+0000","last_fullsized":"2024-02-10T07:47:07.212674+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:45:52.483293+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:45:52.483293+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:19.879373+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":593,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.199319+0000","last_change":"2024-02-10T07:46:10.721405+0000","last_active":"2024-02-10T07:47:07.199319+0000","last_peered":"2024-02-10T07:47:07.199319+0000","last_clean":"2024-02-10T07:47:07.199319+0000","last_became_active":"2024-02-10T07:46:10.720680+0000","last_became_peered":"2024-02-10T07:46:10.720680+0000","last_unstale":"2024-02-10T07:47:07.199319+0000","last_undegraded":"2024-02-10T07:47:07.199319+0000","last_fullsized":"2024-02-10T07:47:07.199319+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:48.097218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:45:48.097218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:07.559293+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"326'13","reported_seq":153,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.195182+0000","last_change":"2024-02-10T07:46:41.125506+0000","last_active":"2024-02-10T07:47:07.195182+0000","last_peered":"2024-02-10T07:47:07.195182+0000","last_clean":"2024-02-10T07:47:07.195182+0000","last_became_active":"2024-02-10T07:46:41.118359+0000","last_became_peered":"2024-02-10T07:46:41.118359+0000","last_unstale":"2024-02-10T07:47:07.195182+0000","last_undegraded":"2024-02-10T07:47:07.195182+0000","last_fullsized":"2024-02-10T07:47:07.195182+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'11","last_scrub_stamp":"2024-02-10T07:46:32.848489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:32.848489+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:39.579598+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":607,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.201935+0000","last_change":"2024-02-10T07:46:40.451635+0000","last_active":"2024-02-10T07:47:07.201935+0000","last_peered":"2024-02-10T07:47:07.201935+0000","last_clean":"2024-02-10T07:47:07.201935+0000","last_became_active":"2024-02-10T07:46:40.127276+0000","last_became_peered":"2024-02-10T07:46:40.127276+0000","last_unstale":"2024-02-10T07:47:07.201935+0000","last_undegraded":"2024-02-10T07:47:07.201935+0000","last_fullsized":"2024-02-10T07:47:07.201935+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:40.451562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:46:40.451562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:41.979328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"250'15","reported_seq":575,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.219444+0000","last_change":"2024-02-10T07:46:40.142310+0000","last_active":"2024-02-10T07:47:07.219444+0000","last_peered":"2024-02-10T07:47:07.219444+0000","last_clean":"2024-02-10T07:47:07.219444+0000","last_became_active":"2024-02-10T07:46:40.127869+0000","last_became_peered":"2024-02-10T07:46:40.127869+0000","last_unstale":"2024-02-10T07:47:07.219444+0000","last_undegraded":"2024-02-10T07:47:07.219444+0000","last_fullsized":"2024-02-10T07:47:07.219444+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:43:02.231670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:02.231670+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":45,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.198145+0000","last_change":"2024-02-10T07:46:10.717193+0000","last_active":"2024-02-10T07:47:07.198145+0000","last_peered":"2024-02-10T07:47:07.198145+0000","last_clean":"2024-02-10T07:47:07.198145+0000","last_became_active":"2024-02-10T07:46:10.716869+0000","last_became_peered":"2024-02-10T07:46:10.716869+0000","last_unstale":"2024-02-10T07:47:07.198145+0000","last_undegraded":"2024-02-10T07:47:07.198145+0000","last_fullsized":"2024-02-10T07:47:07.198145+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:46.781458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:44:46.781458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":80,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.206044+0000","last_change":"2024-02-10T07:46:40.117500+0000","last_active":"2024-02-10T07:47:07.206044+0000","last_peered":"2024-02-10T07:47:07.206044+0000","last_clean":"2024-02-10T07:47:07.206044+0000","last_became_active":"2024-02-10T07:46:40.117173+0000","last_became_peered":"2024-02-10T07:46:40.117173+0000","last_unstale":"2024-02-10T07:47:07.206044+0000","last_undegraded":"2024-02-10T07:47:07.206044+0000","last_fullsized":"2024-02-10T07:47:07.206044+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:58.411270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:44:58.411270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":598,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.195652+0000","last_change":"2024-02-10T07:45:09.502115+0000","last_active":"2024-02-10T07:47:07.195652+0000","last_peered":"2024-02-10T07:47:07.195652+0000","last_clean":"2024-02-10T07:47:07.195652+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:47:07.195652+0000","last_undegraded":"2024-02-10T07:47:07.195652+0000","last_fullsized":"2024-02-10T07:47:07.195652+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:09.501891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:45:09.501891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"250'19","reported_seq":642,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202810+0000","last_change":"2024-02-10T07:46:40.142685+0000","last_active":"2024-02-10T07:47:07.202810+0000","last_peered":"2024-02-10T07:47:07.202810+0000","last_clean":"2024-02-10T07:47:07.202810+0000","last_became_active":"2024-02-10T07:46:40.127170+0000","last_became_peered":"2024-02-10T07:46:40.127170+0000","last_unstale":"2024-02-10T07:47:07.202810+0000","last_undegraded":"2024-02-10T07:47:07.202810+0000","last_fullsized":"2024-02-10T07:47:07.202810+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:44:08.130707+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:44:08.130707+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":586,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.194477+0000","last_change":"2024-02-10T07:46:34.945757+0000","last_active":"2024-02-10T07:47:07.194477+0000","last_peered":"2024-02-10T07:47:07.194477+0000","last_clean":"2024-02-10T07:47:07.194477+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:47:07.194477+0000","last_undegraded":"2024-02-10T07:47:07.194477+0000","last_fullsized":"2024-02-10T07:47:07.194477+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:34.945654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:46:34.945654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:34.991597+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":548,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.213486+0000","last_change":"2024-02-10T07:46:40.126717+0000","last_active":"2024-02-10T07:47:07.213486+0000","last_peered":"2024-02-10T07:47:07.213486+0000","last_clean":"2024-02-10T07:47:07.213486+0000","last_became_active":"2024-02-10T07:46:40.126367+0000","last_became_peered":"2024-02-10T07:46:40.126367+0000","last_unstale":"2024-02-10T07:47:07.213486+0000","last_undegraded":"2024-02-10T07:47:07.213486+0000","last_fullsized":"2024-02-10T07:47:07.213486+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:55.314766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:44:55.314766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":622,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.201577+0000","last_change":"2024-02-10T07:45:58.296611+0000","last_active":"2024-02-10T07:47:07.201577+0000","last_peered":"2024-02-10T07:47:07.201577+0000","last_clean":"2024-02-10T07:47:07.201577+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:47:07.201577+0000","last_undegraded":"2024-02-10T07:47:07.201577+0000","last_fullsized":"2024-02-10T07:47:07.201577+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:58.296537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:45:58.296537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"250'12","reported_seq":505,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.196531+0000","last_change":"2024-02-10T07:46:40.136142+0000","last_active":"2024-02-10T07:47:07.196531+0000","last_peered":"2024-02-10T07:47:07.196531+0000","last_clean":"2024-02-10T07:47:07.196531+0000","last_became_active":"2024-02-10T07:46:40.120521+0000","last_became_peered":"2024-02-10T07:46:40.120521+0000","last_unstale":"2024-02-10T07:47:07.196531+0000","last_undegraded":"2024-02-10T07:47:07.196531+0000","last_fullsized":"2024-02-10T07:47:07.196531+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'12","last_scrub_stamp":"2024-02-10T07:44:50.336184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:44:50.336184+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.6","version":"330'4176","reported_seq":4228,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.301596+0000","last_change":"2024-02-10T07:46:06.701692+0000","last_active":"2024-02-10T07:47:07.301596+0000","last_peered":"2024-02-10T07:47:07.301596+0000","last_clean":"2024-02-10T07:47:07.301596+0000","last_became_active":"2024-02-10T07:46:06.701322+0000","last_became_peered":"2024-02-10T07:46:06.701322+0000","last_unstale":"2024-02-10T07:47:07.301596+0000","last_undegraded":"2024-02-10T07:47:07.301596+0000","last_fullsized":"2024-02-10T07:47:07.301596+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4176,"log_dups_size":0,"ondisk_log_size":4176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:33.973481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273678336,"num_objects":4176,"num_object_clones":0,"num_object_copies":12528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4176,"num_write_kb":267264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":551,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.215835+0000","last_change":"2024-02-10T07:45:46.432142+0000","last_active":"2024-02-10T07:47:07.215835+0000","last_peered":"2024-02-10T07:47:07.215835+0000","last_clean":"2024-02-10T07:47:07.215835+0000","last_became_active":"2024-02-10T07:36:33.763431+0000","last_became_peered":"2024-02-10T07:36:33.763431+0000","last_unstale":"2024-02-10T07:47:07.215835+0000","last_undegraded":"2024-02-10T07:47:07.215835+0000","last_fullsized":"2024-02-10T07:47:07.215835+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:46.432039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:45:46.432039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"245'4","reported_seq":620,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.196429+0000","last_change":"2024-02-10T07:46:35.848544+0000","last_active":"2024-02-10T07:47:07.196429+0000","last_peered":"2024-02-10T07:47:07.196429+0000","last_clean":"2024-02-10T07:47:07.196429+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:47:07.196429+0000","last_undegraded":"2024-02-10T07:47:07.196429+0000","last_fullsized":"2024-02-10T07:47:07.196429+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'4","last_scrub_stamp":"2024-02-10T07:46:35.848398+0000","last_deep_scrub":"245'4","last_deep_scrub_stamp":"2024-02-10T07:41:23.717923+0000","last_clean_scrub_stamp":"2024-02-10T07:46:35.848398+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:51.208817+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"250'34","reported_seq":567,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.216195+0000","last_change":"2024-02-10T07:45:49.390421+0000","last_active":"2024-02-10T07:47:07.216195+0000","last_peered":"2024-02-10T07:47:07.216195+0000","last_clean":"2024-02-10T07:47:07.216195+0000","last_became_active":"2024-02-10T07:36:33.754134+0000","last_became_peered":"2024-02-10T07:36:33.754134+0000","last_unstale":"2024-02-10T07:47:07.216195+0000","last_undegraded":"2024-02-10T07:47:07.216195+0000","last_fullsized":"2024-02-10T07:47:07.216195+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:45:49.390328+0000","last_deep_scrub":"246'22","last_deep_scrub_stamp":"2024-02-10T07:41:26.771112+0000","last_clean_scrub_stamp":"2024-02-10T07:45:49.390328+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:08.812647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":791,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.196098+0000","last_change":"2024-02-10T07:46:00.666508+0000","last_active":"2024-02-10T07:47:07.196098+0000","last_peered":"2024-02-10T07:47:07.196098+0000","last_clean":"2024-02-10T07:47:07.196098+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:47:07.196098+0000","last_undegraded":"2024-02-10T07:47:07.196098+0000","last_fullsized":"2024-02-10T07:47:07.196098+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:46:00.666417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:00.666417+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"330'4328","reported_seq":6000,"reported_epoch":330,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-10T07:47:08.082064+0000","last_change":"2024-02-10T07:46:41.239276+0000","last_active":"2024-02-10T07:47:08.082064+0000","last_peered":"2024-02-10T07:47:08.082064+0000","last_clean":"2024-02-10T07:46:05.681946+0000","last_became_active":"2024-02-10T07:46:41.224441+0000","last_became_peered":"2024-02-10T07:46:41.224441+0000","last_unstale":"2024-02-10T07:47:08.082064+0000","last_undegraded":"2024-02-10T07:46:41.138911+0000","last_fullsized":"2024-02-10T07:46:41.112475+0000","mapping_epoch":325,"log_start":"317'100","ondisk_log_start":"317'100","created":309,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4228,"log_dups_size":0,"ondisk_log_size":4228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283639808,"num_objects":4328,"num_object_clones":0,"num_object_copies":12984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1854,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4328,"num_write_kb":276992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":810,"num_bytes_recovered":53084160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":2474}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":74,"reported_epoch":330,"state":"active+undersized","last_fresh":"2024-02-10T07:47:07.194570+0000","last_change":"2024-02-10T07:45:03.355274+0000","last_active":"2024-02-10T07:47:07.194570+0000","last_peered":"2024-02-10T07:47:07.194570+0000","last_clean":"2024-02-10T07:31:56.282886+0000","last_became_active":"2024-02-10T07:45:03.355274+0000","last_became_peered":"2024-02-10T07:45:03.355274+0000","last_unstale":"2024-02-10T07:47:07.194570+0000","last_undegraded":"2024-02-10T07:47:07.194570+0000","last_fullsized":"2024-02-10T07:45:03.336384+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:02.251208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:43:02.251208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":598,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.195973+0000","last_change":"2024-02-10T07:45:42.650614+0000","last_active":"2024-02-10T07:47:07.195973+0000","last_peered":"2024-02-10T07:47:07.195973+0000","last_clean":"2024-02-10T07:47:07.195973+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:47:07.195973+0000","last_undegraded":"2024-02-10T07:47:07.195973+0000","last_fullsized":"2024-02-10T07:47:07.195973+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:42.650422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:45:42.650422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:10.410830+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"250'23","reported_seq":679,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.193239+0000","last_change":"2024-02-10T07:46:41.138850+0000","last_active":"2024-02-10T07:47:07.193239+0000","last_peered":"2024-02-10T07:47:07.193239+0000","last_clean":"2024-02-10T07:47:07.193239+0000","last_became_active":"2024-02-10T07:46:41.120409+0000","last_became_peered":"2024-02-10T07:46:41.120409+0000","last_unstale":"2024-02-10T07:47:07.193239+0000","last_undegraded":"2024-02-10T07:47:07.193239+0000","last_fullsized":"2024-02-10T07:47:07.193239+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-02-10T07:44:45.144119+0000","last_deep_scrub":"245'17","last_deep_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_clean_scrub_stamp":"2024-02-10T07:44:45.144119+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":46,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.201144+0000","last_change":"2024-02-10T07:46:10.724798+0000","last_active":"2024-02-10T07:47:07.201144+0000","last_peered":"2024-02-10T07:47:07.201144+0000","last_clean":"2024-02-10T07:47:07.201144+0000","last_became_active":"2024-02-10T07:46:10.723006+0000","last_became_peered":"2024-02-10T07:46:10.723006+0000","last_unstale":"2024-02-10T07:47:07.201144+0000","last_undegraded":"2024-02-10T07:47:07.201144+0000","last_fullsized":"2024-02-10T07:47:07.201144+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:44.509694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:44.509694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"330'4315","reported_seq":4364,"reported_epoch":330,"state":"active+undersized+degraded","last_fresh":"2024-02-10T07:47:09.082554+0000","last_change":"2024-02-10T07:46:20.963834+0000","last_active":"2024-02-10T07:47:09.082554+0000","last_peered":"2024-02-10T07:47:09.082554+0000","last_clean":"2024-02-10T07:46:05.681946+0000","last_became_active":"2024-02-10T07:46:06.693964+0000","last_became_peered":"2024-02-10T07:46:06.693964+0000","last_unstale":"2024-02-10T07:47:09.082554+0000","last_undegraded":"2024-02-10T07:46:20.959159+0000","last_fullsized":"2024-02-10T07:46:06.691770+0000","mapping_epoch":309,"log_start":"317'100","ondisk_log_start":"317'100","created":309,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4215,"log_dups_size":0,"ondisk_log_size":4215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:11.311363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282787840,"num_objects":4315,"num_object_clones":0,"num_object_copies":12945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4315,"num_write_kb":276160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4315}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":608,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.192942+0000","last_change":"2024-02-10T07:46:41.124909+0000","last_active":"2024-02-10T07:47:07.192942+0000","last_peered":"2024-02-10T07:47:07.192942+0000","last_clean":"2024-02-10T07:47:07.192942+0000","last_became_active":"2024-02-10T07:46:41.124345+0000","last_became_peered":"2024-02-10T07:46:41.124345+0000","last_unstale":"2024-02-10T07:47:07.192942+0000","last_undegraded":"2024-02-10T07:47:07.192942+0000","last_fullsized":"2024-02-10T07:47:07.192942+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:31.861542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:36.741855+0000","last_clean_scrub_stamp":"2024-02-10T07:46:31.861542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:52.515652+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":246,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.201894+0000","last_change":"2024-02-10T07:46:10.719905+0000","last_active":"2024-02-10T07:47:07.201894+0000","last_peered":"2024-02-10T07:47:07.201894+0000","last_clean":"2024-02-10T07:47:07.201894+0000","last_became_active":"2024-02-10T07:46:10.719610+0000","last_became_peered":"2024-02-10T07:46:10.719610+0000","last_unstale":"2024-02-10T07:47:07.201894+0000","last_undegraded":"2024-02-10T07:47:07.201894+0000","last_fullsized":"2024-02-10T07:47:07.201894+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:08.684696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:08.684696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":607,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.195930+0000","last_change":"2024-02-10T07:45:08.468897+0000","last_active":"2024-02-10T07:47:07.195930+0000","last_peered":"2024-02-10T07:47:07.195930+0000","last_clean":"2024-02-10T07:47:07.195930+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:47:07.195930+0000","last_undegraded":"2024-02-10T07:47:07.195930+0000","last_fullsized":"2024-02-10T07:47:07.195930+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:08.468798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:45:08.468798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"250'35","reported_seq":665,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.205679+0000","last_change":"2024-02-10T07:46:32.439686+0000","last_active":"2024-02-10T07:47:07.205679+0000","last_peered":"2024-02-10T07:47:07.205679+0000","last_clean":"2024-02-10T07:47:07.205679+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:47:07.205679+0000","last_undegraded":"2024-02-10T07:47:07.205679+0000","last_fullsized":"2024-02-10T07:47:07.205679+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'35","last_scrub_stamp":"2024-02-10T07:46:32.439157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:46:32.439157+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:47.002277+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"330'4409","reported_seq":7775,"reported_epoch":330,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-10T07:47:09.107867+0000","last_change":"2024-02-10T07:46:41.244172+0000","last_active":"2024-02-10T07:47:09.107867+0000","last_peered":"2024-02-10T07:47:09.107867+0000","last_clean":"2024-02-10T07:46:39.106295+0000","last_became_active":"2024-02-10T07:46:41.211476+0000","last_became_peered":"2024-02-10T07:46:41.211476+0000","last_unstale":"2024-02-10T07:47:09.107867+0000","last_undegraded":"2024-02-10T07:47:09.107867+0000","last_fullsized":"2024-02-10T07:46:41.106520+0000","mapping_epoch":325,"log_start":"317'200","ondisk_log_start":"317'200","created":309,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288948224,"num_objects":4409,"num_object_clones":0,"num_object_copies":13227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":991,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4409,"num_write_kb":282176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1658,"num_bytes_recovered":108658688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":991},{"shards":"0,7","objects":3418}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":597,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.201689+0000","last_change":"2024-02-10T07:46:41.126327+0000","last_active":"2024-02-10T07:47:07.201689+0000","last_peered":"2024-02-10T07:47:07.201689+0000","last_clean":"2024-02-10T07:47:07.201689+0000","last_became_active":"2024-02-10T07:46:41.125757+0000","last_became_peered":"2024-02-10T07:46:41.125757+0000","last_unstale":"2024-02-10T07:47:07.201689+0000","last_undegraded":"2024-02-10T07:47:07.201689+0000","last_fullsized":"2024-02-10T07:47:07.201689+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:12.239727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:45:12.239727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":577,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.191477+0000","last_change":"2024-02-10T07:46:41.121167+0000","last_active":"2024-02-10T07:47:07.191477+0000","last_peered":"2024-02-10T07:47:07.191477+0000","last_clean":"2024-02-10T07:47:07.191477+0000","last_became_active":"2024-02-10T07:46:41.120629+0000","last_became_peered":"2024-02-10T07:46:41.120629+0000","last_unstale":"2024-02-10T07:47:07.191477+0000","last_undegraded":"2024-02-10T07:47:07.191477+0000","last_fullsized":"2024-02-10T07:47:07.191477+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:28.822700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:28.822700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:34.219671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":556,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.212510+0000","last_change":"2024-02-10T07:46:40.121526+0000","last_active":"2024-02-10T07:47:07.212510+0000","last_peered":"2024-02-10T07:47:07.212510+0000","last_clean":"2024-02-10T07:47:07.212510+0000","last_became_active":"2024-02-10T07:46:40.121189+0000","last_became_peered":"2024-02-10T07:46:40.121189+0000","last_unstale":"2024-02-10T07:47:07.212510+0000","last_undegraded":"2024-02-10T07:47:07.212510+0000","last_fullsized":"2024-02-10T07:47:07.212510+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:59.363500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:43:59.363500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"250'15","reported_seq":619,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202546+0000","last_change":"2024-02-10T07:45:11.162197+0000","last_active":"2024-02-10T07:47:07.202546+0000","last_peered":"2024-02-10T07:47:07.202546+0000","last_clean":"2024-02-10T07:47:07.202546+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:47:07.202546+0000","last_undegraded":"2024-02-10T07:47:07.202546+0000","last_fullsized":"2024-02-10T07:47:07.202546+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:45:11.161953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:45:11.161953+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"330'4313","reported_seq":4364,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:08.057764+0000","last_change":"2024-02-10T07:46:06.701491+0000","last_active":"2024-02-10T07:47:08.057764+0000","last_peered":"2024-02-10T07:47:08.057764+0000","last_clean":"2024-02-10T07:47:08.057764+0000","last_became_active":"2024-02-10T07:46:06.701226+0000","last_became_peered":"2024-02-10T07:46:06.701226+0000","last_unstale":"2024-02-10T07:47:08.057764+0000","last_undegraded":"2024-02-10T07:47:08.057764+0000","last_fullsized":"2024-02-10T07:47:08.057764+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4313,"log_dups_size":0,"ondisk_log_size":4313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:12.054166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282656768,"num_objects":4313,"num_object_clones":0,"num_object_copies":12939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4313,"num_write_kb":276032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":80,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.195156+0000","last_change":"2024-02-10T07:46:40.132072+0000","last_active":"2024-02-10T07:47:07.195156+0000","last_peered":"2024-02-10T07:47:07.195156+0000","last_clean":"2024-02-10T07:47:07.195156+0000","last_became_active":"2024-02-10T07:46:40.131207+0000","last_became_peered":"2024-02-10T07:46:40.131207+0000","last_unstale":"2024-02-10T07:47:07.195156+0000","last_undegraded":"2024-02-10T07:47:07.195156+0000","last_fullsized":"2024-02-10T07:47:07.195156+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:41.991625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:43:41.991625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":539,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.215606+0000","last_change":"2024-02-10T07:45:44.361729+0000","last_active":"2024-02-10T07:47:07.215606+0000","last_peered":"2024-02-10T07:47:07.215606+0000","last_clean":"2024-02-10T07:47:07.215606+0000","last_became_active":"2024-02-10T07:36:33.759656+0000","last_became_peered":"2024-02-10T07:36:33.759656+0000","last_unstale":"2024-02-10T07:47:07.215606+0000","last_undegraded":"2024-02-10T07:47:07.215606+0000","last_fullsized":"2024-02-10T07:47:07.215606+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:45:44.361605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:45:44.361605+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":591,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.192040+0000","last_change":"2024-02-10T07:45:04.220439+0000","last_active":"2024-02-10T07:47:07.192040+0000","last_peered":"2024-02-10T07:47:07.192040+0000","last_clean":"2024-02-10T07:47:07.192040+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:47:07.192040+0000","last_undegraded":"2024-02-10T07:47:07.192040+0000","last_fullsized":"2024-02-10T07:47:07.192040+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:04.220328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:04.220328+0000","last_clean_scrub_stamp":"2024-02-10T07:45:04.220328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"326'45","reported_seq":904,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.205839+0000","last_change":"2024-02-10T07:46:41.157810+0000","last_active":"2024-02-10T07:47:07.205839+0000","last_peered":"2024-02-10T07:47:07.205839+0000","last_clean":"2024-02-10T07:47:07.205839+0000","last_became_active":"2024-02-10T07:46:41.122806+0000","last_became_peered":"2024-02-10T07:46:41.122806+0000","last_unstale":"2024-02-10T07:47:07.205839+0000","last_undegraded":"2024-02-10T07:47:07.205839+0000","last_fullsized":"2024-02-10T07:47:07.205839+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'43","last_scrub_stamp":"2024-02-10T07:46:29.401404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:46:29.401404+0000","objects_scrubbed":10,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:40.393108+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":599,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202328+0000","last_change":"2024-02-10T07:45:47.080393+0000","last_active":"2024-02-10T07:47:07.202328+0000","last_peered":"2024-02-10T07:47:07.202328+0000","last_clean":"2024-02-10T07:47:07.202328+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:47:07.202328+0000","last_undegraded":"2024-02-10T07:47:07.202328+0000","last_fullsized":"2024-02-10T07:47:07.202328+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:47.080312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:22.786949+0000","last_clean_scrub_stamp":"2024-02-10T07:45:47.080312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"330'4265","reported_seq":4323,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.243695+0000","last_change":"2024-02-10T07:46:10.717776+0000","last_active":"2024-02-10T07:47:07.243695+0000","last_peered":"2024-02-10T07:47:07.243695+0000","last_clean":"2024-02-10T07:47:07.243695+0000","last_became_active":"2024-02-10T07:46:10.717353+0000","last_became_peered":"2024-02-10T07:46:10.717353+0000","last_unstale":"2024-02-10T07:47:07.243695+0000","last_undegraded":"2024-02-10T07:47:07.243695+0000","last_fullsized":"2024-02-10T07:47:07.243695+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:15.215416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279511040,"num_objects":4265,"num_object_clones":0,"num_object_copies":12795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4265,"num_write_kb":272960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":582,"reported_epoch":330,"state":"active+undersized","last_fresh":"2024-02-10T07:47:07.196187+0000","last_change":"2024-02-10T07:45:03.342368+0000","last_active":"2024-02-10T07:47:07.196187+0000","last_peered":"2024-02-10T07:47:07.196187+0000","last_clean":"2024-02-10T07:44:59.290650+0000","last_became_active":"2024-02-10T07:45:03.342368+0000","last_became_peered":"2024-02-10T07:45:03.342368+0000","last_unstale":"2024-02-10T07:47:07.196187+0000","last_undegraded":"2024-02-10T07:47:07.196187+0000","last_fullsized":"2024-02-10T07:45:03.334859+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:14.465247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:42.730875+0000","last_clean_scrub_stamp":"2024-02-10T07:43:14.465247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":611,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.205592+0000","last_change":"2024-02-10T07:46:33.322279+0000","last_active":"2024-02-10T07:47:07.205592+0000","last_peered":"2024-02-10T07:47:07.205592+0000","last_clean":"2024-02-10T07:47:07.205592+0000","last_became_active":"2024-02-10T07:46:10.726265+0000","last_became_peered":"2024-02-10T07:46:10.726265+0000","last_unstale":"2024-02-10T07:47:07.205592+0000","last_undegraded":"2024-02-10T07:47:07.205592+0000","last_fullsized":"2024-02-10T07:47:07.205592+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:33.322197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:46:33.322197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:37.273764+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":588,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.201282+0000","last_change":"2024-02-10T07:46:41.118288+0000","last_active":"2024-02-10T07:47:07.201282+0000","last_peered":"2024-02-10T07:47:07.201282+0000","last_clean":"2024-02-10T07:47:07.201282+0000","last_became_active":"2024-02-10T07:46:41.117865+0000","last_became_peered":"2024-02-10T07:46:41.117865+0000","last_unstale":"2024-02-10T07:47:07.201282+0000","last_undegraded":"2024-02-10T07:47:07.201282+0000","last_fullsized":"2024-02-10T07:47:07.201282+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:56.767291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:44:56.767291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"250'17","reported_seq":394,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.208779+0000","last_change":"2024-02-10T07:46:30.354780+0000","last_active":"2024-02-10T07:47:07.208779+0000","last_peered":"2024-02-10T07:47:07.208779+0000","last_clean":"2024-02-10T07:47:07.208779+0000","last_became_active":"2024-02-10T07:37:06.196787+0000","last_became_peered":"2024-02-10T07:37:06.196787+0000","last_unstale":"2024-02-10T07:47:07.208779+0000","last_undegraded":"2024-02-10T07:47:07.208779+0000","last_fullsized":"2024-02-10T07:47:07.208779+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'17","last_scrub_stamp":"2024-02-10T07:46:30.354628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:46:30.354628+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:32.827705+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":254,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.213405+0000","last_change":"2024-02-10T07:46:39.609662+0000","last_active":"2024-02-10T07:47:07.213405+0000","last_peered":"2024-02-10T07:47:07.213405+0000","last_clean":"2024-02-10T07:47:07.213405+0000","last_became_active":"2024-02-10T07:46:10.715491+0000","last_became_peered":"2024-02-10T07:46:10.715491+0000","last_unstale":"2024-02-10T07:47:07.213405+0000","last_undegraded":"2024-02-10T07:47:07.213405+0000","last_fullsized":"2024-02-10T07:47:07.213405+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:39.609589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:39.609589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:44.569885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"330'4276","reported_seq":4327,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:08.173398+0000","last_change":"2024-02-10T07:46:06.701790+0000","last_active":"2024-02-10T07:47:08.173398+0000","last_peered":"2024-02-10T07:47:08.173398+0000","last_clean":"2024-02-10T07:47:08.173398+0000","last_became_active":"2024-02-10T07:46:06.701471+0000","last_became_peered":"2024-02-10T07:46:06.701471+0000","last_unstale":"2024-02-10T07:47:08.173398+0000","last_undegraded":"2024-02-10T07:47:08.173398+0000","last_fullsized":"2024-02-10T07:47:08.173398+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:23.512995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280231936,"num_objects":4276,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4276,"num_write_kb":273664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":538,"reported_epoch":330,"state":"active+undersized+degraded","last_fresh":"2024-02-10T07:47:07.215467+0000","last_change":"2024-02-10T07:45:03.353807+0000","last_active":"2024-02-10T07:47:07.215467+0000","last_peered":"2024-02-10T07:47:07.215467+0000","last_clean":"2024-02-10T07:44:59.293398+0000","last_became_active":"2024-02-10T07:45:03.353807+0000","last_became_peered":"2024-02-10T07:45:03.353807+0000","last_unstale":"2024-02-10T07:47:07.215467+0000","last_undegraded":"2024-02-10T07:45:03.346592+0000","last_fullsized":"2024-02-10T07:45:03.346477+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_clean_scrub_stamp":"2024-02-10T07:42:17.190492+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":844,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.193683+0000","last_change":"2024-02-10T07:45:00.391442+0000","last_active":"2024-02-10T07:47:07.193683+0000","last_peered":"2024-02-10T07:47:07.193683+0000","last_clean":"2024-02-10T07:47:07.193683+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:47:07.193683+0000","last_undegraded":"2024-02-10T07:47:07.193683+0000","last_fullsized":"2024-02-10T07:47:07.193683+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:45:00.391254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:45:00.391254+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":282,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.215754+0000","last_change":"2024-02-10T07:46:40.122597+0000","last_active":"2024-02-10T07:47:07.215754+0000","last_peered":"2024-02-10T07:47:07.215754+0000","last_clean":"2024-02-10T07:47:07.215754+0000","last_became_active":"2024-02-10T07:46:40.122018+0000","last_became_peered":"2024-02-10T07:46:40.122018+0000","last_unstale":"2024-02-10T07:47:07.215754+0000","last_undegraded":"2024-02-10T07:47:07.215754+0000","last_fullsized":"2024-02-10T07:47:07.215754+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:54.302617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:44:54.302617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"250'36","reported_seq":834,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.203416+0000","last_change":"2024-02-10T07:45:08.296056+0000","last_active":"2024-02-10T07:47:07.203416+0000","last_peered":"2024-02-10T07:47:07.203416+0000","last_clean":"2024-02-10T07:47:07.203416+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:47:07.203416+0000","last_undegraded":"2024-02-10T07:47:07.203416+0000","last_fullsized":"2024-02-10T07:47:07.203416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'36","last_scrub_stamp":"2024-02-10T07:45:08.295916+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:45:08.295916+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":614,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.195406+0000","last_change":"2024-02-10T07:46:30.869162+0000","last_active":"2024-02-10T07:47:07.195406+0000","last_peered":"2024-02-10T07:47:07.195406+0000","last_clean":"2024-02-10T07:47:07.195406+0000","last_became_active":"2024-02-10T07:46:10.714440+0000","last_became_peered":"2024-02-10T07:46:10.714440+0000","last_unstale":"2024-02-10T07:47:07.195406+0000","last_undegraded":"2024-02-10T07:47:07.195406+0000","last_fullsized":"2024-02-10T07:47:07.195406+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:30.869034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.736901+0000","last_clean_scrub_stamp":"2024-02-10T07:46:30.869034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:54.637143+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"330'4323","reported_seq":4381,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:09.080557+0000","last_change":"2024-02-10T07:46:10.721112+0000","last_active":"2024-02-10T07:47:09.080557+0000","last_peered":"2024-02-10T07:47:09.080557+0000","last_clean":"2024-02-10T07:47:09.080557+0000","last_became_active":"2024-02-10T07:46:10.720735+0000","last_became_peered":"2024-02-10T07:46:10.720735+0000","last_unstale":"2024-02-10T07:47:09.080557+0000","last_undegraded":"2024-02-10T07:47:09.080557+0000","last_fullsized":"2024-02-10T07:47:09.080557+0000","mapping_epoch":312,"log_start":"317'100","ondisk_log_start":"317'100","created":309,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4223,"log_dups_size":0,"ondisk_log_size":4223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283312128,"num_objects":4323,"num_object_clones":0,"num_object_copies":12969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4323,"num_write_kb":276672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":377,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.204937+0000","last_change":"2024-02-10T07:46:40.124554+0000","last_active":"2024-02-10T07:47:07.204937+0000","last_peered":"2024-02-10T07:47:07.204937+0000","last_clean":"2024-02-10T07:47:07.204937+0000","last_became_active":"2024-02-10T07:46:40.124220+0000","last_became_peered":"2024-02-10T07:46:40.124220+0000","last_unstale":"2024-02-10T07:47:07.204937+0000","last_undegraded":"2024-02-10T07:47:07.204937+0000","last_fullsized":"2024-02-10T07:47:07.204937+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:49.224791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:44:49.224791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":600,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.201364+0000","last_change":"2024-02-10T07:46:40.131814+0000","last_active":"2024-02-10T07:47:07.201364+0000","last_peered":"2024-02-10T07:47:07.201364+0000","last_clean":"2024-02-10T07:47:07.201364+0000","last_became_active":"2024-02-10T07:46:40.131499+0000","last_became_peered":"2024-02-10T07:46:40.131499+0000","last_unstale":"2024-02-10T07:47:07.201364+0000","last_undegraded":"2024-02-10T07:47:07.201364+0000","last_fullsized":"2024-02-10T07:47:07.201364+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:07.146214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:37.574311+0000","last_clean_scrub_stamp":"2024-02-10T07:44:07.146214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1521,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.193632+0000","last_change":"2024-02-10T07:45:06.409043+0000","last_active":"2024-02-10T07:47:07.193632+0000","last_peered":"2024-02-10T07:47:07.193632+0000","last_clean":"2024-02-10T07:47:07.193632+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:47:07.193632+0000","last_undegraded":"2024-02-10T07:47:07.193632+0000","last_fullsized":"2024-02-10T07:47:07.193632+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:45:06.408929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:45:06.408929+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":910,"num_read_kb":905,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"250'30","reported_seq":657,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202469+0000","last_change":"2024-02-10T07:46:10.740692+0000","last_active":"2024-02-10T07:47:07.202469+0000","last_peered":"2024-02-10T07:47:07.202469+0000","last_clean":"2024-02-10T07:47:07.202469+0000","last_became_active":"2024-02-10T07:46:10.724393+0000","last_became_peered":"2024-02-10T07:46:10.724393+0000","last_unstale":"2024-02-10T07:47:07.202469+0000","last_undegraded":"2024-02-10T07:47:07.202469+0000","last_fullsized":"2024-02-10T07:47:07.202469+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:45:58.123359+0000","last_deep_scrub":"237'18","last_deep_scrub_stamp":"2024-02-10T07:40:31.005376+0000","last_clean_scrub_stamp":"2024-02-10T07:45:58.123359+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:14.268465+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":537,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.215968+0000","last_change":"2024-02-10T07:45:08.443685+0000","last_active":"2024-02-10T07:47:07.215968+0000","last_peered":"2024-02-10T07:47:07.215968+0000","last_clean":"2024-02-10T07:47:07.215968+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:47:07.215968+0000","last_undegraded":"2024-02-10T07:47:07.215968+0000","last_fullsized":"2024-02-10T07:47:07.215968+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:08.443588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:08.443588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"330'4300","reported_seq":4351,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:08.042255+0000","last_change":"2024-02-10T07:46:06.701630+0000","last_active":"2024-02-10T07:47:08.042255+0000","last_peered":"2024-02-10T07:47:08.042255+0000","last_clean":"2024-02-10T07:47:08.042255+0000","last_became_active":"2024-02-10T07:46:06.700757+0000","last_became_peered":"2024-02-10T07:46:06.700757+0000","last_unstale":"2024-02-10T07:47:08.042255+0000","last_undegraded":"2024-02-10T07:47:08.042255+0000","last_fullsized":"2024-02-10T07:47:08.042255+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4300,"log_dups_size":0,"ondisk_log_size":4300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:19.322021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281804800,"num_objects":4300,"num_object_clones":0,"num_object_copies":12900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4300,"num_write_kb":275200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":599,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.204725+0000","last_change":"2024-02-10T07:46:39.341399+0000","last_active":"2024-02-10T07:47:07.204725+0000","last_peered":"2024-02-10T07:47:07.204725+0000","last_clean":"2024-02-10T07:47:07.204725+0000","last_became_active":"2024-02-10T07:46:10.716852+0000","last_became_peered":"2024-02-10T07:46:10.716852+0000","last_unstale":"2024-02-10T07:47:07.204725+0000","last_undegraded":"2024-02-10T07:47:07.204725+0000","last_fullsized":"2024-02-10T07:47:07.204725+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:39.341326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:46:39.341326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:59.077153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":612,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.205800+0000","last_change":"2024-02-10T07:46:10.718679+0000","last_active":"2024-02-10T07:47:07.205800+0000","last_peered":"2024-02-10T07:47:07.205800+0000","last_clean":"2024-02-10T07:47:07.205800+0000","last_became_active":"2024-02-10T07:46:10.718340+0000","last_became_peered":"2024-02-10T07:46:10.718340+0000","last_unstale":"2024-02-10T07:47:07.205800+0000","last_undegraded":"2024-02-10T07:47:07.205800+0000","last_fullsized":"2024-02-10T07:47:07.205800+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:43.511591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:45:43.511591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"271'37","reported_seq":663,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202260+0000","last_change":"2024-02-10T07:46:28.324286+0000","last_active":"2024-02-10T07:47:07.202260+0000","last_peered":"2024-02-10T07:47:07.202260+0000","last_clean":"2024-02-10T07:47:07.202260+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:47:07.202260+0000","last_undegraded":"2024-02-10T07:47:07.202260+0000","last_fullsized":"2024-02-10T07:47:07.202260+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'37","last_scrub_stamp":"2024-02-10T07:46:28.324161+0000","last_deep_scrub":"271'37","last_deep_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_clean_scrub_stamp":"2024-02-10T07:46:28.324161+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:39.058621+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":615,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.205726+0000","last_change":"2024-02-10T07:46:00.292423+0000","last_active":"2024-02-10T07:47:07.205726+0000","last_peered":"2024-02-10T07:47:07.205726+0000","last_clean":"2024-02-10T07:47:07.205726+0000","last_became_active":"2024-02-10T07:36:33.765998+0000","last_became_peered":"2024-02-10T07:36:33.765998+0000","last_unstale":"2024-02-10T07:47:07.205726+0000","last_undegraded":"2024-02-10T07:47:07.205726+0000","last_fullsized":"2024-02-10T07:47:07.205726+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:00.292320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:00.292320+0000","last_clean_scrub_stamp":"2024-02-10T07:46:00.292320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:09.084838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"330'4245","reported_seq":4296,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:08.003330+0000","last_change":"2024-02-10T07:46:06.696224+0000","last_active":"2024-02-10T07:47:08.003330+0000","last_peered":"2024-02-10T07:47:08.003330+0000","last_clean":"2024-02-10T07:47:08.003330+0000","last_became_active":"2024-02-10T07:46:06.695925+0000","last_became_peered":"2024-02-10T07:46:06.695925+0000","last_unstale":"2024-02-10T07:47:08.003330+0000","last_undegraded":"2024-02-10T07:47:08.003330+0000","last_fullsized":"2024-02-10T07:47:08.003330+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4245,"log_dups_size":0,"ondisk_log_size":4245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:21.096136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278200320,"num_objects":4245,"num_object_clones":0,"num_object_copies":12735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4245,"num_write_kb":271680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":473,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.201424+0000","last_change":"2024-02-10T07:45:13.380284+0000","last_active":"2024-02-10T07:47:07.201424+0000","last_peered":"2024-02-10T07:47:07.201424+0000","last_clean":"2024-02-10T07:47:07.201424+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:47:07.201424+0000","last_undegraded":"2024-02-10T07:47:07.201424+0000","last_fullsized":"2024-02-10T07:47:07.201424+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:13.379984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:45:13.379984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":573,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.216011+0000","last_change":"2024-02-10T07:46:41.118176+0000","last_active":"2024-02-10T07:47:07.216011+0000","last_peered":"2024-02-10T07:47:07.216011+0000","last_clean":"2024-02-10T07:47:07.216011+0000","last_became_active":"2024-02-10T07:46:41.117867+0000","last_became_peered":"2024-02-10T07:46:41.117867+0000","last_unstale":"2024-02-10T07:47:07.216011+0000","last_undegraded":"2024-02-10T07:47:07.216011+0000","last_fullsized":"2024-02-10T07:47:07.216011+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:33.538497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:46:33.538497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:39.250480+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"250'20","reported_seq":403,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.205973+0000","last_change":"2024-02-10T07:45:58.418445+0000","last_active":"2024-02-10T07:47:07.205973+0000","last_peered":"2024-02-10T07:47:07.205973+0000","last_clean":"2024-02-10T07:47:07.205973+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:47:07.205973+0000","last_undegraded":"2024-02-10T07:47:07.205973+0000","last_fullsized":"2024-02-10T07:47:07.205973+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:45:58.418313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:45:58.418313+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:26.942699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"158'10","reported_seq":793,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202658+0000","last_change":"2024-02-10T07:45:50.057298+0000","last_active":"2024-02-10T07:47:07.202658+0000","last_peered":"2024-02-10T07:47:07.202658+0000","last_clean":"2024-02-10T07:47:07.202658+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:47:07.202658+0000","last_undegraded":"2024-02-10T07:47:07.202658+0000","last_fullsized":"2024-02-10T07:47:07.202658+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:45:50.057213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:50.057213+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:19.384265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"330'4344","reported_seq":6244,"reported_epoch":330,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-10T07:47:08.164502+0000","last_change":"2024-02-10T07:46:41.263511+0000","last_active":"2024-02-10T07:47:08.164502+0000","last_peered":"2024-02-10T07:47:08.164502+0000","last_clean":"2024-02-10T07:46:39.097064+0000","last_became_active":"2024-02-10T07:46:41.227073+0000","last_became_peered":"2024-02-10T07:46:41.227073+0000","last_unstale":"2024-02-10T07:47:08.164502+0000","last_undegraded":"2024-02-10T07:47:08.164502+0000","last_fullsized":"2024-02-10T07:46:41.112776+0000","mapping_epoch":325,"log_start":"317'100","ondisk_log_start":"317'100","created":309,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4244,"log_dups_size":0,"ondisk_log_size":4244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284688384,"num_objects":4344,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1697,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4344,"num_write_kb":278016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":60358656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":1697},{"shards":"4,6","objects":2647}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":593,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.194471+0000","last_change":"2024-02-10T07:46:10.719848+0000","last_active":"2024-02-10T07:47:07.194471+0000","last_peered":"2024-02-10T07:47:07.194471+0000","last_clean":"2024-02-10T07:47:07.194471+0000","last_became_active":"2024-02-10T07:46:10.719534+0000","last_became_peered":"2024-02-10T07:46:10.719534+0000","last_unstale":"2024-02-10T07:47:07.194471+0000","last_undegraded":"2024-02-10T07:47:07.194471+0000","last_fullsized":"2024-02-10T07:47:07.194471+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:45.707189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:45:45.707189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"296'2","reported_seq":609,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202522+0000","last_change":"2024-02-10T07:46:29.295971+0000","last_active":"2024-02-10T07:47:07.202522+0000","last_peered":"2024-02-10T07:47:07.202522+0000","last_clean":"2024-02-10T07:47:07.202522+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:47:07.202522+0000","last_undegraded":"2024-02-10T07:47:07.202522+0000","last_fullsized":"2024-02-10T07:47:07.202522+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'2","last_scrub_stamp":"2024-02-10T07:46:29.295573+0000","last_deep_scrub":"296'2","last_deep_scrub_stamp":"2024-02-10T07:45:03.233375+0000","last_clean_scrub_stamp":"2024-02-10T07:46:29.295573+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:40.722008+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"250'28","reported_seq":638,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.193107+0000","last_change":"2024-02-10T07:45:10.185368+0000","last_active":"2024-02-10T07:47:07.193107+0000","last_peered":"2024-02-10T07:47:07.193107+0000","last_clean":"2024-02-10T07:47:07.193107+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:47:07.193107+0000","last_undegraded":"2024-02-10T07:47:07.193107+0000","last_fullsized":"2024-02-10T07:47:07.193107+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-02-10T07:45:10.185239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:45:10.185239+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"325'11","reported_seq":570,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.204492+0000","last_change":"2024-02-10T07:46:40.131958+0000","last_active":"2024-02-10T07:47:07.204492+0000","last_peered":"2024-02-10T07:47:07.204492+0000","last_clean":"2024-02-10T07:47:07.204492+0000","last_became_active":"2024-02-10T07:46:40.121401+0000","last_became_peered":"2024-02-10T07:46:40.121401+0000","last_unstale":"2024-02-10T07:47:07.204492+0000","last_undegraded":"2024-02-10T07:47:07.204492+0000","last_fullsized":"2024-02-10T07:47:07.204492+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'9","last_scrub_stamp":"2024-02-10T07:44:55.376986+0000","last_deep_scrub":"227'9","last_deep_scrub_stamp":"2024-02-10T07:39:53.440221+0000","last_clean_scrub_stamp":"2024-02-10T07:44:55.376986+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"330'4322","reported_seq":4378,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:08.041514+0000","last_change":"2024-02-10T07:46:10.721591+0000","last_active":"2024-02-10T07:47:08.041514+0000","last_peered":"2024-02-10T07:47:08.041514+0000","last_clean":"2024-02-10T07:47:08.041514+0000","last_became_active":"2024-02-10T07:46:10.721275+0000","last_became_peered":"2024-02-10T07:46:10.721275+0000","last_unstale":"2024-02-10T07:47:08.041514+0000","last_undegraded":"2024-02-10T07:47:08.041514+0000","last_fullsized":"2024-02-10T07:47:08.041514+0000","mapping_epoch":312,"log_start":"319'500","ondisk_log_start":"319'500","created":309,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":3822,"log_dups_size":0,"ondisk_log_size":3822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:09.301353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283246592,"num_objects":4322,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4322,"num_write_kb":276608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":592,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.191193+0000","last_change":"2024-02-10T07:46:41.118534+0000","last_active":"2024-02-10T07:47:07.191193+0000","last_peered":"2024-02-10T07:47:07.191193+0000","last_clean":"2024-02-10T07:47:07.191193+0000","last_became_active":"2024-02-10T07:46:41.118200+0000","last_became_peered":"2024-02-10T07:46:41.118200+0000","last_unstale":"2024-02-10T07:47:07.191193+0000","last_undegraded":"2024-02-10T07:47:07.191193+0000","last_fullsized":"2024-02-10T07:47:07.191193+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:27.819612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:46:27.819612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:47.550307+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":74,"reported_epoch":330,"state":"active+undersized","last_fresh":"2024-02-10T07:47:07.195923+0000","last_change":"2024-02-10T07:45:03.340655+0000","last_active":"2024-02-10T07:47:07.195923+0000","last_peered":"2024-02-10T07:47:07.195923+0000","last_clean":"2024-02-10T07:31:50.269547+0000","last_became_active":"2024-02-10T07:45:03.340655+0000","last_became_peered":"2024-02-10T07:45:03.340655+0000","last_unstale":"2024-02-10T07:47:07.195923+0000","last_undegraded":"2024-02-10T07:47:07.195923+0000","last_fullsized":"2024-02-10T07:45:03.334163+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:06.020588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:44:06.020588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"250'30","reported_seq":548,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.194202+0000","last_change":"2024-02-10T07:46:40.146353+0000","last_active":"2024-02-10T07:47:07.194202+0000","last_peered":"2024-02-10T07:47:07.194202+0000","last_clean":"2024-02-10T07:47:07.194202+0000","last_became_active":"2024-02-10T07:46:40.127999+0000","last_became_peered":"2024-02-10T07:46:40.127999+0000","last_unstale":"2024-02-10T07:47:07.194202+0000","last_undegraded":"2024-02-10T07:47:07.194202+0000","last_fullsized":"2024-02-10T07:47:07.194202+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:44:08.032066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:44:08.032066+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":592,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.192901+0000","last_change":"2024-02-10T07:46:40.132937+0000","last_active":"2024-02-10T07:47:07.192901+0000","last_peered":"2024-02-10T07:47:07.192901+0000","last_clean":"2024-02-10T07:47:07.192901+0000","last_became_active":"2024-02-10T07:46:40.132156+0000","last_became_peered":"2024-02-10T07:46:40.132156+0000","last_unstale":"2024-02-10T07:47:07.192901+0000","last_undegraded":"2024-02-10T07:47:07.192901+0000","last_fullsized":"2024-02-10T07:47:07.192901+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:47.483009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:43:47.483009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"330'4334","reported_seq":4385,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:08.009635+0000","last_change":"2024-02-10T07:46:06.699530+0000","last_active":"2024-02-10T07:47:08.009635+0000","last_peered":"2024-02-10T07:47:08.009635+0000","last_clean":"2024-02-10T07:47:08.009635+0000","last_became_active":"2024-02-10T07:46:06.699247+0000","last_became_peered":"2024-02-10T07:46:06.699247+0000","last_unstale":"2024-02-10T07:47:08.009635+0000","last_undegraded":"2024-02-10T07:47:08.009635+0000","last_fullsized":"2024-02-10T07:47:08.009635+0000","mapping_epoch":309,"log_start":"319'500","ondisk_log_start":"319'500","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":3834,"log_dups_size":0,"ondisk_log_size":3834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:22.607657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284033024,"num_objects":4334,"num_object_clones":0,"num_object_copies":13002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4334,"num_write_kb":277376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":537,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.219533+0000","last_change":"2024-02-10T07:45:45.420020+0000","last_active":"2024-02-10T07:47:07.219533+0000","last_peered":"2024-02-10T07:47:07.219533+0000","last_clean":"2024-02-10T07:47:07.219533+0000","last_became_active":"2024-02-10T07:36:33.772094+0000","last_became_peered":"2024-02-10T07:36:33.772094+0000","last_unstale":"2024-02-10T07:47:07.219533+0000","last_undegraded":"2024-02-10T07:47:07.219533+0000","last_fullsized":"2024-02-10T07:47:07.219533+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:45.419912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:49.704024+0000","last_clean_scrub_stamp":"2024-02-10T07:45:45.419912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"158'10","reported_seq":784,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.195615+0000","last_change":"2024-02-10T07:45:06.247795+0000","last_active":"2024-02-10T07:47:07.195615+0000","last_peered":"2024-02-10T07:47:07.195615+0000","last_clean":"2024-02-10T07:47:07.195615+0000","last_became_active":"2024-02-10T07:36:33.755111+0000","last_became_peered":"2024-02-10T07:36:33.755111+0000","last_unstale":"2024-02-10T07:47:07.195615+0000","last_undegraded":"2024-02-10T07:47:07.195615+0000","last_fullsized":"2024-02-10T07:47:07.195615+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:45:06.247355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:06.247355+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":80,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.205157+0000","last_change":"2024-02-10T07:46:40.116568+0000","last_active":"2024-02-10T07:47:07.205157+0000","last_peered":"2024-02-10T07:47:07.205157+0000","last_clean":"2024-02-10T07:47:07.205157+0000","last_became_active":"2024-02-10T07:46:40.116253+0000","last_became_peered":"2024-02-10T07:46:40.116253+0000","last_unstale":"2024-02-10T07:47:07.205157+0000","last_undegraded":"2024-02-10T07:47:07.205157+0000","last_fullsized":"2024-02-10T07:47:07.205157+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:48.313261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:44:48.313261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"250'27","reported_seq":638,"reported_epoch":330,"state":"active+undersized+degraded","last_fresh":"2024-02-10T07:47:07.205825+0000","last_change":"2024-02-10T07:45:03.349183+0000","last_active":"2024-02-10T07:47:07.205825+0000","last_peered":"2024-02-10T07:47:07.205825+0000","last_clean":"2024-02-10T07:44:59.300405+0000","last_became_active":"2024-02-10T07:45:03.349183+0000","last_became_peered":"2024-02-10T07:45:03.349183+0000","last_unstale":"2024-02-10T07:47:07.205825+0000","last_undegraded":"2024-02-10T07:45:03.336565+0000","last_fullsized":"2024-02-10T07:45:03.336456+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:43:58.327027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:43:58.327027+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"330'4343","reported_seq":4394,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:08.084772+0000","last_change":"2024-02-10T07:46:06.699422+0000","last_active":"2024-02-10T07:47:08.084772+0000","last_peered":"2024-02-10T07:47:08.084772+0000","last_clean":"2024-02-10T07:47:08.084772+0000","last_became_active":"2024-02-10T07:46:06.699140+0000","last_became_peered":"2024-02-10T07:46:06.699140+0000","last_unstale":"2024-02-10T07:47:08.084772+0000","last_undegraded":"2024-02-10T07:47:08.084772+0000","last_fullsized":"2024-02-10T07:47:08.084772+0000","mapping_epoch":309,"log_start":"317'100","ondisk_log_start":"317'100","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:35.313669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284622848,"num_objects":4343,"num_object_clones":0,"num_object_copies":13029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":277952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":277,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.201199+0000","last_change":"2024-02-10T07:46:40.124370+0000","last_active":"2024-02-10T07:47:07.201199+0000","last_peered":"2024-02-10T07:47:07.201199+0000","last_clean":"2024-02-10T07:47:07.201199+0000","last_became_active":"2024-02-10T07:46:40.124029+0000","last_became_peered":"2024-02-10T07:46:40.124029+0000","last_unstale":"2024-02-10T07:47:07.201199+0000","last_undegraded":"2024-02-10T07:47:07.201199+0000","last_fullsized":"2024-02-10T07:47:07.201199+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:19.934260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_clean_scrub_stamp":"2024-02-10T07:43:19.934260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":519,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.215724+0000","last_change":"2024-02-10T07:45:47.344122+0000","last_active":"2024-02-10T07:47:07.215724+0000","last_peered":"2024-02-10T07:47:07.215724+0000","last_clean":"2024-02-10T07:47:07.215724+0000","last_became_active":"2024-02-10T07:36:33.759566+0000","last_became_peered":"2024-02-10T07:36:33.759566+0000","last_unstale":"2024-02-10T07:47:07.215724+0000","last_undegraded":"2024-02-10T07:47:07.215724+0000","last_fullsized":"2024-02-10T07:47:07.215724+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:47.343978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:47.343978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:12.537455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":605,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.192990+0000","last_change":"2024-02-10T07:46:37.995480+0000","last_active":"2024-02-10T07:47:07.192990+0000","last_peered":"2024-02-10T07:47:07.192990+0000","last_clean":"2024-02-10T07:47:07.192990+0000","last_became_active":"2024-02-10T07:46:10.721886+0000","last_became_peered":"2024-02-10T07:46:10.721886+0000","last_unstale":"2024-02-10T07:47:07.192990+0000","last_undegraded":"2024-02-10T07:47:07.192990+0000","last_fullsized":"2024-02-10T07:47:07.192990+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:37.995362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.995362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:48.497594+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"250'24","reported_seq":658,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.195330+0000","last_change":"2024-02-10T07:45:05.256634+0000","last_active":"2024-02-10T07:47:07.195330+0000","last_peered":"2024-02-10T07:47:07.195330+0000","last_clean":"2024-02-10T07:47:07.195330+0000","last_became_active":"2024-02-10T07:37:06.193386+0000","last_became_peered":"2024-02-10T07:37:06.193386+0000","last_unstale":"2024-02-10T07:47:07.195330+0000","last_undegraded":"2024-02-10T07:47:07.195330+0000","last_fullsized":"2024-02-10T07:47:07.195330+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:45:05.256278+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:45:05.256278+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"330'4214","reported_seq":6614,"reported_epoch":330,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-10T07:47:07.297111+0000","last_change":"2024-02-10T07:46:41.236312+0000","last_active":"2024-02-10T07:47:07.297111+0000","last_peered":"2024-02-10T07:47:07.297111+0000","last_clean":"2024-02-10T07:46:39.083851+0000","last_became_active":"2024-02-10T07:46:41.217135+0000","last_became_peered":"2024-02-10T07:46:41.217135+0000","last_unstale":"2024-02-10T07:47:07.297111+0000","last_undegraded":"2024-02-10T07:47:07.297111+0000","last_fullsized":"2024-02-10T07:46:41.107158+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4214,"log_dups_size":0,"ondisk_log_size":4214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276168704,"num_objects":4214,"num_object_clones":0,"num_object_copies":12642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1413,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4214,"num_write_kb":269696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1175,"num_bytes_recovered":77004800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4],"avail_no_missing":["5","2","4"],"object_location_counts":[{"shards":"2,4,5","objects":1413},{"shards":"4,5","objects":2801}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":565,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.199108+0000","last_change":"2024-02-10T07:46:41.216563+0000","last_active":"2024-02-10T07:47:07.199108+0000","last_peered":"2024-02-10T07:47:07.199108+0000","last_clean":"2024-02-10T07:47:07.199108+0000","last_became_active":"2024-02-10T07:46:41.216189+0000","last_became_peered":"2024-02-10T07:46:41.216189+0000","last_unstale":"2024-02-10T07:47:07.199108+0000","last_undegraded":"2024-02-10T07:47:07.199108+0000","last_fullsized":"2024-02-10T07:47:07.199108+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:59.300487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:43:59.300487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":588,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.205902+0000","last_change":"2024-02-10T07:45:50.420105+0000","last_active":"2024-02-10T07:47:07.205902+0000","last_peered":"2024-02-10T07:47:07.205902+0000","last_clean":"2024-02-10T07:47:07.205902+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:47:07.205902+0000","last_undegraded":"2024-02-10T07:47:07.205902+0000","last_fullsized":"2024-02-10T07:47:07.205902+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:50.420019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:50.420019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":614,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.205834+0000","last_change":"2024-02-10T07:45:57.343470+0000","last_active":"2024-02-10T07:47:07.205834+0000","last_peered":"2024-02-10T07:47:07.205834+0000","last_clean":"2024-02-10T07:47:07.205834+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:47:07.205834+0000","last_undegraded":"2024-02-10T07:47:07.205834+0000","last_fullsized":"2024-02-10T07:47:07.205834+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:57.343387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:44.067302+0000","last_clean_scrub_stamp":"2024-02-10T07:45:57.343387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:08.357952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"250'39","reported_seq":667,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.192688+0000","last_change":"2024-02-10T07:46:10.733638+0000","last_active":"2024-02-10T07:47:07.192688+0000","last_peered":"2024-02-10T07:47:07.192688+0000","last_clean":"2024-02-10T07:47:07.192688+0000","last_became_active":"2024-02-10T07:46:10.721474+0000","last_became_peered":"2024-02-10T07:46:10.721474+0000","last_unstale":"2024-02-10T07:47:07.192688+0000","last_undegraded":"2024-02-10T07:47:07.192688+0000","last_fullsized":"2024-02-10T07:47:07.192688+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T07:44:55.232884+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_clean_scrub_stamp":"2024-02-10T07:44:55.232884+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"330'4398","reported_seq":4449,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:09.080147+0000","last_change":"2024-02-10T07:46:06.698262+0000","last_active":"2024-02-10T07:47:09.080147+0000","last_peered":"2024-02-10T07:47:09.080147+0000","last_clean":"2024-02-10T07:47:09.080147+0000","last_became_active":"2024-02-10T07:46:06.697863+0000","last_became_peered":"2024-02-10T07:46:06.697863+0000","last_unstale":"2024-02-10T07:47:09.080147+0000","last_undegraded":"2024-02-10T07:47:09.080147+0000","last_fullsized":"2024-02-10T07:47:09.080147+0000","mapping_epoch":309,"log_start":"319'600","ondisk_log_start":"319'600","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":3798,"log_dups_size":0,"ondisk_log_size":3798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:26.341062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288227328,"num_objects":4398,"num_object_clones":0,"num_object_copies":13194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4398,"num_write_kb":281472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":584,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.195708+0000","last_change":"2024-02-10T07:46:10.718207+0000","last_active":"2024-02-10T07:47:07.195708+0000","last_peered":"2024-02-10T07:47:07.195708+0000","last_clean":"2024-02-10T07:47:07.195708+0000","last_became_active":"2024-02-10T07:46:10.717081+0000","last_became_peered":"2024-02-10T07:46:10.717081+0000","last_unstale":"2024-02-10T07:47:07.195708+0000","last_undegraded":"2024-02-10T07:47:07.195708+0000","last_fullsized":"2024-02-10T07:47:07.195708+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:41.997290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:45:41.997290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"313'11","reported_seq":97,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.195846+0000","last_change":"2024-02-10T07:46:38.873286+0000","last_active":"2024-02-10T07:47:07.195846+0000","last_peered":"2024-02-10T07:47:07.195846+0000","last_clean":"2024-02-10T07:47:07.195846+0000","last_became_active":"2024-02-10T07:46:10.719380+0000","last_became_peered":"2024-02-10T07:46:10.719380+0000","last_unstale":"2024-02-10T07:47:07.195846+0000","last_undegraded":"2024-02-10T07:47:07.195846+0000","last_fullsized":"2024-02-10T07:47:07.195846+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'11","last_scrub_stamp":"2024-02-10T07:46:38.873049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:38.873049+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:47:56.042414+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":81,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.202455+0000","last_change":"2024-02-10T07:46:40.129803+0000","last_active":"2024-02-10T07:47:07.202455+0000","last_peered":"2024-02-10T07:47:07.202455+0000","last_clean":"2024-02-10T07:47:07.202455+0000","last_became_active":"2024-02-10T07:46:40.129473+0000","last_became_peered":"2024-02-10T07:46:40.129473+0000","last_unstale":"2024-02-10T07:47:07.202455+0000","last_undegraded":"2024-02-10T07:47:07.202455+0000","last_fullsized":"2024-02-10T07:47:07.202455+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_clean_scrub_stamp":"2024-02-10T07:43:09.950640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"250'20","reported_seq":637,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-10T07:47:07.197982+0000","last_change":"2024-02-10T07:46:40.137915+0000","last_active":"2024-02-10T07:47:07.197982+0000","last_peered":"2024-02-10T07:47:07.197982+0000","last_clean":"2024-02-10T07:47:07.197982+0000","last_became_active":"2024-02-10T07:46:40.116204+0000","last_became_peered":"2024-02-10T07:46:40.116204+0000","last_unstale":"2024-02-10T07:47:07.197982+0000","last_undegraded":"2024-02-10T07:47:07.197982+0000","last_fullsized":"2024-02-10T07:47:07.197982+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:45:00.168073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:45:00.168073+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4515758080,"num_objects":68905,"num_object_clones":0,"num_object_copies":206715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6169,"num_objects_misplaced":4101,"num_objects_unfound":0,"num_objects_dirty":68905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68905,"num_write_kb":4409920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4564,"num_bytes_recovered":299106304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13407879168,"data_stored":13407879168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66605,"ondisk_log_size":66605,"up":47,"acting":43,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":910,"num_read_kb":905,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":93,"acting":93,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":3906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":13472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":767,"ondisk_log_size":767,"up":93,"acting":93,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037948,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1941020,"kb_used_data":1364244,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":91804132,"statfs":{"total":95995035648,"available":94007431168,"internally_reserved":0,"allocated":1396985856,"data_stored":1396519744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":590608842},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:46:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.22,"15min":1.758},"min":{"1min":0.246,"5min":0.246,"15min":0.20699999999999999},"max":{"1min":4.1349999999999998,"5min":10.529,"15min":52.591999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.2529999999999999,"15min":1.641},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.23100000000000001},"max":{"1min":3.2959999999999998,"5min":10.640000000000001,"15min":52.494},"last":0.41699999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:46:06 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":2.0089999999999999,"15min":1.6200000000000001},"min":{"1min":0.35699999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.2640000000000002,"5min":30.228000000000002,"15min":35.459000000000003},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.819,"15min":1.607},"min":{"1min":0.29299999999999998,"5min":0.26500000000000001,"15min":0.24399999999999999},"max":{"1min":4.7640000000000002,"5min":23.239000000000001,"15min":35.225000000000001},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.93000000000000005,"15min":1.161},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":4.1760000000000002,"5min":5.9450000000000003,"15min":19.710000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.071,"15min":1.2989999999999999},"min":{"1min":0.33000000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.2320000000000002,"5min":4.6210000000000004,"15min":19.643000000000001},"last":0.34599999999999997}]},{"osd":3,"last update":"Sat Feb 10 07:45:00 2024","interfaces":[{"interface":"back","average":{"1min":3.0830000000000002,"5min":2.6190000000000002,"15min":1.889},"min":{"1min":0.29299999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":19.184000000000001,"5min":43.573999999999998,"15min":64.120999999999995},"last":4.5250000000000004},{"interface":"front","average":{"1min":2.577,"5min":2.5459999999999998,"15min":2.0019999999999998},"min":{"1min":0.35299999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":17.806999999999999,"5min":43.451000000000001,"15min":64.400000000000006},"last":0.57799999999999996}]},{"osd":4,"last update":"Sat Feb 10 07:46:06 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.82599999999999996,"15min":0.66000000000000003},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":2.238,"5min":3.2330000000000001,"15min":5.8499999999999996},"last":3.5289999999999999},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.80600000000000005,"15min":0.66900000000000004},"min":{"1min":0.28299999999999997,"5min":0.26600000000000001,"15min":0.24099999999999999},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.6980000000000004},"last":0.58099999999999996}]},{"osd":5,"last update":"Sat Feb 10 07:46:52 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.98099999999999998,"15min":0.84199999999999997},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":3.173,"5min":6.1050000000000004,"15min":6.1050000000000004},"last":11.472},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.90600000000000003,"15min":0.86399999999999999},"min":{"1min":0.34000000000000002,"5min":0.32300000000000001,"15min":0.29999999999999999},"max":{"1min":3.109,"5min":4.0369999999999999,"15min":4.4939999999999998},"last":3.6440000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:46:06 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":0.999,"15min":0.77900000000000003},"min":{"1min":0.36899999999999999,"5min":0.35699999999999998,"15min":0.28199999999999997},"max":{"1min":3.9700000000000002,"5min":9.7720000000000002,"15min":9.7720000000000002},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.2370000000000001,"5min":0.92800000000000005,"15min":0.78600000000000003},"min":{"1min":0.33700000000000002,"5min":0.29799999999999999,"15min":0.27400000000000002},"max":{"1min":4.218,"5min":4.218,"15min":6.6109999999999998},"last":0.62}]}]},{"osd":6,"up_from":50,"seq":214748364993,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2345976,"kb_used_data":1927472,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":91399176,"statfs":{"total":95995035648,"available":93592756224,"internally_reserved":0,"allocated":1973731328,"data_stored":1973245731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":428538310},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:46:38 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.99199999999999999,"15min":1.401},"min":{"1min":0.30399999999999999,"5min":0.23200000000000001,"15min":0.217},"max":{"1min":2.5819999999999999,"5min":4.5129999999999999,"15min":63.405999999999999},"last":5.6820000000000004},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.77700000000000002,"15min":1.5229999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":3.4990000000000001,"5min":4.0140000000000002,"15min":67.748000000000005},"last":2.556}]},{"osd":1,"last update":"Sat Feb 10 07:46:38 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":1.8560000000000001,"15min":1.528},"min":{"1min":0.26200000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.548,"5min":37.659999999999997,"15min":37.659999999999997},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.786,"15min":1.5129999999999999},"min":{"1min":0.35699999999999998,"5min":0.27000000000000002,"15min":0.24399999999999999},"max":{"1min":2.8820000000000001,"5min":37.75,"15min":37.75},"last":5.7679999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:46:38 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.93899999999999995,"15min":1.4039999999999999},"min":{"1min":0.34100000000000003,"5min":0.253,"15min":0.253},"max":{"1min":2.8250000000000002,"5min":4.556,"15min":65.278999999999996},"last":5.8040000000000003},{"interface":"front","average":{"1min":0.879,"5min":0.81399999999999995,"15min":1.45},"min":{"1min":0.35299999999999998,"5min":0.28399999999999997,"15min":0.25800000000000001},"max":{"1min":3.5489999999999999,"5min":5.6779999999999999,"15min":65.265000000000001},"last":0.61899999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.899,"5min":1.8240000000000001,"15min":1.766},"min":{"1min":0.34799999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":11.103999999999999,"5min":20.603999999999999,"15min":62.835999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":2.274,"5min":1.925,"15min":1.4259999999999999},"min":{"1min":0.36399999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":11.304,"5min":20.478999999999999,"15min":63.878999999999998},"last":0.46400000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:46:38 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.68700000000000006,"15min":0.61699999999999999},"min":{"1min":0.32300000000000001,"5min":0.26900000000000002,"15min":0.23400000000000001},"max":{"1min":3.5859999999999999,"5min":4.7489999999999997,"15min":7.7839999999999998},"last":2.629},{"interface":"front","average":{"1min":1.121,"5min":0.69799999999999995,"15min":0.621},"min":{"1min":0.39500000000000002,"5min":0.28000000000000003,"15min":0.26000000000000001},"max":{"1min":7.0300000000000002,"5min":7.0300000000000002,"15min":7.0300000000000002},"last":3.4620000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:46:52 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.351,"15min":0.94199999999999995},"min":{"1min":0.29599999999999999,"5min":0.26500000000000001,"15min":0.24099999999999999},"max":{"1min":3.6579999999999999,"5min":44.689,"15min":44.689},"last":2.5950000000000002},{"interface":"front","average":{"1min":1.175,"5min":1.375,"15min":0.91300000000000003},"min":{"1min":0.34100000000000003,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.7509999999999999,"5min":44.465000000000003,"15min":44.465000000000003},"last":3.4220000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:47:05 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.5840000000000001,"15min":0.98499999999999999},"min":{"1min":0.372,"5min":0.32200000000000001,"15min":0.28100000000000003},"max":{"1min":8.4499999999999993,"5min":42.631999999999998,"15min":42.631999999999998},"last":5.7329999999999997},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.524,"15min":0.96299999999999997},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.29699999999999999},"max":{"1min":3.6240000000000001,"5min":44.792999999999999,"15min":44.792999999999999},"last":0.54300000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575067,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1579852,"kb_used_data":1003076,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92165300,"statfs":{"total":95995035648,"available":94377267200,"internally_reserved":0,"allocated":1027149824,"data_stored":1026642918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":590608838},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":1.2310000000000001,"15min":1.748},"min":{"1min":0.19,"5min":0.19,"15min":0.182},"max":{"1min":0.996,"5min":7.2850000000000001,"15min":114.27},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":1.2390000000000001,"15min":1.9319999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.96299999999999997,"5min":7.524,"15min":119.499},"last":0.38500000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:47:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":1.1399999999999999,"15min":1.1859999999999999},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":0.84099999999999997,"5min":8.4819999999999993,"15min":8.9209999999999994},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":1.0369999999999999,"15min":1.3220000000000001},"min":{"1min":0.251,"5min":0.224,"15min":0.224},"max":{"1min":0.70799999999999996,"5min":8.8300000000000001,"15min":8.8300000000000001},"last":0.308}]},{"osd":3,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.8340000000000001,"5min":2.0939999999999999,"15min":1.46},"min":{"1min":0.30299999999999999,"5min":0.222,"15min":0.21099999999999999},"max":{"1min":8.4139999999999997,"5min":11.535,"15min":11.535},"last":1.671},{"interface":"front","average":{"1min":2.6379999999999999,"5min":1.8859999999999999,"15min":1.3979999999999999},"min":{"1min":0.59599999999999997,"5min":0.24199999999999999,"15min":0.23799999999999999},"max":{"1min":8.6289999999999996,"5min":9.516,"15min":11.698},"last":1.7170000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:46:58 2024","interfaces":[{"interface":"back","average":{"1min":2.9199999999999999,"5min":1.6919999999999999,"15min":2.0139999999999998},"min":{"1min":0.34799999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":31.908999999999999,"5min":31.908999999999999,"15min":207.45699999999999},"last":0.75},{"interface":"front","average":{"1min":2.4409999999999998,"5min":1.3080000000000001,"15min":1.1519999999999999},"min":{"1min":0.38300000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":32.201000000000001,"5min":32.201000000000001,"15min":32.201000000000001},"last":2.2989999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:46:58 2024","interfaces":[{"interface":"back","average":{"1min":2.7440000000000002,"5min":1.841,"15min":2.0649999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":31.824999999999999,"5min":31.824999999999999,"15min":98.703000000000003},"last":0.82799999999999996},{"interface":"front","average":{"1min":3.0590000000000002,"5min":1.9430000000000001,"15min":2.0390000000000001},"min":{"1min":0.435,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":31.841999999999999,"5min":31.841999999999999,"15min":98.664000000000001},"last":0.84299999999999997}]},{"osd":6,"last update":"Sat Feb 10 07:46:42 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":1.085,"15min":1.115},"min":{"1min":0.374,"5min":0.32800000000000001,"15min":0.308},"max":{"1min":3.4009999999999998,"5min":6.7869999999999999,"15min":11.244999999999999},"last":1.4390000000000001},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.3580000000000001,"15min":1.3089999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.33300000000000002},"max":{"1min":3.9159999999999999,"5min":8.6020000000000003,"15min":9.0090000000000003},"last":1.3759999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:47:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.5109999999999999,"15min":1.3520000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":1.2689999999999999,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.109,"5min":1.2569999999999999,"15min":1.224},"min":{"1min":0.379,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.673,"5min":7.7089999999999996,"15min":11.212},"last":2.403}]}]},{"osd":0,"up_from":8,"seq":34359738592,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3526568,"kb_used_data":2976160,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":90218584,"statfs":{"total":95995035648,"available":92383830016,"internally_reserved":0,"allocated":3047587840,"data_stored":3047118115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:46:32 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":3.3300000000000001,"15min":2.1789999999999998},"min":{"1min":0.193,"5min":0.182,"15min":0.182},"max":{"1min":0.72699999999999998,"5min":156.16200000000001,"15min":156.16200000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":3.573,"15min":2.2040000000000002},"min":{"1min":0.25700000000000001,"5min":0.19,"15min":0.183},"max":{"1min":0.48799999999999999,"5min":156.941,"15min":156.941},"last":0.38200000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:46:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":1.101,"15min":1.444},"min":{"1min":0.21299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.42899999999999999,"5min":9.0899999999999999,"15min":28.163},"last":0.313},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.92900000000000005,"15min":1.25},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.0029999999999999,"5min":5.3010000000000002,"15min":24.059999999999999},"last":0.32900000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:44:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9640000000000004,"5min":3.6070000000000002,"15min":2.2069999999999999},"min":{"1min":0.25700000000000001,"5min":0.222,"15min":0.222},"max":{"1min":97.677000000000007,"5min":97.677000000000007,"15min":97.677000000000007},"last":2.8130000000000002},{"interface":"front","average":{"1min":5.7160000000000002,"5min":2.5670000000000002,"15min":1.784},"min":{"1min":0.32400000000000001,"5min":0.246,"15min":0.246},"max":{"1min":49.344000000000001,"5min":49.344000000000001,"15min":49.344000000000001},"last":6.6559999999999997}]},{"osd":4,"last update":"Sat Feb 10 07:46:53 2024","interfaces":[{"interface":"back","average":{"1min":3.1080000000000001,"5min":1.3700000000000001,"15min":1.1839999999999999},"min":{"1min":0.30299999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":37.286000000000001,"5min":37.286000000000001,"15min":37.286000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":3.0339999999999998,"5min":1.3149999999999999,"15min":1.3660000000000001},"min":{"1min":0.46400000000000002,"5min":0.29199999999999998,"15min":0.26900000000000002},"max":{"1min":34.683,"5min":34.683,"15min":34.683},"last":0.54700000000000004}]},{"osd":5,"last update":"Sat Feb 10 07:46:53 2024","interfaces":[{"interface":"back","average":{"1min":3.1389999999999998,"5min":1.982,"15min":2.5249999999999999},"min":{"1min":0.373,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":37.377000000000002,"5min":37.377000000000002,"15min":37.377000000000002},"last":41.429000000000002},{"interface":"front","average":{"1min":2.988,"5min":1.982,"15min":2.8359999999999999},"min":{"1min":0.41199999999999998,"5min":0.33500000000000002,"15min":0.308},"max":{"1min":34.743000000000002,"5min":35.771000000000001,"15min":35.771000000000001},"last":41.456000000000003}]},{"osd":6,"last update":"Sat Feb 10 07:46:32 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.92100000000000004,"15min":1.1719999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.29599999999999999},"max":{"1min":2.71,"5min":5.3310000000000004,"15min":19.16},"last":41.587000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.95999999999999996,"15min":1.1930000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.9099999999999999,"5min":5.1509999999999998,"15min":16.431000000000001},"last":41.256}]},{"osd":7,"last update":"Sat Feb 10 07:46:13 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.097,"15min":1.48},"min":{"1min":0.38100000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":3.2050000000000001,"5min":6.7430000000000003,"15min":19.178000000000001},"last":0.499},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.069,"15min":1.169},"min":{"1min":0.41599999999999998,"5min":0.38400000000000001,"15min":0.376},"max":{"1min":3.7829999999999999,"5min":5.3550000000000004,"15min":8.5700000000000003},"last":0.62}]}]},{"osd":2,"up_from":0,"seq":77309411541,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":26,"seq":111669149878,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2383868,"kb_used_data":1883316,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":91361284,"statfs":{"total":95995035648,"available":93553954816,"internally_reserved":0,"allocated":1928515584,"data_stored":1928193923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":512424391},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:44:31 2024","interfaces":[{"interface":"back","average":{"1min":2.056,"5min":2.0369999999999999,"15min":1.595},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.17499999999999999},"max":{"1min":7.181,"5min":15.446,"15min":25.277000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.903,"5min":1.671,"15min":1.4810000000000001},"min":{"1min":0.215,"5min":0.21299999999999999,"15min":0.18099999999999999},"max":{"1min":6.4000000000000004,"5min":9.0459999999999994,"15min":28.390000000000001},"last":0.55500000000000005}]},{"osd":1,"last update":"Sat Feb 10 07:44:31 2024","interfaces":[{"interface":"back","average":{"1min":2.056,"5min":2.0699999999999998,"15min":1.5129999999999999},"min":{"1min":0.36099999999999999,"5min":0.219,"15min":0.20999999999999999},"max":{"1min":6.5289999999999999,"5min":25.105,"15min":25.105},"last":0.75},{"interface":"front","average":{"1min":2.4470000000000001,"5min":2.6230000000000002,"15min":1.823},"min":{"1min":0.29599999999999999,"5min":0.251,"15min":0.23300000000000001},"max":{"1min":6.2489999999999997,"5min":28.555,"15min":28.555},"last":1.744}]},{"osd":2,"last update":"Sat Feb 10 07:44:31 2024","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.2069999999999999,"15min":2.0790000000000002},"min":{"1min":0.25900000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":5.9560000000000004,"5min":13.209,"15min":163.441},"last":1.853},{"interface":"front","average":{"1min":1.7589999999999999,"5min":1.865,"15min":2.044},"min":{"1min":0.42499999999999999,"5min":0.313,"15min":0.218},"max":{"1min":5.4630000000000001,"5min":9.3989999999999991,"15min":166.82400000000001},"last":0.93600000000000005}]},{"osd":4,"last update":"Sat Feb 10 07:44:51 2024","interfaces":[{"interface":"back","average":{"1min":2.0630000000000002,"5min":1.5980000000000001,"15min":1.2210000000000001},"min":{"1min":0.34300000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.2549999999999999,"5min":9.6609999999999996,"15min":9.6609999999999996},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.8340000000000001,"5min":2.0209999999999999,"15min":1.4319999999999999},"min":{"1min":0.29899999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.9180000000000001,"5min":13.276,"15min":13.276},"last":1.2470000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:44:51 2024","interfaces":[{"interface":"back","average":{"1min":2.1019999999999999,"5min":2.4060000000000001,"15min":1.536},"min":{"1min":0.308,"5min":0.308,"15min":0.30499999999999999},"max":{"1min":4.3079999999999998,"5min":62.817,"15min":62.817},"last":1.0269999999999999},{"interface":"front","average":{"1min":2.4750000000000001,"5min":3.0299999999999998,"15min":1.8360000000000001},"min":{"1min":0.33800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":8.0109999999999992,"5min":62.865000000000002,"15min":62.865000000000002},"last":1.1220000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:44:33 2024","interfaces":[{"interface":"back","average":{"1min":2.0409999999999999,"5min":2.1120000000000001,"15min":1.5860000000000001},"min":{"1min":0.33800000000000002,"5min":0.318,"15min":0.318},"max":{"1min":5.9219999999999997,"5min":13.307,"15min":32.048999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.665,"15min":1.3660000000000001},"min":{"1min":0.437,"5min":0.38300000000000001,"15min":0.33600000000000002},"max":{"1min":5.6340000000000003,"5min":10.191000000000001,"15min":25.937000000000001},"last":0.997}]},{"osd":7,"last update":"Sat Feb 10 07:44:03 2024","interfaces":[{"interface":"back","average":{"1min":3.8559999999999999,"5min":1.954,"15min":1.444},"min":{"1min":0.47599999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":31.257999999999999,"5min":31.257999999999999,"15min":31.257999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":5.609,"5min":2.5720000000000001,"15min":1.698},"min":{"1min":0.46999999999999997,"5min":0.39300000000000002,"15min":0.33600000000000002},"max":{"1min":30.5,"5min":30.5,"15min":30.5},"last":1.3200000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953675,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3651248,"kb_used_data":3173028,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":90093904,"statfs":{"total":95995035648,"available":92256157696,"internally_reserved":0,"allocated":3249180672,"data_stored":3248705914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:46:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.3129999999999999,"15min":2.0030000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.4020000000000001,"5min":45.610999999999997,"15min":74.533000000000001},"last":1.665},{"interface":"front","average":{"1min":0.88500000000000001,"5min":1.3149999999999999,"15min":1.911},"min":{"1min":0.34100000000000003,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.7839999999999998,"5min":45.524999999999999,"15min":74.340999999999994},"last":0.98199999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:46:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":2.2170000000000001,"15min":1.9730000000000001},"min":{"1min":0.27200000000000002,"5min":0.26600000000000001,"15min":0.23200000000000001},"max":{"1min":7.3440000000000003,"5min":53.472000000000001,"15min":69.852000000000004},"last":1.7310000000000001},{"interface":"front","average":{"1min":1.1359999999999999,"5min":2.532,"15min":2.0209999999999999},"min":{"1min":0.33900000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.3280000000000003,"5min":53.536000000000001,"15min":70.692999999999998},"last":3.3100000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:46:53 2024","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":0.80200000000000005,"15min":1.2709999999999999},"min":{"1min":0.38600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.3689999999999998,"5min":4.3689999999999998,"15min":50.162999999999997},"last":3.2690000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.80800000000000005,"15min":1.212},"min":{"1min":0.38100000000000001,"5min":0.30099999999999999,"15min":0.27700000000000002},"max":{"1min":2.8530000000000002,"5min":4.3200000000000003,"15min":50.453000000000003},"last":1.02}]},{"osd":3,"last update":"Sat Feb 10 07:44:52 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.5369999999999999,"15min":1.6519999999999999},"min":{"1min":0.311,"5min":0.30499999999999999,"15min":0.28499999999999998},"max":{"1min":3.714,"5min":40.710000000000001,"15min":69.813000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.6160000000000001,"15min":1.6930000000000001},"min":{"1min":0.33400000000000002,"5min":0.30099999999999999,"15min":0.27900000000000003},"max":{"1min":4.609,"5min":40.981000000000002,"15min":69.760999999999996},"last":0.82999999999999996}]},{"osd":5,"last update":"Sat Feb 10 07:46:53 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":0.94399999999999995,"15min":0.70999999999999996},"min":{"1min":0.32300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":5.5049999999999999,"5min":5.5049999999999999,"15min":7.4969999999999999},"last":1.077},{"interface":"front","average":{"1min":1.6439999999999999,"5min":0.92100000000000004,"15min":0.70699999999999996},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":7.298,"5min":7.298,"15min":7.298},"last":1.6240000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:46:35 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.72099999999999997,"15min":0.623},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.25900000000000001},"max":{"1min":2.702,"5min":2.8690000000000002,"15min":3.347},"last":1.05},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.76000000000000001,"15min":0.65500000000000003},"min":{"1min":0.29499999999999998,"5min":0.26600000000000001,"15min":0.24399999999999999},"max":{"1min":2.262,"5min":3.5310000000000001,"15min":3.7109999999999999},"last":3.3479999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:47:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.026,"15min":0.72299999999999998},"min":{"1min":0.34499999999999997,"5min":0.315,"15min":0.29099999999999998},"max":{"1min":6.5640000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.794,"5min":1.052,"15min":0.79000000000000004},"min":{"1min":0.35999999999999999,"5min":0.30199999999999999,"15min":0.29399999999999998},"max":{"1min":7.3719999999999999,"5min":7.3719999999999999,"15min":7.3719999999999999},"last":1.7030000000000001}]}]},{"osd":5,"up_from":156,"seq":670014898304,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2617420,"kb_used_data":2087428,"kb_used_omap":4,"kb_used_meta":529979,"kb_avail":91127732,"statfs":{"total":95995035648,"available":93314797568,"internally_reserved":0,"allocated":2137526272,"data_stored":2137076421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":542699170},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:46:47 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":1.4690000000000001,"15min":1.6679999999999999},"min":{"1min":0.24099999999999999,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":3.8719999999999999,"5min":52.938000000000002,"15min":52.938000000000002},"last":1.8340000000000001},{"interface":"front","average":{"1min":0.75700000000000001,"5min":2.1269999999999998,"15min":1.9610000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.1890000000000001,"5min":52.805,"15min":52.805},"last":5.2480000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:46:47 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.891,"15min":2.6560000000000001},"min":{"1min":0.28000000000000003,"5min":0.26500000000000001,"15min":0.245},"max":{"1min":2.2490000000000001,"5min":54.298000000000002,"15min":54.298000000000002},"last":5.2910000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":1.514,"15min":2.593},"min":{"1min":0.38800000000000001,"5min":0.246,"15min":0.246},"max":{"1min":1.0509999999999999,"5min":52.969999999999999,"15min":52.969999999999999},"last":3.5720000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:46:47 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.105,"15min":1.417},"min":{"1min":0.35499999999999998,"5min":0.29399999999999998,"15min":0.28999999999999998},"max":{"1min":2.218,"5min":5.9660000000000002,"15min":11.601000000000001},"last":5.3150000000000004},{"interface":"front","average":{"1min":1.444,"5min":1.0549999999999999,"15min":1.4710000000000001},"min":{"1min":0.40899999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":13.662000000000001,"5min":13.662000000000001,"15min":13.662000000000001},"last":3.6160000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:44:46 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":2.7999999999999998,"15min":1.9350000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":10,"5min":53.366999999999997,"15min":53.366999999999997},"last":5.6200000000000001},{"interface":"front","average":{"1min":1.669,"5min":2.9550000000000001,"15min":2.1880000000000002},"min":{"1min":0.34999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":6.4039999999999999,"5min":52.731000000000002,"15min":52.731000000000002},"last":0.629}]},{"osd":4,"last update":"Sat Feb 10 07:46:47 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.78800000000000003,"15min":0.67900000000000005},"min":{"1min":0.252,"5min":0.246,"15min":0.246},"max":{"1min":3.2450000000000001,"5min":4.1840000000000002,"15min":5.4050000000000002},"last":8.952},{"interface":"front","average":{"1min":1.0780000000000001,"5min":0.81499999999999995,"15min":0.66700000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.25900000000000001},"max":{"1min":3.9089999999999998,"5min":5.5670000000000002,"15min":5.5670000000000002},"last":10.180999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:46:47 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.81499999999999995,"15min":0.68899999999999995},"min":{"1min":0.28699999999999998,"5min":0.27900000000000003,"15min":0.23999999999999999},"max":{"1min":2.089,"5min":4.0759999999999996,"15min":4.3860000000000001},"last":9.0229999999999997},{"interface":"front","average":{"1min":1.04,"5min":0.78100000000000003,"15min":0.75700000000000001},"min":{"1min":0.30099999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.9340000000000002,"5min":3.9340000000000002,"15min":4.2839999999999998},"last":1.8660000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:46:47 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.2969999999999999,"15min":0.88200000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.4550000000000001,"5min":8.8059999999999992,"15min":8.8059999999999992},"last":5.3490000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.98099999999999998,"15min":0.78400000000000003},"min":{"1min":0.33500000000000002,"5min":0.33300000000000002,"15min":0.27800000000000002},"max":{"1min":1.9259999999999999,"5min":4.2160000000000002,"15min":4.3600000000000003},"last":4.21}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3046572032,"data_stored":3046572032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1040121856,"data_stored":1040121856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":514719744,"data_stored":514719744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3257794560,"data_stored":3257794560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2156658688,"data_stored":2156658688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1982201856,"data_stored":1982201856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1409810432,"data_stored":1409810432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:47:13.158 INFO:tasks.thrashosds.thrasher:Setting 3.18 to [6, 3, 0, 2, 7, 5] 2024-02-10T07:47:13.158 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.18', '6', '3', '0', '2', '7', '5'] 2024-02-10T07:47:13.158 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pg-upmap-items 3.18 6 3 0 2 7 5 2024-02-10T07:47:13.175 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:13.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:13.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:13.277 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:13.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:13.316 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:13.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:13.417 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:13.454 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:13.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:13.555 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:13.583 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:13.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:13.685 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:13.726 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:13.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:13.828 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:13.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:13 smithi008 ceph-mon[29816]: pgmap v692: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 43 MiB/s wr, 693 op/s; 6184/207309 objects degraded (2.983%); 4101/207309 objects misplaced (1.978%); 9.4 MiB/s, 150 objects/s recovering 2024-02-10T07:47:13.831 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:13 smithi008 ceph-mon[40743]: pgmap v692: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 43 MiB/s wr, 693 op/s; 6184/207309 objects degraded (2.983%); 4101/207309 objects misplaced (1.978%); 9.4 MiB/s, 150 objects/s recovering 2024-02-10T07:47:13.856 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:13.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:13.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:13 smithi170 ceph-mon[33394]: pgmap v692: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 43 MiB/s wr, 693 op/s; 6184/207309 objects degraded (2.983%); 4101/207309 objects misplaced (1.978%); 9.4 MiB/s, 150 objects/s recovering 2024-02-10T07:47:13.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:47:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:47:13.949 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 53 16 74339 74323 87.6365 61.375 0.0355694 0.0114059 2024-02-10T07:47:13.957 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:14.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:14.003 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:14.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:14.104 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:14.146 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:14.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:14.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:14.186 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:14.248 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:14.272 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:47:14.290 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:14.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:14.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:14.299 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:47:14.328 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:47:14.360 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:47:14.392 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:14.395 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:47:14.421 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:14.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:14.441 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:47:14.482 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:47:14.483 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:47:14.483 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:47:14.483 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:47:14.483 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-10T07:47:14.483 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:47:14.483 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:47:14.483 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:47:14.483 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:47:14.524 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:14.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:14 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1305934890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:47:14.524 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:14 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1305934890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:47:14.560 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:14.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:14.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:14.662 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:14.690 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:14.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:14.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:14.792 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:14.831 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:14.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:14.882 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:47:14.882 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:47:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:14 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1305934890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:47:14.933 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:14.949 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 54 16 75343 75327 87.1756 62.75 0.0231675 0.0114654 2024-02-10T07:47:14.973 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:14.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:14.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:15.074 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:15.104 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:15.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:15.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:15.206 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:15.237 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:15.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:15.338 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:15.379 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:15.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:15.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:15.480 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:15.518 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:15.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:15.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:15.620 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:15.643 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:15.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:15.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:15.698 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:15 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:15.698 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:15 smithi008 ceph-mon[29816]: pgmap v693: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 988 op/s; 6033/217830 objects degraded (2.770%); 3596/217830 objects misplaced (1.651%); 12 MiB/s, 196 objects/s recovering 2024-02-10T07:47:15.699 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:15 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:15.699 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:15 smithi008 ceph-mon[40743]: pgmap v693: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 988 op/s; 6033/217830 objects degraded (2.770%); 3596/217830 objects misplaced (1.651%); 12 MiB/s, 196 objects/s recovering 2024-02-10T07:47:15.736 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:47:15.737 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:47:15.745 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:15.772 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:15.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:15.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:15.874 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:15 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:15 smithi170 ceph-mon[33394]: pgmap v693: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 988 op/s; 6033/217830 objects degraded (2.770%); 3596/217830 objects misplaced (1.651%); 12 MiB/s, 196 objects/s recovering 2024-02-10T07:47:15.949 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 55 16 76290 76274 86.6666 59.1875 0.0214843 0.0115341 2024-02-10T07:47:16.066 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:16.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:16.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:16.167 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:16.196 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:16.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:16.298 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:16.335 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:16.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:16.437 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:16.463 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-10T07:47:16.481 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:16.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:16.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:16.582 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:16.615 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:16.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:16.716 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:16.751 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:16.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:16.853 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:16.890 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:16.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:16.949 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 56 16 77278 77262 86.2216 61.75 0.0168139 0.0115918 2024-02-10T07:47:16.992 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:17.030 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:17.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:17.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:17.132 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:17.174 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:17.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:17.276 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:17.314 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:17.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:17.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:17.416 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:17.443 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:17.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:17.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:17.546 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:17.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:17.576 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:17.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:17.579 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:17.645 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:17.678 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:17.706 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:17.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:17.720 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:17 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:47:17.720 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:17 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:47:17.720 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:17 smithi008 ceph-mon[29816]: pgmap v694: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 6335/221520 objects degraded (2.860%); 3220/221520 objects misplaced (1.454%); 14 MiB/s, 220 objects/s recovering 2024-02-10T07:47:17.721 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:17 smithi008 ceph-mon[29816]: osdmap e332: 8 total, 7 up, 7 in 2024-02-10T07:47:17.721 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:17 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 6335/221520 objects degraded (2.860%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:17.721 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:17 smithi008 ceph-mon[29816]: osdmap e333: 8 total, 7 up, 7 in 2024-02-10T07:47:17.721 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:17 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:47:17.721 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:17 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:47:17.721 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:17 smithi008 ceph-mon[40743]: pgmap v694: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 6335/221520 objects degraded (2.860%); 3220/221520 objects misplaced (1.454%); 14 MiB/s, 220 objects/s recovering 2024-02-10T07:47:17.721 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:17 smithi008 ceph-mon[40743]: osdmap e332: 8 total, 7 up, 7 in 2024-02-10T07:47:17.721 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:17 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 6335/221520 objects degraded (2.860%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:17.721 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:17 smithi008 ceph-mon[40743]: osdmap e333: 8 total, 7 up, 7 in 2024-02-10T07:47:17.738 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:47:17.808 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:17.849 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:17.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:17.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:17 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:47:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:17 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:47:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:17 smithi170 ceph-mon[33394]: pgmap v694: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 6335/221520 objects degraded (2.860%); 3220/221520 objects misplaced (1.454%); 14 MiB/s, 220 objects/s recovering 2024-02-10T07:47:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:17 smithi170 ceph-mon[33394]: osdmap e332: 8 total, 7 up, 7 in 2024-02-10T07:47:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:17 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 6335/221520 objects degraded (2.860%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:17 smithi170 ceph-mon[33394]: osdmap e333: 8 total, 7 up, 7 in 2024-02-10T07:47:17.949 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 57 16 78263 78247 85.7887 61.5625 0.00927132 0.0116511 2024-02-10T07:47:17.951 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:17.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:17.989 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:17.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:18.090 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:18.122 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:18.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:18.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:18.224 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:18.255 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:18.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:18.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:18.357 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:18.387 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:18.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:18.489 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:18.527 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:18.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:18.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:18.631 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:18.674 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:18.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:18.776 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:18.807 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:18.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:18.909 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:18.941 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:18.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:18.949 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 58 16 79284 79268 85.4097 63.8125 0.0253231 0.0117024 2024-02-10T07:47:19.043 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:19.080 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:19.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:19.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:47:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:47:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:47:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:18 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:47:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:18 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:47:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:18 smithi008 ceph-mon[29816]: osdmap e334: 8 total, 7 up, 7 in 2024-02-10T07:47:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:18 smithi008 ceph-mon[29816]: pgmap v698: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 59 MiB/s wr, 938 op/s; 6335/224190 objects degraded (2.826%); 2886/224190 objects misplaced (1.287%); 14 MiB/s, 227 objects/s recovering 2024-02-10T07:47:19.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:47:19.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:18 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:47:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:18 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:47:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:18 smithi008 ceph-mon[40743]: osdmap e334: 8 total, 7 up, 7 in 2024-02-10T07:47:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:18 smithi008 ceph-mon[40743]: pgmap v698: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 59 MiB/s wr, 938 op/s; 6335/224190 objects degraded (2.826%); 2886/224190 objects misplaced (1.287%); 14 MiB/s, 227 objects/s recovering 2024-02-10T07:47:19.083 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:47:19.182 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:19.221 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:19.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:19.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:19.322 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:19.365 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:19.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:19.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:18 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:47:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:18 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:47:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:18 smithi170 ceph-mon[33394]: osdmap e334: 8 total, 7 up, 7 in 2024-02-10T07:47:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:18 smithi170 ceph-mon[33394]: pgmap v698: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 59 MiB/s wr, 938 op/s; 6335/224190 objects degraded (2.826%); 2886/224190 objects misplaced (1.287%); 14 MiB/s, 227 objects/s recovering 2024-02-10T07:47:19.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:47:19.466 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:19.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:19.510 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:19.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:19.611 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:19.646 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:19.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:19.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:19.747 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:19.788 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:19.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:19.890 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:19.916 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:19.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:19.949 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 59 16 80287 80271 85.0245 62.6875 0.027125 0.0117563 2024-02-10T07:47:20.018 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:20.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:20.058 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:20.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:20.159 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:20.190 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:20.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:20.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:20.291 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:20.336 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:20.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:20.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:20.438 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:20.482 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:20.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:20.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:20.584 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:20.664 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:20.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:20.765 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:20.795 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:20.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:20.897 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:20.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:20.939 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:20.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:20.949 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:47:20.950682+0000 min lat: 0.00147754 max lat: 2.14964 avg lat: 0.0118105 2024-02-10T07:47:20.950 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:47:20.950 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 60 16 81275 81259 84.6365 61.75 0.0174225 0.0118105 2024-02-10T07:47:21.040 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:21.158 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:21.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:21.260 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:21.287 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:21.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:21.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:21 smithi170 ceph-mon[33394]: pgmap v699: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s; 6484/236319 objects degraded (2.744%); 2244/236319 objects misplaced (0.950%); 16 MiB/s, 252 objects/s recovering 2024-02-10T07:47:21.389 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:21.416 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:21.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:21.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:21.464 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:21 smithi008 ceph-mon[29816]: pgmap v699: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s; 6484/236319 objects degraded (2.744%); 2244/236319 objects misplaced (0.950%); 16 MiB/s, 252 objects/s recovering 2024-02-10T07:47:21.465 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:21 smithi008 ceph-mon[40743]: pgmap v699: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s; 6484/236319 objects degraded (2.744%); 2244/236319 objects misplaced (0.950%); 16 MiB/s, 252 objects/s recovering 2024-02-10T07:47:21.518 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:21.543 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:21.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:21.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:21.645 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:21.671 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:21.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:21.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:21.773 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:21.799 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:21.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:21.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:21.901 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:21.926 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:21.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:21.950 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 61 16 81662 81646 83.6455 24.1875 0.00438305 0.0118151 2024-02-10T07:47:22.028 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:22.055 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:22.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:22.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:22.121 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:47:22.157 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:22.200 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:22.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:22.301 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:22.342 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:22.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:22.382 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:22 smithi170 ceph-mon[33394]: osdmap e335: 8 total, 7 up, 7 in 2024-02-10T07:47:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:22 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3700412073' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:47:22.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:22 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 6484/236319 objects degraded (2.744%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:22.442 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:22.476 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:22.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:22.491 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:22 smithi008 ceph-mon[29816]: osdmap e335: 8 total, 7 up, 7 in 2024-02-10T07:47:22.491 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:22 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3700412073' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:47:22.491 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:22 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 6484/236319 objects degraded (2.744%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:22.491 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:22 smithi008 ceph-mon[40743]: osdmap e335: 8 total, 7 up, 7 in 2024-02-10T07:47:22.492 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:22 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3700412073' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:47:22.492 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:22 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 6484/236319 objects degraded (2.744%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:22.579 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:22.608 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:22.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:22.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:22.646 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:47:22.647 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:47:22.711 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:22.750 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:22.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:22.852 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:22.889 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:22.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:22.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:22.950 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 62 16 82508 82492 83.1492 52.875 0.00720479 0.0120206 2024-02-10T07:47:22.991 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:23.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:23.036 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:23.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:23.137 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:23.173 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:23.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:23.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:23.275 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:23.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:23.316 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:23.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:23.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:23 smithi008 ceph-mon[29816]: pgmap v701: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 915 B/s rd, 55 MiB/s wr, 882 op/s; 6484/236319 objects degraded (2.744%); 2244/236319 objects misplaced (0.950%); 13 MiB/s, 203 objects/s recovering 2024-02-10T07:47:23.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:23 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:47:23.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:23 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3700412073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:47:23.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:23 smithi008 ceph-mon[29816]: osdmap e336: 8 total, 7 up, 7 in 2024-02-10T07:47:23.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:23 smithi008 ceph-mon[40743]: pgmap v701: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 915 B/s rd, 55 MiB/s wr, 882 op/s; 6484/236319 objects degraded (2.744%); 2244/236319 objects misplaced (0.950%); 13 MiB/s, 203 objects/s recovering 2024-02-10T07:47:23.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:23 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:47:23.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:23 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3700412073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:47:23.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:23 smithi008 ceph-mon[40743]: osdmap e336: 8 total, 7 up, 7 in 2024-02-10T07:47:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:23 smithi170 ceph-mon[33394]: pgmap v701: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 915 B/s rd, 55 MiB/s wr, 882 op/s; 6484/236319 objects degraded (2.744%); 2244/236319 objects misplaced (0.950%); 13 MiB/s, 203 objects/s recovering 2024-02-10T07:47:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:23 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:47:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:23 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3700412073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:47:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:23 smithi170 ceph-mon[33394]: osdmap e336: 8 total, 7 up, 7 in 2024-02-10T07:47:23.417 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:23.429 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-10T07:47:23.446 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:23.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:23.549 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:23.594 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:23.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:23.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:23.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:23.740 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:23.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:23.842 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:23.876 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:23.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:23.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:47:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:47:23.950 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 63 16 83577 83561 82.8897 66.8125 0.00779466 0.0120589 2024-02-10T07:47:23.978 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:24.007 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:24.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:24.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:24.109 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:24.141 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:24.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:24.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:24.243 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:24.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:24.280 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:24.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:24.380 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:24.411 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:24.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:24.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:24.513 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:24.537 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:24.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:24.639 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:24.640 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:24.667 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:24.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:24.769 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:24.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:24.801 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:24.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:24.902 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:24.926 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:24.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:24.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:24.950 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 64 16 83944 83928 81.953 22.9375 0.00382183 0.0120647 2024-02-10T07:47:25.029 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:25.115 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:25.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:25.147 INFO:teuthology.orchestra.run.smithi008.stderr:set 3.18 pg_upmap_items mapping to [6->3,0->2,7->5] 2024-02-10T07:47:25.217 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:25.279 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:25.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:25.381 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:25 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:25 smithi170 ceph-mon[33394]: pgmap v703: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 6182/245850 objects degraded (2.515%); 1249/245850 objects misplaced (0.508%); 22 MiB/s, 344 objects/s recovering 2024-02-10T07:47:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:25 smithi170 ceph-mon[33394]: 10.c scrub starts 2024-02-10T07:47:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:25 smithi170 ceph-mon[33394]: osdmap e337: 8 total, 7 up, 7 in 2024-02-10T07:47:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:25 smithi170 ceph-mon[33394]: 3.b scrub starts 2024-02-10T07:47:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:25 smithi170 ceph-mon[33394]: 3.b scrub ok 2024-02-10T07:47:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:25 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1586792549' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [6, 3, 0, 2, 7, 5]}]: dispatch 2024-02-10T07:47:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:25 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [6, 3, 0, 2, 7, 5]}]: dispatch 2024-02-10T07:47:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:25 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:25 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:25.414 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:25.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:25.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:25.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:25.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[40743]: pgmap v703: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 6182/245850 objects degraded (2.515%); 1249/245850 objects misplaced (0.508%); 22 MiB/s, 344 objects/s recovering 2024-02-10T07:47:25.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[40743]: 10.c scrub starts 2024-02-10T07:47:25.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[40743]: osdmap e337: 8 total, 7 up, 7 in 2024-02-10T07:47:25.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[40743]: 3.b scrub starts 2024-02-10T07:47:25.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[40743]: 3.b scrub ok 2024-02-10T07:47:25.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1586792549' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [6, 3, 0, 2, 7, 5]}]: dispatch 2024-02-10T07:47:25.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [6, 3, 0, 2, 7, 5]}]: dispatch 2024-02-10T07:47:25.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:25.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:25.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:25.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[29816]: pgmap v703: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 3 active+recovering+undersized+remapped, 4 active+undersized, 134 active+clean; 5.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s; 6182/245850 objects degraded (2.515%); 1249/245850 objects misplaced (0.508%); 22 MiB/s, 344 objects/s recovering 2024-02-10T07:47:25.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[29816]: 10.c scrub starts 2024-02-10T07:47:25.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[29816]: osdmap e337: 8 total, 7 up, 7 in 2024-02-10T07:47:25.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[29816]: 3.b scrub starts 2024-02-10T07:47:25.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[29816]: 3.b scrub ok 2024-02-10T07:47:25.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1586792549' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [6, 3, 0, 2, 7, 5]}]: dispatch 2024-02-10T07:47:25.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [6, 3, 0, 2, 7, 5]}]: dispatch 2024-02-10T07:47:25.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:25.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:25 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:25.516 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:25.543 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:25.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:25.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:25.645 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:25.674 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:25.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:25.776 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:25.805 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:25.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:25.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:25.907 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:25.935 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:25.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:25.950 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 65 16 84807 84791 81.522 53.9375 0.021171 0.0122606 2024-02-10T07:47:26.036 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:26.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:26.079 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:26.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:26.180 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:26.222 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:26.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:26.323 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:26.363 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:26.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:26.464 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:26.498 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:26.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:26.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[29816]: 10.c scrub ok 2024-02-10T07:47:26.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [6, 3, 0, 2, 7, 5]}]': finished 2024-02-10T07:47:26.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[29816]: osdmap e338: 8 total, 7 up, 7 in 2024-02-10T07:47:26.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:47:26.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:47:26.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:47:26.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:47:26.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:26.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[40743]: 10.c scrub ok 2024-02-10T07:47:26.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [6, 3, 0, 2, 7, 5]}]': finished 2024-02-10T07:47:26.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[40743]: osdmap e338: 8 total, 7 up, 7 in 2024-02-10T07:47:26.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:47:26.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:47:26.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:47:26.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:47:26.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:26 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:26.599 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:26 smithi170 ceph-mon[33394]: 10.c scrub ok 2024-02-10T07:47:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:26 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [6, 3, 0, 2, 7, 5]}]': finished 2024-02-10T07:47:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:26 smithi170 ceph-mon[33394]: osdmap e338: 8 total, 7 up, 7 in 2024-02-10T07:47:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:26 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:47:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:26 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:47:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:26 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:47:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:26 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:47:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:26 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:26.634 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:26.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:26.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:26.648 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:26.736 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:26.772 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:26.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:26.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:26.874 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:26.901 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:26.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:26.950 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 66 16 85814 85798 81.2403 62.9375 0.00456279 0.0123041 2024-02-10T07:47:27.003 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:27.042 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:27.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:27.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:27.144 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:27.171 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:27.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:27.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:27.207 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:47:27.273 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:27.310 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:27.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:27.405 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:27 smithi008 ceph-mon[29816]: pgmap v706: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 2 active+recovering+undersized+remapped, 4 active+undersized, 135 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 54 MiB/s wr, 866 op/s; 6460/249162 objects degraded (2.593%); 1064/249162 objects misplaced (0.427%); 19 MiB/s, 299 objects/s recovering 2024-02-10T07:47:27.405 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:27 smithi008 ceph-mon[29816]: osdmap e339: 8 total, 7 up, 7 in 2024-02-10T07:47:27.405 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:27 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 6460/249162 objects degraded (2.593%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:27.406 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:27 smithi008 ceph-mon[40743]: pgmap v706: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 2 active+recovering+undersized+remapped, 4 active+undersized, 135 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 54 MiB/s wr, 866 op/s; 6460/249162 objects degraded (2.593%); 1064/249162 objects misplaced (0.427%); 19 MiB/s, 299 objects/s recovering 2024-02-10T07:47:27.406 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:27 smithi008 ceph-mon[40743]: osdmap e339: 8 total, 7 up, 7 in 2024-02-10T07:47:27.406 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:27 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 6460/249162 objects degraded (2.593%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:27.412 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:27.446 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:27.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:27.547 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:27.575 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:27.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:27.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:27.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:27 smithi170 ceph-mon[33394]: pgmap v706: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 2 active+recovering+undersized+remapped, 4 active+undersized, 135 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 54 MiB/s wr, 866 op/s; 6460/249162 objects degraded (2.593%); 1064/249162 objects misplaced (0.427%); 19 MiB/s, 299 objects/s recovering 2024-02-10T07:47:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:27 smithi170 ceph-mon[33394]: osdmap e339: 8 total, 7 up, 7 in 2024-02-10T07:47:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:27 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 6460/249162 objects degraded (2.593%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:27.677 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:27.704 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:27.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:27.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:27.806 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:27.845 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:27.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:27.947 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:27.950 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 67 16 86896 86880 81.0369 67.625 0.0115768 0.0123326 2024-02-10T07:47:27.970 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:27.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:27.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:28.072 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:28.100 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:28.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:28.202 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:28.228 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:28.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:28.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:28.330 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:28.358 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:28.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:28.460 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:28.499 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:28.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:28.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:28.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:28 smithi008 ceph-mon[29816]: 4.11 scrub starts 2024-02-10T07:47:28.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:28 smithi008 ceph-mon[29816]: 4.11 scrub ok 2024-02-10T07:47:28.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:28 smithi008 ceph-mon[40743]: 4.11 scrub starts 2024-02-10T07:47:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:28 smithi008 ceph-mon[40743]: 4.11 scrub ok 2024-02-10T07:47:28.601 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:28.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:28 smithi170 ceph-mon[33394]: 4.11 scrub starts 2024-02-10T07:47:28.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:28 smithi170 ceph-mon[33394]: 4.11 scrub ok 2024-02-10T07:47:28.643 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:28.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:28.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:28.745 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:28.746 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:28.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:28.787 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:28.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:28.888 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:28.894 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:47:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:47:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:47:28.922 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:28.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:28.950 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 68 16 87870 87854 80.7404 60.875 0.0107581 0.0123802 2024-02-10T07:47:29.025 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:29.062 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:29.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:29.064 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:47:29.096 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:47:29.123 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:47:29.154 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:47:29.163 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:29.193 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:47:29.205 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:29.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:29.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:29.223 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:47:29.252 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:47:29.252 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:47:29.252 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:47:29.253 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:47:29.253 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-10T07:47:29.253 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:47:29.253 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:47:29.253 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:47:29.253 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:47:29.306 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:29.345 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:29.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:29.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:29.434 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:29 smithi008 ceph-mon[29816]: pgmap v708: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+undersized+remapped, 4 active+undersized, 136 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 862 B/s rd, 54 MiB/s wr, 860 op/s; 6460/251625 objects degraded (2.567%); 926/251625 objects misplaced (0.368%); 17 MiB/s, 273 objects/s recovering 2024-02-10T07:47:29.434 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:29 smithi008 ceph-mon[29816]: 6.1b scrub starts 2024-02-10T07:47:29.434 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:29 smithi008 ceph-mon[29816]: 6.1b scrub ok 2024-02-10T07:47:29.434 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:29 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1686885821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:47:29.435 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:29 smithi008 ceph-mon[40743]: pgmap v708: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+undersized+remapped, 4 active+undersized, 136 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 862 B/s rd, 54 MiB/s wr, 860 op/s; 6460/251625 objects degraded (2.567%); 926/251625 objects misplaced (0.368%); 17 MiB/s, 273 objects/s recovering 2024-02-10T07:47:29.435 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:29 smithi008 ceph-mon[40743]: 6.1b scrub starts 2024-02-10T07:47:29.435 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:29 smithi008 ceph-mon[40743]: 6.1b scrub ok 2024-02-10T07:47:29.435 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:29 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1686885821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:47:29.447 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:29.479 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:29.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:29.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:29.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:29 smithi170 ceph-mon[33394]: pgmap v708: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+undersized+remapped, 4 active+undersized, 136 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 862 B/s rd, 54 MiB/s wr, 860 op/s; 6460/251625 objects degraded (2.567%); 926/251625 objects misplaced (0.368%); 17 MiB/s, 273 objects/s recovering 2024-02-10T07:47:29.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:29 smithi170 ceph-mon[33394]: 6.1b scrub starts 2024-02-10T07:47:29.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:29 smithi170 ceph-mon[33394]: 6.1b scrub ok 2024-02-10T07:47:29.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:29 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1686885821' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:47:29.581 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:29.623 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:29.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:29.724 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:29.763 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:29.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:29.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:29.865 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:29.892 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:29.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:29.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:29.950 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 69 16 88971 88955 80.5674 68.8125 0.0116897 0.0124073 2024-02-10T07:47:29.994 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:30.034 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:30.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:30.136 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:30.177 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:30.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:30.278 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:30.306 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:30.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:30.408 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:30.448 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:30.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:30.550 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:30.590 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:30.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:30.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:30.692 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:30.719 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:30.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:30.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:30.820 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:30.858 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:30.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:30.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:30.904 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [3] live_osds: [0, 1, 2, 4, 6, 7, 5] 2024-02-10T07:47:30.904 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:47:30.904 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-02-10T07:47:30.905 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2024-02-10T07:47:30.905 DEBUG:teuthology.orchestra.run.smithi008:osd.3> sudo journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.3.service 2024-02-10T07:47:30.907 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl start ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.3 2024-02-10T07:47:30.950 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 70 16 89991 89975 80.3271 63.75 0.00906893 0.0124436 2024-02-10T07:47:30.959 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:30.989 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:30.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:31.091 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:31.131 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:31.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:31.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:31.233 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:31.278 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:31.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:31.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:31.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:31 smithi008 ceph-mon[29816]: pgmap v709: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+undersized+remapped, 4 active+undersized, 136 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 904 op/s; 6367/262113 objects degraded (2.429%); 391/262113 objects misplaced (0.149%); 13 MiB/s, 202 objects/s recovering 2024-02-10T07:47:31.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:31 smithi008 ceph-mon[40743]: pgmap v709: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+undersized+remapped, 4 active+undersized, 136 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 904 op/s; 6367/262113 objects degraded (2.429%); 391/262113 objects misplaced (0.149%); 13 MiB/s, 202 objects/s recovering 2024-02-10T07:47:31.380 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:31.422 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:31.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:31.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:31.523 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:31.565 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:31.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:31.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:31 smithi170 ceph-mon[33394]: pgmap v709: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+undersized+remapped, 4 active+undersized, 136 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 904 op/s; 6367/262113 objects degraded (2.429%); 391/262113 objects misplaced (0.149%); 13 MiB/s, 202 objects/s recovering 2024-02-10T07:47:31.666 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:31.695 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:31.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:31.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:31.797 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:31.824 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:31.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:31.926 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:31.950 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 71 16 91017 91001 80.0988 64.125 0.0285012 0.0124787 2024-02-10T07:47:31.955 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:31.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:32.057 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:32.094 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:32.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:32.196 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:32.225 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:32.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:32.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:32.326 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:32.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:32 smithi008 ceph-mon[29816]: 6.11 scrub starts 2024-02-10T07:47:32.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:32 smithi008 ceph-mon[29816]: 6.11 scrub ok 2024-02-10T07:47:32.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:32 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 6367/262113 objects degraded (2.429%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:32 smithi008 ceph-mon[40743]: 6.11 scrub starts 2024-02-10T07:47:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:32 smithi008 ceph-mon[40743]: 6.11 scrub ok 2024-02-10T07:47:32.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:32 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 6367/262113 objects degraded (2.429%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:32.368 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:32.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:32.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:32.470 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:32.517 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:32.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:32.619 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:32.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:32 smithi170 ceph-mon[33394]: 6.11 scrub starts 2024-02-10T07:47:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:32 smithi170 ceph-mon[33394]: 6.11 scrub ok 2024-02-10T07:47:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:32 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 6367/262113 objects degraded (2.429%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:32.659 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:32.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:32.761 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:32.789 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:32.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:32.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:32.891 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:32.921 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:32.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:32.951 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 72 16 92030 92014 79.8656 63.3125 0.0166001 0.0125154 2024-02-10T07:47:33.023 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:33.060 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:33.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:33.162 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:33.189 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:33.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:33.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:33.291 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:33.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:33.330 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:33.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:33.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:33 smithi008 ceph-mon[29816]: pgmap v710: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+undersized+remapped, 4 active+undersized, 136 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 646 B/s rd, 41 MiB/s wr, 653 op/s; 6367/262113 objects degraded (2.429%); 391/262113 objects misplaced (0.149%); 9.6 MiB/s, 153 objects/s recovering 2024-02-10T07:47:33.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:33 smithi008 ceph-mon[29816]: 6.13 scrub starts 2024-02-10T07:47:33.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:33 smithi008 ceph-mon[29816]: 4.0 scrub starts 2024-02-10T07:47:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:33 smithi008 ceph-mon[29816]: 4.0 scrub ok 2024-02-10T07:47:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:33 smithi008 ceph-mon[29816]: 6.13 scrub ok 2024-02-10T07:47:33.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:47:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:33 smithi008 ceph-mon[40743]: pgmap v710: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+undersized+remapped, 4 active+undersized, 136 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 646 B/s rd, 41 MiB/s wr, 653 op/s; 6367/262113 objects degraded (2.429%); 391/262113 objects misplaced (0.149%); 9.6 MiB/s, 153 objects/s recovering 2024-02-10T07:47:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:33 smithi008 ceph-mon[40743]: 6.13 scrub starts 2024-02-10T07:47:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:33 smithi008 ceph-mon[40743]: 4.0 scrub starts 2024-02-10T07:47:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:33 smithi008 ceph-mon[40743]: 4.0 scrub ok 2024-02-10T07:47:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:33 smithi008 ceph-mon[40743]: 6.13 scrub ok 2024-02-10T07:47:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:47:33.431 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:33.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:33.469 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:33.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:33.569 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:33.596 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:33.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:33.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:33.609 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:33 smithi170 ceph-mon[33394]: pgmap v710: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 1 active+recovering+undersized+remapped, 4 active+undersized, 136 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 646 B/s rd, 41 MiB/s wr, 653 op/s; 6367/262113 objects degraded (2.429%); 391/262113 objects misplaced (0.149%); 9.6 MiB/s, 153 objects/s recovering 2024-02-10T07:47:33.609 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:33 smithi170 ceph-mon[33394]: 6.13 scrub starts 2024-02-10T07:47:33.609 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:33 smithi170 ceph-mon[33394]: 4.0 scrub starts 2024-02-10T07:47:33.609 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:33 smithi170 ceph-mon[33394]: 4.0 scrub ok 2024-02-10T07:47:33.609 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:33 smithi170 ceph-mon[33394]: 6.13 scrub ok 2024-02-10T07:47:33.609 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:47:33.699 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:33.724 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:33.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:33.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:33.729 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:47:33.729 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:47:33.825 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:33.849 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:33.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:33.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:47:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:47:33.951 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:33.951 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 73 16 92561 92545 79.2261 33.1875 0.00361557 0.0125243 2024-02-10T07:47:33.974 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:33.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:34.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:34.101 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:34.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:34.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:34.203 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:34.233 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:34.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:34.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:34.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:34 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:47:34.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:34 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:47:34.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:34 smithi008 ceph-mon[29816]: 4.17 scrub starts 2024-02-10T07:47:34.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:34 smithi008 ceph-mon[29816]: 4.17 scrub ok 2024-02-10T07:47:34.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:34 smithi008 ceph-mon[29816]: osdmap e340: 8 total, 7 up, 7 in 2024-02-10T07:47:34.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:34 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:47:34.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:34 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:47:34.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:34 smithi008 ceph-mon[40743]: 4.17 scrub starts 2024-02-10T07:47:34.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:34 smithi008 ceph-mon[40743]: 4.17 scrub ok 2024-02-10T07:47:34.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:34 smithi008 ceph-mon[40743]: osdmap e340: 8 total, 7 up, 7 in 2024-02-10T07:47:34.335 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:34.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:34.363 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:34.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:34 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:47:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:34 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:47:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:34 smithi170 ceph-mon[33394]: 4.17 scrub starts 2024-02-10T07:47:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:34 smithi170 ceph-mon[33394]: 4.17 scrub ok 2024-02-10T07:47:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:34 smithi170 ceph-mon[33394]: osdmap e340: 8 total, 7 up, 7 in 2024-02-10T07:47:34.465 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:34.500 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:34.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:34.604 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:34.635 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:34.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:34.736 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:34.763 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:34.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:34.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:34.865 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:34.901 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:34.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:34.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:34.951 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 74 16 93437 93421 78.8953 54.75 0.0059936 0.0126706 2024-02-10T07:47:34.995 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:34 smithi008 podman[168847]: 2024-02-10 07:47:34.506150223 +0000 UTC m=+0.450774558 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:47:34.995 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:34 smithi008 podman[168847]: 2024-02-10 07:47:34.662503099 +0000 UTC m=+0.607127424 container create 7c37657227e5956ea59658c8b492a579dd428e17a2874223149f37a522324943 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , RELEASE=main-290212a) 2024-02-10T07:47:35.003 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:35.033 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:35.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:35.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:35.134 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:35.167 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:35.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:35.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:35.266 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:35.266 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-mon[29816]: pgmap v712: 148 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 136 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 6111/273822 objects degraded (2.232%); 13 MiB/s, 209 objects/s recovering 2024-02-10T07:47:35.266 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-mon[29816]: 6.1c scrub starts 2024-02-10T07:47:35.266 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-mon[29816]: 6.1c scrub ok 2024-02-10T07:47:35.266 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-mon[29816]: osdmap e341: 8 total, 7 up, 7 in 2024-02-10T07:47:35.266 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:34 smithi008 podman[168847]: 2024-02-10 07:47:34.995543019 +0000 UTC m=+0.940167361 container init 7c37657227e5956ea59658c8b492a579dd428e17a2874223149f37a522324943 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-10T07:47:35.267 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 podman[168847]: 2024-02-10 07:47:35.000424395 +0000 UTC m=+0.945048735 container start 7c37657227e5956ea59658c8b492a579dd428e17a2874223149f37a522324943 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-10T07:47:35.267 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 podman[168847]: 2024-02-10 07:47:35.107685834 +0000 UTC m=+1.052310166 container attach 7c37657227e5956ea59658c8b492a579dd428e17a2874223149f37a522324943 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux ) 2024-02-10T07:47:35.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:35.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-mon[40743]: pgmap v712: 148 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 136 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 6111/273822 objects degraded (2.232%); 13 MiB/s, 209 objects/s recovering 2024-02-10T07:47:35.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-mon[40743]: 6.1c scrub starts 2024-02-10T07:47:35.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-mon[40743]: 6.1c scrub ok 2024-02-10T07:47:35.267 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-mon[40743]: osdmap e341: 8 total, 7 up, 7 in 2024-02-10T07:47:35.268 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:35.307 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:35.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:35.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:35.409 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:35.447 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:35.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:35.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:35.549 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:35.589 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:35.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:35.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:35.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:35 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:35.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:35 smithi170 ceph-mon[33394]: pgmap v712: 148 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 136 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 6111/273822 objects degraded (2.232%); 13 MiB/s, 209 objects/s recovering 2024-02-10T07:47:35.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:35 smithi170 ceph-mon[33394]: 6.1c scrub starts 2024-02-10T07:47:35.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:35 smithi170 ceph-mon[33394]: 6.1c scrub ok 2024-02-10T07:47:35.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:35 smithi170 ceph-mon[33394]: osdmap e341: 8 total, 7 up, 7 in 2024-02-10T07:47:35.691 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:35.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:35.729 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:35.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:35.730 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:47:35.831 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:35.866 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:35.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:35.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:35.924 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-activate[168946]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T07:47:35.924 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-activate[168946]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T07:47:35.924 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 bash[168847]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T07:47:35.924 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 bash[168847]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-10T07:47:35.924 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 bash[168847]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T07:47:35.924 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 bash[168847]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T07:47:35.924 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 bash[168847]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-10T07:47:35.924 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 bash[168847]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T07:47:35.925 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 bash[168847]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-10T07:47:35.925 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-activate[168946]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-10T07:47:35.925 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-activate[168946]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-10T07:47:35.925 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-activate[168946]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-10T07:47:35.925 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-activate[168946]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-10T07:47:35.925 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:35 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-activate[168946]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-10T07:47:35.953 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 75 15 94614 94599 78.8231 73.625 0.0153636 0.0126819 2024-02-10T07:47:35.968 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:36.006 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:36.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:36.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:36.108 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:36.143 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:36.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:36.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:36.245 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:36.262 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:47:36.262 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:47:36.262 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-02-10T07:47:36.263 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-02-10T07:47:36.263 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:47:36.285 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:36.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:36.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:36.387 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:36.415 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:36.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:36.497 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:36 smithi008 ceph-mon[29816]: 4.10 scrub starts 2024-02-10T07:47:36.497 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:36 smithi008 ceph-mon[29816]: 4.10 scrub ok 2024-02-10T07:47:36.498 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:36 smithi008 podman[168847]: 2024-02-10 07:47:36.211682386 +0000 UTC m=+2.156306708 container died 7c37657227e5956ea59658c8b492a579dd428e17a2874223149f37a522324943 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-activate, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-10T07:47:36.498 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:36 smithi008 ceph-mon[40743]: 4.10 scrub starts 2024-02-10T07:47:36.498 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:36 smithi008 ceph-mon[40743]: 4.10 scrub ok 2024-02-10T07:47:36.517 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:36.566 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:36.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:36.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:36.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:36 smithi170 ceph-mon[33394]: 4.10 scrub starts 2024-02-10T07:47:36.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:36 smithi170 ceph-mon[33394]: 4.10 scrub ok 2024-02-10T07:47:36.668 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:36.698 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:36.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:36.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:36.799 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:36.827 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:36.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:36.929 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:36.953 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 76 16 95785 95769 78.7481 73.125 0.0165947 0.0126933 2024-02-10T07:47:36.976 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:36.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:37.078 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:37.102 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:36 smithi008 podman[169152]: 2024-02-10 07:47:36.8389083 +0000 UTC m=+1.148975201 container remove 7c37657227e5956ea59658c8b492a579dd428e17a2874223149f37a522324943 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-activate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:47:37.114 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:37.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:37.216 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:37.254 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:37.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:37.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:37.356 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:37.390 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:37.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:37.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:37.492 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:37.532 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:37.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:37.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:37 smithi008 ceph-mon[29816]: pgmap v714: 148 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 136 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 6380/277137 objects degraded (2.302%); 12 MiB/s, 192 objects/s recovering 2024-02-10T07:47:37.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:37 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 6380/277137 objects degraded (2.302%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:37.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:37 smithi008 ceph-mon[40743]: pgmap v714: 148 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 136 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 6380/277137 objects degraded (2.302%); 12 MiB/s, 192 objects/s recovering 2024-02-10T07:47:37.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:37 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 6380/277137 objects degraded (2.302%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:37 smithi170 ceph-mon[33394]: pgmap v714: 148 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 136 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 6380/277137 objects degraded (2.302%); 12 MiB/s, 192 objects/s recovering 2024-02-10T07:47:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:37 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 6380/277137 objects degraded (2.302%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:37.634 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:37.665 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:37.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:37.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:37.767 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:37.796 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-10T07:47:37.799 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:37.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:37.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:37.902 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:37.941 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:37.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:37.953 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 77 16 96935 96919 78.6587 71.875 0.00920439 0.012708 2024-02-10T07:47:38.042 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:38.082 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:38.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:38.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:38.152 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:38.176 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:38.184 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:38.218 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:38.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:38.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:38.319 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:38.361 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:38.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:38.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:38.462 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:38.502 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:38.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:38.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:38.604 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:38.633 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:38.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:38.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:38.735 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:38.781 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:38.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:38.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:38.883 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:38.913 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:38.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:38.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:38.953 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 78 15 98091 98076 78.5773 72.3125 0.0140149 0.0127218 2024-02-10T07:47:39.016 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:39.041 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:47:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:47:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:47:39.045 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:39.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:39.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:39.147 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:39.175 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:39.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:39.229 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:47:39.277 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:39.305 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:39.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:39.407 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:39.437 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:39.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:39.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:39.528 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:39 smithi008 ceph-mon[40743]: pgmap v715: 148 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 136 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 46 MiB/s wr, 735 op/s; 6380/279759 objects degraded (2.281%); 5.1 MiB/s, 80 objects/s recovering 2024-02-10T07:47:39.528 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:39 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/910209381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:39.528 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:39 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:47:39.528 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:39 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:47:39.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:39 smithi008 ceph-mon[29816]: pgmap v715: 148 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 136 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 46 MiB/s wr, 735 op/s; 6380/279759 objects degraded (2.281%); 5.1 MiB/s, 80 objects/s recovering 2024-02-10T07:47:39.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:39 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/910209381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:39.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:39 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:47:39.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:39 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:47:39.539 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:39.569 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:39.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:39 smithi170 ceph-mon[33394]: pgmap v715: 148 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 136 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 46 MiB/s wr, 735 op/s; 6380/279759 objects degraded (2.281%); 5.1 MiB/s, 80 objects/s recovering 2024-02-10T07:47:39.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:39 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/910209381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:39.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:39 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:47:39.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:39 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:47:39.670 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:39.698 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:39.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:39.801 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:39.832 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:39.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:39.933 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:39.953 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 79 16 99279 99263 78.5217 74.1875 0.0174943 0.0127302 2024-02-10T07:47:39.974 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:39.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:39.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:40.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:40.108 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:39 smithi008 podman[169494]: 2024-02-10 07:47:39.762897477 +0000 UTC m=+1.596636044 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:47:40.112 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:40.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:40.214 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:40.250 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:40.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:40.352 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:40.365 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:40 smithi008 ceph-mon[29816]: 3.d scrub starts 2024-02-10T07:47:40.365 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:40 smithi008 ceph-mon[29816]: 3.d scrub ok 2024-02-10T07:47:40.365 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:40 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:47:40.365 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:40 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/910209381' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:47:40.365 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:40 smithi008 ceph-mon[29816]: osdmap e342: 8 total, 7 up, 7 in 2024-02-10T07:47:40.365 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:40 smithi008 podman[169494]: 2024-02-10 07:47:40.216592873 +0000 UTC m=+2.050331519 container create e65d1d5399fbdba4f0c3e3332fe06d71c789e0a1011249073785e3bf4da79cbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-02-10T07:47:40.366 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:40 smithi008 ceph-mon[40743]: 3.d scrub starts 2024-02-10T07:47:40.366 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:40 smithi008 ceph-mon[40743]: 3.d scrub ok 2024-02-10T07:47:40.366 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:40 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:47:40.366 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:40 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/910209381' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:47:40.366 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:40 smithi008 ceph-mon[40743]: osdmap e342: 8 total, 7 up, 7 in 2024-02-10T07:47:40.377 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:40.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:40.478 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:40.502 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:40.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:40.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:40.603 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:40.627 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:40.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:40.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:40 smithi170 ceph-mon[33394]: 3.d scrub starts 2024-02-10T07:47:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:40 smithi170 ceph-mon[33394]: 3.d scrub ok 2024-02-10T07:47:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:40 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:47:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:40 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/910209381' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:47:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:40 smithi170 ceph-mon[33394]: osdmap e342: 8 total, 7 up, 7 in 2024-02-10T07:47:40.729 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:40.752 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:40.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:40.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:40.856 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:40.880 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:40.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:40.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:40.953 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:47:40.954191+0000 min lat: 0.00147754 max lat: 2.14964 avg lat: 0.0127289 2024-02-10T07:47:40.953 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:47:40.953 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 80 16 99795 99779 77.9432 32.25 0.00260254 0.0127289 2024-02-10T07:47:40.982 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:41.006 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:41.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:41.103 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-10T07:47:41.108 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:41.128 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:40 smithi008 podman[169494]: 2024-02-10 07:47:40.998589511 +0000 UTC m=+2.832328071 container init e65d1d5399fbdba4f0c3e3332fe06d71c789e0a1011249073785e3bf4da79cbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:47:41.128 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:41 smithi008 podman[169494]: 2024-02-10 07:47:41.001758683 +0000 UTC m=+2.835497243 container start e65d1d5399fbdba4f0c3e3332fe06d71c789e0a1011249073785e3bf4da79cbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:47:41.128 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:41 smithi008 ceph-osd[169788]: -- 172.21.15.8:0/2557403140 <== mon.2 v2:172.21.15.8:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556dd3bd31e0 con 0x556dd3ba4c00 2024-02-10T07:47:41.129 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:41 smithi008 bash[169494]: e65d1d5399fbdba4f0c3e3332fe06d71c789e0a1011249073785e3bf4da79cbf 2024-02-10T07:47:41.129 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:41 smithi008 systemd[1]: Started Ceph osd.3 for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:47:41.152 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:41.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:41.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:41.253 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:41.282 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:41.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:41.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:41.384 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:41.425 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:41.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:41.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:41.527 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:41.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:41 smithi008 ceph-mon[29816]: 4.15 scrub starts 2024-02-10T07:47:41.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:41 smithi008 ceph-mon[29816]: 4.15 scrub ok 2024-02-10T07:47:41.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:41 smithi008 ceph-mon[29816]: pgmap v717: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 137 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 743 B/s rd, 74 MiB/s wr, 1.19k op/s; 6252/292578 objects degraded (2.137%); 11 MiB/s, 170 objects/s recovering 2024-02-10T07:47:41.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:41 smithi008 ceph-mon[29816]: 5.13 scrub starts 2024-02-10T07:47:41.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:41 smithi008 ceph-mon[29816]: 5.13 scrub ok 2024-02-10T07:47:41.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:41 smithi008 ceph-mon[29816]: osdmap e343: 8 total, 7 up, 7 in 2024-02-10T07:47:41.528 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:41 smithi008 ceph-mon[40743]: 4.15 scrub starts 2024-02-10T07:47:41.528 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:41 smithi008 ceph-mon[40743]: 4.15 scrub ok 2024-02-10T07:47:41.529 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:41 smithi008 ceph-mon[40743]: pgmap v717: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 137 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 743 B/s rd, 74 MiB/s wr, 1.19k op/s; 6252/292578 objects degraded (2.137%); 11 MiB/s, 170 objects/s recovering 2024-02-10T07:47:41.529 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:41 smithi008 ceph-mon[40743]: 5.13 scrub starts 2024-02-10T07:47:41.529 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:41 smithi008 ceph-mon[40743]: 5.13 scrub ok 2024-02-10T07:47:41.529 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:41 smithi008 ceph-mon[40743]: osdmap e343: 8 total, 7 up, 7 in 2024-02-10T07:47:41.567 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:41.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:41.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:41.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:41 smithi170 ceph-mon[33394]: 4.15 scrub starts 2024-02-10T07:47:41.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:41 smithi170 ceph-mon[33394]: 4.15 scrub ok 2024-02-10T07:47:41.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:41 smithi170 ceph-mon[33394]: pgmap v717: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 137 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 743 B/s rd, 74 MiB/s wr, 1.19k op/s; 6252/292578 objects degraded (2.137%); 11 MiB/s, 170 objects/s recovering 2024-02-10T07:47:41.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:41 smithi170 ceph-mon[33394]: 5.13 scrub starts 2024-02-10T07:47:41.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:41 smithi170 ceph-mon[33394]: 5.13 scrub ok 2024-02-10T07:47:41.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:41 smithi170 ceph-mon[33394]: osdmap e343: 8 total, 7 up, 7 in 2024-02-10T07:47:41.668 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:41.705 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:41.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:41.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:41.779 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:41 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3[169715]: 2024-02-10T07:47:41.540+0000 7fb4a67277c0 -1 Falling back to public interface 2024-02-10T07:47:41.807 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:41.845 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:41.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:41.946 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:41.954 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 81 16 100597 100581 77.5993 50.125 0.0140175 0.012882 2024-02-10T07:47:41.979 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:41.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:41.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:42.081 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:42.125 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:42.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:42.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:42.227 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:42.263 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:42.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:42.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:42.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[29816]: 3.c scrub starts 2024-02-10T07:47:42.316 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[29816]: 3.c scrub ok 2024-02-10T07:47:42.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[29816]: 3.1 deep-scrub starts 2024-02-10T07:47:42.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[29816]: 3.1 deep-scrub ok 2024-02-10T07:47:42.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[29816]: 6.c scrub starts 2024-02-10T07:47:42.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[29816]: 6.c scrub ok 2024-02-10T07:47:42.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[29816]: osdmap e344: 8 total, 7 up, 7 in 2024-02-10T07:47:42.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[29816]: 6.6 scrub starts 2024-02-10T07:47:42.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[29816]: 6.6 scrub ok 2024-02-10T07:47:42.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 6252/292578 objects degraded (2.137%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:42.317 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[29816]: osdmap e345: 8 total, 7 up, 7 in 2024-02-10T07:47:42.318 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:47:42.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[40743]: 3.c scrub starts 2024-02-10T07:47:42.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[40743]: 3.c scrub ok 2024-02-10T07:47:42.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[40743]: 3.1 deep-scrub starts 2024-02-10T07:47:42.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[40743]: 3.1 deep-scrub ok 2024-02-10T07:47:42.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[40743]: 6.c scrub starts 2024-02-10T07:47:42.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[40743]: 6.c scrub ok 2024-02-10T07:47:42.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[40743]: osdmap e344: 8 total, 7 up, 7 in 2024-02-10T07:47:42.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[40743]: 6.6 scrub starts 2024-02-10T07:47:42.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[40743]: 6.6 scrub ok 2024-02-10T07:47:42.319 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 6252/292578 objects degraded (2.137%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:42.319 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:42 smithi008 ceph-mon[40743]: osdmap e345: 8 total, 7 up, 7 in 2024-02-10T07:47:42.345 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:47:42.365 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:42.375 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:47:42.395 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:42.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:42.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:42.408 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:47:42.439 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:47:42.472 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:47:42.499 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:42.503 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:47:42.503 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:47:42.503 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:47:42.503 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:47:42.503 INFO:teuthology.orchestra.run.smithi008.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-02-10T07:47:42.503 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:47:42.503 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:47:42.503 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:47:42.503 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:47:42.537 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:42.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:42 smithi170 ceph-mon[33394]: 3.c scrub starts 2024-02-10T07:47:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:42 smithi170 ceph-mon[33394]: 3.c scrub ok 2024-02-10T07:47:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:42 smithi170 ceph-mon[33394]: 3.1 deep-scrub starts 2024-02-10T07:47:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:42 smithi170 ceph-mon[33394]: 3.1 deep-scrub ok 2024-02-10T07:47:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:42 smithi170 ceph-mon[33394]: 6.c scrub starts 2024-02-10T07:47:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:42 smithi170 ceph-mon[33394]: 6.c scrub ok 2024-02-10T07:47:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:42 smithi170 ceph-mon[33394]: osdmap e344: 8 total, 7 up, 7 in 2024-02-10T07:47:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:42 smithi170 ceph-mon[33394]: 6.6 scrub starts 2024-02-10T07:47:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:42 smithi170 ceph-mon[33394]: 6.6 scrub ok 2024-02-10T07:47:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:42 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 6252/292578 objects degraded (2.137%), 7 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:42.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:42 smithi170 ceph-mon[33394]: osdmap e345: 8 total, 7 up, 7 in 2024-02-10T07:47:42.639 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:42.669 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:42.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:42.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:42.770 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:42.807 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:42.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:42.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:42.909 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:42.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:42.933 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:42.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:42.954 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 82 16 101652 101636 77.457 65.9375 0.00324826 0.0128963 2024-02-10T07:47:43.033 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:43.073 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:43.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:43.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:43.175 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:43.218 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:43.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:43.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:43.319 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:43.348 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:43.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:43.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:43.449 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:43.486 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:43.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:43.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:43.494 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[29816]: 6.19 scrub starts 2024-02-10T07:47:43.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[29816]: 6.19 scrub ok 2024-02-10T07:47:43.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[29816]: pgmap v721: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 137 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 6252/292578 objects degraded (2.137%); 8.2 MiB/s, 130 objects/s recovering 2024-02-10T07:47:43.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[29816]: 2.0 scrub starts 2024-02-10T07:47:43.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[29816]: 2.0 scrub ok 2024-02-10T07:47:43.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[29816]: 4.16 scrub starts 2024-02-10T07:47:43.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[29816]: 4.16 scrub ok 2024-02-10T07:47:43.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[29816]: 6.1f scrub starts 2024-02-10T07:47:43.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4053024329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:47:43.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[29816]: 4.1d scrub starts 2024-02-10T07:47:43.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[29816]: 4.1d scrub ok 2024-02-10T07:47:43.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[29816]: 5.1 scrub starts 2024-02-10T07:47:43.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[29816]: 5.1 scrub ok 2024-02-10T07:47:43.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[40743]: 6.19 scrub starts 2024-02-10T07:47:43.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[40743]: 6.19 scrub ok 2024-02-10T07:47:43.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[40743]: pgmap v721: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 137 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 6252/292578 objects degraded (2.137%); 8.2 MiB/s, 130 objects/s recovering 2024-02-10T07:47:43.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[40743]: 2.0 scrub starts 2024-02-10T07:47:43.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[40743]: 2.0 scrub ok 2024-02-10T07:47:43.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[40743]: 4.16 scrub starts 2024-02-10T07:47:43.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[40743]: 4.16 scrub ok 2024-02-10T07:47:43.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[40743]: 6.1f scrub starts 2024-02-10T07:47:43.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4053024329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:47:43.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[40743]: 4.1d scrub starts 2024-02-10T07:47:43.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[40743]: 4.1d scrub ok 2024-02-10T07:47:43.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[40743]: 5.1 scrub starts 2024-02-10T07:47:43.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:43 smithi008 ceph-mon[40743]: 5.1 scrub ok 2024-02-10T07:47:43.571 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:47:43.588 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:43.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:43 smithi170 ceph-mon[33394]: 6.19 scrub starts 2024-02-10T07:47:43.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:43 smithi170 ceph-mon[33394]: 6.19 scrub ok 2024-02-10T07:47:43.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:43 smithi170 ceph-mon[33394]: pgmap v721: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+undersized+degraded, 4 active+undersized, 137 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 6252/292578 objects degraded (2.137%); 8.2 MiB/s, 130 objects/s recovering 2024-02-10T07:47:43.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:43 smithi170 ceph-mon[33394]: 2.0 scrub starts 2024-02-10T07:47:43.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:43 smithi170 ceph-mon[33394]: 2.0 scrub ok 2024-02-10T07:47:43.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:43 smithi170 ceph-mon[33394]: 4.16 scrub starts 2024-02-10T07:47:43.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:43 smithi170 ceph-mon[33394]: 4.16 scrub ok 2024-02-10T07:47:43.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:43 smithi170 ceph-mon[33394]: 6.1f scrub starts 2024-02-10T07:47:43.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:43 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4053024329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:47:43.622 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:43 smithi170 ceph-mon[33394]: 4.1d scrub starts 2024-02-10T07:47:43.623 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:43 smithi170 ceph-mon[33394]: 4.1d scrub ok 2024-02-10T07:47:43.623 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:43 smithi170 ceph-mon[33394]: 5.1 scrub starts 2024-02-10T07:47:43.623 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:43 smithi170 ceph-mon[33394]: 5.1 scrub ok 2024-02-10T07:47:43.626 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:43.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:43.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:43.727 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:43.758 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:43.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:43.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:43.859 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:43.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:47:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:47:43.895 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:43.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:43.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:43.954 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 83 16 102815 102799 77.3994 72.6875 0.0127263 0.0129157 2024-02-10T07:47:43.996 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:44.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:44.035 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:44.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:44.136 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:44.165 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:44.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:44.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:44.266 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:44.275 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-10T07:47:44.275 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-10T07:47:44.300 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:44.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:44.402 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:44.440 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:44.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:44.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:44.542 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:44.563 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:44 smithi008 ceph-mon[29816]: 6.1f scrub ok 2024-02-10T07:47:44.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:44 smithi008 ceph-mon[29816]: 6.0 scrub starts 2024-02-10T07:47:44.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:44 smithi008 ceph-mon[29816]: 6.0 scrub ok 2024-02-10T07:47:44.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:44 smithi008 ceph-mon[29816]: 6.16 scrub starts 2024-02-10T07:47:44.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:44 smithi008 ceph-mon[29816]: 6.16 scrub ok 2024-02-10T07:47:44.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:44 smithi008 ceph-mon[29816]: 4.13 scrub starts 2024-02-10T07:47:44.564 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:44 smithi008 ceph-mon[29816]: 4.13 scrub ok 2024-02-10T07:47:44.564 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:44 smithi008 ceph-mon[40743]: 6.1f scrub ok 2024-02-10T07:47:44.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:44 smithi008 ceph-mon[40743]: 6.0 scrub starts 2024-02-10T07:47:44.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:44 smithi008 ceph-mon[40743]: 6.0 scrub ok 2024-02-10T07:47:44.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:44 smithi008 ceph-mon[40743]: 6.16 scrub starts 2024-02-10T07:47:44.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:44 smithi008 ceph-mon[40743]: 6.16 scrub ok 2024-02-10T07:47:44.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:44 smithi008 ceph-mon[40743]: 4.13 scrub starts 2024-02-10T07:47:44.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:44 smithi008 ceph-mon[40743]: 4.13 scrub ok 2024-02-10T07:47:44.565 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:47:44.565 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:47:44.578 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:44.582 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:44.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:44.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:44.584 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:44 smithi170 ceph-mon[33394]: 6.1f scrub ok 2024-02-10T07:47:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:44 smithi170 ceph-mon[33394]: 6.0 scrub starts 2024-02-10T07:47:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:44 smithi170 ceph-mon[33394]: 6.0 scrub ok 2024-02-10T07:47:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:44 smithi170 ceph-mon[33394]: 6.16 scrub starts 2024-02-10T07:47:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:44 smithi170 ceph-mon[33394]: 6.16 scrub ok 2024-02-10T07:47:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:44 smithi170 ceph-mon[33394]: 4.13 scrub starts 2024-02-10T07:47:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:44 smithi170 ceph-mon[33394]: 4.13 scrub ok 2024-02-10T07:47:44.684 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:44.718 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:44.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:44.820 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:44.859 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:44.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:44.954 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 84 16 103992 103976 77.3536 73.5625 0.0110182 0.012923 2024-02-10T07:47:44.961 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:44.998 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:44.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:45.100 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:45.132 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:45.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:45.234 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:45.269 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:45.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:45.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:45.371 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:45.384 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-10T07:47:45.400 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:45.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:45.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:45.502 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:45.529 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:45.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:45.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:45.581 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:45 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3[169715]: 2024-02-10T07:47:45.327+0000 7fb4a67277c0 -1 osd.3 295 log_to_monitors true 2024-02-10T07:47:45.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:45 smithi008 ceph-mon[40743]: 5.16 scrub starts 2024-02-10T07:47:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:45 smithi008 ceph-mon[40743]: 5.16 scrub ok 2024-02-10T07:47:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:45 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:45 smithi008 ceph-mon[40743]: pgmap v722: 148 pgs: 6 active+undersized+degraded, 4 active+undersized, 138 active+clean; 6.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 46 MiB/s wr, 733 op/s; 6153/303189 objects degraded (2.029%); 1.3 MiB/s, 20 objects/s recovering 2024-02-10T07:47:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:45 smithi008 ceph-mon[40743]: 5.1f scrub starts 2024-02-10T07:47:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:45 smithi008 ceph-mon[40743]: 5.1f scrub ok 2024-02-10T07:47:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:45 smithi008 ceph-mon[29816]: 5.16 scrub starts 2024-02-10T07:47:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:45 smithi008 ceph-mon[29816]: 5.16 scrub ok 2024-02-10T07:47:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:45 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:45 smithi008 ceph-mon[29816]: pgmap v722: 148 pgs: 6 active+undersized+degraded, 4 active+undersized, 138 active+clean; 6.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 46 MiB/s wr, 733 op/s; 6153/303189 objects degraded (2.029%); 1.3 MiB/s, 20 objects/s recovering 2024-02-10T07:47:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:45 smithi008 ceph-mon[29816]: 5.1f scrub starts 2024-02-10T07:47:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:45 smithi008 ceph-mon[29816]: 5.1f scrub ok 2024-02-10T07:47:45.631 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:45.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:45 smithi170 ceph-mon[33394]: 5.16 scrub starts 2024-02-10T07:47:45.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:45 smithi170 ceph-mon[33394]: 5.16 scrub ok 2024-02-10T07:47:45.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:45 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:45.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:45 smithi170 ceph-mon[33394]: pgmap v722: 148 pgs: 6 active+undersized+degraded, 4 active+undersized, 138 active+clean; 6.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 46 MiB/s wr, 733 op/s; 6153/303189 objects degraded (2.029%); 1.3 MiB/s, 20 objects/s recovering 2024-02-10T07:47:45.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:45 smithi170 ceph-mon[33394]: 5.1f scrub starts 2024-02-10T07:47:45.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:45 smithi170 ceph-mon[33394]: 5.1f scrub ok 2024-02-10T07:47:45.678 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:45.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:45.780 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:45.808 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:45.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:45.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:45.910 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:45.952 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:45.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:45.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:45.954 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 85 16 105133 105117 77.2825 71.3125 0.0167621 0.0129348 2024-02-10T07:47:46.053 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:46.084 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:46.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:46.185 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:46.213 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:46.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:46.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:46.276 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:47:46.315 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:46.360 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:46.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:46.462 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:46.503 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:46.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:46.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:46.536 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:46 smithi008 ceph-mon[29816]: 3.18 scrub starts 2024-02-10T07:47:46.536 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:46 smithi008 ceph-mon[29816]: 3.18 scrub ok 2024-02-10T07:47:46.536 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:46 smithi008 ceph-mon[29816]: from='osd.3 [v2:172.21.15.8:6826/4266121075,v1:172.21.15.8:6827/4266121075]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:47:46.536 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:46 smithi008 ceph-mon[29816]: 6.f scrub starts 2024-02-10T07:47:46.536 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:47:46 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3[169715]: 2024-02-10T07:47:46.393+0000 7fb493b77700 -1 osd.3 346 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T07:47:46.537 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:46 smithi008 ceph-mon[40743]: 3.18 scrub starts 2024-02-10T07:47:46.537 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:46 smithi008 ceph-mon[40743]: 3.18 scrub ok 2024-02-10T07:47:46.537 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:46 smithi008 ceph-mon[40743]: from='osd.3 [v2:172.21.15.8:6826/4266121075,v1:172.21.15.8:6827/4266121075]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:47:46.537 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:46 smithi008 ceph-mon[40743]: 6.f scrub starts 2024-02-10T07:47:46.605 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:46.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:46 smithi170 ceph-mon[33394]: 3.18 scrub starts 2024-02-10T07:47:46.721 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:46 smithi170 ceph-mon[33394]: 3.18 scrub ok 2024-02-10T07:47:46.721 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:46 smithi170 ceph-mon[33394]: from='osd.3 [v2:172.21.15.8:6826/4266121075,v1:172.21.15.8:6827/4266121075]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T07:47:46.721 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:46 smithi170 ceph-mon[33394]: 6.f scrub starts 2024-02-10T07:47:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:46.723 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:46.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:46.824 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:46.833 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:47:46.884 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:46.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:46.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:46.954 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 86 16 106257 106241 77.2007 70.25 0.0127035 0.0129479 2024-02-10T07:47:46.986 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:47.028 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:47.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:47.130 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:47.167 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:47.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:47.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:47.269 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:47.293 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:47.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:47.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:47.395 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:47.418 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:47.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:47.515 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[40743]: 6.f scrub ok 2024-02-10T07:47:47.515 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[40743]: pgmap v723: 148 pgs: 6 active+undersized+degraded, 4 active+undersized, 138 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 49 MiB/s wr, 779 op/s; 6458/306597 objects degraded (2.106%); 1.0 MiB/s, 16 objects/s recovering 2024-02-10T07:47:47.515 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[40743]: from='osd.3 [v2:172.21.15.8:6826/4266121075,v1:172.21.15.8:6827/4266121075]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T07:47:47.515 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[40743]: osdmap e346: 8 total, 7 up, 7 in 2024-02-10T07:47:47.516 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[40743]: from='osd.3 [v2:172.21.15.8:6826/4266121075,v1:172.21.15.8:6827/4266121075]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:47:47.516 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 6458/306597 objects degraded (2.106%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:47.516 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[40743]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:47:47.516 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[40743]: osd.3 [v2:172.21.15.8:6826/4266121075,v1:172.21.15.8:6827/4266121075] boot 2024-02-10T07:47:47.516 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[40743]: osdmap e347: 8 total, 8 up, 7 in 2024-02-10T07:47:47.516 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:47:47.516 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[29816]: 6.f scrub ok 2024-02-10T07:47:47.516 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[29816]: pgmap v723: 148 pgs: 6 active+undersized+degraded, 4 active+undersized, 138 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 49 MiB/s wr, 779 op/s; 6458/306597 objects degraded (2.106%); 1.0 MiB/s, 16 objects/s recovering 2024-02-10T07:47:47.516 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[29816]: from='osd.3 [v2:172.21.15.8:6826/4266121075,v1:172.21.15.8:6827/4266121075]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T07:47:47.516 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[29816]: osdmap e346: 8 total, 7 up, 7 in 2024-02-10T07:47:47.516 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[29816]: from='osd.3 [v2:172.21.15.8:6826/4266121075,v1:172.21.15.8:6827/4266121075]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:47:47.517 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 6458/306597 objects degraded (2.106%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:47.517 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[29816]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:47:47.517 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[29816]: osd.3 [v2:172.21.15.8:6826/4266121075,v1:172.21.15.8:6827/4266121075] boot 2024-02-10T07:47:47.517 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[29816]: osdmap e347: 8 total, 8 up, 7 in 2024-02-10T07:47:47.517 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:47 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:47:47.520 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:47.545 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:47.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:47.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:47 smithi170 ceph-mon[33394]: 6.f scrub ok 2024-02-10T07:47:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:47 smithi170 ceph-mon[33394]: pgmap v723: 148 pgs: 6 active+undersized+degraded, 4 active+undersized, 138 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 49 MiB/s wr, 779 op/s; 6458/306597 objects degraded (2.106%); 1.0 MiB/s, 16 objects/s recovering 2024-02-10T07:47:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:47 smithi170 ceph-mon[33394]: from='osd.3 [v2:172.21.15.8:6826/4266121075,v1:172.21.15.8:6827/4266121075]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T07:47:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:47 smithi170 ceph-mon[33394]: osdmap e346: 8 total, 7 up, 7 in 2024-02-10T07:47:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:47 smithi170 ceph-mon[33394]: from='osd.3 [v2:172.21.15.8:6826/4266121075,v1:172.21.15.8:6827/4266121075]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi008", "root=default"]}]: dispatch 2024-02-10T07:47:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:47 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 6458/306597 objects degraded (2.106%), 6 pgs degraded, 10 pgs undersized (PG_DEGRADED) 2024-02-10T07:47:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:47 smithi170 ceph-mon[33394]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-10T07:47:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:47 smithi170 ceph-mon[33394]: osd.3 [v2:172.21.15.8:6826/4266121075,v1:172.21.15.8:6827/4266121075] boot 2024-02-10T07:47:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:47 smithi170 ceph-mon[33394]: osdmap e347: 8 total, 8 up, 7 in 2024-02-10T07:47:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:47 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T07:47:47.647 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:47.670 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:47.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:47.772 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:47.796 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:47.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:47.898 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:47.922 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:47.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:47.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:47.954 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 87 16 106556 106540 76.5281 18.6875 0.00276863 0.0129383 2024-02-10T07:47:48.024 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:48.047 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:48.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:48.149 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:48.173 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:48.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:48.274 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:48.298 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:48.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:48 smithi170 ceph-mon[33394]: 3.11 scrub starts 2024-02-10T07:47:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:48 smithi170 ceph-mon[33394]: 3.11 scrub ok 2024-02-10T07:47:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:48 smithi170 ceph-mon[33394]: 4.5 scrub starts 2024-02-10T07:47:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:48 smithi170 ceph-mon[33394]: 4.5 scrub ok 2024-02-10T07:47:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:48 smithi170 ceph-mon[33394]: 3.8 scrub starts 2024-02-10T07:47:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:48 smithi170 ceph-mon[33394]: 3.8 scrub ok 2024-02-10T07:47:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:48 smithi170 ceph-mon[33394]: osdmap e348: 8 total, 8 up, 7 in 2024-02-10T07:47:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:48 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:48.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:48 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:47:48.400 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:48.424 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:48.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:48.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:48.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[29816]: 3.11 scrub starts 2024-02-10T07:47:48.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[29816]: 3.11 scrub ok 2024-02-10T07:47:48.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[29816]: 4.5 scrub starts 2024-02-10T07:47:48.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[29816]: 4.5 scrub ok 2024-02-10T07:47:48.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[29816]: 3.8 scrub starts 2024-02-10T07:47:48.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[29816]: 3.8 scrub ok 2024-02-10T07:47:48.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[29816]: osdmap e348: 8 total, 8 up, 7 in 2024-02-10T07:47:48.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:48.523 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:47:48.524 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[40743]: 3.11 scrub starts 2024-02-10T07:47:48.524 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[40743]: 3.11 scrub ok 2024-02-10T07:47:48.524 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[40743]: 4.5 scrub starts 2024-02-10T07:47:48.524 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[40743]: 4.5 scrub ok 2024-02-10T07:47:48.524 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[40743]: 3.8 scrub starts 2024-02-10T07:47:48.524 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[40743]: 3.8 scrub ok 2024-02-10T07:47:48.524 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[40743]: osdmap e348: 8 total, 8 up, 7 in 2024-02-10T07:47:48.524 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:47:48.524 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:47:48.526 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:48.550 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:48.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:48.552 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:48.577 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:47:48.577 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:47:48.577 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-02-10T07:47:48.577 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:47:48.651 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:48.676 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:48.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:48.777 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:48.800 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:48.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:48.902 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:48.926 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:48.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:48.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:48.954 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 88 16 106556 106540 75.6584 0 - 0.0129383 2024-02-10T07:47:49.028 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:49.052 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:49.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:49.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:49.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:47:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:47:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:47:49.140 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:47:49.140 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:47:49.154 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:49.186 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:49.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:49.288 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:49.329 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:49.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:49 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:47:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:49 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:47:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:49 smithi170 ceph-mon[33394]: pgmap v727: 148 pgs: 1 peering, 5 active+undersized+degraded, 4 active+undersized, 138 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 60 MiB/s wr, 956 op/s; 6457/309780 objects degraded (2.084%); 1.0 MiB/s, 16 objects/s recovering 2024-02-10T07:47:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:49 smithi170 ceph-mon[33394]: osdmap e349: 8 total, 8 up, 7 in 2024-02-10T07:47:49.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:49 smithi170 ceph-mon[33394]: 10.4 starting backfill to osd.3 from (0'0,0'0] MAX to 346'6665 2024-02-10T07:47:49.431 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:49.464 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:49.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:49.565 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:49.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:49 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:47:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:49 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:47:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:49 smithi008 ceph-mon[29816]: pgmap v727: 148 pgs: 1 peering, 5 active+undersized+degraded, 4 active+undersized, 138 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 60 MiB/s wr, 956 op/s; 6457/309780 objects degraded (2.084%); 1.0 MiB/s, 16 objects/s recovering 2024-02-10T07:47:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:49 smithi008 ceph-mon[29816]: osdmap e349: 8 total, 8 up, 7 in 2024-02-10T07:47:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:49 smithi008 ceph-mon[29816]: 10.4 starting backfill to osd.3 from (0'0,0'0] MAX to 346'6665 2024-02-10T07:47:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:49 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:47:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:49 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:47:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:49 smithi008 ceph-mon[40743]: pgmap v727: 148 pgs: 1 peering, 5 active+undersized+degraded, 4 active+undersized, 138 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 60 MiB/s wr, 956 op/s; 6457/309780 objects degraded (2.084%); 1.0 MiB/s, 16 objects/s recovering 2024-02-10T07:47:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:49 smithi008 ceph-mon[40743]: osdmap e349: 8 total, 8 up, 7 in 2024-02-10T07:47:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:49 smithi008 ceph-mon[40743]: 10.4 starting backfill to osd.3 from (0'0,0'0] MAX to 346'6665 2024-02-10T07:47:49.600 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:49.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:49.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:49.703 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:49.742 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:49.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:49.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:49.843 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:49.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:49.885 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:49.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:49.954 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 89 16 107536 107520 75.4965 30.625 0.0159719 0.013241 2024-02-10T07:47:49.962 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-10T07:47:49.987 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:50.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:50.059 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:50.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:50.161 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:50.194 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:50.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:50.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:50.296 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:50.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:50.355 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:50.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:50.455 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:47:50.456 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:50.458 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:50 smithi008 ceph-mon[29816]: 3.15 scrub starts 2024-02-10T07:47:50.458 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:50 smithi008 ceph-mon[29816]: 3.15 scrub ok 2024-02-10T07:47:50.458 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:50 smithi008 ceph-mon[29816]: 5.11 scrub starts 2024-02-10T07:47:50.458 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:50 smithi008 ceph-mon[29816]: 5.11 scrub ok 2024-02-10T07:47:50.458 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:50 smithi008 ceph-mon[29816]: 4.1a scrub starts 2024-02-10T07:47:50.458 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:50 smithi008 ceph-mon[29816]: 4.1a scrub ok 2024-02-10T07:47:50.458 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:50 smithi008 ceph-mon[40743]: 3.15 scrub starts 2024-02-10T07:47:50.458 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:50 smithi008 ceph-mon[40743]: 3.15 scrub ok 2024-02-10T07:47:50.459 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:50 smithi008 ceph-mon[40743]: 5.11 scrub starts 2024-02-10T07:47:50.459 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:50 smithi008 ceph-mon[40743]: 5.11 scrub ok 2024-02-10T07:47:50.459 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:50 smithi008 ceph-mon[40743]: 4.1a scrub starts 2024-02-10T07:47:50.459 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:50 smithi008 ceph-mon[40743]: 4.1a scrub ok 2024-02-10T07:47:50.502 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:50.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:50.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:50.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:50 smithi170 ceph-mon[33394]: 3.15 scrub starts 2024-02-10T07:47:50.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:50 smithi170 ceph-mon[33394]: 3.15 scrub ok 2024-02-10T07:47:50.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:50 smithi170 ceph-mon[33394]: 5.11 scrub starts 2024-02-10T07:47:50.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:50 smithi170 ceph-mon[33394]: 5.11 scrub ok 2024-02-10T07:47:50.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:50 smithi170 ceph-mon[33394]: 4.1a scrub starts 2024-02-10T07:47:50.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:50 smithi170 ceph-mon[33394]: 4.1a scrub ok 2024-02-10T07:47:50.604 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:50.703 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:50.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:50.804 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:50.846 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:50.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:50.948 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:50.955 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Total time run: 90.004 2024-02-10T07:47:50.955 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Total writes made: 108458 2024-02-10T07:47:50.955 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Write size: 65536 2024-02-10T07:47:50.955 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Object size: 65536 2024-02-10T07:47:50.955 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Bandwidth (MB/sec): 75.3147 2024-02-10T07:47:50.955 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev Bandwidth: 41.2931 2024-02-10T07:47:50.955 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max bandwidth (MB/sec): 216.688 2024-02-10T07:47:50.955 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min bandwidth (MB/sec): 0 2024-02-10T07:47:50.955 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Average IOPS: 1205 2024-02-10T07:47:50.955 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev IOPS: 660.69 2024-02-10T07:47:50.955 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max IOPS: 3467 2024-02-10T07:47:50.955 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min IOPS: 0 2024-02-10T07:47:50.956 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Average Latency(s): 0.0132739 2024-02-10T07:47:50.956 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev Latency(s): 0.0417633 2024-02-10T07:47:50.956 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max latency(s): 2.14964 2024-02-10T07:47:50.956 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min latency(s): 0.00147754 2024-02-10T07:47:50.958 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Cleaning up (deleting benchmark objects) 2024-02-10T07:47:50.979 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:50.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:51.081 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:51.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:51.122 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:51.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:51.223 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:51.253 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:51.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:51.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:51.288 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:47:51.355 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:51.387 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:51.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:51.489 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:51.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:51.527 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:51.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:51 smithi008 ceph-mon[29816]: 6.7 scrub starts 2024-02-10T07:47:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:51 smithi008 ceph-mon[29816]: 6.7 scrub ok 2024-02-10T07:47:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:51 smithi008 ceph-mon[29816]: 6.5 scrub starts 2024-02-10T07:47:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:51 smithi008 ceph-mon[29816]: pgmap v729: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating, 1 peering, 1 active+undersized, 144 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 6650/320340 objects degraded (2.076%); 608 KiB/s, 9 objects/s recovering 2024-02-10T07:47:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:51 smithi008 ceph-mon[29816]: 6.5 scrub ok 2024-02-10T07:47:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:51 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1302690946' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:47:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:51 smithi008 ceph-mon[40743]: 6.7 scrub starts 2024-02-10T07:47:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:51 smithi008 ceph-mon[40743]: 6.7 scrub ok 2024-02-10T07:47:51.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:51 smithi008 ceph-mon[40743]: 6.5 scrub starts 2024-02-10T07:47:51.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:51 smithi008 ceph-mon[40743]: pgmap v729: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating, 1 peering, 1 active+undersized, 144 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 6650/320340 objects degraded (2.076%); 608 KiB/s, 9 objects/s recovering 2024-02-10T07:47:51.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:51 smithi008 ceph-mon[40743]: 6.5 scrub ok 2024-02-10T07:47:51.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:51 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1302690946' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:47:51.620 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:51 smithi170 ceph-mon[33394]: 6.7 scrub starts 2024-02-10T07:47:51.620 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:51 smithi170 ceph-mon[33394]: 6.7 scrub ok 2024-02-10T07:47:51.620 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:51 smithi170 ceph-mon[33394]: 6.5 scrub starts 2024-02-10T07:47:51.620 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:51 smithi170 ceph-mon[33394]: pgmap v729: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating, 1 peering, 1 active+undersized, 144 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 6650/320340 objects degraded (2.076%); 608 KiB/s, 9 objects/s recovering 2024-02-10T07:47:51.620 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:51 smithi170 ceph-mon[33394]: 6.5 scrub ok 2024-02-10T07:47:51.621 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:51 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1302690946' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:47:51.629 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:51.663 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:51.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:51.765 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:51.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:51.802 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:51.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:51.903 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:51.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:51.941 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:51.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:52.042 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:52.077 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:52.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:52.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:52.180 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:52.214 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:52.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:52.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:52.316 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:52.330 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[29816]: 3.7 scrub starts 2024-02-10T07:47:52.330 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[29816]: 3.7 scrub ok 2024-02-10T07:47:52.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[29816]: 3.10 scrub starts 2024-02-10T07:47:52.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[29816]: 3.10 scrub ok 2024-02-10T07:47:52.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:47:52.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1302690946' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:47:52.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[29816]: osdmap e350: 8 total, 8 up, 7 in 2024-02-10T07:47:52.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 6650/320340 objects degraded (2.076%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-10T07:47:52.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[29816]: osdmap e351: 8 total, 8 up, 7 in 2024-02-10T07:47:52.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[40743]: 3.7 scrub starts 2024-02-10T07:47:52.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[40743]: 3.7 scrub ok 2024-02-10T07:47:52.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[40743]: 3.10 scrub starts 2024-02-10T07:47:52.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[40743]: 3.10 scrub ok 2024-02-10T07:47:52.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:47:52.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1302690946' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:47:52.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[40743]: osdmap e350: 8 total, 8 up, 7 in 2024-02-10T07:47:52.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 6650/320340 objects degraded (2.076%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-10T07:47:52.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:52 smithi008 ceph-mon[40743]: osdmap e351: 8 total, 8 up, 7 in 2024-02-10T07:47:52.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:52.372 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:52.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:52.473 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:47:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:52.514 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:52.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:47:52.560 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:52 smithi170 ceph-mon[33394]: 3.7 scrub starts 2024-02-10T07:47:52.560 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:52 smithi170 ceph-mon[33394]: 3.7 scrub ok 2024-02-10T07:47:52.561 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:52 smithi170 ceph-mon[33394]: 3.10 scrub starts 2024-02-10T07:47:52.561 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:52 smithi170 ceph-mon[33394]: 3.10 scrub ok 2024-02-10T07:47:52.561 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:52 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:47:52.561 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:52 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1302690946' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:47:52.561 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:52 smithi170 ceph-mon[33394]: osdmap e350: 8 total, 8 up, 7 in 2024-02-10T07:47:52.561 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:52 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 6650/320340 objects degraded (2.076%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-02-10T07:47:52.561 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:52 smithi170 ceph-mon[33394]: osdmap e351: 8 total, 8 up, 7 in 2024-02-10T07:47:52.615 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:47:52.651 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:52.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:47:52.670 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:47:52.701 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:47:52.732 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:47:52.753 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:47:52.766 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:47:52.782 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:52.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:52.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:47:52.803 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:47:52.837 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:47:52.884 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:52.885 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:47:52.920 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:47:52.922 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:47:52.923 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:47:52.923 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:47:52.923 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-02-10T07:47:52.923 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:47:52.923 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:47:52.923 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:47:52.923 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:47:52.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:52.925 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:52.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:53.026 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:53.057 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:53.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:53.087 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:47:53.088 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2024-02-10T07:47:53.088 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2024-02-10T07:47:53.088 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2024-02-10T07:47:53.088 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:47:53.158 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:53.189 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:53.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:53.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:53.291 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:53.321 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:53.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:53.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:53.423 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:53.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:53.464 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:53.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:53.565 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:53.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:53 smithi008 ceph-mon[29816]: pgmap v732: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating, 1 peering, 145 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 859 op/s; 6650/320340 objects degraded (2.076%); 593 KiB/s, 9 objects/s recovering 2024-02-10T07:47:53.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:53 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:47:53.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:53 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:47:53.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:53 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2319205722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:47:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:53 smithi008 ceph-mon[40743]: pgmap v732: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating, 1 peering, 145 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 859 op/s; 6650/320340 objects degraded (2.076%); 593 KiB/s, 9 objects/s recovering 2024-02-10T07:47:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:53 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:47:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:53 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:47:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:53 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2319205722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:47:53.605 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:53.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:53.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:53.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:53 smithi170 ceph-mon[33394]: pgmap v732: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating, 1 peering, 145 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 859 op/s; 6650/320340 objects degraded (2.076%); 593 KiB/s, 9 objects/s recovering 2024-02-10T07:47:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:53 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:47:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:53 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:47:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:53 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2319205722' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:47:53.707 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:53.742 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:53.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:53.842 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-10T07:47:53.843 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:53.872 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:53.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:53.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:53.924 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:47:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:47:53.974 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:54.008 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:54.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:54.109 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:54.109 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:47:54.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:54.143 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:54.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:54.245 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:54.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:54.283 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:54.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:54.384 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:54.421 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:54.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:54.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:54.480 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:54.524 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:54.542 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:47:54.567 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:54.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:54.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:54 smithi008 ceph-mon[29816]: osdmap e352: 8 total, 8 up, 7 in 2024-02-10T07:47:54.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:54 smithi008 ceph-mon[40743]: osdmap e352: 8 total, 8 up, 7 in 2024-02-10T07:47:54.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:54 smithi170 ceph-mon[33394]: osdmap e352: 8 total, 8 up, 7 in 2024-02-10T07:47:54.669 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:54.701 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:54.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:54.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:54.803 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:54.837 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:54.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:54.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:54.939 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:54.976 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:54.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:54.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:55.078 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:55.108 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:55.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:55.209 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:55.235 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 4, 6, 7, 5, 3] 2024-02-10T07:47:55.235 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:47:55.235 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-10T07:47:55.235 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.226452 2024-02-10T07:47:55.235 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 5 0.22645164871857304 2024-02-10T07:47:55.240 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:55.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:55.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:55.293 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:55 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:55.342 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:55.377 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:55.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:55.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:55.479 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:55.521 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:55.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:55.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:55 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:55.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:55 smithi008 ceph-mon[29816]: pgmap v734: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 809 op/s; 6650/319152 objects degraded (2.084%); 477 KiB/s, 7 objects/s recovering 2024-02-10T07:47:55.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:55 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:47:55.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:55 smithi008 ceph-mon[40743]: pgmap v734: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 809 op/s; 6650/319152 objects degraded (2.084%); 477 KiB/s, 7 objects/s recovering 2024-02-10T07:47:55.622 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:55 smithi170 ceph-mon[33394]: pgmap v734: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 28 MiB/s wr, 809 op/s; 6650/319152 objects degraded (2.084%); 477 KiB/s, 7 objects/s recovering 2024-02-10T07:47:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:55.658 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:55.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:55.759 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:55.763 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:47:55.793 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:55.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:55.895 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:55.926 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:55.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:55.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:56.028 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:56.071 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:56.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:56.172 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:56.184 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:56.216 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:56.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:56.318 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:56.361 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:56.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:56.463 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:56.503 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:56.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:56.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:56.604 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:56.638 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:56.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:56.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:56.718 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:47:56.739 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:56.782 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:56.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:56.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:56.884 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:47:56.914 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:56.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:56.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:47:57.016 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:47:57.053 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:57.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:57.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:47:57.155 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:57.201 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:57.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:57.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:57.303 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:57.342 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:57.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:57.410 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:57 smithi170 ceph-mon[33394]: pgmap v735: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 20 MiB/s wr, 770 op/s; 4009/317751 objects degraded (1.262%); 27 MiB/s, 430 objects/s recovering 2024-02-10T07:47:57.410 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:57 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 4009/317751 objects degraded (1.262%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:47:57.410 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:57 smithi170 ceph-mon[33394]: osdmap e353: 8 total, 8 up, 7 in 2024-02-10T07:47:57.410 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:57 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1266769839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:57.410 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:57 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:57.444 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:57.478 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:57.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:57.533 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:57 smithi008 ceph-mon[29816]: pgmap v735: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 20 MiB/s wr, 770 op/s; 4009/317751 objects degraded (1.262%); 27 MiB/s, 430 objects/s recovering 2024-02-10T07:47:57.533 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:57 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 4009/317751 objects degraded (1.262%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:47:57.533 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:57 smithi008 ceph-mon[29816]: osdmap e353: 8 total, 8 up, 7 in 2024-02-10T07:47:57.533 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:57 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1266769839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:57.533 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:57 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:57.534 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:57 smithi008 ceph-mon[40743]: pgmap v735: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 20 MiB/s wr, 770 op/s; 4009/317751 objects degraded (1.262%); 27 MiB/s, 430 objects/s recovering 2024-02-10T07:47:57.534 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:57 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 4009/317751 objects degraded (1.262%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:47:57.534 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:57 smithi008 ceph-mon[40743]: osdmap e353: 8 total, 8 up, 7 in 2024-02-10T07:47:57.534 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:57 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1266769839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:57.534 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:57 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:47:57.580 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:47:57.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:57.612 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:57.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:47:57.687 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:47:57.687 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2024-02-10T07:47:57.687 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2024-02-10T07:47:57.687 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:47:57.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.687 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:144630 8.2 8:4cb0000a:::benchmark_data_smithi008_109566_object19543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-02-10T07:47:57.687 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.476248+0000", 2024-02-10T07:47:57.687 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 463.85175024199998, 2024-02-10T07:47:57.687 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11699762599999999, 2024-02-10T07:47:57.687 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.687 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.687 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.688 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:47:57.688 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:47:57.688 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 144630 2024-02-10T07:47:57.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.688 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476248+0000", 2024-02-10T07:47:57.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476248+0000", 2024-02-10T07:47:57.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476250+0000", 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9120000000000001e-06 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476256+0000", 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5339999999999999e-06 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476268+0000", 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2394e-05 2024-02-10T07:47:57.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476276+0000", 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6000000000000001e-06 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476326+0000", 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0649999999999998e-05 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476409+0000", 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2434000000000004e-05 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,5", 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.476477+0000", 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8449999999999997e-05 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.478509+0000", 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020322869999999998 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.482094+0000", 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0035841499999999999 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.691 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.593144+0000", 2024-02-10T07:47:57.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.111050597 2024-02-10T07:47:57.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.692 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.692 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.593204+0000", 2024-02-10T07:47:57.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9373000000000002e-05 2024-02-10T07:47:57.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.692 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.692 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.593246+0000", 2024-02-10T07:47:57.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2245e-05 2024-02-10T07:47:57.692 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.692 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.692 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:144635 8.2 8:4c7aedb6:::benchmark_data_smithi008_109566_object19548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.480454+0000", 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 463.84754475199998, 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.119664639, 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 144635 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480454+0000", 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480454+0000", 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480455+0000", 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.2999999999999999e-07 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480458+0000", 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.574e-06 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480467+0000", 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3040000000000005e-06 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480472+0000", 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.237e-06 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.695 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.696 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480517+0000", 2024-02-10T07:47:57.696 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5034000000000003e-05 2024-02-10T07:47:57.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.696 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.696 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480591+0000", 2024-02-10T07:47:57.696 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4116999999999995e-05 2024-02-10T07:47:57.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.696 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,5", 2024-02-10T07:47:57.696 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.480658+0000", 2024-02-10T07:47:57.696 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6737999999999997e-05 2024-02-10T07:47:57.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.482174+0000", 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0015163170000000001 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.484133+0000", 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001959396 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.599951+0000", 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.115818009 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.600073+0000", 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000121647 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.600118+0000", 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5336000000000001e-05 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:144651 8.2 8:4e1c7b18:::benchmark_data_smithi008_109566_object19564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:13.499604+0000", 2024-02-10T07:47:57.698 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 463.82839470800002, 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.102912694, 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 144651 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.499604+0000", 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.699 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.499604+0000", 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.499605+0000", 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6509999999999999e-06 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.499612+0000", 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1539999999999999e-06 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.700 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.499626+0000", 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4684e-05 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.499635+0000", 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3119999999999994e-06 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.499685+0000", 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0263999999999999e-05 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.499775+0000", 2024-02-10T07:47:57.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.9742999999999998e-05 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,5", 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.499845+0000", 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0340999999999996e-05 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.501695+0000", 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001850436 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.502121+0000", 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00042538400000000001 2024-02-10T07:47:57.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.602329+0000", 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.10020788799999999 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.602471+0000", 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000142447 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:13.602516+0000", 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5389999999999997e-05 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.703 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:164429 8.2 8:4c05036b:::benchmark_data_smithi008_109566_object39342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:31.020334+0000", 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 446.30766419100001, 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.090744155000000007, 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 164429 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.020334+0000", 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.020334+0000", 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.020336+0000", 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.153e-06 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.020347+0000", 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.058e-05 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.020372+0000", 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.4893e-05 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.020388+0000", 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5631999999999999e-05 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.020504+0000", 2024-02-10T07:47:57.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011669500000000001 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.020691+0000", 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000186319 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,5", 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.020830+0000", 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013972100000000001 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.022964+0000", 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002134103 2024-02-10T07:47:57.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.024853+0000", 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00188908 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.110943+0000", 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.086089409000000006 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.111035+0000", 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1992999999999998e-05 2024-02-10T07:47:57.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.111078+0000", 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3577e-05 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:164436 8.2 8:497ec52f:::benchmark_data_smithi008_109566_object39349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:31.032458+0000", 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 446.29554034, 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.082790855999999996, 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.709 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 164436 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.032458+0000", 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.032458+0000", 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.032459+0000", 2024-02-10T07:47:57.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.017e-06 2024-02-10T07:47:57.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.032463+0000", 2024-02-10T07:47:57.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6689999999999999e-06 2024-02-10T07:47:57.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.032473+0000", 2024-02-10T07:47:57.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0105000000000001e-05 2024-02-10T07:47:57.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.032478+0000", 2024-02-10T07:47:57.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.4790000000000004e-06 2024-02-10T07:47:57.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.032526+0000", 2024-02-10T07:47:57.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7457000000000003e-05 2024-02-10T07:47:57.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.713 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.032611+0000", 2024-02-10T07:47:57.713 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5265000000000004e-05 2024-02-10T07:47:57.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.713 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.713 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,5", 2024-02-10T07:47:57.713 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.032682+0000", 2024-02-10T07:47:57.713 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0835999999999997e-05 2024-02-10T07:47:57.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.713 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.713 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.713 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.034463+0000", 2024-02-10T07:47:57.713 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001781539 2024-02-10T07:47:57.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.713 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.035111+0000", 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00064801599999999996 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.115147+0000", 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.080035311999999997 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.115204+0000", 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6869e-05 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:31.115249+0000", 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5291999999999999e-05 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15087.0:168182 8.2 8:4ef5a55f:::benchmark_data_smithi008_109566_object43095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:40:34.253590+0000", 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 443.07440885099999, 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.098550459000000007, 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15087", 2024-02-10T07:47:57.715 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1969748604", 2024-02-10T07:47:57.716 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 168182 2024-02-10T07:47:57.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.716 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.716 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.716 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.716 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.253590+0000", 2024-02-10T07:47:57.716 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.716 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.716 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.716 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.253590+0000", 2024-02-10T07:47:57.716 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.716 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.253591+0000", 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.584e-06 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.253596+0000", 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9949999999999996e-06 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.253610+0000", 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.332e-05 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.253617+0000", 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4039999999999998e-06 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.253667+0000", 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0065000000000003e-05 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.253759+0000", 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.2200999999999994e-05 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.718 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,5", 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.253831+0000", 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.1950000000000001e-05 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.255806+0000", 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0019745800000000001 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.256100+0000", 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00029414000000000003 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.719 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.352020+0000", 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.095920200999999997 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.352071+0000", 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1230999999999998e-05 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:40:34.352140+0000", 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8788000000000006e-05 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.720 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:34419 10.e 10:7c735169:::benchmark_data_smithi008_161452_object34418:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:34.958719+0000", 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 82.369279657000007, 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.081639767000000002, 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 34419 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:34.958719+0000", 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:34.958719+0000", 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:34.958720+0000", 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2109999999999999e-06 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:34.958791+0000", 2024-02-10T07:47:57.722 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0568000000000006e-05 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:34.958800+0000", 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.7960000000000007e-06 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:34.958806+0000", 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.7230000000000001e-06 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:34.958866+0000", 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9966000000000002e-05 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.724 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.724 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:34.958998+0000", 2024-02-10T07:47:57.724 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013158199999999999 2024-02-10T07:47:57.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.724 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:47:57.724 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:34.959146+0000", 2024-02-10T07:47:57.724 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000148033 2024-02-10T07:47:57.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.724 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.724 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:34.960523+0000", 2024-02-10T07:47:57.724 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001377571 2024-02-10T07:47:57.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:34.960766+0000", 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00024304099999999999 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:35.040265+0000", 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.079498316999999999 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:35.040322+0000", 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6984999999999997e-05 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:35.040359+0000", 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6974000000000002e-05 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:60964 10.e 10:79096cdd:::benchmark_data_smithi008_161452_object60963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:59.266787+0000", 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 58.061211745999998, 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.066934024999999994, 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.726 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 60964 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:59.266787+0000", 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:59.266787+0000", 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:59.266788+0000", 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.601e-06 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:59.266828+0000", 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9968000000000001e-05 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:59.266846+0000", 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7649999999999999e-05 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:59.266854+0000", 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8520000000000004e-06 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:59.267017+0000", 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000163246 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:59.267195+0000", 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000177589 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:59.267650+0000", 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000455227 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:59.271711+0000", 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0040615240000000004 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:59.275139+0000", 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.003427696 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.730 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:59.333560+0000", 2024-02-10T07:47:57.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.058420773000000002 2024-02-10T07:47:57.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.731 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.731 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.731 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:59.333660+0000", 2024-02-10T07:47:57.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000100176 2024-02-10T07:47:57.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.731 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.731 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.731 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:59.333721+0000", 2024-02-10T07:47:57.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0723000000000001e-05 2024-02-10T07:47:57.731 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.731 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:68069 10.e 10:71252399:::benchmark_data_smithi008_161452_object68068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:07.094055+0000", 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 50.233943754000002, 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.070551244999999999, 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 68069 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.732 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.733 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.733 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.733 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.094055+0000", 2024-02-10T07:47:57.733 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.733 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.733 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.733 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.094055+0000", 2024-02-10T07:47:57.733 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.733 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.733 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.733 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.094056+0000", 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2720000000000001e-06 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.094089+0000", 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3256999999999997e-05 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.094100+0000", 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0811e-05 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.094107+0000", 2024-02-10T07:47:57.734 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2819999999999999e-06 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.094209+0000", 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000101214 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.094344+0000", 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000135217 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.094482+0000", 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000138236 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.097242+0000", 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0027602289999999999 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.098276+0000", 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001033908 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.164499+0000", 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.066223263000000004 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.736 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.164568+0000", 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8540000000000004e-05 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.164606+0000", 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8016000000000003e-05 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:68700 10.e 10:788e2c60:::benchmark_data_smithi008_161452_object68699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:07.864554+0000", 2024-02-10T07:47:57.737 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 49.463444213000002, 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11747103, 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 68700 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.864554+0000", 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.738 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.864554+0000", 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.864556+0000", 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.513e-06 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.864604+0000", 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.778e-05 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.739 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.864615+0000", 2024-02-10T07:47:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1843e-05 2024-02-10T07:47:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.864624+0000", 2024-02-10T07:47:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2069999999999995e-06 2024-02-10T07:47:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.864690+0000", 2024-02-10T07:47:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6841999999999995e-05 2024-02-10T07:47:57.740 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.864825+0000", 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013462500000000001 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.864981+0000", 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015586800000000001 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.867069+0000", 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020883479999999999 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.867345+0000", 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00027550799999999999 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.981931+0000", 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11458618700000001 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.981989+0000", 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8457e-05 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.742 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:07.982025+0000", 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5852000000000002e-05 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:68832 10.e 10:774f369d:::benchmark_data_smithi008_161452_object68831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:08.099479+0000", 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 49.228519026999997, 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.070665839999999994, 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:47:57.743 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 68832 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.099479+0000", 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.099479+0000", 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.744 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.099481+0000", 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.553e-06 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.099516+0000", 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.4703000000000003e-05 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.099529+0000", 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3043e-05 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.099536+0000", 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6690000000000006e-06 2024-02-10T07:47:57.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.099590+0000", 2024-02-10T07:47:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3922000000000003e-05 2024-02-10T07:47:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.099939+0000", 2024-02-10T07:47:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000348654 2024-02-10T07:47:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:47:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.100084+0000", 2024-02-10T07:47:57.746 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014465300000000001 2024-02-10T07:47:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.102392+0000", 2024-02-10T07:47:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0023088499999999999 2024-02-10T07:47:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.103376+0000", 2024-02-10T07:47:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0009840230000000001 2024-02-10T07:47:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.747 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.170005+0000", 2024-02-10T07:47:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.066628085000000004 2024-02-10T07:47:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.170104+0000", 2024-02-10T07:47:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9667000000000006e-05 2024-02-10T07:47:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.170145+0000", 2024-02-10T07:47:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1017999999999999e-05 2024-02-10T07:47:57.748 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:68837 10.e 10:75028024:::benchmark_data_smithi008_161452_object68836:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:08.103549+0000", 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 49.224449233000001, 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.076280638999999997, 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 68837 2024-02-10T07:47:57.749 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.103549+0000", 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.103549+0000", 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.103550+0000", 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2449999999999999e-06 2024-02-10T07:47:57.750 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.751 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.751 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.751 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.103597+0000", 2024-02-10T07:47:57.751 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6814999999999998e-05 2024-02-10T07:47:57.751 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.751 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.751 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.751 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.103607+0000", 2024-02-10T07:47:57.751 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5650000000000007e-06 2024-02-10T07:47:57.751 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.751 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.751 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.751 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.103613+0000", 2024-02-10T07:47:57.751 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6089999999999997e-06 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.103648+0000", 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5043999999999999e-05 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.103752+0000", 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010357900000000001 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.103843+0000", 2024-02-10T07:47:57.752 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0654000000000003e-05 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.105996+0000", 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0021530569999999999 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.109453+0000", 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.003456822 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.179718+0000", 2024-02-10T07:47:57.753 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.070265561000000004 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.179791+0000", 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3159999999999997e-05 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:08.179830+0000", 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8528000000000001e-05 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.754 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:92277 10.1 10:8ef6f4a3:::benchmark_data_smithi008_161452_object92276:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:33.175380+0000", 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 24.152618755999999, 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.10283505900000001, 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 92277 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.175380+0000", 2024-02-10T07:47:57.755 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.756 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.756 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.756 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.756 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.175380+0000", 2024-02-10T07:47:57.756 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.756 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.756 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.756 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.756 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.175381+0000", 2024-02-10T07:47:57.756 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.246e-06 2024-02-10T07:47:57.756 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.756 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.756 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.756 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.175572+0000", 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000190793 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.175585+0000", 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3219e-05 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.175597+0000", 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2459e-05 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.182126+0000", 2024-02-10T07:47:57.757 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0065286670000000002 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.182278+0000", 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015145299999999999 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,7", 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.182454+0000", 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017670899999999999 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.192845+0000", 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010390396999999999 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.759 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.759 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.193149+0000", 2024-02-10T07:47:57.759 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00030391000000000001 2024-02-10T07:47:57.759 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.759 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.759 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.759 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.278123+0000", 2024-02-10T07:47:57.759 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.084974292000000007 2024-02-10T07:47:57.759 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.759 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.759 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.759 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.278178+0000", 2024-02-10T07:47:57.759 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5195000000000003e-05 2024-02-10T07:47:57.759 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.278215+0000", 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6718999999999999e-05 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:92292 10.1 10:8a8b4096:::benchmark_data_smithi008_161452_object92291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:33.196080+0000", 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 24.131918356, 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.082238921000000006, 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.760 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 92292 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.196080+0000", 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.196080+0000", 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.761 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.196082+0000", 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.548e-06 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.196183+0000", 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000101832 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.196196+0000", 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2051e-05 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.762 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.196202+0000", 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3389999999999998e-06 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.196266+0000", 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3958e-05 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.196439+0000", 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017328600000000001 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.763 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,7", 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.196611+0000", 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017143599999999999 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.198678+0000", 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020669830000000001 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.199129+0000", 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00045123400000000001 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.764 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.278249+0000", 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.079119941999999999 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.278290+0000", 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1579999999999998e-05 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.278319+0000", 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8731999999999999e-05 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.765 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:115576 10.e 10:75cb4c29:::benchmark_data_smithi008_161452_object7115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:56.891450+0000", 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.43654892099999998, 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.077518862999999993, 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 115576 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.766 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:56.891450+0000", 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:56.891450+0000", 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:56.891451+0000", 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4649999999999999e-06 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:56.891456+0000", 2024-02-10T07:47:57.767 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.074e-06 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:56.891468+0000", 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1552000000000001e-05 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:56.891475+0000", 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5070000000000003e-06 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:56.891790+0000", 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00031513400000000001 2024-02-10T07:47:57.768 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:56.891906+0000", 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000115606 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:56.892014+0000", 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000108119 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:56.894002+0000", 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0019878679999999998 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.769 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:56.894239+0000", 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000236674 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:56.968769+0000", 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.074530788000000001 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:56.968924+0000", 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015423699999999999 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.770 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:56.968968+0000", 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4839000000000002e-05 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15630.0:115868 10.c e353/309 10:34f78da8:::benchmark_data_smithi008_161452_object7407:head v 353'7156, mlcod=353'7154)", 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:57.189747+0000", 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.138251557, 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.071513430000000003, 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.771 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.189747+0000", 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.189747+0000", 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.189747+0000", 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2399999999999998e-07 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.772 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.189754+0000", 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1070000000000001e-06 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.189763+0000", 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0429999999999996e-06 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.189768+0000", 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.891e-06 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.773 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.191829+0000", 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020601180000000001 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.191847+0000", 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8193e-05 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.261211+0000", 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.069364354000000003 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.774 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.261260+0000", 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9200000000000003e-05 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15630.0:115870 10.c e353/309 10:30970b19:::benchmark_data_smithi008_161452_object7409:head v 353'7157, mlcod=353'7154)", 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:57.192197+0000", 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.135801215, 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.074294101000000001, 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.775 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.192197+0000", 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.192197+0000", 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.192198+0000", 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1600000000000001e-07 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.776 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.192204+0000", 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6139999999999997e-06 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.192212+0000", 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5800000000000003e-06 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.192218+0000", 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9340000000000003e-06 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.195685+0000", 2024-02-10T07:47:57.777 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0034669330000000002 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.195705+0000", 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0582e-05 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.266300+0000", 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.070594527000000004 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.266491+0000", 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019141500000000001 2024-02-10T07:47:57.778 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15630.0:115875 10.c e353/309 10:371b2c10:::benchmark_data_smithi008_161452_object7414:head v 353'7158, mlcod=353'7154)", 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:57.197316+0000", 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.13068210999999999, 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.070317363999999993, 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.197316+0000", 2024-02-10T07:47:57.779 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.197316+0000", 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.197317+0000", 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1999999999999998e-07 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.197324+0000", 2024-02-10T07:47:57.780 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9970000000000001e-06 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.197334+0000", 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0227000000000001e-05 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.197342+0000", 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.401e-06 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.261292+0000", 2024-02-10T07:47:57.781 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.063950558000000005 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.261314+0000", 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1668e-05 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.267344+0000", 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0060302899999999998 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.267634+0000", 2024-02-10T07:47:57.782 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00028930299999999998 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15630.0:115877 10.3 e353/337 10:c5908577:::benchmark_data_smithi008_161452_object7416:head v 353'7247, mlcod=353'7244)", 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:57.197490+0000", 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.13050809599999999, 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.066885427999999997, 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.783 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.197490+0000", 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.197490+0000", 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.197491+0000", 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.86e-07 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.784 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.197493+0000", 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8260000000000001e-06 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.197498+0000", 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5249999999999999e-06 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.197501+0000", 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5650000000000002e-06 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.785 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.261420+0000", 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.063918864000000006 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.261440+0000", 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.003e-05 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.264336+0000", 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0028951929999999999 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.786 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.264376+0000", 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.0238999999999999e-05 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:115883 10.e 10:7b47a638:::benchmark_data_smithi008_161452_object7422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:57.201288+0000", 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.126710287, 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.085160445000000001, 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:47:57.787 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:47:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 115883 2024-02-10T07:47:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:47:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:47:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.201288+0000", 2024-02-10T07:47:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:47:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.201288+0000", 2024-02-10T07:47:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:47:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.788 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.201290+0000", 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.421e-06 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.201294+0000", 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4190000000000004e-06 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.261628+0000", 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.060333699999999997 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.789 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.261637+0000", 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0440000000000001e-06 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.263802+0000", 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0021653890000000002 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.263961+0000", 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000158479 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.790 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 5,6", 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.264242+0000", 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00028121299999999999 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.267366+0000", 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0031237050000000001 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.269626+0000", 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0022607399999999998 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:47:57.791 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.286352+0000", 2024-02-10T07:47:57.792 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016725633 2024-02-10T07:47:57.792 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.792 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.792 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:47:57.792 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.286407+0000", 2024-02-10T07:47:57.792 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5507999999999998e-05 2024-02-10T07:47:57.792 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:47:57.792 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:47:57.792 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:47:57.792 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:57.286449+0000", 2024-02-10T07:47:57.792 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1193999999999999e-05 2024-02-10T07:47:57.792 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.792 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.792 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.793 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:47:57.793 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:47:57.793 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:47:57.793 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:47:57.824 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:57.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:57.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:47:57.926 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:57.964 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:57.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:57.969 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:47:58.065 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:58.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:58.098 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:58.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:47:58.199 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:58.235 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:58.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:58.337 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:47:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:58.374 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:58.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:47:58.475 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:58.506 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:58.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:58.559 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:58 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:47:58.559 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:58 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:47:58.559 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:58 smithi008 ceph-mon[29816]: osdmap e354: 8 total, 8 up, 7 in 2024-02-10T07:47:58.559 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:58 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:47:58.559 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:58 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:47:58.559 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:58 smithi008 ceph-mon[40743]: osdmap e354: 8 total, 8 up, 7 in 2024-02-10T07:47:58.608 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:58.611 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:58 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:47:58.612 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:58 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:47:58.612 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:58 smithi170 ceph-mon[33394]: osdmap e354: 8 total, 8 up, 7 in 2024-02-10T07:47:58.642 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:58.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:58.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:58.744 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:47:58.788 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:58.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:58.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:47:58.828 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:47:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:47:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:47:58.890 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:58.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:58.926 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:58.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:59.027 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:59.059 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:59.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:59.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:59.161 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:59.192 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:59.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:59.294 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:47:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:59.330 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:59.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:47:59.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:47:59 smithi170 ceph-mon[33394]: pgmap v738: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 20 MiB/s wr, 950 op/s; 4009/314502 objects degraded (1.275%); 27 MiB/s, 430 objects/s recovering 2024-02-10T07:47:59.431 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:47:59.461 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:47:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:59.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:47:59.563 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:59.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:47:59 smithi008 ceph-mon[29816]: pgmap v738: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 20 MiB/s wr, 950 op/s; 4009/314502 objects degraded (1.275%); 27 MiB/s, 430 objects/s recovering 2024-02-10T07:47:59.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:47:59 smithi008 ceph-mon[40743]: pgmap v738: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 20 MiB/s wr, 950 op/s; 4009/314502 objects degraded (1.275%); 27 MiB/s, 430 objects/s recovering 2024-02-10T07:47:59.602 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:59.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:59.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:59.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-10T07:47:59.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:47:59.755 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:59.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:47:59.857 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:47:59.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:47:59.900 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:47:59.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:00.001 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:00.032 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:00.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:00.134 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:00.165 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:00.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:00.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:00.183 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:48:00.266 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:00.317 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:00.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:00.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:00.418 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:00.464 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:00.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:00.566 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:00.599 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:00.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:00.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:00.701 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:00.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:00.733 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:00.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:00.834 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:00.867 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:00.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:00.968 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:01.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:01.003 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:01.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:01.104 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:01.144 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:01.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:01.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:01.246 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:01.286 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:01.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:01.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:01.318 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.5 primary-affinity to 0.226452 (8148402) 2024-02-10T07:48:01.388 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:01.411 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:01.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:01.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:01.514 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:01.540 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:01.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:01.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:01.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:01 smithi008 ceph-mon[29816]: pgmap v739: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 906 B/s rd, 11 MiB/s wr, 1.33k op/s; 1553/300084 objects degraded (0.518%); 45 MiB/s, 718 objects/s recovering 2024-02-10T07:48:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:01 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3074886313' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.22645164871857304}]: dispatch 2024-02-10T07:48:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:01 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.22645164871857304}]: dispatch 2024-02-10T07:48:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:01 smithi008 ceph-mon[40743]: pgmap v739: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 906 B/s rd, 11 MiB/s wr, 1.33k op/s; 1553/300084 objects degraded (0.518%); 45 MiB/s, 718 objects/s recovering 2024-02-10T07:48:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:01 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3074886313' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.22645164871857304}]: dispatch 2024-02-10T07:48:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:01 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.22645164871857304}]: dispatch 2024-02-10T07:48:01.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:01 smithi170 ceph-mon[33394]: pgmap v739: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 906 B/s rd, 11 MiB/s wr, 1.33k op/s; 1553/300084 objects degraded (0.518%); 45 MiB/s, 718 objects/s recovering 2024-02-10T07:48:01.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:01 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3074886313' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.22645164871857304}]: dispatch 2024-02-10T07:48:01.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:01 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.22645164871857304}]: dispatch 2024-02-10T07:48:01.642 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:01.665 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:01.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:01.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:01.767 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:01.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:01.792 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:01.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:01.893 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:01.917 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:01.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:02.018 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:02.044 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:02.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:02.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:02.093 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:48:02.137 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:48:02.137 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2024-02-10T07:48:02.137 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 0 2024-02-10T07:48:02.137 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:48:02.147 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:02.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:02.175 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:02.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:02.276 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:02.301 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:02.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:02.402 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:02.428 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:02.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:02.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:02.530 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:02.555 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:02.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:02.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:02.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:02 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.22645164871857304}]': finished 2024-02-10T07:48:02.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:02 smithi008 ceph-mon[29816]: osdmap e355: 8 total, 8 up, 7 in 2024-02-10T07:48:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:02 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 1553/300084 objects degraded (0.518%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:48:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:02 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.22645164871857304}]': finished 2024-02-10T07:48:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:02 smithi008 ceph-mon[40743]: osdmap e355: 8 total, 8 up, 7 in 2024-02-10T07:48:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:02 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 1553/300084 objects degraded (0.518%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:48:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:02 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.22645164871857304}]': finished 2024-02-10T07:48:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:02 smithi170 ceph-mon[33394]: osdmap e355: 8 total, 8 up, 7 in 2024-02-10T07:48:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:02 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 1553/300084 objects degraded (0.518%), 1 pg degraded (PG_DEGRADED) 2024-02-10T07:48:02.657 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:02.681 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:02.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:02.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:02.783 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:02.807 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:02.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:02.908 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:02.912 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-10T07:48:02.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:02.933 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:02.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:03.034 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:03.059 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:03.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:03.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:03.161 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:03.171 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:48:03.185 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:03.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:03.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:03.287 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:03.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:03.311 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:03.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:03.412 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:03.444 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:03.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:03.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:03.545 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:03 smithi008 ceph-mon[29816]: pgmap v741: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 981 op/s; 1553/300084 objects degraded (0.518%); 24 MiB/s, 380 objects/s recovering 2024-02-10T07:48:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:03 smithi008 ceph-mon[29816]: osdmap e356: 8 total, 8 up, 7 in 2024-02-10T07:48:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:48:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:03 smithi008 ceph-mon[40743]: pgmap v741: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 981 op/s; 1553/300084 objects degraded (0.518%); 24 MiB/s, 380 objects/s recovering 2024-02-10T07:48:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:03 smithi008 ceph-mon[40743]: osdmap e356: 8 total, 8 up, 7 in 2024-02-10T07:48:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:48:03.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:03.586 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:03.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:03 smithi170 ceph-mon[33394]: pgmap v741: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 6.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 981 op/s; 1553/300084 objects degraded (0.518%); 24 MiB/s, 380 objects/s recovering 2024-02-10T07:48:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:03 smithi170 ceph-mon[33394]: osdmap e356: 8 total, 8 up, 7 in 2024-02-10T07:48:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:48:03.687 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:03.718 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:03.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:03.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:03.823 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:03.854 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:03.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:03.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:03.956 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:03.984 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:03.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:03.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:04.086 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:04.115 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:04.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:04.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:04.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:48:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:48:04.217 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:04.244 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:04.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:04.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:04.346 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:04.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:04.379 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:04.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:04.480 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:04.511 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:04.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:04.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:04 smithi008 ceph-mon[29816]: osdmap e357: 8 total, 8 up, 7 in 2024-02-10T07:48:04.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:04 smithi008 ceph-mon[40743]: osdmap e357: 8 total, 8 up, 7 in 2024-02-10T07:48:04.613 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:04 smithi170 ceph-mon[33394]: osdmap e357: 8 total, 8 up, 7 in 2024-02-10T07:48:04.647 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:04.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:04.749 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:04.778 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:04.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:04.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:04.879 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:04.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:04.909 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:04.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:05.010 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:05.037 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:05.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:05.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:05.139 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:05.177 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:05.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:05.279 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:05.308 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:05.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:05.410 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:05.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:05.445 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:05.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:05.546 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:05.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:05 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:05.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:05 smithi008 ceph-mon[29816]: pgmap v744: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.27k op/s; 1553/291642 objects degraded (0.533%); 24 MiB/s, 380 objects/s recovering 2024-02-10T07:48:05.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:05 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:05.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:05 smithi008 ceph-mon[40743]: pgmap v744: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.27k op/s; 1553/291642 objects degraded (0.533%); 24 MiB/s, 380 objects/s recovering 2024-02-10T07:48:05.583 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:05.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:05.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:05 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:05 smithi170 ceph-mon[33394]: pgmap v744: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.27k op/s; 1553/291642 objects degraded (0.533%); 24 MiB/s, 380 objects/s recovering 2024-02-10T07:48:05.686 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:05.713 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:05.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:05.815 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:05.843 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:05.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:05.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:05.945 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:05.975 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:05.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:05.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:06.076 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:06.100 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:06.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:06.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:06.202 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:06.228 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:06.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:06.330 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:06.368 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:06.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:06.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:06.470 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:06.507 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:06.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:06.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:06.609 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:06.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:06 smithi170 ceph-mon[33394]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1553/291642 objects degraded (0.533%), 1 pg degraded) 2024-02-10T07:48:06.637 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:06.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:06.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:06.738 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:06.761 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:06.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:06.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:06.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:06 smithi008 ceph-mon[29816]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1553/291642 objects degraded (0.533%), 1 pg degraded) 2024-02-10T07:48:06.831 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:06 smithi008 ceph-mon[40743]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1553/291642 objects degraded (0.533%), 1 pg degraded) 2024-02-10T07:48:06.863 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:06.890 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:06.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:06.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:06.992 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:07.032 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:07.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:07.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:07.134 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:07.161 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:07.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:07.263 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:07.302 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:07.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:07.403 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:07.426 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:07.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:07.528 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:07.563 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:07.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:07 smithi170 ceph-mon[33394]: pgmap v745: 148 pgs: 148 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 694 op/s; 16 MiB/s, 253 objects/s recovering 2024-02-10T07:48:07.642 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:07 smithi008 ceph-mon[29816]: pgmap v745: 148 pgs: 148 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 694 op/s; 16 MiB/s, 253 objects/s recovering 2024-02-10T07:48:07.642 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:07 smithi008 ceph-mon[40743]: pgmap v745: 148 pgs: 148 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 341 B/s rd, 0 B/s wr, 694 op/s; 16 MiB/s, 253 objects/s recovering 2024-02-10T07:48:07.665 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:07.699 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:07.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:07.800 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:07.830 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:07.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:07.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:07.932 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:07.970 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:07.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:08.037 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:08.072 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:08.101 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:48:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:08.109 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:08.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:08.129 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:48:08.155 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:48:08.181 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:48:08.211 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:08.217 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:48:08.247 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:48:08.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:08.263 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:08.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:08.276 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:48:08.312 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:48:08.317 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:48:08.317 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:48:08.317 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:48:08.317 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:48:08.317 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:48:08.317 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:48:08.317 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:48:08.317 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:48:08.364 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:08.404 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:08.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:08.433 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:08 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2008581379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:08.433 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:08 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2008581379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:08.507 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:08.535 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:08.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:08.637 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:08.668 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:08.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:08.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:08.739 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:08 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2008581379' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:08.770 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:08.798 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:08.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:08.900 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:08.927 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:08.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:08.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:08.947 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:48:08.947 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:48:09.029 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:09.065 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:09.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:09.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:48:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:48:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:48:09.168 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:09.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:09.198 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:09.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:09.299 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:09.337 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:09.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:09.438 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:09.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:09.477 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:09.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:09.578 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:09.613 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:09.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:09.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:09.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:09 smithi170 ceph-mon[33394]: pgmap v746: 148 pgs: 148 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 303 B/s rd, 0 B/s wr, 616 op/s; 14 MiB/s, 225 objects/s recovering 2024-02-10T07:48:09.715 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:09.744 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:09.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:09.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:09 smithi008 ceph-mon[29816]: pgmap v746: 148 pgs: 148 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 303 B/s rd, 0 B/s wr, 616 op/s; 14 MiB/s, 225 objects/s recovering 2024-02-10T07:48:09.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:09 smithi008 ceph-mon[40743]: pgmap v746: 148 pgs: 148 active+clean; 5.8 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 303 B/s rd, 0 B/s wr, 616 op/s; 14 MiB/s, 225 objects/s recovering 2024-02-10T07:48:09.846 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:09.878 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:09.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:09.966 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-10T07:48:09.980 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:10.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:10.054 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:10.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:10.155 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:10.201 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:10.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:10.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:10.303 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:10.349 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:10.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:10.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:10.450 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:10.466 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:48:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:10.490 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:10.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:10.591 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:10.633 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:10.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:10.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:10.735 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:10.763 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:10.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:10.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:10.865 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:10.897 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:10.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:10.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:11.000 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:11.046 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:11.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:11.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:11.148 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:11.181 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:11.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:11.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:11.283 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:11.323 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:11.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:11.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:11.401 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:48:11.425 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:11.459 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:11.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:11.561 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:11.600 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:11.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:11.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:11.656 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:11 smithi170 ceph-mon[33394]: pgmap v747: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 895 B/s rd, 0 B/s wr, 1.21k op/s; 12 MiB/s, 190 objects/s recovering 2024-02-10T07:48:11.656 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:11 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4215035025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:48:11.656 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:11 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:48:11.702 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:11.733 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:11.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:11.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:11.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:11 smithi008 ceph-mon[29816]: pgmap v747: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 895 B/s rd, 0 B/s wr, 1.21k op/s; 12 MiB/s, 190 objects/s recovering 2024-02-10T07:48:11.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:11 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4215035025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:48:11.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:11 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:48:11.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:11 smithi008 ceph-mon[40743]: pgmap v747: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 895 B/s rd, 0 B/s wr, 1.21k op/s; 12 MiB/s, 190 objects/s recovering 2024-02-10T07:48:11.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:11 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4215035025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:48:11.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:11 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:48:11.835 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:11.873 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:11.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:11.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:11.975 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:12.038 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:12.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:12.140 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:12.184 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:12.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:12.285 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:12.334 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:12.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:12.346 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:48:12.437 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:12.464 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 4, 6, 7, 5, 3] 2024-02-10T07:48:12.464 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:48:12.464 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-02-10T07:48:12.464 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-10T07:48:12.464 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-10T07:48:12.464 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-02-10T07:48:12.473 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:12.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:12.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:12.575 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:12.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:12 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:48:12.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:12 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:48:12.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:12 smithi008 ceph-mon[29816]: osdmap e358: 8 total, 8 up, 7 in 2024-02-10T07:48:12.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:12 smithi008 ceph-mon[29816]: 4.1b scrub starts 2024-02-10T07:48:12.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:12 smithi008 ceph-mon[29816]: 4.1b scrub ok 2024-02-10T07:48:12.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:12 smithi008 ceph-mon[29816]: osdmap e359: 8 total, 8 up, 7 in 2024-02-10T07:48:12.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:12 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:48:12.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:12 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:48:12.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:12 smithi008 ceph-mon[40743]: osdmap e358: 8 total, 8 up, 7 in 2024-02-10T07:48:12.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:12 smithi008 ceph-mon[40743]: 4.1b scrub starts 2024-02-10T07:48:12.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:12 smithi008 ceph-mon[40743]: 4.1b scrub ok 2024-02-10T07:48:12.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:12 smithi008 ceph-mon[40743]: osdmap e359: 8 total, 8 up, 7 in 2024-02-10T07:48:12.611 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:12.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:12.682 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:48:12.682 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:48:12.683 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:12 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:48:12.683 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:12 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:48:12.683 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:12 smithi170 ceph-mon[33394]: osdmap e358: 8 total, 8 up, 7 in 2024-02-10T07:48:12.683 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:12 smithi170 ceph-mon[33394]: 4.1b scrub starts 2024-02-10T07:48:12.683 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:12 smithi170 ceph-mon[33394]: 4.1b scrub ok 2024-02-10T07:48:12.683 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:12 smithi170 ceph-mon[33394]: osdmap e359: 8 total, 8 up, 7 in 2024-02-10T07:48:12.684 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:12.714 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:12.741 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:12.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:12.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:12.843 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:12.872 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:12.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:12.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:12.974 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:13.015 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:13.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:13.116 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:13.157 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:13.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:13.257 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:13.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:13.293 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:13.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:13.394 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:13.400 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:13 smithi008 ceph-mon[29816]: 4.8 scrub starts 2024-02-10T07:48:13.424 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:13.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:13.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:13.526 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:13.554 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:13.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:13.656 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:13 smithi170 ceph-mon[33394]: 4.8 scrub starts 2024-02-10T07:48:13.656 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:13 smithi170 ceph-mon[33394]: 4.8 scrub ok 2024-02-10T07:48:13.656 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:13 smithi170 ceph-mon[33394]: pgmap v750: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 854 op/s; 12 MiB/s, 190 objects/s recovering 2024-02-10T07:48:13.656 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:13 smithi170 ceph-mon[33394]: 4.1e scrub starts 2024-02-10T07:48:13.656 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:13 smithi170 ceph-mon[33394]: 4.1e scrub ok 2024-02-10T07:48:13.656 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:13 smithi170 ceph-mon[33394]: 6.b scrub starts 2024-02-10T07:48:13.656 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:13 smithi170 ceph-mon[33394]: 6.b scrub ok 2024-02-10T07:48:13.656 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:13.692 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:13.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:13.789 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-10T07:48:13.794 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:13.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:13 smithi008 ceph-mon[29816]: 4.8 scrub ok 2024-02-10T07:48:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:13 smithi008 ceph-mon[29816]: pgmap v750: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 854 op/s; 12 MiB/s, 190 objects/s recovering 2024-02-10T07:48:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:13 smithi008 ceph-mon[29816]: 4.1e scrub starts 2024-02-10T07:48:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:13 smithi008 ceph-mon[29816]: 4.1e scrub ok 2024-02-10T07:48:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:13 smithi008 ceph-mon[29816]: 6.b scrub starts 2024-02-10T07:48:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:13 smithi008 ceph-mon[29816]: 6.b scrub ok 2024-02-10T07:48:13.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:13 smithi008 ceph-mon[40743]: 4.8 scrub starts 2024-02-10T07:48:13.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:13 smithi008 ceph-mon[40743]: 4.8 scrub ok 2024-02-10T07:48:13.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:13 smithi008 ceph-mon[40743]: pgmap v750: 148 pgs: 148 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 854 op/s; 12 MiB/s, 190 objects/s recovering 2024-02-10T07:48:13.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:13 smithi008 ceph-mon[40743]: 4.1e scrub starts 2024-02-10T07:48:13.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:13 smithi008 ceph-mon[40743]: 4.1e scrub ok 2024-02-10T07:48:13.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:13 smithi008 ceph-mon[40743]: 6.b scrub starts 2024-02-10T07:48:13.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:13 smithi008 ceph-mon[40743]: 6.b scrub ok 2024-02-10T07:48:13.894 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:13.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:13.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:13.899 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:48:13.996 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:14.049 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:48:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:48:14.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:14.058 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:14.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:14.159 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:14.205 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:14.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:14.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:14.307 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:14.333 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:48:14.335 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:14.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:14.437 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:14.479 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:14.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:14.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:14.580 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:14.620 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:14.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:14.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:14.679 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:14.722 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:14.755 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:14.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:14.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:14.782 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:14 smithi008 ceph-mon[29816]: 4.19 scrub starts 2024-02-10T07:48:14.782 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:14 smithi008 ceph-mon[29816]: 4.19 scrub ok 2024-02-10T07:48:14.782 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:14 smithi008 ceph-mon[29816]: 6.1 scrub starts 2024-02-10T07:48:14.782 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:14 smithi008 ceph-mon[29816]: 6.1 scrub ok 2024-02-10T07:48:14.782 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:14 smithi008 ceph-mon[29816]: 6.1e scrub starts 2024-02-10T07:48:14.782 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:14 smithi008 ceph-mon[29816]: 6.1e scrub ok 2024-02-10T07:48:14.782 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:14 smithi008 ceph-mon[40743]: 4.19 scrub starts 2024-02-10T07:48:14.782 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:14 smithi008 ceph-mon[40743]: 4.19 scrub ok 2024-02-10T07:48:14.783 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:14 smithi008 ceph-mon[40743]: 6.1 scrub starts 2024-02-10T07:48:14.783 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:14 smithi008 ceph-mon[40743]: 6.1 scrub ok 2024-02-10T07:48:14.783 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:14 smithi008 ceph-mon[40743]: 6.1e scrub starts 2024-02-10T07:48:14.783 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:14 smithi008 ceph-mon[40743]: 6.1e scrub ok 2024-02-10T07:48:14.857 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:14 smithi170 ceph-mon[33394]: 4.19 scrub starts 2024-02-10T07:48:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:14 smithi170 ceph-mon[33394]: 4.19 scrub ok 2024-02-10T07:48:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:14 smithi170 ceph-mon[33394]: 6.1 scrub starts 2024-02-10T07:48:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:14 smithi170 ceph-mon[33394]: 6.1 scrub ok 2024-02-10T07:48:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:14 smithi170 ceph-mon[33394]: 6.1e scrub starts 2024-02-10T07:48:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:14 smithi170 ceph-mon[33394]: 6.1e scrub ok 2024-02-10T07:48:14.894 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:14.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:14.996 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:15.026 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:15.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:15.128 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:15.170 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:15.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:15.271 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:15.309 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:15.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:15.366 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:48:15.394 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:48:15.411 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:15.427 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:48:15.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:15.445 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:15.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:15.456 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:48:15.482 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:48:15.513 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:48:15.513 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:48:15.538 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:48:15.547 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:15.576 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:15.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:15.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:15.578 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:48:15.580 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:48:15.580 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:48:15.580 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:48:15.580 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:48:15.580 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:48:15.580 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:48:15.580 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:48:15.580 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:48:15.653 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:15 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:15.653 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:15 smithi170 ceph-mon[33394]: pgmap v751: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-02-10T07:48:15.653 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:15 smithi170 ceph-mon[33394]: 5.7 scrub starts 2024-02-10T07:48:15.654 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:15 smithi170 ceph-mon[33394]: 5.7 scrub ok 2024-02-10T07:48:15.654 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:15 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/689228226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:15.678 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:15.723 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:15.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:15.824 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:15.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:15 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:15.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:15 smithi008 ceph-mon[29816]: pgmap v751: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-02-10T07:48:15.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:15 smithi008 ceph-mon[29816]: 5.7 scrub starts 2024-02-10T07:48:15.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:15 smithi008 ceph-mon[29816]: 5.7 scrub ok 2024-02-10T07:48:15.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:15 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/689228226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:15.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:15 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:15.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:15 smithi008 ceph-mon[40743]: pgmap v751: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-02-10T07:48:15.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:15 smithi008 ceph-mon[40743]: 5.7 scrub starts 2024-02-10T07:48:15.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:15 smithi008 ceph-mon[40743]: 5.7 scrub ok 2024-02-10T07:48:15.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:15 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/689228226' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:15.863 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:15.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:15.964 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:15.995 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:15.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:16.096 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:16.144 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:16.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:16.245 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:16.281 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:16.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:16.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:16.383 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:16.414 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:16.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:16.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:16.516 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:16.543 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:16.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:16.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:16.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[29816]: 1.0 scrub starts 2024-02-10T07:48:16.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[29816]: 1.0 scrub ok 2024-02-10T07:48:16.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[29816]: 2.1 scrub starts 2024-02-10T07:48:16.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[29816]: 2.1 scrub ok 2024-02-10T07:48:16.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[29816]: 4.c scrub starts 2024-02-10T07:48:16.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[29816]: 4.c scrub ok 2024-02-10T07:48:16.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[29816]: 5.1b scrub starts 2024-02-10T07:48:16.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[29816]: 5.1b scrub ok 2024-02-10T07:48:16.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[40743]: 1.0 scrub starts 2024-02-10T07:48:16.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[40743]: 1.0 scrub ok 2024-02-10T07:48:16.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[40743]: 2.1 scrub starts 2024-02-10T07:48:16.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[40743]: 2.1 scrub ok 2024-02-10T07:48:16.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[40743]: 4.c scrub starts 2024-02-10T07:48:16.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[40743]: 4.c scrub ok 2024-02-10T07:48:16.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[40743]: 5.1b scrub starts 2024-02-10T07:48:16.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:16 smithi008 ceph-mon[40743]: 5.1b scrub ok 2024-02-10T07:48:16.645 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:16.689 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:16.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:16.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:16.734 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:48:16.734 INFO:teuthology.orchestra.run.smithi170.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-10T07:48:16.734 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:48:16.734 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:16 smithi170 ceph-mon[33394]: 1.0 scrub starts 2024-02-10T07:48:16.734 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:16 smithi170 ceph-mon[33394]: 1.0 scrub ok 2024-02-10T07:48:16.734 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:16 smithi170 ceph-mon[33394]: 2.1 scrub starts 2024-02-10T07:48:16.734 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:16 smithi170 ceph-mon[33394]: 2.1 scrub ok 2024-02-10T07:48:16.734 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:16 smithi170 ceph-mon[33394]: 4.c scrub starts 2024-02-10T07:48:16.735 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:16 smithi170 ceph-mon[33394]: 4.c scrub ok 2024-02-10T07:48:16.735 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:16 smithi170 ceph-mon[33394]: 5.1b scrub starts 2024-02-10T07:48:16.735 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:16 smithi170 ceph-mon[33394]: 5.1b scrub ok 2024-02-10T07:48:16.790 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:16.833 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:16.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:16.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:16.935 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:16.978 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:16.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:16.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:17.080 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:17.108 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:17.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:17.210 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:17.254 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:17.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:17.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:17.355 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:17.404 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:17.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:17.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:17.506 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:17.544 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:17.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:17.645 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:17.674 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:17 smithi008 ceph-mon[29816]: pgmap v752: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-02-10T07:48:17.675 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:17 smithi008 ceph-mon[29816]: 4.f scrub starts 2024-02-10T07:48:17.675 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:17 smithi008 ceph-mon[29816]: 4.f scrub ok 2024-02-10T07:48:17.675 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:17 smithi008 ceph-mon[29816]: 3.16 scrub starts 2024-02-10T07:48:17.675 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:17 smithi008 ceph-mon[29816]: 3.16 scrub ok 2024-02-10T07:48:17.675 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:17 smithi008 ceph-mon[29816]: 3.2 scrub starts 2024-02-10T07:48:17.675 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:17 smithi008 ceph-mon[29816]: 3.2 scrub ok 2024-02-10T07:48:17.675 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:17 smithi008 ceph-mon[40743]: pgmap v752: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-02-10T07:48:17.675 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:17 smithi008 ceph-mon[40743]: 4.f scrub starts 2024-02-10T07:48:17.675 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:17 smithi008 ceph-mon[40743]: 4.f scrub ok 2024-02-10T07:48:17.675 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:17 smithi008 ceph-mon[40743]: 3.16 scrub starts 2024-02-10T07:48:17.675 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:17 smithi008 ceph-mon[40743]: 3.16 scrub ok 2024-02-10T07:48:17.676 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:17 smithi008 ceph-mon[40743]: 3.2 scrub starts 2024-02-10T07:48:17.676 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:17 smithi008 ceph-mon[40743]: 3.2 scrub ok 2024-02-10T07:48:17.684 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:17.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:17.693 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:17 smithi170 ceph-mon[33394]: pgmap v752: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-02-10T07:48:17.693 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:17 smithi170 ceph-mon[33394]: 4.f scrub starts 2024-02-10T07:48:17.693 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:17 smithi170 ceph-mon[33394]: 4.f scrub ok 2024-02-10T07:48:17.693 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:17 smithi170 ceph-mon[33394]: 3.16 scrub starts 2024-02-10T07:48:17.693 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:17 smithi170 ceph-mon[33394]: 3.16 scrub ok 2024-02-10T07:48:17.693 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:17 smithi170 ceph-mon[33394]: 3.2 scrub starts 2024-02-10T07:48:17.693 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:17 smithi170 ceph-mon[33394]: 3.2 scrub ok 2024-02-10T07:48:17.786 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:17.827 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:17.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:17.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:17.958 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:17.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:18.060 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:18.099 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:18.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:18.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:18.236 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:18.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:18.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:18.338 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:18.367 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:18.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:18.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:18.469 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:18.502 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:18.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:18.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:18.604 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:18.628 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:18.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:18.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:18.730 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:18.759 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:18.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:18.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:18 smithi008 ceph-mon[29816]: 3.1f scrub starts 2024-02-10T07:48:18.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:18 smithi008 ceph-mon[29816]: 3.1f scrub ok 2024-02-10T07:48:18.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:18 smithi008 ceph-mon[29816]: 5.17 scrub starts 2024-02-10T07:48:18.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:18 smithi008 ceph-mon[29816]: 5.17 scrub ok 2024-02-10T07:48:18.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:18.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:48:18.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:18 smithi008 ceph-mon[40743]: 3.1f scrub starts 2024-02-10T07:48:18.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:18 smithi008 ceph-mon[40743]: 3.1f scrub ok 2024-02-10T07:48:18.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:18 smithi008 ceph-mon[40743]: 5.17 scrub starts 2024-02-10T07:48:18.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:18 smithi008 ceph-mon[40743]: 5.17 scrub ok 2024-02-10T07:48:18.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:18.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:48:18.862 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:18.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:18 smithi170 ceph-mon[33394]: 3.1f scrub starts 2024-02-10T07:48:18.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:18 smithi170 ceph-mon[33394]: 3.1f scrub ok 2024-02-10T07:48:18.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:18 smithi170 ceph-mon[33394]: 5.17 scrub starts 2024-02-10T07:48:18.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:18 smithi170 ceph-mon[33394]: 5.17 scrub ok 2024-02-10T07:48:18.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:18.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:48:18.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:18.903 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:18.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:19.004 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:19.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:19.028 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:19.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:19.048 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:48:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:48:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:48:19.129 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:19.169 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:19.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:19.271 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:19.306 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:19.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:19.408 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:19.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:19.448 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:19.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:19.549 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:19.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:19 smithi008 ceph-mon[29816]: 5.1a scrub starts 2024-02-10T07:48:19.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:19 smithi008 ceph-mon[29816]: pgmap v753: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s 2024-02-10T07:48:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:19 smithi008 ceph-mon[29816]: 5.1a scrub ok 2024-02-10T07:48:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:19 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:48:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:19 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:48:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:19 smithi008 ceph-mon[29816]: 6.a scrub starts 2024-02-10T07:48:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:19 smithi008 ceph-mon[29816]: 6.a scrub ok 2024-02-10T07:48:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:19 smithi008 ceph-mon[40743]: 5.1a scrub starts 2024-02-10T07:48:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:19 smithi008 ceph-mon[40743]: pgmap v753: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s 2024-02-10T07:48:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:19 smithi008 ceph-mon[40743]: 5.1a scrub ok 2024-02-10T07:48:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:19 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:48:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:19 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:48:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:19 smithi008 ceph-mon[40743]: 6.a scrub starts 2024-02-10T07:48:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:19 smithi008 ceph-mon[40743]: 6.a scrub ok 2024-02-10T07:48:19.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:19.584 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:19.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:19 smithi170 ceph-mon[33394]: 5.1a scrub starts 2024-02-10T07:48:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:19 smithi170 ceph-mon[33394]: pgmap v753: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s 2024-02-10T07:48:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:19 smithi170 ceph-mon[33394]: 5.1a scrub ok 2024-02-10T07:48:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:19 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:48:19.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:19 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:48:19.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:19 smithi170 ceph-mon[33394]: 6.a scrub starts 2024-02-10T07:48:19.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:19 smithi170 ceph-mon[33394]: 6.a scrub ok 2024-02-10T07:48:19.685 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:19.723 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:19.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:19.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:19.825 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:19.856 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:19.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:19.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:19.958 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:19.991 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:19.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:20.093 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:20.124 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:20.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:20.226 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:20.262 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:20.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:20.364 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:20.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:20.396 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:20.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:20.454 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:48:20.497 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:20.522 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:48:20.538 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:20.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:20.550 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[29816]: 6.14 scrub starts 2024-02-10T07:48:20.550 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[29816]: 6.14 scrub ok 2024-02-10T07:48:20.550 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[29816]: 5.12 scrub starts 2024-02-10T07:48:20.550 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[29816]: 5.12 scrub ok 2024-02-10T07:48:20.550 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[29816]: osdmap e360: 8 total, 8 up, 7 in 2024-02-10T07:48:20.550 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3138887221' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:48:20.550 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:48:20.550 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[29816]: 3.a scrub starts 2024-02-10T07:48:20.550 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[29816]: 3.a scrub ok 2024-02-10T07:48:20.551 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[40743]: 6.14 scrub starts 2024-02-10T07:48:20.551 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[40743]: 6.14 scrub ok 2024-02-10T07:48:20.551 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[40743]: 5.12 scrub starts 2024-02-10T07:48:20.551 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[40743]: 5.12 scrub ok 2024-02-10T07:48:20.551 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[40743]: osdmap e360: 8 total, 8 up, 7 in 2024-02-10T07:48:20.551 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3138887221' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:48:20.551 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:48:20.551 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[40743]: 3.a scrub starts 2024-02-10T07:48:20.551 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:20 smithi008 ceph-mon[40743]: 3.a scrub ok 2024-02-10T07:48:20.640 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:20.670 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:20.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:20.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:20.747 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:20 smithi170 ceph-mon[33394]: 6.14 scrub starts 2024-02-10T07:48:20.747 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:20 smithi170 ceph-mon[33394]: 6.14 scrub ok 2024-02-10T07:48:20.748 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:20 smithi170 ceph-mon[33394]: 5.12 scrub starts 2024-02-10T07:48:20.748 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:20 smithi170 ceph-mon[33394]: 5.12 scrub ok 2024-02-10T07:48:20.748 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:20 smithi170 ceph-mon[33394]: osdmap e360: 8 total, 8 up, 7 in 2024-02-10T07:48:20.748 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:20 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3138887221' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:48:20.748 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:20 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:48:20.748 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:20 smithi170 ceph-mon[33394]: 3.a scrub starts 2024-02-10T07:48:20.748 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:20 smithi170 ceph-mon[33394]: 3.a scrub ok 2024-02-10T07:48:20.772 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:20.813 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:20.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:20.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:20.917 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:20.966 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:20.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:20.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:21.068 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:21.115 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:21.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:21.217 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:21.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:21.248 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:21.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:21.324 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:48:21.324 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:48:21.324 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.324 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:141968 10.c 10:3b0e6bee:::benchmark_data_smithi008_161452_object33507:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-02-10T07:48:21.324 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:21.313645+0000", 2024-02-10T07:48:21.324 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.0095904990000000006, 2024-02-10T07:48:21.324 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0096243809999999996, 2024-02-10T07:48:21.324 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:48:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:48:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:48:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:48:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:48:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 141968 2024-02-10T07:48:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:48:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:48:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.313645+0000", 2024-02-10T07:48:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.325 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:48:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.313645+0000", 2024-02-10T07:48:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:48:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.313646+0000", 2024-02-10T07:48:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1799999999999995e-07 2024-02-10T07:48:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:48:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.313649+0000", 2024-02-10T07:48:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2969999999999999e-06 2024-02-10T07:48:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.326 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.313656+0000", 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2089999999999999e-06 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.313662+0000", 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2369999999999997e-06 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.313740+0000", 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7383000000000001e-05 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.327 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:48:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.313959+0000", 2024-02-10T07:48:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000219197 2024-02-10T07:48:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,4", 2024-02-10T07:48:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.314089+0000", 2024-02-10T07:48:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012971099999999999 2024-02-10T07:48:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:48:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.318869+0000", 2024-02-10T07:48:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0047800400000000002 2024-02-10T07:48:21.328 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.321292+0000", 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0024230900000000001 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:141976 10.9 10:9df8a32e:::benchmark_data_smithi008_161452_object33515:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e361)", 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:21.322003+0000", 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.0012325579999999999, 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001357818, 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:48:21.329 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 141976 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.322003+0000", 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.322003+0000", 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:21.330 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.322005+0000", 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2089999999999999e-06 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.322009+0000", 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.048e-06 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.322018+0000", 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.2399999999999996e-06 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.331 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.322025+0000", 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.156e-06 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.322628+0000", 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00060307199999999996 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.322729+0000", 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000101403 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:21.332 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:21.333 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:48:21.333 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:21.322841+0000", 2024-02-10T07:48:21.333 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011168899999999999 2024-02-10T07:48:21.333 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:21.333 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:48:21.333 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:21.333 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:21.333 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2024-02-10T07:48:21.333 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 2 2024-02-10T07:48:21.333 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:48:21.348 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:21.381 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:21.392 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:21.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:21.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:21.494 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:21.508 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:21 smithi008 ceph-mon[29816]: pgmap v755: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-02-10T07:48:21.508 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:21 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:48:21.508 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:21 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:48:21.508 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:21 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:48:21.508 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:21 smithi008 ceph-mon[29816]: osdmap e361: 8 total, 8 up, 7 in 2024-02-10T07:48:21.509 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:21 smithi008 ceph-mon[40743]: pgmap v755: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-02-10T07:48:21.509 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:21 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:48:21.509 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:21 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:48:21.509 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:21 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:48:21.509 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:21 smithi008 ceph-mon[40743]: osdmap e361: 8 total, 8 up, 7 in 2024-02-10T07:48:21.523 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:21.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:21.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:21.624 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:21.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:21.668 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:21.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:21.769 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:21.803 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:21.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:21.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:21.830 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:21 smithi170 ceph-mon[33394]: pgmap v755: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-02-10T07:48:21.830 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:21 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:48:21.830 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:21 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:48:21.830 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:21 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:48:21.830 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:21 smithi170 ceph-mon[33394]: osdmap e361: 8 total, 8 up, 7 in 2024-02-10T07:48:21.906 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:21.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:21.945 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:21.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:22.047 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:22.076 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:22.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:22.178 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:22.213 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:22.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:22.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:22.315 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:22.341 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:22.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:22.345 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-10T07:48:22.442 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:22.469 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:22.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:22.571 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:22.598 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:22.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:22.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:22.700 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:22.742 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 4, 6, 7, 5, 3] 2024-02-10T07:48:22.742 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:48:22.742 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-02-10T07:48:22.742 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-02-10T07:48:22.742 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:48:22.755 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:22.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:22.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:22.791 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:48:22.857 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:22.907 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:22.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:22.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:22.975 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:22 smithi008 ceph-mon[29816]: 5.b scrub starts 2024-02-10T07:48:22.975 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:22 smithi008 ceph-mon[29816]: 5.b scrub ok 2024-02-10T07:48:22.975 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:22 smithi008 ceph-mon[29816]: 4.9 scrub starts 2024-02-10T07:48:22.975 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:22 smithi008 ceph-mon[29816]: 4.9 scrub ok 2024-02-10T07:48:22.975 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:22 smithi008 ceph-mon[29816]: pgmap v757: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 984 op/s 2024-02-10T07:48:23.009 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:23.036 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:23.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:23.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:23.131 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:22 smithi170 ceph-mon[33394]: 5.b scrub starts 2024-02-10T07:48:23.131 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:22 smithi170 ceph-mon[33394]: 5.b scrub ok 2024-02-10T07:48:23.131 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:22 smithi170 ceph-mon[33394]: 4.9 scrub starts 2024-02-10T07:48:23.131 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:22 smithi170 ceph-mon[33394]: 4.9 scrub ok 2024-02-10T07:48:23.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:22 smithi170 ceph-mon[33394]: pgmap v757: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 984 op/s 2024-02-10T07:48:23.138 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:23.166 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:23.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:23.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:23.268 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:23.286 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:48:23.310 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:23.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:23.311 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:22 smithi008 ceph-mon[40743]: 5.b scrub starts 2024-02-10T07:48:23.311 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:22 smithi008 ceph-mon[40743]: 5.b scrub ok 2024-02-10T07:48:23.311 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:22 smithi008 ceph-mon[40743]: 4.9 scrub starts 2024-02-10T07:48:23.312 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:22 smithi008 ceph-mon[40743]: 4.9 scrub ok 2024-02-10T07:48:23.312 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:22 smithi008 ceph-mon[40743]: pgmap v757: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 984 op/s 2024-02-10T07:48:23.411 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:23.452 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:23.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:23.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:23.554 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:23.593 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:23.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:23.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:23.695 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:23.729 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:23.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:23.769 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:23.831 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:23.837 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:48:23.860 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:23.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:23.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:23.865 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:48:23.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:48:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:48:23.890 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:48:23.915 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:48:23.950 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:48:23.961 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:23.982 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:48:23.999 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:24.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:24.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:24.013 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:48:24.048 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:48:24.050 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:48:24.051 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:48:24.051 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:48:24.051 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:48:24.051 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:48:24.051 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:48:24.051 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:48:24.051 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:48:24.101 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:24.137 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:24.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:24.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:24.167 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:23 smithi008 ceph-mon[29816]: 6.3 deep-scrub starts 2024-02-10T07:48:24.167 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:23 smithi008 ceph-mon[29816]: 6.3 deep-scrub ok 2024-02-10T07:48:24.167 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:23 smithi008 ceph-mon[29816]: 5.e scrub starts 2024-02-10T07:48:24.167 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:23 smithi008 ceph-mon[29816]: 5.e scrub ok 2024-02-10T07:48:24.168 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:23 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3820589082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:24.168 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:23 smithi008 ceph-mon[40743]: 6.3 deep-scrub starts 2024-02-10T07:48:24.168 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:23 smithi008 ceph-mon[40743]: 6.3 deep-scrub ok 2024-02-10T07:48:24.168 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:23 smithi008 ceph-mon[40743]: 5.e scrub starts 2024-02-10T07:48:24.168 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:23 smithi008 ceph-mon[40743]: 5.e scrub ok 2024-02-10T07:48:24.168 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:23 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3820589082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:24.238 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:24.273 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:24.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:24.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:24.351 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:23 smithi170 ceph-mon[33394]: 6.3 deep-scrub starts 2024-02-10T07:48:24.352 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:23 smithi170 ceph-mon[33394]: 6.3 deep-scrub ok 2024-02-10T07:48:24.352 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:23 smithi170 ceph-mon[33394]: 5.e scrub starts 2024-02-10T07:48:24.352 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:23 smithi170 ceph-mon[33394]: 5.e scrub ok 2024-02-10T07:48:24.352 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:23 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3820589082' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:24.375 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:24.404 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:24.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:24.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:24.505 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:24.533 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:24.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:24.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:24.635 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:24.667 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:24.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:24.769 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:24.795 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:24.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:24.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:24.896 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:24.925 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:24.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:25.027 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:25.054 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:25.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:25.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:25.120 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:48:25.120 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:48:25.156 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:25.184 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:25.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:25.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:25.286 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:25.317 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:25.317 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:25.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:25.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:24 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:24 smithi008 ceph-mon[29816]: pgmap v758: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-10T07:48:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:24 smithi008 ceph-mon[29816]: 5.10 scrub starts 2024-02-10T07:48:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:24 smithi008 ceph-mon[29816]: 5.10 scrub ok 2024-02-10T07:48:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:24 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:24 smithi008 ceph-mon[40743]: pgmap v758: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-10T07:48:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:24 smithi008 ceph-mon[40743]: 5.10 scrub starts 2024-02-10T07:48:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:24 smithi008 ceph-mon[40743]: 5.10 scrub ok 2024-02-10T07:48:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:24 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:24 smithi170 ceph-mon[33394]: pgmap v758: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-10T07:48:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:24 smithi170 ceph-mon[33394]: 5.10 scrub starts 2024-02-10T07:48:25.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:24 smithi170 ceph-mon[33394]: 5.10 scrub ok 2024-02-10T07:48:25.419 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:25.447 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:25.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:25.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:25.549 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:25.579 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:25.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:25.681 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:25.712 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:25.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:25.814 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:25.865 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:25.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:25.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:25.966 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:26.015 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:26.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:26.117 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:26.142 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-10T07:48:26.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:26.145 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:26.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:26.203 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:25 smithi008 ceph-mon[29816]: 5.15 scrub starts 2024-02-10T07:48:26.204 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:25 smithi008 ceph-mon[29816]: 5.15 scrub ok 2024-02-10T07:48:26.204 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:25 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:48:26.204 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:25 smithi008 ceph-mon[40743]: 5.15 scrub starts 2024-02-10T07:48:26.204 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:25 smithi008 ceph-mon[40743]: 5.15 scrub ok 2024-02-10T07:48:26.204 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:25 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:48:26.246 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:26.273 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:26.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:26.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:26.375 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:25 smithi170 ceph-mon[33394]: 5.15 scrub starts 2024-02-10T07:48:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:25 smithi170 ceph-mon[33394]: 5.15 scrub ok 2024-02-10T07:48:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:25 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:48:26.408 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:26.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:26.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:26.510 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:26.546 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:26.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:26.630 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:48:26.647 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:26.681 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:26.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:26.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:26.783 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:26.817 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:26.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:26.867 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:48:26.870 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:48:26.920 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:26.967 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:26.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:27.068 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:27.108 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:27.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:27.209 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:27.244 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:27.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:27.345 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:27.373 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:27.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:27.474 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:27.504 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:27.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:27.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:27.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:27 smithi008 ceph-mon[40743]: pgmap v759: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-02-10T07:48:27.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:27 smithi008 ceph-mon[40743]: 6.4 scrub starts 2024-02-10T07:48:27.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:27 smithi008 ceph-mon[40743]: 6.4 scrub ok 2024-02-10T07:48:27.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:27 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:27.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:27 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:27.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:27 smithi008 ceph-mon[29816]: pgmap v759: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-02-10T07:48:27.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:27 smithi008 ceph-mon[29816]: 6.4 scrub starts 2024-02-10T07:48:27.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:27 smithi008 ceph-mon[29816]: 6.4 scrub ok 2024-02-10T07:48:27.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:27 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:27.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:27 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:27.606 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:27.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:27.643 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:27.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:27.744 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:27.773 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:27.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:27.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:27.817 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:27 smithi170 ceph-mon[33394]: pgmap v759: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-02-10T07:48:27.817 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:27 smithi170 ceph-mon[33394]: 6.4 scrub starts 2024-02-10T07:48:27.817 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:27 smithi170 ceph-mon[33394]: 6.4 scrub ok 2024-02-10T07:48:27.817 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:27 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:27.818 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:27 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:27.875 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:27.903 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:27.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:27.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:27.930 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:48:28.005 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:28.035 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:28.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:28.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:28.137 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:28.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:28.185 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:28.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:28.286 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:28.320 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:28.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:28.422 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:28.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:28.453 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:28.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:28.554 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:28.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:28 smithi008 ceph-mon[29816]: from='client.25691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:48:28.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:28 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:48:28.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:28 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:48:28.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:28 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:28 smithi008 ceph-mon[40743]: from='client.25691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:48:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:28 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:48:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:28 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:48:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:28 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:28.588 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:28.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:28.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:28 smithi170 ceph-mon[33394]: from='client.25691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:48:28.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:28 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:48:28.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:28 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:48:28.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:28 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:48:28.689 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:28.720 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:28.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:28.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:28.822 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:28.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:28.865 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:28.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:28.966 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:28.995 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:28.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:29.047 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:48:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:48:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:48:29.092 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":759,"stamp":"2024-02-10T07:48:26.069710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4743960948,"num_objects":72577,"num_object_clones":0,"num_object_copies":217731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72577,"num_whiteouts":0,"num_read":2334,"num_read_kb":2180,"num_write":145440,"num_write_kb":6943774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17132,"num_bytes_recovered":1112676879,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67642,"ondisk_log_size":67642,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17745220,"kb_used_data":14081220,"kb_used_omap":17,"kb_used_meta":3663598,"kb_avail":638470844,"statfs":{"total":671965249536,"available":653794144256,"internally_reserved":0,"allocated":14419169280,"data_stored":14415000170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18039,"internal_metadata":3751524745},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,6,3,11,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-774373376,"num_objects":-11816,"num_object_clones":0,"num_object_copies":-35448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11816,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11816,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001739"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":649,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452225+0000","last_change":"2024-02-10T07:45:43.013900+0000","last_active":"2024-02-10T07:48:20.452225+0000","last_peered":"2024-02-10T07:48:20.452225+0000","last_clean":"2024-02-10T07:48:20.452225+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:48:20.452225+0000","last_undegraded":"2024-02-10T07:48:20.452225+0000","last_fullsized":"2024-02-10T07:48:20.452225+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:43.013774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:45:43.013774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":659,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.467247+0000","last_change":"2024-02-10T07:45:53.422092+0000","last_active":"2024-02-10T07:48:20.467247+0000","last_peered":"2024-02-10T07:48:20.467247+0000","last_clean":"2024-02-10T07:48:20.467247+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:48:20.467247+0000","last_undegraded":"2024-02-10T07:48:20.467247+0000","last_fullsized":"2024-02-10T07:48:20.467247+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:53.421994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:53.421994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":686,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451860+0000","last_change":"2024-02-10T07:48:17.198789+0000","last_active":"2024-02-10T07:48:20.451860+0000","last_peered":"2024-02-10T07:48:20.451860+0000","last_clean":"2024-02-10T07:48:20.451860+0000","last_became_active":"2024-02-10T07:46:10.718945+0000","last_became_peered":"2024-02-10T07:46:10.718945+0000","last_unstale":"2024-02-10T07:48:20.451860+0000","last_undegraded":"2024-02-10T07:48:20.451860+0000","last_fullsized":"2024-02-10T07:48:20.451860+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:17.198701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:17.198701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:18.440746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"287'19","reported_seq":717,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.450359+0000","last_change":"2024-02-10T07:46:37.008978+0000","last_active":"2024-02-10T07:48:20.450359+0000","last_peered":"2024-02-10T07:48:20.450359+0000","last_clean":"2024-02-10T07:48:20.450359+0000","last_became_active":"2024-02-10T07:46:10.717475+0000","last_became_peered":"2024-02-10T07:46:10.717475+0000","last_unstale":"2024-02-10T07:48:20.450359+0000","last_undegraded":"2024-02-10T07:48:20.450359+0000","last_fullsized":"2024-02-10T07:48:20.450359+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'19","last_scrub_stamp":"2024-02-10T07:46:37.008841+0000","last_deep_scrub":"287'19","last_deep_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.008841+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":151,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.456639+0000","last_change":"2024-02-10T07:47:28.300203+0000","last_active":"2024-02-10T07:48:20.456639+0000","last_peered":"2024-02-10T07:48:20.456639+0000","last_clean":"2024-02-10T07:48:20.456639+0000","last_became_active":"2024-02-10T07:46:40.116294+0000","last_became_peered":"2024-02-10T07:46:40.116294+0000","last_unstale":"2024-02-10T07:48:20.456639+0000","last_undegraded":"2024-02-10T07:48:20.456639+0000","last_fullsized":"2024-02-10T07:48:20.456639+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:28.300090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:28.300090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:31.446577+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":652,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.450261+0000","last_change":"2024-02-10T07:45:55.951770+0000","last_active":"2024-02-10T07:48:20.450261+0000","last_peered":"2024-02-10T07:48:20.450261+0000","last_clean":"2024-02-10T07:48:20.450261+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:48:20.450261+0000","last_undegraded":"2024-02-10T07:48:20.450261+0000","last_fullsized":"2024-02-10T07:48:20.450261+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:55.951689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:55.951689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":155,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454870+0000","last_change":"2024-02-10T07:46:41.128244+0000","last_active":"2024-02-10T07:48:20.454870+0000","last_peered":"2024-02-10T07:48:20.454870+0000","last_clean":"2024-02-10T07:48:20.454870+0000","last_became_active":"2024-02-10T07:46:41.127905+0000","last_became_peered":"2024-02-10T07:46:41.127905+0000","last_unstale":"2024-02-10T07:48:20.454870+0000","last_undegraded":"2024-02-10T07:48:20.454870+0000","last_fullsized":"2024-02-10T07:48:20.454870+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:31.386263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:46:31.386263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"250'27","reported_seq":602,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451837+0000","last_change":"2024-02-10T07:48:13.230990+0000","last_active":"2024-02-10T07:48:20.451837+0000","last_peered":"2024-02-10T07:48:20.451837+0000","last_clean":"2024-02-10T07:48:20.451837+0000","last_became_active":"2024-02-10T07:46:41.117335+0000","last_became_peered":"2024-02-10T07:46:41.117335+0000","last_unstale":"2024-02-10T07:48:20.451837+0000","last_undegraded":"2024-02-10T07:48:20.451837+0000","last_fullsized":"2024-02-10T07:48:20.451837+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:48:13.230908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.230908+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:38.278534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":648,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451633+0000","last_change":"2024-02-10T07:46:27.392670+0000","last_active":"2024-02-10T07:48:20.451633+0000","last_peered":"2024-02-10T07:48:20.451633+0000","last_clean":"2024-02-10T07:48:20.451633+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:48:20.451633+0000","last_undegraded":"2024-02-10T07:48:20.451633+0000","last_fullsized":"2024-02-10T07:48:20.451633+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:27.392544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:46:27.392544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":621,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.459851+0000","last_change":"2024-02-10T07:48:15.696091+0000","last_active":"2024-02-10T07:48:20.459851+0000","last_peered":"2024-02-10T07:48:20.459851+0000","last_clean":"2024-02-10T07:48:20.459851+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:48:20.459851+0000","last_undegraded":"2024-02-10T07:48:20.459851+0000","last_fullsized":"2024-02-10T07:48:20.459851+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:15.695954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.695954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:37.080036+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":190,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451965+0000","last_change":"2024-02-10T07:48:02.333301+0000","last_active":"2024-02-10T07:48:20.451965+0000","last_peered":"2024-02-10T07:48:20.451965+0000","last_clean":"2024-02-10T07:48:20.451965+0000","last_became_active":"2024-02-10T07:48:02.332960+0000","last_became_peered":"2024-02-10T07:48:02.332960+0000","last_unstale":"2024-02-10T07:48:20.451965+0000","last_undegraded":"2024-02-10T07:48:20.451965+0000","last_fullsized":"2024-02-10T07:48:20.451965+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:45:09.404822+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:34:49.349360+0000","last_clean_scrub_stamp":"2024-02-10T07:45:09.404822+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"250'15","reported_seq":682,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452479+0000","last_change":"2024-02-10T07:47:49.553645+0000","last_active":"2024-02-10T07:48:20.452479+0000","last_peered":"2024-02-10T07:48:20.452479+0000","last_clean":"2024-02-10T07:48:20.452479+0000","last_became_active":"2024-02-10T07:47:47.965930+0000","last_became_peered":"2024-02-10T07:47:47.965930+0000","last_unstale":"2024-02-10T07:48:20.452479+0000","last_undegraded":"2024-02-10T07:48:20.452479+0000","last_fullsized":"2024-02-10T07:48:20.452479+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:49.553517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.553517+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:17.619792+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":555,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.467426+0000","last_change":"2024-02-10T07:48:02.367573+0000","last_active":"2024-02-10T07:48:20.467426+0000","last_peered":"2024-02-10T07:48:20.467426+0000","last_clean":"2024-02-10T07:48:20.467426+0000","last_became_active":"2024-02-10T07:48:02.367257+0000","last_became_peered":"2024-02-10T07:48:02.367257+0000","last_unstale":"2024-02-10T07:48:20.467426+0000","last_undegraded":"2024-02-10T07:48:20.467426+0000","last_fullsized":"2024-02-10T07:48:20.467426+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:41.604664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:41.604664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:46.589237+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":660,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.456494+0000","last_change":"2024-02-10T07:48:18.069773+0000","last_active":"2024-02-10T07:48:20.456494+0000","last_peered":"2024-02-10T07:48:20.456494+0000","last_clean":"2024-02-10T07:48:20.456494+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:48:20.456494+0000","last_undegraded":"2024-02-10T07:48:20.456494+0000","last_fullsized":"2024-02-10T07:48:20.456494+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:18.069641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:18.069641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:31.062588+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":178,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452623+0000","last_change":"2024-02-10T07:48:02.330152+0000","last_active":"2024-02-10T07:48:20.452623+0000","last_peered":"2024-02-10T07:48:20.452623+0000","last_clean":"2024-02-10T07:48:20.452623+0000","last_became_active":"2024-02-10T07:48:02.329859+0000","last_became_peered":"2024-02-10T07:48:02.329859+0000","last_unstale":"2024-02-10T07:48:20.452623+0000","last_undegraded":"2024-02-10T07:48:20.452623+0000","last_fullsized":"2024-02-10T07:48:20.452623+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:12.464185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.473686+0000","last_clean_scrub_stamp":"2024-02-10T07:45:12.464185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"250'7","reported_seq":670,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454191+0000","last_change":"2024-02-10T07:48:11.611397+0000","last_active":"2024-02-10T07:48:20.454191+0000","last_peered":"2024-02-10T07:48:20.454191+0000","last_clean":"2024-02-10T07:48:20.454191+0000","last_became_active":"2024-02-10T07:47:47.966093+0000","last_became_peered":"2024-02-10T07:47:47.966093+0000","last_unstale":"2024-02-10T07:48:20.454191+0000","last_undegraded":"2024-02-10T07:48:20.454191+0000","last_fullsized":"2024-02-10T07:48:20.454191+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7","last_scrub_stamp":"2024-02-10T07:48:11.608499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.608499+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:35.908442+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":659,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452246+0000","last_change":"2024-02-10T07:48:13.621135+0000","last_active":"2024-02-10T07:48:20.452246+0000","last_peered":"2024-02-10T07:48:20.452246+0000","last_clean":"2024-02-10T07:48:20.452246+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:48:20.452246+0000","last_undegraded":"2024-02-10T07:48:20.452246+0000","last_fullsized":"2024-02-10T07:48:20.452246+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:13.620938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.620938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:24.503618+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":658,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.468228+0000","last_change":"2024-02-10T07:46:32.360289+0000","last_active":"2024-02-10T07:48:20.468228+0000","last_peered":"2024-02-10T07:48:20.468228+0000","last_clean":"2024-02-10T07:48:20.468228+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:48:20.468228+0000","last_undegraded":"2024-02-10T07:48:20.468228+0000","last_fullsized":"2024-02-10T07:48:20.468228+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:32.360208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:32.360208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":660,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454142+0000","last_change":"2024-02-10T07:45:56.038392+0000","last_active":"2024-02-10T07:48:20.454142+0000","last_peered":"2024-02-10T07:48:20.454142+0000","last_clean":"2024-02-10T07:48:20.454142+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:48:20.454142+0000","last_undegraded":"2024-02-10T07:48:20.454142+0000","last_fullsized":"2024-02-10T07:48:20.454142+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:56.038313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:45:56.038313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"250'34","reported_seq":607,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.468902+0000","last_change":"2024-02-10T07:46:41.158963+0000","last_active":"2024-02-10T07:48:20.468902+0000","last_peered":"2024-02-10T07:48:20.468902+0000","last_clean":"2024-02-10T07:48:20.468902+0000","last_became_active":"2024-02-10T07:46:41.123507+0000","last_became_peered":"2024-02-10T07:46:41.123507+0000","last_unstale":"2024-02-10T07:48:20.468902+0000","last_undegraded":"2024-02-10T07:48:20.468902+0000","last_fullsized":"2024-02-10T07:48:20.468902+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:46:27.370078+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:46:27.370078+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":340,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.455230+0000","last_change":"2024-02-10T07:47:42.394213+0000","last_active":"2024-02-10T07:48:20.455230+0000","last_peered":"2024-02-10T07:48:20.455230+0000","last_clean":"2024-02-10T07:48:20.455230+0000","last_became_active":"2024-02-10T07:46:40.124797+0000","last_became_peered":"2024-02-10T07:46:40.124797+0000","last_unstale":"2024-02-10T07:48:20.455230+0000","last_undegraded":"2024-02-10T07:48:20.455230+0000","last_fullsized":"2024-02-10T07:48:20.455230+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:42.394111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.394111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:00.351838+0000","scrub_duration":221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":657,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.453076+0000","last_change":"2024-02-10T07:46:41.121554+0000","last_active":"2024-02-10T07:48:20.453076+0000","last_peered":"2024-02-10T07:48:20.453076+0000","last_clean":"2024-02-10T07:48:20.453076+0000","last_became_active":"2024-02-10T07:46:41.121201+0000","last_became_peered":"2024-02-10T07:46:41.121201+0000","last_unstale":"2024-02-10T07:48:20.453076+0000","last_undegraded":"2024-02-10T07:48:20.453076+0000","last_fullsized":"2024-02-10T07:48:20.453076+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:56.206287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_clean_scrub_stamp":"2024-02-10T07:44:56.206287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":666,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452623+0000","last_change":"2024-02-10T07:46:10.717885+0000","last_active":"2024-02-10T07:48:20.452623+0000","last_peered":"2024-02-10T07:48:20.452623+0000","last_clean":"2024-02-10T07:48:20.452623+0000","last_became_active":"2024-02-10T07:46:10.717041+0000","last_became_peered":"2024-02-10T07:46:10.717041+0000","last_unstale":"2024-02-10T07:48:20.452623+0000","last_undegraded":"2024-02-10T07:48:20.452623+0000","last_fullsized":"2024-02-10T07:48:20.452623+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:44.989380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:45:44.989380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"250'18","reported_seq":601,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.467905+0000","last_change":"2024-02-10T07:47:42.354040+0000","last_active":"2024-02-10T07:48:20.467905+0000","last_peered":"2024-02-10T07:48:20.467905+0000","last_clean":"2024-02-10T07:48:20.467905+0000","last_became_active":"2024-02-10T07:46:40.123579+0000","last_became_peered":"2024-02-10T07:46:40.123579+0000","last_unstale":"2024-02-10T07:48:20.467905+0000","last_undegraded":"2024-02-10T07:48:20.467905+0000","last_fullsized":"2024-02-10T07:48:20.467905+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'18","last_scrub_stamp":"2024-02-10T07:47:42.353952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.353952+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:58.016548+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":672,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.457329+0000","last_change":"2024-02-10T07:47:34.074855+0000","last_active":"2024-02-10T07:48:20.457329+0000","last_peered":"2024-02-10T07:48:20.457329+0000","last_clean":"2024-02-10T07:48:20.457329+0000","last_became_active":"2024-02-10T07:46:41.116241+0000","last_became_peered":"2024-02-10T07:46:41.116241+0000","last_unstale":"2024-02-10T07:48:20.457329+0000","last_undegraded":"2024-02-10T07:48:20.457329+0000","last_fullsized":"2024-02-10T07:48:20.457329+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:47:34.074746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:34.074746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:03.392709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":886,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454858+0000","last_change":"2024-02-10T07:47:44.170718+0000","last_active":"2024-02-10T07:48:20.454858+0000","last_peered":"2024-02-10T07:48:20.454858+0000","last_clean":"2024-02-10T07:48:20.454858+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:48:20.454858+0000","last_undegraded":"2024-02-10T07:48:20.454858+0000","last_fullsized":"2024-02-10T07:48:20.454858+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:47:44.170583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:44.170583+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:07.982096+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":668,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.467152+0000","last_change":"2024-02-10T07:47:47.966498+0000","last_active":"2024-02-10T07:48:20.467152+0000","last_peered":"2024-02-10T07:48:20.467152+0000","last_clean":"2024-02-10T07:48:20.467152+0000","last_became_active":"2024-02-10T07:47:47.966127+0000","last_became_peered":"2024-02-10T07:47:47.966127+0000","last_unstale":"2024-02-10T07:48:20.467152+0000","last_undegraded":"2024-02-10T07:48:20.467152+0000","last_fullsized":"2024-02-10T07:48:20.467152+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:44:54.243093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:44:54.243093+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"250'25","reported_seq":722,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452007+0000","last_change":"2024-02-10T07:48:12.279112+0000","last_active":"2024-02-10T07:48:20.452007+0000","last_peered":"2024-02-10T07:48:20.452007+0000","last_clean":"2024-02-10T07:48:20.452007+0000","last_became_active":"2024-02-10T07:46:40.127716+0000","last_became_peered":"2024-02-10T07:46:40.127716+0000","last_unstale":"2024-02-10T07:48:20.452007+0000","last_undegraded":"2024-02-10T07:48:20.452007+0000","last_fullsized":"2024-02-10T07:48:20.452007+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:48:12.278982+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-02-10T07:42:17.820959+0000","last_clean_scrub_stamp":"2024-02-10T07:48:12.278982+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:39.032968+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":655,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.468656+0000","last_change":"2024-02-10T07:45:57.490792+0000","last_active":"2024-02-10T07:48:20.468656+0000","last_peered":"2024-02-10T07:48:20.468656+0000","last_clean":"2024-02-10T07:48:20.468656+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:48:20.468656+0000","last_undegraded":"2024-02-10T07:48:20.468656+0000","last_fullsized":"2024-02-10T07:48:20.468656+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:57.490690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_clean_scrub_stamp":"2024-02-10T07:45:57.490690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":652,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452332+0000","last_change":"2024-02-10T07:46:10.724526+0000","last_active":"2024-02-10T07:48:20.452332+0000","last_peered":"2024-02-10T07:48:20.452332+0000","last_clean":"2024-02-10T07:48:20.452332+0000","last_became_active":"2024-02-10T07:46:10.724252+0000","last_became_peered":"2024-02-10T07:46:10.724252+0000","last_unstale":"2024-02-10T07:48:20.452332+0000","last_undegraded":"2024-02-10T07:48:20.452332+0000","last_fullsized":"2024-02-10T07:48:20.452332+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:06.200009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:06.200009+0000","last_clean_scrub_stamp":"2024-02-10T07:45:06.200009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":158,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451549+0000","last_change":"2024-02-10T07:47:45.224639+0000","last_active":"2024-02-10T07:48:20.451549+0000","last_peered":"2024-02-10T07:48:20.451549+0000","last_clean":"2024-02-10T07:48:20.451549+0000","last_became_active":"2024-02-10T07:47:26.155591+0000","last_became_peered":"2024-02-10T07:47:26.155591+0000","last_unstale":"2024-02-10T07:48:20.451549+0000","last_undegraded":"2024-02-10T07:48:20.451549+0000","last_fullsized":"2024-02-10T07:48:20.451549+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:45.224482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:47:45.224482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:50.095464+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"250'19","reported_seq":704,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.455091+0000","last_change":"2024-02-10T07:46:37.460129+0000","last_active":"2024-02-10T07:48:20.455091+0000","last_peered":"2024-02-10T07:48:20.455091+0000","last_clean":"2024-02-10T07:48:20.455091+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:48:20.455091+0000","last_undegraded":"2024-02-10T07:48:20.455091+0000","last_fullsized":"2024-02-10T07:48:20.455091+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:46:37.459988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.459988+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":659,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.456397+0000","last_change":"2024-02-10T07:46:41.115159+0000","last_active":"2024-02-10T07:48:20.456397+0000","last_peered":"2024-02-10T07:48:20.456397+0000","last_clean":"2024-02-10T07:48:20.456397+0000","last_became_active":"2024-02-10T07:46:41.114829+0000","last_became_peered":"2024-02-10T07:46:41.114829+0000","last_unstale":"2024-02-10T07:48:20.456397+0000","last_undegraded":"2024-02-10T07:48:20.456397+0000","last_fullsized":"2024-02-10T07:48:20.456397+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:48.648483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:45:48.648483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":668,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454454+0000","last_change":"2024-02-10T07:47:49.116910+0000","last_active":"2024-02-10T07:48:20.454454+0000","last_peered":"2024-02-10T07:48:20.454454+0000","last_clean":"2024-02-10T07:48:20.454454+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:48:20.454454+0000","last_undegraded":"2024-02-10T07:48:20.454454+0000","last_fullsized":"2024-02-10T07:48:20.454454+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:49.116752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.116752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:02.808272+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":678,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452342+0000","last_change":"2024-02-10T07:46:40.128287+0000","last_active":"2024-02-10T07:48:20.452342+0000","last_peered":"2024-02-10T07:48:20.452342+0000","last_clean":"2024-02-10T07:48:20.452342+0000","last_became_active":"2024-02-10T07:46:40.127951+0000","last_became_peered":"2024-02-10T07:46:40.127951+0000","last_unstale":"2024-02-10T07:48:20.452342+0000","last_undegraded":"2024-02-10T07:48:20.452342+0000","last_fullsized":"2024-02-10T07:48:20.452342+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:55.277457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:44:55.277457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"250'10","reported_seq":570,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.467590+0000","last_change":"2024-02-10T07:47:35.328681+0000","last_active":"2024-02-10T07:48:20.467590+0000","last_peered":"2024-02-10T07:48:20.467590+0000","last_clean":"2024-02-10T07:48:20.467590+0000","last_became_active":"2024-02-10T07:46:40.124444+0000","last_became_peered":"2024-02-10T07:46:40.124444+0000","last_unstale":"2024-02-10T07:48:20.467590+0000","last_undegraded":"2024-02-10T07:48:20.467590+0000","last_fullsized":"2024-02-10T07:48:20.467590+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'10","last_scrub_stamp":"2024-02-10T07:47:35.328559+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:47:35.328559+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:44.629298+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":152,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.453752+0000","last_change":"2024-02-10T07:47:32.258437+0000","last_active":"2024-02-10T07:48:20.453752+0000","last_peered":"2024-02-10T07:48:20.453752+0000","last_clean":"2024-02-10T07:48:20.453752+0000","last_became_active":"2024-02-10T07:46:40.131402+0000","last_became_peered":"2024-02-10T07:46:40.131402+0000","last_unstale":"2024-02-10T07:48:20.453752+0000","last_undegraded":"2024-02-10T07:48:20.453752+0000","last_fullsized":"2024-02-10T07:48:20.453752+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:32.258321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:32.258321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:57.034722+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":653,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.457133+0000","last_change":"2024-02-10T07:45:46.640690+0000","last_active":"2024-02-10T07:48:20.457133+0000","last_peered":"2024-02-10T07:48:20.457133+0000","last_clean":"2024-02-10T07:48:20.457133+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:48:20.457133+0000","last_undegraded":"2024-02-10T07:48:20.457133+0000","last_fullsized":"2024-02-10T07:48:20.457133+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:46.640569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:46.640569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":593,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.461755+0000","last_change":"2024-02-10T07:48:16.719023+0000","last_active":"2024-02-10T07:48:20.461755+0000","last_peered":"2024-02-10T07:48:20.461755+0000","last_clean":"2024-02-10T07:48:20.461755+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:48:20.461755+0000","last_undegraded":"2024-02-10T07:48:20.461755+0000","last_fullsized":"2024-02-10T07:48:20.461755+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:16.718907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:48:16.718907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:30.392111+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"250'19","reported_seq":601,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.461177+0000","last_change":"2024-02-10T07:47:27.163722+0000","last_active":"2024-02-10T07:48:20.461177+0000","last_peered":"2024-02-10T07:48:20.461177+0000","last_clean":"2024-02-10T07:48:20.461177+0000","last_became_active":"2024-02-10T07:46:40.123712+0000","last_became_peered":"2024-02-10T07:46:40.123712+0000","last_unstale":"2024-02-10T07:48:20.461177+0000","last_undegraded":"2024-02-10T07:48:20.461177+0000","last_fullsized":"2024-02-10T07:48:20.461177+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:47:27.163553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:27.163553+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:40.035381+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":650,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451304+0000","last_change":"2024-02-10T07:45:59.179456+0000","last_active":"2024-02-10T07:48:20.451304+0000","last_peered":"2024-02-10T07:48:20.451304+0000","last_clean":"2024-02-10T07:48:20.451304+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:48:20.451304+0000","last_undegraded":"2024-02-10T07:48:20.451304+0000","last_fullsized":"2024-02-10T07:48:20.451304+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:59.179342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:59.179342+0000","last_clean_scrub_stamp":"2024-02-10T07:45:59.179342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":152,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.453933+0000","last_change":"2024-02-10T07:47:40.140671+0000","last_active":"2024-02-10T07:48:20.453933+0000","last_peered":"2024-02-10T07:48:20.453933+0000","last_clean":"2024-02-10T07:48:20.453933+0000","last_became_active":"2024-02-10T07:46:40.123780+0000","last_became_peered":"2024-02-10T07:46:40.123780+0000","last_unstale":"2024-02-10T07:48:20.453933+0000","last_undegraded":"2024-02-10T07:48:20.453933+0000","last_fullsized":"2024-02-10T07:48:20.453933+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:40.140557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:40.140557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:40.897693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":669,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.458396+0000","last_change":"2024-02-10T07:47:49.030015+0000","last_active":"2024-02-10T07:48:20.458396+0000","last_peered":"2024-02-10T07:48:20.458396+0000","last_clean":"2024-02-10T07:48:20.458396+0000","last_became_active":"2024-02-10T07:47:47.966472+0000","last_became_peered":"2024-02-10T07:47:47.966472+0000","last_unstale":"2024-02-10T07:48:20.458396+0000","last_undegraded":"2024-02-10T07:48:20.458396+0000","last_fullsized":"2024-02-10T07:48:20.458396+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:49.029944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.029944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:54.336294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"250'19","reported_seq":477,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.467758+0000","last_change":"2024-02-10T07:46:10.740738+0000","last_active":"2024-02-10T07:48:20.467758+0000","last_peered":"2024-02-10T07:48:20.467758+0000","last_clean":"2024-02-10T07:48:20.467758+0000","last_became_active":"2024-02-10T07:46:10.726359+0000","last_became_peered":"2024-02-10T07:46:10.726359+0000","last_unstale":"2024-02-10T07:48:20.467758+0000","last_undegraded":"2024-02-10T07:48:20.467758+0000","last_fullsized":"2024-02-10T07:48:20.467758+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:45:59.430820+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-02-10T07:39:32.934269+0000","last_clean_scrub_stamp":"2024-02-10T07:45:59.430820+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":340,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451723+0000","last_change":"2024-02-10T07:47:31.118096+0000","last_active":"2024-02-10T07:48:20.451723+0000","last_peered":"2024-02-10T07:48:20.451723+0000","last_clean":"2024-02-10T07:48:20.451723+0000","last_became_active":"2024-02-10T07:46:40.129733+0000","last_became_peered":"2024-02-10T07:46:40.129733+0000","last_unstale":"2024-02-10T07:48:20.451723+0000","last_undegraded":"2024-02-10T07:48:20.451723+0000","last_fullsized":"2024-02-10T07:48:20.451723+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:31.117976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:31.117976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:58.105240+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":677,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.467736+0000","last_change":"2024-02-10T07:48:19.263555+0000","last_active":"2024-02-10T07:48:20.467736+0000","last_peered":"2024-02-10T07:48:20.467736+0000","last_clean":"2024-02-10T07:48:20.467736+0000","last_became_active":"2024-02-10T07:46:40.121948+0000","last_became_peered":"2024-02-10T07:46:40.121948+0000","last_unstale":"2024-02-10T07:48:20.467736+0000","last_undegraded":"2024-02-10T07:48:20.467736+0000","last_fullsized":"2024-02-10T07:48:20.467736+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:19.263442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:19.263442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:31.568687+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":661,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452770+0000","last_change":"2024-02-10T07:45:54.972676+0000","last_active":"2024-02-10T07:48:20.452770+0000","last_peered":"2024-02-10T07:48:20.452770+0000","last_clean":"2024-02-10T07:48:20.452770+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:48:20.452770+0000","last_undegraded":"2024-02-10T07:48:20.452770+0000","last_fullsized":"2024-02-10T07:48:20.452770+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:54.972600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:45:54.972600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"250'24","reported_seq":729,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454015+0000","last_change":"2024-02-10T07:47:43.494877+0000","last_active":"2024-02-10T07:48:20.454015+0000","last_peered":"2024-02-10T07:48:20.454015+0000","last_clean":"2024-02-10T07:48:20.454015+0000","last_became_active":"2024-02-10T07:46:10.721336+0000","last_became_peered":"2024-02-10T07:46:10.721336+0000","last_unstale":"2024-02-10T07:48:20.454015+0000","last_undegraded":"2024-02-10T07:48:20.454015+0000","last_fullsized":"2024-02-10T07:48:20.454015+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:47:43.494350+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.494350+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:55.351104+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":657,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451701+0000","last_change":"2024-02-10T07:47:43.186880+0000","last_active":"2024-02-10T07:48:20.451701+0000","last_peered":"2024-02-10T07:48:20.451701+0000","last_clean":"2024-02-10T07:48:20.451701+0000","last_became_active":"2024-02-10T07:46:40.128084+0000","last_became_peered":"2024-02-10T07:46:40.128084+0000","last_unstale":"2024-02-10T07:48:20.451701+0000","last_undegraded":"2024-02-10T07:48:20.451701+0000","last_fullsized":"2024-02-10T07:48:20.451701+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:43.186809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.186809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:53.187567+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"356'11","reported_seq":96,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.468160+0000","last_change":"2024-02-10T07:48:02.372211+0000","last_active":"2024-02-10T07:48:20.468160+0000","last_peered":"2024-02-10T07:48:20.468160+0000","last_clean":"2024-02-10T07:48:20.468160+0000","last_became_active":"2024-02-10T07:48:02.371888+0000","last_became_peered":"2024-02-10T07:48:02.371888+0000","last_unstale":"2024-02-10T07:48:20.468160+0000","last_undegraded":"2024-02-10T07:48:20.468160+0000","last_fullsized":"2024-02-10T07:48:20.468160+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:45:04.392999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:04.392999+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":675,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.456728+0000","last_change":"2024-02-10T07:46:36.855577+0000","last_active":"2024-02-10T07:48:20.456728+0000","last_peered":"2024-02-10T07:48:20.456728+0000","last_clean":"2024-02-10T07:48:20.456728+0000","last_became_active":"2024-02-10T07:46:10.716647+0000","last_became_peered":"2024-02-10T07:46:10.716647+0000","last_unstale":"2024-02-10T07:48:20.456728+0000","last_undegraded":"2024-02-10T07:48:20.456728+0000","last_fullsized":"2024-02-10T07:48:20.456728+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:36.855461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:46:36.855461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"250'25","reported_seq":282,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.468725+0000","last_change":"2024-02-10T07:48:02.329423+0000","last_active":"2024-02-10T07:48:20.468725+0000","last_peered":"2024-02-10T07:48:20.468725+0000","last_clean":"2024-02-10T07:48:20.468725+0000","last_became_active":"2024-02-10T07:48:02.329187+0000","last_became_peered":"2024-02-10T07:48:02.329187+0000","last_unstale":"2024-02-10T07:48:20.468725+0000","last_undegraded":"2024-02-10T07:48:20.468725+0000","last_fullsized":"2024-02-10T07:48:20.468725+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:45:52.483293+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:45:52.483293+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":655,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.460728+0000","last_change":"2024-02-10T07:46:10.721405+0000","last_active":"2024-02-10T07:48:20.460728+0000","last_peered":"2024-02-10T07:48:20.460728+0000","last_clean":"2024-02-10T07:48:20.460728+0000","last_became_active":"2024-02-10T07:46:10.720680+0000","last_became_peered":"2024-02-10T07:46:10.720680+0000","last_unstale":"2024-02-10T07:48:20.460728+0000","last_undegraded":"2024-02-10T07:48:20.460728+0000","last_fullsized":"2024-02-10T07:48:20.460728+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:48.097218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:45:48.097218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"326'13","reported_seq":231,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.457346+0000","last_change":"2024-02-10T07:46:41.125506+0000","last_active":"2024-02-10T07:48:20.457346+0000","last_peered":"2024-02-10T07:48:20.457346+0000","last_clean":"2024-02-10T07:48:20.457346+0000","last_became_active":"2024-02-10T07:46:41.118359+0000","last_became_peered":"2024-02-10T07:46:41.118359+0000","last_unstale":"2024-02-10T07:48:20.457346+0000","last_undegraded":"2024-02-10T07:48:20.457346+0000","last_fullsized":"2024-02-10T07:48:20.457346+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'11","last_scrub_stamp":"2024-02-10T07:46:32.848489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:32.848489+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":670,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452274+0000","last_change":"2024-02-10T07:46:40.451635+0000","last_active":"2024-02-10T07:48:20.452274+0000","last_peered":"2024-02-10T07:48:20.452274+0000","last_clean":"2024-02-10T07:48:20.452274+0000","last_became_active":"2024-02-10T07:46:40.127276+0000","last_became_peered":"2024-02-10T07:46:40.127276+0000","last_unstale":"2024-02-10T07:48:20.452274+0000","last_undegraded":"2024-02-10T07:48:20.452274+0000","last_fullsized":"2024-02-10T07:48:20.452274+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:40.451562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:46:40.451562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"250'15","reported_seq":406,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.458720+0000","last_change":"2024-02-10T07:48:02.330897+0000","last_active":"2024-02-10T07:48:20.458720+0000","last_peered":"2024-02-10T07:48:20.458720+0000","last_clean":"2024-02-10T07:48:20.458720+0000","last_became_active":"2024-02-10T07:48:02.330517+0000","last_became_peered":"2024-02-10T07:48:02.330517+0000","last_unstale":"2024-02-10T07:48:20.458720+0000","last_undegraded":"2024-02-10T07:48:20.458720+0000","last_fullsized":"2024-02-10T07:48:20.458720+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:39.610017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:39.610017+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:04.740834+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":115,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.453160+0000","last_change":"2024-02-10T07:48:18.750657+0000","last_active":"2024-02-10T07:48:20.453160+0000","last_peered":"2024-02-10T07:48:20.453160+0000","last_clean":"2024-02-10T07:48:20.453160+0000","last_became_active":"2024-02-10T07:46:10.716869+0000","last_became_peered":"2024-02-10T07:46:10.716869+0000","last_unstale":"2024-02-10T07:48:20.453160+0000","last_undegraded":"2024-02-10T07:48:20.453160+0000","last_fullsized":"2024-02-10T07:48:20.453160+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:18.750347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:48:18.750347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:43.265856+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":151,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.469208+0000","last_change":"2024-02-10T07:48:17.294814+0000","last_active":"2024-02-10T07:48:20.469208+0000","last_peered":"2024-02-10T07:48:20.469208+0000","last_clean":"2024-02-10T07:48:20.469208+0000","last_became_active":"2024-02-10T07:46:40.117173+0000","last_became_peered":"2024-02-10T07:46:40.117173+0000","last_unstale":"2024-02-10T07:48:20.469208+0000","last_undegraded":"2024-02-10T07:48:20.469208+0000","last_fullsized":"2024-02-10T07:48:20.469208+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:17.294700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:48:17.294700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:40.075748+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":669,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.461050+0000","last_change":"2024-02-10T07:47:47.065963+0000","last_active":"2024-02-10T07:48:20.461050+0000","last_peered":"2024-02-10T07:48:20.461050+0000","last_clean":"2024-02-10T07:48:20.461050+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:48:20.461050+0000","last_undegraded":"2024-02-10T07:48:20.461050+0000","last_fullsized":"2024-02-10T07:48:20.461050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:47.065863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:47:47.065863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:53.744548+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"250'19","reported_seq":712,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452177+0000","last_change":"2024-02-10T07:47:42.150745+0000","last_active":"2024-02-10T07:48:20.452177+0000","last_peered":"2024-02-10T07:48:20.452177+0000","last_clean":"2024-02-10T07:48:20.452177+0000","last_became_active":"2024-02-10T07:46:40.127170+0000","last_became_peered":"2024-02-10T07:46:40.127170+0000","last_unstale":"2024-02-10T07:48:20.452177+0000","last_undegraded":"2024-02-10T07:48:20.452177+0000","last_fullsized":"2024-02-10T07:48:20.452177+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:47:42.150605+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.150605+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:48.602794+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":648,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.458271+0000","last_change":"2024-02-10T07:46:34.945757+0000","last_active":"2024-02-10T07:48:20.458271+0000","last_peered":"2024-02-10T07:48:20.458271+0000","last_clean":"2024-02-10T07:48:20.458271+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:48:20.458271+0000","last_undegraded":"2024-02-10T07:48:20.458271+0000","last_fullsized":"2024-02-10T07:48:20.458271+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:34.945654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:46:34.945654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":551,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.468998+0000","last_change":"2024-02-10T07:48:02.328360+0000","last_active":"2024-02-10T07:48:20.468998+0000","last_peered":"2024-02-10T07:48:20.468998+0000","last_clean":"2024-02-10T07:48:20.468998+0000","last_became_active":"2024-02-10T07:48:02.328091+0000","last_became_peered":"2024-02-10T07:48:02.328091+0000","last_unstale":"2024-02-10T07:48:20.468998+0000","last_undegraded":"2024-02-10T07:48:20.468998+0000","last_fullsized":"2024-02-10T07:48:20.468998+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:43.595609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.595609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:55.265080+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":693,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454057+0000","last_change":"2024-02-10T07:47:51.170585+0000","last_active":"2024-02-10T07:48:20.454057+0000","last_peered":"2024-02-10T07:48:20.454057+0000","last_clean":"2024-02-10T07:48:20.454057+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:48:20.454057+0000","last_undegraded":"2024-02-10T07:48:20.454057+0000","last_fullsized":"2024-02-10T07:48:20.454057+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:51.170362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:47:51.170362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:59.828983+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"250'12","reported_seq":576,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.458266+0000","last_change":"2024-02-10T07:47:33.129508+0000","last_active":"2024-02-10T07:48:20.458266+0000","last_peered":"2024-02-10T07:48:20.458266+0000","last_clean":"2024-02-10T07:48:20.458266+0000","last_became_active":"2024-02-10T07:46:40.120521+0000","last_became_peered":"2024-02-10T07:46:40.120521+0000","last_unstale":"2024-02-10T07:48:20.458266+0000","last_undegraded":"2024-02-10T07:48:20.458266+0000","last_fullsized":"2024-02-10T07:48:20.458266+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'12","last_scrub_stamp":"2024-02-10T07:47:33.129377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:33.129377+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:50.468325+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.6","version":"361'8913","reported_seq":9041,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:24.128834+0000","last_change":"2024-02-10T07:48:02.435528+0000","last_active":"2024-02-10T07:48:24.128834+0000","last_peered":"2024-02-10T07:48:24.128834+0000","last_clean":"2024-02-10T07:48:24.128834+0000","last_became_active":"2024-02-10T07:48:02.384751+0000","last_became_peered":"2024-02-10T07:48:02.384751+0000","last_unstale":"2024-02-10T07:48:24.128834+0000","last_undegraded":"2024-02-10T07:48:24.128834+0000","last_fullsized":"2024-02-10T07:48:24.128834+0000","mapping_epoch":355,"log_start":"333'4740","ondisk_log_start":"333'4740","created":309,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288686080,"num_objects":4405,"num_object_clones":0,"num_object_copies":13215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8913,"num_write_kb":426176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":182,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454103+0000","last_change":"2024-02-10T07:48:19.178760+0000","last_active":"2024-02-10T07:48:20.454103+0000","last_peered":"2024-02-10T07:48:20.454103+0000","last_clean":"2024-02-10T07:48:20.454103+0000","last_became_active":"2024-02-10T07:48:02.366002+0000","last_became_peered":"2024-02-10T07:48:02.366002+0000","last_unstale":"2024-02-10T07:48:20.454103+0000","last_undegraded":"2024-02-10T07:48:20.454103+0000","last_fullsized":"2024-02-10T07:48:20.454103+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:19.178651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:19.178651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:27.830824+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"245'4","reported_seq":683,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.458180+0000","last_change":"2024-02-10T07:46:35.848544+0000","last_active":"2024-02-10T07:48:20.458180+0000","last_peered":"2024-02-10T07:48:20.458180+0000","last_clean":"2024-02-10T07:48:20.458180+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:48:20.458180+0000","last_undegraded":"2024-02-10T07:48:20.458180+0000","last_fullsized":"2024-02-10T07:48:20.458180+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'4","last_scrub_stamp":"2024-02-10T07:46:35.848398+0000","last_deep_scrub":"245'4","last_deep_scrub_stamp":"2024-02-10T07:41:23.717923+0000","last_clean_scrub_stamp":"2024-02-10T07:46:35.848398+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"250'34","reported_seq":417,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.456638+0000","last_change":"2024-02-10T07:48:11.967227+0000","last_active":"2024-02-10T07:48:20.456638+0000","last_peered":"2024-02-10T07:48:20.456638+0000","last_clean":"2024-02-10T07:48:20.456638+0000","last_became_active":"2024-02-10T07:48:02.330130+0000","last_became_peered":"2024-02-10T07:48:02.330130+0000","last_unstale":"2024-02-10T07:48:20.456638+0000","last_undegraded":"2024-02-10T07:48:20.456638+0000","last_fullsized":"2024-02-10T07:48:20.456638+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:48:11.966957+0000","last_deep_scrub":"246'22","last_deep_scrub_stamp":"2024-02-10T07:41:26.771112+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.966957+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:12.386691+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":868,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.461292+0000","last_change":"2024-02-10T07:46:00.666508+0000","last_active":"2024-02-10T07:48:20.461292+0000","last_peered":"2024-02-10T07:48:20.461292+0000","last_clean":"2024-02-10T07:48:20.461292+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:48:20.461292+0000","last_undegraded":"2024-02-10T07:48:20.461292+0000","last_fullsized":"2024-02-10T07:48:20.461292+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:46:00.666417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:00.666417+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"361'9095","reported_seq":14541,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:23.305967+0000","last_change":"2024-02-10T07:47:41.250067+0000","last_active":"2024-02-10T07:48:23.305967+0000","last_peered":"2024-02-10T07:48:23.305967+0000","last_clean":"2024-02-10T07:48:23.305967+0000","last_became_active":"2024-02-10T07:47:41.245911+0000","last_became_peered":"2024-02-10T07:47:41.245911+0000","last_unstale":"2024-02-10T07:48:23.305967+0000","last_undegraded":"2024-02-10T07:48:23.305967+0000","last_fullsized":"2024-02-10T07:48:23.305967+0000","mapping_epoch":343,"log_start":"331'4800","ondisk_log_start":"331'4800","created":309,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4295,"log_dups_size":0,"ondisk_log_size":4295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294191104,"num_objects":4489,"num_object_clones":0,"num_object_copies":13467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":434688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2664,"num_bytes_recovered":174587904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":487,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.445906+0000","last_change":"2024-02-10T07:48:12.693292+0000","last_active":"2024-02-10T07:48:20.445906+0000","last_peered":"2024-02-10T07:48:20.445906+0000","last_clean":"2024-02-10T07:48:20.445906+0000","last_became_active":"2024-02-10T07:47:47.966183+0000","last_became_peered":"2024-02-10T07:47:47.966183+0000","last_unstale":"2024-02-10T07:48:20.445906+0000","last_undegraded":"2024-02-10T07:48:20.445906+0000","last_fullsized":"2024-02-10T07:48:20.445906+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:12.693200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:12.693200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:41.139290+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":661,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.456961+0000","last_change":"2024-02-10T07:45:42.650614+0000","last_active":"2024-02-10T07:48:20.456961+0000","last_peered":"2024-02-10T07:48:20.456961+0000","last_clean":"2024-02-10T07:48:20.456961+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:48:20.456961+0000","last_undegraded":"2024-02-10T07:48:20.456961+0000","last_fullsized":"2024-02-10T07:48:20.456961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:42.650422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:45:42.650422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"250'23","reported_seq":749,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:21.673333+0000","last_change":"2024-02-10T07:48:21.673459+0000","last_active":"2024-02-10T07:48:21.673333+0000","last_peered":"2024-02-10T07:48:21.673333+0000","last_clean":"2024-02-10T07:48:21.673333+0000","last_became_active":"2024-02-10T07:46:41.120409+0000","last_became_peered":"2024-02-10T07:46:41.120409+0000","last_unstale":"2024-02-10T07:48:21.673333+0000","last_undegraded":"2024-02-10T07:48:21.673333+0000","last_fullsized":"2024-02-10T07:48:21.673333+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-02-10T07:48:21.673122+0000","last_deep_scrub":"245'17","last_deep_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_clean_scrub_stamp":"2024-02-10T07:48:21.673122+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:45.249522+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":109,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451755+0000","last_change":"2024-02-10T07:46:10.724798+0000","last_active":"2024-02-10T07:48:20.451755+0000","last_peered":"2024-02-10T07:48:20.451755+0000","last_clean":"2024-02-10T07:48:20.451755+0000","last_became_active":"2024-02-10T07:46:10.723006+0000","last_became_peered":"2024-02-10T07:46:10.723006+0000","last_unstale":"2024-02-10T07:48:20.451755+0000","last_undegraded":"2024-02-10T07:48:20.451755+0000","last_fullsized":"2024-02-10T07:48:20.451755+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:44.509694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:44.509694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"361'9067","reported_seq":22054,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:24.156512+0000","last_change":"2024-02-10T07:48:03.334406+0000","last_active":"2024-02-10T07:48:24.156512+0000","last_peered":"2024-02-10T07:48:24.156512+0000","last_clean":"2024-02-10T07:48:24.156512+0000","last_became_active":"2024-02-10T07:48:03.326525+0000","last_became_peered":"2024-02-10T07:48:03.326525+0000","last_unstale":"2024-02-10T07:48:24.156512+0000","last_undegraded":"2024-02-10T07:48:24.156512+0000","last_fullsized":"2024-02-10T07:48:24.156512+0000","mapping_epoch":356,"log_start":"334'4900","ondisk_log_start":"334'4900","created":309,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4167,"log_dups_size":0,"ondisk_log_size":4167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295763968,"num_objects":4513,"num_object_clones":0,"num_object_copies":13539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9067,"num_write_kb":434560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6431,"num_bytes_recovered":421462016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":670,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.456236+0000","last_change":"2024-02-10T07:46:41.124909+0000","last_active":"2024-02-10T07:48:20.456236+0000","last_peered":"2024-02-10T07:48:20.456236+0000","last_clean":"2024-02-10T07:48:20.456236+0000","last_became_active":"2024-02-10T07:46:41.124345+0000","last_became_peered":"2024-02-10T07:46:41.124345+0000","last_unstale":"2024-02-10T07:48:20.456236+0000","last_undegraded":"2024-02-10T07:48:20.456236+0000","last_fullsized":"2024-02-10T07:48:20.456236+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:31.861542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:36.741855+0000","last_clean_scrub_stamp":"2024-02-10T07:46:31.861542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":316,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:21.172653+0000","last_change":"2024-02-10T07:48:21.172677+0000","last_active":"2024-02-10T07:48:21.172653+0000","last_peered":"2024-02-10T07:48:21.172653+0000","last_clean":"2024-02-10T07:48:21.172653+0000","last_became_active":"2024-02-10T07:46:10.719610+0000","last_became_peered":"2024-02-10T07:46:10.719610+0000","last_unstale":"2024-02-10T07:48:21.172653+0000","last_undegraded":"2024-02-10T07:48:21.172653+0000","last_fullsized":"2024-02-10T07:48:21.172653+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:21.172601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:21.172601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:23.224718+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":678,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.458650+0000","last_change":"2024-02-10T07:47:39.184439+0000","last_active":"2024-02-10T07:48:20.458650+0000","last_peered":"2024-02-10T07:48:20.458650+0000","last_clean":"2024-02-10T07:48:20.458650+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:48:20.458650+0000","last_undegraded":"2024-02-10T07:48:20.458650+0000","last_fullsized":"2024-02-10T07:48:20.458650+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:39.184193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:47:39.184193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:01.101035+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"250'35","reported_seq":727,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454331+0000","last_change":"2024-02-10T07:46:32.439686+0000","last_active":"2024-02-10T07:48:20.454331+0000","last_peered":"2024-02-10T07:48:20.454331+0000","last_clean":"2024-02-10T07:48:20.454331+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:48:20.454331+0000","last_undegraded":"2024-02-10T07:48:20.454331+0000","last_fullsized":"2024-02-10T07:48:20.454331+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'35","last_scrub_stamp":"2024-02-10T07:46:32.439157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:46:32.439157+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"361'9151","reported_seq":14573,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:24.148413+0000","last_change":"2024-02-10T07:47:22.181013+0000","last_active":"2024-02-10T07:48:24.148413+0000","last_peered":"2024-02-10T07:48:24.148413+0000","last_clean":"2024-02-10T07:48:24.148413+0000","last_became_active":"2024-02-10T07:47:22.138566+0000","last_became_peered":"2024-02-10T07:47:22.138566+0000","last_unstale":"2024-02-10T07:48:24.148413+0000","last_undegraded":"2024-02-10T07:48:24.148413+0000","last_fullsized":"2024-02-10T07:48:24.148413+0000","mapping_epoch":335,"log_start":"333'4900","ondisk_log_start":"333'4900","created":309,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4251,"log_dups_size":0,"ondisk_log_size":4251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296157184,"num_objects":4519,"num_object_clones":0,"num_object_copies":13557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":437440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2650,"num_bytes_recovered":173670400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":659,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451943+0000","last_change":"2024-02-10T07:46:41.126327+0000","last_active":"2024-02-10T07:48:20.451943+0000","last_peered":"2024-02-10T07:48:20.451943+0000","last_clean":"2024-02-10T07:48:20.451943+0000","last_became_active":"2024-02-10T07:46:41.125757+0000","last_became_peered":"2024-02-10T07:46:41.125757+0000","last_unstale":"2024-02-10T07:48:20.451943+0000","last_undegraded":"2024-02-10T07:48:20.451943+0000","last_fullsized":"2024-02-10T07:48:20.451943+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:12.239727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:45:12.239727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":640,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452805+0000","last_change":"2024-02-10T07:46:41.121167+0000","last_active":"2024-02-10T07:48:20.452805+0000","last_peered":"2024-02-10T07:48:20.452805+0000","last_clean":"2024-02-10T07:48:20.452805+0000","last_became_active":"2024-02-10T07:46:41.120629+0000","last_became_peered":"2024-02-10T07:46:41.120629+0000","last_unstale":"2024-02-10T07:48:20.452805+0000","last_undegraded":"2024-02-10T07:48:20.452805+0000","last_fullsized":"2024-02-10T07:48:20.452805+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:28.822700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:28.822700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":559,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454793+0000","last_change":"2024-02-10T07:48:02.329521+0000","last_active":"2024-02-10T07:48:20.454793+0000","last_peered":"2024-02-10T07:48:20.454793+0000","last_clean":"2024-02-10T07:48:20.454793+0000","last_became_active":"2024-02-10T07:48:02.329213+0000","last_became_peered":"2024-02-10T07:48:02.329213+0000","last_unstale":"2024-02-10T07:48:20.454793+0000","last_undegraded":"2024-02-10T07:48:20.454793+0000","last_fullsized":"2024-02-10T07:48:20.454793+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:40.558483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:47:40.558483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:53.360127+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"250'15","reported_seq":682,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454275+0000","last_change":"2024-02-10T07:45:11.162197+0000","last_active":"2024-02-10T07:48:20.454275+0000","last_peered":"2024-02-10T07:48:20.454275+0000","last_clean":"2024-02-10T07:48:20.454275+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:48:20.454275+0000","last_undegraded":"2024-02-10T07:48:20.454275+0000","last_fullsized":"2024-02-10T07:48:20.454275+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:45:11.161953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:45:11.161953+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"361'9111","reported_seq":9225,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:23.319750+0000","last_change":"2024-02-10T07:46:06.701491+0000","last_active":"2024-02-10T07:48:23.319750+0000","last_peered":"2024-02-10T07:48:23.319750+0000","last_clean":"2024-02-10T07:48:23.319750+0000","last_became_active":"2024-02-10T07:46:06.701226+0000","last_became_peered":"2024-02-10T07:46:06.701226+0000","last_unstale":"2024-02-10T07:48:23.319750+0000","last_undegraded":"2024-02-10T07:48:23.319750+0000","last_fullsized":"2024-02-10T07:48:23.319750+0000","mapping_epoch":309,"log_start":"331'4800","ondisk_log_start":"331'4800","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4311,"log_dups_size":0,"ondisk_log_size":4311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301400064,"num_objects":4599,"num_object_clones":0,"num_object_copies":13797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9111,"num_write_kb":438720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":151,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451929+0000","last_change":"2024-02-10T07:47:41.538310+0000","last_active":"2024-02-10T07:48:20.451929+0000","last_peered":"2024-02-10T07:48:20.451929+0000","last_clean":"2024-02-10T07:48:20.451929+0000","last_became_active":"2024-02-10T07:46:40.131207+0000","last_became_peered":"2024-02-10T07:46:40.131207+0000","last_unstale":"2024-02-10T07:48:20.451929+0000","last_undegraded":"2024-02-10T07:48:20.451929+0000","last_fullsized":"2024-02-10T07:48:20.451929+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:41.537113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:47:41.537113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:43.366082+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":81,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.467178+0000","last_change":"2024-02-10T07:48:02.328550+0000","last_active":"2024-02-10T07:48:20.467178+0000","last_peered":"2024-02-10T07:48:20.467178+0000","last_clean":"2024-02-10T07:48:20.467178+0000","last_became_active":"2024-02-10T07:48:02.328316+0000","last_became_peered":"2024-02-10T07:48:02.328316+0000","last_unstale":"2024-02-10T07:48:20.467178+0000","last_undegraded":"2024-02-10T07:48:20.467178+0000","last_fullsized":"2024-02-10T07:48:20.467178+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:45:44.361605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:45:44.361605+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":653,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.453896+0000","last_change":"2024-02-10T07:45:04.220439+0000","last_active":"2024-02-10T07:48:20.453896+0000","last_peered":"2024-02-10T07:48:20.453896+0000","last_clean":"2024-02-10T07:48:20.453896+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:48:20.453896+0000","last_undegraded":"2024-02-10T07:48:20.453896+0000","last_fullsized":"2024-02-10T07:48:20.453896+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:04.220328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:04.220328+0000","last_clean_scrub_stamp":"2024-02-10T07:45:04.220328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"326'45","reported_seq":981,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.469204+0000","last_change":"2024-02-10T07:46:41.157810+0000","last_active":"2024-02-10T07:48:20.469204+0000","last_peered":"2024-02-10T07:48:20.469204+0000","last_clean":"2024-02-10T07:48:20.469204+0000","last_became_active":"2024-02-10T07:46:41.122806+0000","last_became_peered":"2024-02-10T07:46:41.122806+0000","last_unstale":"2024-02-10T07:48:20.469204+0000","last_undegraded":"2024-02-10T07:48:20.469204+0000","last_fullsized":"2024-02-10T07:48:20.469204+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'43","last_scrub_stamp":"2024-02-10T07:46:29.401404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:46:29.401404+0000","objects_scrubbed":10,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":661,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452114+0000","last_change":"2024-02-10T07:45:47.080393+0000","last_active":"2024-02-10T07:48:20.452114+0000","last_peered":"2024-02-10T07:48:20.452114+0000","last_clean":"2024-02-10T07:48:20.452114+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:48:20.452114+0000","last_undegraded":"2024-02-10T07:48:20.452114+0000","last_fullsized":"2024-02-10T07:48:20.452114+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:47.080312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:22.786949+0000","last_clean_scrub_stamp":"2024-02-10T07:45:47.080312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"361'8931","reported_seq":9059,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:23.148468+0000","last_change":"2024-02-10T07:48:02.352917+0000","last_active":"2024-02-10T07:48:23.148468+0000","last_peered":"2024-02-10T07:48:23.148468+0000","last_clean":"2024-02-10T07:48:23.148468+0000","last_became_active":"2024-02-10T07:48:02.330032+0000","last_became_peered":"2024-02-10T07:48:02.330032+0000","last_unstale":"2024-02-10T07:48:23.148468+0000","last_undegraded":"2024-02-10T07:48:23.148468+0000","last_fullsized":"2024-02-10T07:48:23.148468+0000","mapping_epoch":355,"log_start":"330'4466","ondisk_log_start":"330'4466","created":309,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4465,"log_dups_size":0,"ondisk_log_size":4465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295239680,"num_objects":4505,"num_object_clones":0,"num_object_copies":13515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8931,"num_write_kb":429952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":658,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.458809+0000","last_change":"2024-02-10T07:47:50.081222+0000","last_active":"2024-02-10T07:48:20.458809+0000","last_peered":"2024-02-10T07:48:20.458809+0000","last_clean":"2024-02-10T07:48:20.458809+0000","last_became_active":"2024-02-10T07:47:48.149256+0000","last_became_peered":"2024-02-10T07:47:48.149256+0000","last_unstale":"2024-02-10T07:48:20.458809+0000","last_undegraded":"2024-02-10T07:48:20.458809+0000","last_fullsized":"2024-02-10T07:48:20.458809+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.080714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:42.730875+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.080714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:01.907879+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":674,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.467532+0000","last_change":"2024-02-10T07:46:33.322279+0000","last_active":"2024-02-10T07:48:20.467532+0000","last_peered":"2024-02-10T07:48:20.467532+0000","last_clean":"2024-02-10T07:48:20.467532+0000","last_became_active":"2024-02-10T07:46:10.726265+0000","last_became_peered":"2024-02-10T07:46:10.726265+0000","last_unstale":"2024-02-10T07:48:20.467532+0000","last_undegraded":"2024-02-10T07:48:20.467532+0000","last_fullsized":"2024-02-10T07:48:20.467532+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:33.322197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:46:33.322197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":659,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451539+0000","last_change":"2024-02-10T07:48:15.234243+0000","last_active":"2024-02-10T07:48:20.451539+0000","last_peered":"2024-02-10T07:48:20.451539+0000","last_clean":"2024-02-10T07:48:20.451539+0000","last_became_active":"2024-02-10T07:46:41.117865+0000","last_became_peered":"2024-02-10T07:46:41.117865+0000","last_unstale":"2024-02-10T07:48:20.451539+0000","last_undegraded":"2024-02-10T07:48:20.451539+0000","last_fullsized":"2024-02-10T07:48:20.451539+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:15.234112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.234112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:38.975362+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"250'17","reported_seq":457,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.469416+0000","last_change":"2024-02-10T07:46:30.354780+0000","last_active":"2024-02-10T07:48:20.469416+0000","last_peered":"2024-02-10T07:48:20.469416+0000","last_clean":"2024-02-10T07:48:20.469416+0000","last_became_active":"2024-02-10T07:37:06.196787+0000","last_became_peered":"2024-02-10T07:37:06.196787+0000","last_unstale":"2024-02-10T07:48:20.469416+0000","last_undegraded":"2024-02-10T07:48:20.469416+0000","last_fullsized":"2024-02-10T07:48:20.469416+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'17","last_scrub_stamp":"2024-02-10T07:46:30.354628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:46:30.354628+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":224,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.457174+0000","last_change":"2024-02-10T07:48:02.331814+0000","last_active":"2024-02-10T07:48:20.457174+0000","last_peered":"2024-02-10T07:48:20.457174+0000","last_clean":"2024-02-10T07:48:20.457174+0000","last_became_active":"2024-02-10T07:48:02.331496+0000","last_became_peered":"2024-02-10T07:48:02.331496+0000","last_unstale":"2024-02-10T07:48:20.457174+0000","last_undegraded":"2024-02-10T07:48:20.457174+0000","last_fullsized":"2024-02-10T07:48:20.457174+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:39.609589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:39.609589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.b","version":"361'8916","reported_seq":9029,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:23.320128+0000","last_change":"2024-02-10T07:46:06.701790+0000","last_active":"2024-02-10T07:48:23.320128+0000","last_peered":"2024-02-10T07:48:23.320128+0000","last_clean":"2024-02-10T07:48:23.320128+0000","last_became_active":"2024-02-10T07:46:06.701471+0000","last_became_peered":"2024-02-10T07:46:06.701471+0000","last_unstale":"2024-02-10T07:48:23.320128+0000","last_undegraded":"2024-02-10T07:48:23.320128+0000","last_fullsized":"2024-02-10T07:48:23.320128+0000","mapping_epoch":309,"log_start":"331'4600","ondisk_log_start":"331'4600","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4316,"log_dups_size":0,"ondisk_log_size":4316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298319872,"num_objects":4552,"num_object_clones":0,"num_object_copies":13656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8916,"num_write_kb":430976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":78,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.460329+0000","last_change":"2024-02-10T07:48:02.330253+0000","last_active":"2024-02-10T07:48:20.460329+0000","last_peered":"2024-02-10T07:48:20.460329+0000","last_clean":"2024-02-10T07:48:20.460329+0000","last_became_active":"2024-02-10T07:48:02.329687+0000","last_became_peered":"2024-02-10T07:48:02.329687+0000","last_unstale":"2024-02-10T07:48:20.460329+0000","last_undegraded":"2024-02-10T07:48:20.460329+0000","last_fullsized":"2024-02-10T07:48:20.460329+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:47:49.750508+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.750508+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:54.086839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":915,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.456486+0000","last_change":"2024-02-10T07:48:15.033077+0000","last_active":"2024-02-10T07:48:20.456486+0000","last_peered":"2024-02-10T07:48:20.456486+0000","last_clean":"2024-02-10T07:48:20.456486+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:48:20.456486+0000","last_undegraded":"2024-02-10T07:48:20.456486+0000","last_fullsized":"2024-02-10T07:48:20.456486+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:48:15.032944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.032944+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:30.583135+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":293,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454584+0000","last_change":"2024-02-10T07:48:17.198463+0000","last_active":"2024-02-10T07:48:20.454584+0000","last_peered":"2024-02-10T07:48:20.454584+0000","last_clean":"2024-02-10T07:48:20.454584+0000","last_became_active":"2024-02-10T07:48:02.330370+0000","last_became_peered":"2024-02-10T07:48:02.330370+0000","last_unstale":"2024-02-10T07:48:20.454584+0000","last_undegraded":"2024-02-10T07:48:20.454584+0000","last_fullsized":"2024-02-10T07:48:20.454584+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:17.198213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:17.198213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:25.232723+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"250'36","reported_seq":919,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.455602+0000","last_change":"2024-02-10T07:47:47.180236+0000","last_active":"2024-02-10T07:48:20.455602+0000","last_peered":"2024-02-10T07:48:20.455602+0000","last_clean":"2024-02-10T07:48:20.455602+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:48:20.455602+0000","last_undegraded":"2024-02-10T07:48:20.455602+0000","last_fullsized":"2024-02-10T07:48:20.455602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'36","last_scrub_stamp":"2024-02-10T07:47:47.180016+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:47:47.180016+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:09.218137+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":676,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.457394+0000","last_change":"2024-02-10T07:46:30.869162+0000","last_active":"2024-02-10T07:48:20.457394+0000","last_peered":"2024-02-10T07:48:20.457394+0000","last_clean":"2024-02-10T07:48:20.457394+0000","last_became_active":"2024-02-10T07:46:10.714440+0000","last_became_peered":"2024-02-10T07:46:10.714440+0000","last_unstale":"2024-02-10T07:48:20.457394+0000","last_undegraded":"2024-02-10T07:48:20.457394+0000","last_fullsized":"2024-02-10T07:48:20.457394+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:30.869034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.736901+0000","last_clean_scrub_stamp":"2024-02-10T07:46:30.869034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"361'9133","reported_seq":9252,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:24.151162+0000","last_change":"2024-02-10T07:46:10.721112+0000","last_active":"2024-02-10T07:48:24.151162+0000","last_peered":"2024-02-10T07:48:24.151162+0000","last_clean":"2024-02-10T07:48:24.151162+0000","last_became_active":"2024-02-10T07:46:10.720735+0000","last_became_peered":"2024-02-10T07:46:10.720735+0000","last_unstale":"2024-02-10T07:48:24.151162+0000","last_undegraded":"2024-02-10T07:48:24.151162+0000","last_fullsized":"2024-02-10T07:48:24.151162+0000","mapping_epoch":312,"log_start":"334'4900","ondisk_log_start":"334'4900","created":309,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295108608,"num_objects":4503,"num_object_clones":0,"num_object_copies":13509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9133,"num_write_kb":436352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":440,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.468199+0000","last_change":"2024-02-10T07:46:40.124554+0000","last_active":"2024-02-10T07:48:20.468199+0000","last_peered":"2024-02-10T07:48:20.468199+0000","last_clean":"2024-02-10T07:48:20.468199+0000","last_became_active":"2024-02-10T07:46:40.124220+0000","last_became_peered":"2024-02-10T07:46:40.124220+0000","last_unstale":"2024-02-10T07:48:20.468199+0000","last_undegraded":"2024-02-10T07:48:20.468199+0000","last_fullsized":"2024-02-10T07:48:20.468199+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:44:49.224791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:44:49.224791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":670,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451627+0000","last_change":"2024-02-10T07:47:41.181852+0000","last_active":"2024-02-10T07:48:20.451627+0000","last_peered":"2024-02-10T07:48:20.451627+0000","last_clean":"2024-02-10T07:48:20.451627+0000","last_became_active":"2024-02-10T07:46:40.131499+0000","last_became_peered":"2024-02-10T07:46:40.131499+0000","last_unstale":"2024-02-10T07:48:20.451627+0000","last_undegraded":"2024-02-10T07:48:20.451627+0000","last_fullsized":"2024-02-10T07:48:20.451627+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_clean_scrub_stamp":"2024-02-10T07:47:41.181770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:43.220644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1667,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:21.765615+0000","last_change":"2024-02-10T07:47:42.096274+0000","last_active":"2024-02-10T07:48:21.765615+0000","last_peered":"2024-02-10T07:48:21.765615+0000","last_clean":"2024-02-10T07:48:21.765615+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:48:21.765615+0000","last_undegraded":"2024-02-10T07:48:21.765615+0000","last_fullsized":"2024-02-10T07:48:21.765615+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:47:42.095508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.095508+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:54.407009+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":985,"num_read_kb":980,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"250'30","reported_seq":719,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451797+0000","last_change":"2024-02-10T07:46:10.740692+0000","last_active":"2024-02-10T07:48:20.451797+0000","last_peered":"2024-02-10T07:48:20.451797+0000","last_clean":"2024-02-10T07:48:20.451797+0000","last_became_active":"2024-02-10T07:46:10.724393+0000","last_became_peered":"2024-02-10T07:46:10.724393+0000","last_unstale":"2024-02-10T07:48:20.451797+0000","last_undegraded":"2024-02-10T07:48:20.451797+0000","last_fullsized":"2024-02-10T07:48:20.451797+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:45:58.123359+0000","last_deep_scrub":"237'18","last_deep_scrub_stamp":"2024-02-10T07:40:31.005376+0000","last_clean_scrub_stamp":"2024-02-10T07:45:58.123359+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":608,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.462608+0000","last_change":"2024-02-10T07:48:14.683211+0000","last_active":"2024-02-10T07:48:20.462608+0000","last_peered":"2024-02-10T07:48:20.462608+0000","last_clean":"2024-02-10T07:48:20.462608+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:48:20.462608+0000","last_undegraded":"2024-02-10T07:48:20.462608+0000","last_fullsized":"2024-02-10T07:48:20.462608+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:14.683089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:14.683089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:30.380839+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"361'9050","reported_seq":9164,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:23.312893+0000","last_change":"2024-02-10T07:46:06.701630+0000","last_active":"2024-02-10T07:48:23.312893+0000","last_peered":"2024-02-10T07:48:23.312893+0000","last_clean":"2024-02-10T07:48:23.312893+0000","last_became_active":"2024-02-10T07:46:06.700757+0000","last_became_peered":"2024-02-10T07:46:06.700757+0000","last_unstale":"2024-02-10T07:48:23.312893+0000","last_undegraded":"2024-02-10T07:48:23.312893+0000","last_fullsized":"2024-02-10T07:48:23.312893+0000","mapping_epoch":309,"log_start":"331'4800","ondisk_log_start":"331'4800","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302841880,"num_objects":4622,"num_object_clones":0,"num_object_copies":13866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4622,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9050,"num_write_kb":437441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":669,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.467333+0000","last_change":"2024-02-10T07:48:13.249120+0000","last_active":"2024-02-10T07:48:20.467333+0000","last_peered":"2024-02-10T07:48:20.467333+0000","last_clean":"2024-02-10T07:48:20.467333+0000","last_became_active":"2024-02-10T07:46:10.716852+0000","last_became_peered":"2024-02-10T07:46:10.716852+0000","last_unstale":"2024-02-10T07:48:20.467333+0000","last_undegraded":"2024-02-10T07:48:20.467333+0000","last_fullsized":"2024-02-10T07:48:20.467333+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:13.249033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.249033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:19.265706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":674,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.469006+0000","last_change":"2024-02-10T07:46:10.718679+0000","last_active":"2024-02-10T07:48:20.469006+0000","last_peered":"2024-02-10T07:48:20.469006+0000","last_clean":"2024-02-10T07:48:20.469006+0000","last_became_active":"2024-02-10T07:46:10.718340+0000","last_became_peered":"2024-02-10T07:46:10.718340+0000","last_unstale":"2024-02-10T07:48:20.469006+0000","last_undegraded":"2024-02-10T07:48:20.469006+0000","last_fullsized":"2024-02-10T07:48:20.469006+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:43.511591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:45:43.511591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"271'37","reported_seq":725,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454059+0000","last_change":"2024-02-10T07:46:28.324286+0000","last_active":"2024-02-10T07:48:20.454059+0000","last_peered":"2024-02-10T07:48:20.454059+0000","last_clean":"2024-02-10T07:48:20.454059+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:48:20.454059+0000","last_undegraded":"2024-02-10T07:48:20.454059+0000","last_fullsized":"2024-02-10T07:48:20.454059+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'37","last_scrub_stamp":"2024-02-10T07:46:28.324161+0000","last_deep_scrub":"271'37","last_deep_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_clean_scrub_stamp":"2024-02-10T07:46:28.324161+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":677,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454431+0000","last_change":"2024-02-10T07:46:00.292423+0000","last_active":"2024-02-10T07:48:20.454431+0000","last_peered":"2024-02-10T07:48:20.454431+0000","last_clean":"2024-02-10T07:48:20.454431+0000","last_became_active":"2024-02-10T07:36:33.765998+0000","last_became_peered":"2024-02-10T07:36:33.765998+0000","last_unstale":"2024-02-10T07:48:20.454431+0000","last_undegraded":"2024-02-10T07:48:20.454431+0000","last_fullsized":"2024-02-10T07:48:20.454431+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:00.292320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:00.292320+0000","last_clean_scrub_stamp":"2024-02-10T07:46:00.292320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"361'8971","reported_seq":9143,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:23.235965+0000","last_change":"2024-02-10T07:47:24.895006+0000","last_active":"2024-02-10T07:48:23.235965+0000","last_peered":"2024-02-10T07:48:23.235965+0000","last_clean":"2024-02-10T07:48:23.235965+0000","last_became_active":"2024-02-10T07:46:06.695925+0000","last_became_peered":"2024-02-10T07:46:06.695925+0000","last_unstale":"2024-02-10T07:48:23.235965+0000","last_undegraded":"2024-02-10T07:48:23.235965+0000","last_fullsized":"2024-02-10T07:48:23.235965+0000","mapping_epoch":309,"log_start":"331'4500","ondisk_log_start":"331'4500","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'5171","last_scrub_stamp":"2024-02-10T07:47:24.894834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:47:24.894834+0000","objects_scrubbed":5171,"log_size":4471,"log_dups_size":0,"ondisk_log_size":4471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:29.263050+0000","scrub_duration":802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294191104,"num_objects":4489,"num_object_clones":0,"num_object_copies":13467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8971,"num_write_kb":430720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":544,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.453867+0000","last_change":"2024-02-10T07:47:43.180122+0000","last_active":"2024-02-10T07:48:20.453867+0000","last_peered":"2024-02-10T07:48:20.453867+0000","last_clean":"2024-02-10T07:48:20.453867+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:48:20.453867+0000","last_undegraded":"2024-02-10T07:48:20.453867+0000","last_fullsized":"2024-02-10T07:48:20.453867+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:43.179398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.179398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:08.620817+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":577,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.455188+0000","last_change":"2024-02-10T07:48:02.330991+0000","last_active":"2024-02-10T07:48:20.455188+0000","last_peered":"2024-02-10T07:48:20.455188+0000","last_clean":"2024-02-10T07:48:20.455188+0000","last_became_active":"2024-02-10T07:48:02.330524+0000","last_became_peered":"2024-02-10T07:48:02.330524+0000","last_unstale":"2024-02-10T07:48:20.455188+0000","last_undegraded":"2024-02-10T07:48:20.455188+0000","last_fullsized":"2024-02-10T07:48:20.455188+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:33.538497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:46:33.538497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"250'20","reported_seq":466,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.467601+0000","last_change":"2024-02-10T07:45:58.418445+0000","last_active":"2024-02-10T07:48:20.467601+0000","last_peered":"2024-02-10T07:48:20.467601+0000","last_clean":"2024-02-10T07:48:20.467601+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:48:20.467601+0000","last_undegraded":"2024-02-10T07:48:20.467601+0000","last_fullsized":"2024-02-10T07:48:20.467601+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:45:58.418313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:45:58.418313+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"158'10","reported_seq":870,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454375+0000","last_change":"2024-02-10T07:45:50.057298+0000","last_active":"2024-02-10T07:48:20.454375+0000","last_peered":"2024-02-10T07:48:20.454375+0000","last_clean":"2024-02-10T07:48:20.454375+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:48:20.454375+0000","last_undegraded":"2024-02-10T07:48:20.454375+0000","last_fullsized":"2024-02-10T07:48:20.454375+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:45:50.057213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:50.057213+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"361'9008","reported_seq":14368,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:23.308666+0000","last_change":"2024-02-10T07:47:34.186577+0000","last_active":"2024-02-10T07:48:23.308666+0000","last_peered":"2024-02-10T07:48:23.308666+0000","last_clean":"2024-02-10T07:48:23.308666+0000","last_became_active":"2024-02-10T07:47:34.183543+0000","last_became_peered":"2024-02-10T07:47:34.183543+0000","last_unstale":"2024-02-10T07:48:23.308666+0000","last_undegraded":"2024-02-10T07:48:23.308666+0000","last_fullsized":"2024-02-10T07:48:23.308666+0000","mapping_epoch":340,"log_start":"331'4700","ondisk_log_start":"331'4700","created":309,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4308,"log_dups_size":0,"ondisk_log_size":4308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300285952,"num_objects":4582,"num_object_clones":0,"num_object_copies":13746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9008,"num_write_kb":434880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2618,"num_bytes_recovered":171573248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":659,"reported_epoch":361,"state":"active+clean+scrubbing+deep","last_fresh":"2024-02-10T07:48:23.133668+0000","last_change":"2024-02-10T07:48:23.113676+0000","last_active":"2024-02-10T07:48:23.133668+0000","last_peered":"2024-02-10T07:48:23.133668+0000","last_clean":"2024-02-10T07:48:23.133668+0000","last_became_active":"2024-02-10T07:46:10.719534+0000","last_became_peered":"2024-02-10T07:46:10.719534+0000","last_unstale":"2024-02-10T07:48:23.133668+0000","last_undegraded":"2024-02-10T07:48:23.133668+0000","last_fullsized":"2024-02-10T07:48:23.133668+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:45.707189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:45:45.707189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"296'2","reported_seq":671,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.451903+0000","last_change":"2024-02-10T07:46:29.295971+0000","last_active":"2024-02-10T07:48:20.451903+0000","last_peered":"2024-02-10T07:48:20.451903+0000","last_clean":"2024-02-10T07:48:20.451903+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:48:20.451903+0000","last_undegraded":"2024-02-10T07:48:20.451903+0000","last_fullsized":"2024-02-10T07:48:20.451903+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'2","last_scrub_stamp":"2024-02-10T07:46:29.295573+0000","last_deep_scrub":"296'2","last_deep_scrub_stamp":"2024-02-10T07:45:03.233375+0000","last_clean_scrub_stamp":"2024-02-10T07:46:29.295573+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"250'28","reported_seq":700,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.463559+0000","last_change":"2024-02-10T07:45:10.185368+0000","last_active":"2024-02-10T07:48:20.463559+0000","last_peered":"2024-02-10T07:48:20.463559+0000","last_clean":"2024-02-10T07:48:20.463559+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:48:20.463559+0000","last_undegraded":"2024-02-10T07:48:20.463559+0000","last_fullsized":"2024-02-10T07:48:20.463559+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-02-10T07:45:10.185239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:45:10.185239+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"325'11","reported_seq":647,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.467099+0000","last_change":"2024-02-10T07:46:40.131958+0000","last_active":"2024-02-10T07:48:20.467099+0000","last_peered":"2024-02-10T07:48:20.467099+0000","last_clean":"2024-02-10T07:48:20.467099+0000","last_became_active":"2024-02-10T07:46:40.121401+0000","last_became_peered":"2024-02-10T07:46:40.121401+0000","last_unstale":"2024-02-10T07:48:20.467099+0000","last_undegraded":"2024-02-10T07:48:20.467099+0000","last_fullsized":"2024-02-10T07:48:20.467099+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'9","last_scrub_stamp":"2024-02-10T07:44:55.376986+0000","last_deep_scrub":"227'9","last_deep_scrub_stamp":"2024-02-10T07:39:53.440221+0000","last_clean_scrub_stamp":"2024-02-10T07:44:55.376986+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"361'8974","reported_seq":9092,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:23.190570+0000","last_change":"2024-02-10T07:46:10.721591+0000","last_active":"2024-02-10T07:48:23.190570+0000","last_peered":"2024-02-10T07:48:23.190570+0000","last_clean":"2024-02-10T07:48:23.190570+0000","last_became_active":"2024-02-10T07:46:10.721275+0000","last_became_peered":"2024-02-10T07:46:10.721275+0000","last_unstale":"2024-02-10T07:48:23.190570+0000","last_undegraded":"2024-02-10T07:48:23.190570+0000","last_fullsized":"2024-02-10T07:48:23.190570+0000","mapping_epoch":312,"log_start":"336'5200","ondisk_log_start":"336'5200","created":309,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":3774,"log_dups_size":0,"ondisk_log_size":3774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295829504,"num_objects":4514,"num_object_clones":0,"num_object_copies":13542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8974,"num_write_kb":431616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":654,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452342+0000","last_change":"2024-02-10T07:46:41.118534+0000","last_active":"2024-02-10T07:48:20.452342+0000","last_peered":"2024-02-10T07:48:20.452342+0000","last_clean":"2024-02-10T07:48:20.452342+0000","last_became_active":"2024-02-10T07:46:41.118200+0000","last_became_peered":"2024-02-10T07:46:41.118200+0000","last_unstale":"2024-02-10T07:48:20.452342+0000","last_undegraded":"2024-02-10T07:48:20.452342+0000","last_fullsized":"2024-02-10T07:48:20.452342+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:27.819612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:46:27.819612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":151,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.456581+0000","last_change":"2024-02-10T07:47:50.948710+0000","last_active":"2024-02-10T07:48:20.456581+0000","last_peered":"2024-02-10T07:48:20.456581+0000","last_clean":"2024-02-10T07:48:20.456581+0000","last_became_active":"2024-02-10T07:47:47.966443+0000","last_became_peered":"2024-02-10T07:47:47.966443+0000","last_unstale":"2024-02-10T07:48:20.456581+0000","last_undegraded":"2024-02-10T07:48:20.456581+0000","last_fullsized":"2024-02-10T07:48:20.456581+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.948624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.948624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:17.232684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"250'30","reported_seq":618,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.457045+0000","last_change":"2024-02-10T07:47:32.154259+0000","last_active":"2024-02-10T07:48:20.457045+0000","last_peered":"2024-02-10T07:48:20.457045+0000","last_clean":"2024-02-10T07:48:20.457045+0000","last_became_active":"2024-02-10T07:46:40.127999+0000","last_became_peered":"2024-02-10T07:46:40.127999+0000","last_unstale":"2024-02-10T07:48:20.457045+0000","last_undegraded":"2024-02-10T07:48:20.457045+0000","last_fullsized":"2024-02-10T07:48:20.457045+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:47:32.153637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:32.153637+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:35.065241+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":662,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.463350+0000","last_change":"2024-02-10T07:47:42.549136+0000","last_active":"2024-02-10T07:48:20.463350+0000","last_peered":"2024-02-10T07:48:20.463350+0000","last_clean":"2024-02-10T07:48:20.463350+0000","last_became_active":"2024-02-10T07:46:40.132156+0000","last_became_peered":"2024-02-10T07:46:40.132156+0000","last_unstale":"2024-02-10T07:48:20.463350+0000","last_undegraded":"2024-02-10T07:48:20.463350+0000","last_fullsized":"2024-02-10T07:48:20.463350+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:42.549018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.549018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:04.448170+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"361'8985","reported_seq":9098,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:23.206270+0000","last_change":"2024-02-10T07:46:06.699530+0000","last_active":"2024-02-10T07:48:23.206270+0000","last_peered":"2024-02-10T07:48:23.206270+0000","last_clean":"2024-02-10T07:48:23.206270+0000","last_became_active":"2024-02-10T07:46:06.699247+0000","last_became_peered":"2024-02-10T07:46:06.699247+0000","last_unstale":"2024-02-10T07:48:23.206270+0000","last_undegraded":"2024-02-10T07:48:23.206270+0000","last_fullsized":"2024-02-10T07:48:23.206270+0000","mapping_epoch":309,"log_start":"336'5200","ondisk_log_start":"336'5200","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":3785,"log_dups_size":0,"ondisk_log_size":3785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295108608,"num_objects":4503,"num_object_clones":0,"num_object_copies":13509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8985,"num_write_kb":431616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":168,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.467971+0000","last_change":"2024-02-10T07:48:02.372236+0000","last_active":"2024-02-10T07:48:20.467971+0000","last_peered":"2024-02-10T07:48:20.467971+0000","last_clean":"2024-02-10T07:48:20.467971+0000","last_became_active":"2024-02-10T07:48:02.371926+0000","last_became_peered":"2024-02-10T07:48:02.371926+0000","last_unstale":"2024-02-10T07:48:20.467971+0000","last_undegraded":"2024-02-10T07:48:20.467971+0000","last_fullsized":"2024-02-10T07:48:20.467971+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:45.419912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:49.704024+0000","last_clean_scrub_stamp":"2024-02-10T07:45:45.419912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"158'10","reported_seq":862,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452377+0000","last_change":"2024-02-10T07:45:06.247795+0000","last_active":"2024-02-10T07:48:20.452377+0000","last_peered":"2024-02-10T07:48:20.452377+0000","last_clean":"2024-02-10T07:48:20.452377+0000","last_became_active":"2024-02-10T07:36:33.755111+0000","last_became_peered":"2024-02-10T07:36:33.755111+0000","last_unstale":"2024-02-10T07:48:20.452377+0000","last_undegraded":"2024-02-10T07:48:20.452377+0000","last_fullsized":"2024-02-10T07:48:20.452377+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:45:06.247355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:06.247355+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":151,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.467720+0000","last_change":"2024-02-10T07:47:47.302669+0000","last_active":"2024-02-10T07:48:20.467720+0000","last_peered":"2024-02-10T07:48:20.467720+0000","last_clean":"2024-02-10T07:48:20.467720+0000","last_became_active":"2024-02-10T07:46:40.116253+0000","last_became_peered":"2024-02-10T07:46:40.116253+0000","last_unstale":"2024-02-10T07:48:20.467720+0000","last_undegraded":"2024-02-10T07:48:20.467720+0000","last_fullsized":"2024-02-10T07:48:20.467720+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:47.302598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:47:47.302598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:05.357565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"250'27","reported_seq":715,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.469183+0000","last_change":"2024-02-10T07:48:16.280198+0000","last_active":"2024-02-10T07:48:20.469183+0000","last_peered":"2024-02-10T07:48:20.469183+0000","last_clean":"2024-02-10T07:48:20.469183+0000","last_became_active":"2024-02-10T07:47:47.967427+0000","last_became_peered":"2024-02-10T07:47:47.967427+0000","last_unstale":"2024-02-10T07:48:20.469183+0000","last_undegraded":"2024-02-10T07:48:20.469183+0000","last_fullsized":"2024-02-10T07:48:20.469183+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:48:16.280030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:16.280030+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:19.284535+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"361'9081","reported_seq":9195,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:23.297104+0000","last_change":"2024-02-10T07:46:06.699422+0000","last_active":"2024-02-10T07:48:23.297104+0000","last_peered":"2024-02-10T07:48:23.297104+0000","last_clean":"2024-02-10T07:48:23.297104+0000","last_became_active":"2024-02-10T07:46:06.699140+0000","last_became_peered":"2024-02-10T07:46:06.699140+0000","last_unstale":"2024-02-10T07:48:23.297104+0000","last_undegraded":"2024-02-10T07:48:23.297104+0000","last_fullsized":"2024-02-10T07:48:23.297104+0000","mapping_epoch":309,"log_start":"331'4800","ondisk_log_start":"331'4800","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":4281,"log_dups_size":0,"ondisk_log_size":4281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294060032,"num_objects":4487,"num_object_clones":0,"num_object_copies":13461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9081,"num_write_kb":434176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":348,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454523+0000","last_change":"2024-02-10T07:47:41.208022+0000","last_active":"2024-02-10T07:48:20.454523+0000","last_peered":"2024-02-10T07:48:20.454523+0000","last_clean":"2024-02-10T07:48:20.454523+0000","last_became_active":"2024-02-10T07:46:40.124029+0000","last_became_peered":"2024-02-10T07:46:40.124029+0000","last_unstale":"2024-02-10T07:48:20.454523+0000","last_undegraded":"2024-02-10T07:48:20.454523+0000","last_fullsized":"2024-02-10T07:48:20.454523+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:41.207934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_clean_scrub_stamp":"2024-02-10T07:47:41.207934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:06.545450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":142,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.460649+0000","last_change":"2024-02-10T07:48:02.334048+0000","last_active":"2024-02-10T07:48:20.460649+0000","last_peered":"2024-02-10T07:48:20.460649+0000","last_clean":"2024-02-10T07:48:20.460649+0000","last_became_active":"2024-02-10T07:48:02.333689+0000","last_became_peered":"2024-02-10T07:48:02.333689+0000","last_unstale":"2024-02-10T07:48:20.460649+0000","last_undegraded":"2024-02-10T07:48:20.460649+0000","last_fullsized":"2024-02-10T07:48:20.460649+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:47.343978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:47.343978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":667,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.463429+0000","last_change":"2024-02-10T07:46:37.995480+0000","last_active":"2024-02-10T07:48:20.463429+0000","last_peered":"2024-02-10T07:48:20.463429+0000","last_clean":"2024-02-10T07:48:20.463429+0000","last_became_active":"2024-02-10T07:46:10.721886+0000","last_became_peered":"2024-02-10T07:46:10.721886+0000","last_unstale":"2024-02-10T07:48:20.463429+0000","last_undegraded":"2024-02-10T07:48:20.463429+0000","last_fullsized":"2024-02-10T07:48:20.463429+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:37.995362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.995362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"250'24","reported_seq":721,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452157+0000","last_change":"2024-02-10T07:45:05.256634+0000","last_active":"2024-02-10T07:48:20.452157+0000","last_peered":"2024-02-10T07:48:20.452157+0000","last_clean":"2024-02-10T07:48:20.452157+0000","last_became_active":"2024-02-10T07:37:06.193386+0000","last_became_peered":"2024-02-10T07:37:06.193386+0000","last_unstale":"2024-02-10T07:48:20.452157+0000","last_undegraded":"2024-02-10T07:48:20.452157+0000","last_fullsized":"2024-02-10T07:48:20.452157+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:45:05.256278+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:45:05.256278+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"361'9001","reported_seq":14302,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:23.181386+0000","last_change":"2024-02-10T07:48:02.387002+0000","last_active":"2024-02-10T07:48:23.181386+0000","last_peered":"2024-02-10T07:48:23.181386+0000","last_clean":"2024-02-10T07:48:23.181386+0000","last_became_active":"2024-02-10T07:48:02.366190+0000","last_became_peered":"2024-02-10T07:48:02.366190+0000","last_unstale":"2024-02-10T07:48:23.181386+0000","last_undegraded":"2024-02-10T07:48:23.181386+0000","last_fullsized":"2024-02-10T07:48:23.181386+0000","mapping_epoch":355,"log_start":"338'5188","ondisk_log_start":"338'5188","created":309,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":3813,"log_dups_size":0,"ondisk_log_size":3813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298254336,"num_objects":4551,"num_object_clones":0,"num_object_copies":13653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9001,"num_write_kb":433664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2588,"num_bytes_recovered":169607168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":635,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.460524+0000","last_change":"2024-02-10T07:47:45.659017+0000","last_active":"2024-02-10T07:48:20.460524+0000","last_peered":"2024-02-10T07:48:20.460524+0000","last_clean":"2024-02-10T07:48:20.460524+0000","last_became_active":"2024-02-10T07:46:41.216189+0000","last_became_peered":"2024-02-10T07:46:41.216189+0000","last_unstale":"2024-02-10T07:48:20.460524+0000","last_undegraded":"2024-02-10T07:48:20.460524+0000","last_fullsized":"2024-02-10T07:48:20.460524+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:45.658749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:47:45.658749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:51.828516+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":651,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.469252+0000","last_change":"2024-02-10T07:45:50.420105+0000","last_active":"2024-02-10T07:48:20.469252+0000","last_peered":"2024-02-10T07:48:20.469252+0000","last_clean":"2024-02-10T07:48:20.469252+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:48:20.469252+0000","last_undegraded":"2024-02-10T07:48:20.469252+0000","last_fullsized":"2024-02-10T07:48:20.469252+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:50.420019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:45:50.420019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":684,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454508+0000","last_change":"2024-02-10T07:48:20.145267+0000","last_active":"2024-02-10T07:48:20.454508+0000","last_peered":"2024-02-10T07:48:20.454508+0000","last_clean":"2024-02-10T07:48:20.454508+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:48:20.454508+0000","last_undegraded":"2024-02-10T07:48:20.454508+0000","last_fullsized":"2024-02-10T07:48:20.454508+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:20.145002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:44.067302+0000","last_clean_scrub_stamp":"2024-02-10T07:48:20.145002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:26.149318+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"250'39","reported_seq":730,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452061+0000","last_change":"2024-02-10T07:46:10.733638+0000","last_active":"2024-02-10T07:48:20.452061+0000","last_peered":"2024-02-10T07:48:20.452061+0000","last_clean":"2024-02-10T07:48:20.452061+0000","last_became_active":"2024-02-10T07:46:10.721474+0000","last_became_peered":"2024-02-10T07:46:10.721474+0000","last_unstale":"2024-02-10T07:48:20.452061+0000","last_undegraded":"2024-02-10T07:48:20.452061+0000","last_fullsized":"2024-02-10T07:48:20.452061+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T07:44:55.232884+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_clean_scrub_stamp":"2024-02-10T07:44:55.232884+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"361'9152","reported_seq":9266,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:24.104522+0000","last_change":"2024-02-10T07:46:06.698262+0000","last_active":"2024-02-10T07:48:24.104522+0000","last_peered":"2024-02-10T07:48:24.104522+0000","last_clean":"2024-02-10T07:48:24.104522+0000","last_became_active":"2024-02-10T07:46:06.697863+0000","last_became_peered":"2024-02-10T07:46:06.697863+0000","last_unstale":"2024-02-10T07:48:24.104522+0000","last_undegraded":"2024-02-10T07:48:24.104522+0000","last_fullsized":"2024-02-10T07:48:24.104522+0000","mapping_epoch":309,"log_start":"338'5400","ondisk_log_start":"338'5400","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:05.681946+0000","last_clean_scrub_stamp":"2024-02-10T07:46:05.681946+0000","objects_scrubbed":0,"log_size":3752,"log_dups_size":0,"ondisk_log_size":3752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297926656,"num_objects":4546,"num_object_clones":0,"num_object_copies":13638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":438336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":647,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452401+0000","last_change":"2024-02-10T07:46:10.718207+0000","last_active":"2024-02-10T07:48:20.452401+0000","last_peered":"2024-02-10T07:48:20.452401+0000","last_clean":"2024-02-10T07:48:20.452401+0000","last_became_active":"2024-02-10T07:46:10.717081+0000","last_became_peered":"2024-02-10T07:46:10.717081+0000","last_unstale":"2024-02-10T07:48:20.452401+0000","last_undegraded":"2024-02-10T07:48:20.452401+0000","last_fullsized":"2024-02-10T07:48:20.452401+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:45:41.997290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:45:41.997290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"313'11","reported_seq":174,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.458607+0000","last_change":"2024-02-10T07:46:38.873286+0000","last_active":"2024-02-10T07:48:20.458607+0000","last_peered":"2024-02-10T07:48:20.458607+0000","last_clean":"2024-02-10T07:48:20.458607+0000","last_became_active":"2024-02-10T07:46:10.719380+0000","last_became_peered":"2024-02-10T07:46:10.719380+0000","last_unstale":"2024-02-10T07:48:20.458607+0000","last_undegraded":"2024-02-10T07:48:20.458607+0000","last_fullsized":"2024-02-10T07:48:20.458607+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'11","last_scrub_stamp":"2024-02-10T07:46:38.873049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:38.873049+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":152,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.454209+0000","last_change":"2024-02-10T07:47:24.220672+0000","last_active":"2024-02-10T07:48:20.454209+0000","last_peered":"2024-02-10T07:48:20.454209+0000","last_clean":"2024-02-10T07:48:20.454209+0000","last_became_active":"2024-02-10T07:46:40.129473+0000","last_became_peered":"2024-02-10T07:46:40.129473+0000","last_unstale":"2024-02-10T07:48:20.454209+0000","last_undegraded":"2024-02-10T07:48:20.454209+0000","last_fullsized":"2024-02-10T07:48:20.454209+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:24.220575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_clean_scrub_stamp":"2024-02-10T07:47:24.220575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:48:24.673830+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"250'20","reported_seq":708,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-02-10T07:48:20.452958+0000","last_change":"2024-02-10T07:48:15.588337+0000","last_active":"2024-02-10T07:48:20.452958+0000","last_peered":"2024-02-10T07:48:20.452958+0000","last_clean":"2024-02-10T07:48:20.452958+0000","last_became_active":"2024-02-10T07:46:40.116204+0000","last_became_peered":"2024-02-10T07:46:40.116204+0000","last_unstale":"2024-02-10T07:48:20.452958+0000","last_undegraded":"2024-02-10T07:48:20.452958+0000","last_fullsized":"2024-02-10T07:48:20.452958+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":325,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:48:15.588199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.588199+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:49:22.399116+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4743364632,"num_objects":72379,"num_object_clones":0,"num_object_copies":217137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72379,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144539,"num_write_kb":6941313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16955,"num_bytes_recovered":1110900736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14440738816,"data_stored":14440726600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66645,"ondisk_log_size":66645,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":985,"num_read_kb":980,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1112,"num_read_kb":869,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":3796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":11904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":767,"ondisk_log_size":767,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037963,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2052828,"kb_used_data":1453716,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":91692324,"statfs":{"total":95995035648,"available":93892939776,"internally_reserved":0,"allocated":1488605184,"data_stored":1488008640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":613349834},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:48:12 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.2090000000000001,"15min":1.831},"min":{"1min":0.215,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":7.5359999999999996,"5min":10.529,"15min":52.591999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.1659999999999999,"15min":1.696},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":8.1980000000000004,"5min":10.640000000000001,"15min":52.494},"last":0.89200000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:48:12 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.6479999999999999,"15min":1.6639999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.24399999999999999},"max":{"1min":4.0300000000000002,"5min":30.228000000000002,"15min":35.459000000000003},"last":0.376},{"interface":"front","average":{"1min":0.88900000000000001,"5min":1.5620000000000001,"15min":1.643},"min":{"1min":0.26500000000000001,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":3.1699999999999999,"5min":23.239000000000001,"15min":35.225000000000001},"last":0.85899999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.93999999999999995,"15min":1.206},"min":{"1min":0.318,"5min":0.255,"15min":0.255},"max":{"1min":3.101,"5min":5.9450000000000003,"15min":19.710000000000001},"last":0.503},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.026,"15min":1.3420000000000001},"min":{"1min":0.32400000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.0010000000000003,"5min":4.2320000000000002,"15min":19.643000000000001},"last":0.41299999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:45:00 2024","interfaces":[{"interface":"back","average":{"1min":3.0830000000000002,"5min":2.6190000000000002,"15min":1.889},"min":{"1min":0.29299999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":19.184000000000001,"5min":43.573999999999998,"15min":64.120999999999995},"last":0.92200000000000004},{"interface":"front","average":{"1min":2.577,"5min":2.5459999999999998,"15min":2.0019999999999998},"min":{"1min":0.35299999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":17.806999999999999,"5min":43.451000000000001,"15min":64.400000000000006},"last":0.47099999999999997}]},{"osd":4,"last update":"Sat Feb 10 07:48:12 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.381,"15min":0.88300000000000001},"min":{"1min":0.39800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":8.6150000000000002,"5min":18.407,"15min":18.407},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.7969999999999999,"5min":1.389,"15min":0.88200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.24099999999999999},"max":{"1min":6.5330000000000004,"5min":18.609000000000002,"15min":18.609000000000002},"last":5.5670000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:47:56 2024","interfaces":[{"interface":"back","average":{"1min":2.7370000000000001,"5min":1.419,"15min":0.97599999999999998},"min":{"1min":0.36599999999999999,"5min":0.29099999999999998,"15min":0.245},"max":{"1min":18.126999999999999,"5min":18.126999999999999,"15min":18.126999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":3.2109999999999999,"5min":1.4199999999999999,"15min":1.03},"min":{"1min":0.34100000000000003,"5min":0.32300000000000001,"15min":0.29999999999999999},"max":{"1min":19.548999999999999,"5min":19.548999999999999,"15min":19.548999999999999},"last":0.95999999999999996}]},{"osd":6,"last update":"Sat Feb 10 07:48:12 2024","interfaces":[{"interface":"back","average":{"1min":2.0270000000000001,"5min":1.385,"15min":0.96799999999999997},"min":{"1min":0.432,"5min":0.34499999999999997,"15min":0.28199999999999997},"max":{"1min":7.726,"5min":17.832000000000001,"15min":17.832000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":2.1699999999999999,"5min":1.6060000000000001,"15min":1.0329999999999999},"min":{"1min":0.504,"5min":0.33700000000000002,"15min":0.27400000000000002},"max":{"1min":7.3920000000000003,"5min":20.449999999999999,"15min":20.449999999999999},"last":5.4009999999999998}]}]},{"osd":6,"up_from":50,"seq":214748365008,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2512176,"kb_used_data":2043688,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":91232976,"statfs":{"total":95995035648,"available":93422567424,"internally_reserved":0,"allocated":2092736512,"data_stored":2092120503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":479721926},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:47:39 2024","interfaces":[{"interface":"back","average":{"1min":3.8140000000000001,"5min":1.5940000000000001,"15min":1.6180000000000001},"min":{"1min":0.29099999999999998,"5min":0.23200000000000001,"15min":0.217},"max":{"1min":24.727,"5min":24.727,"15min":63.405999999999999},"last":4.8369999999999997},{"interface":"front","average":{"1min":3.4510000000000001,"5min":1.327,"15min":1.7170000000000001},"min":{"1min":0.26800000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":29.084,"5min":29.084,"15min":67.748000000000005},"last":0.29499999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:47:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.5069999999999999,"15min":1.571},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.8340000000000001,"5min":20.573,"15min":37.659999999999997},"last":0.436},{"interface":"front","average":{"1min":2.6230000000000002,"5min":1.7170000000000001,"15min":1.649},"min":{"1min":0.317,"5min":0.27000000000000002,"15min":0.24399999999999999},"max":{"1min":8.5190000000000001,"5min":20.795000000000002,"15min":37.75},"last":4.8819999999999997}]},{"osd":2,"last update":"Sat Feb 10 07:47:39 2024","interfaces":[{"interface":"back","average":{"1min":2.786,"5min":1.323,"15min":1.55},"min":{"1min":0.34599999999999997,"5min":0.28899999999999998,"15min":0.253},"max":{"1min":8.7490000000000006,"5min":8.7490000000000006,"15min":65.278999999999996},"last":4.9119999999999999},{"interface":"front","average":{"1min":1.855,"5min":1.05,"15min":1.5349999999999999},"min":{"1min":0.30399999999999999,"5min":0.28399999999999997,"15min":0.25800000000000001},"max":{"1min":6.0190000000000001,"5min":6.0190000000000001,"15min":65.265000000000001},"last":1.6919999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.899,"5min":1.8240000000000001,"15min":1.766},"min":{"1min":0.34799999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":11.103999999999999,"5min":20.603999999999999,"15min":62.835999999999999},"last":1.8180000000000001},{"interface":"front","average":{"1min":2.274,"5min":1.925,"15min":1.4259999999999999},"min":{"1min":0.36399999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":11.304,"5min":20.478999999999999,"15min":63.878999999999998},"last":0.53800000000000003}]},{"osd":4,"last update":"Sat Feb 10 07:47:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5470000000000002,"5min":1.0920000000000001,"15min":0.755},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23400000000000001},"max":{"1min":6.0590000000000002,"5min":6.0590000000000002,"15min":7.7839999999999998},"last":1.7549999999999999},{"interface":"front","average":{"1min":2.0110000000000001,"5min":0.997,"15min":0.72399999999999998},"min":{"1min":0.28699999999999998,"5min":0.28000000000000003,"15min":0.26000000000000001},"max":{"1min":5.9210000000000003,"5min":7.0300000000000002,"15min":7.0300000000000002},"last":0.50900000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:47:52 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":1.8120000000000001,"15min":1.083},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.24099999999999999},"max":{"1min":7.4470000000000001,"5min":44.689,"15min":44.689},"last":1.79},{"interface":"front","average":{"1min":2.3359999999999999,"5min":1.7250000000000001,"15min":1.03},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.29199999999999998},"max":{"1min":9.2249999999999996,"5min":44.465000000000003,"15min":44.465000000000003},"last":1.889}]},{"osd":7,"last update":"Sat Feb 10 07:48:06 2024","interfaces":[{"interface":"back","average":{"1min":3.1269999999999998,"5min":2.0329999999999999,"15min":1.159},"min":{"1min":0.36699999999999999,"5min":0.32200000000000001,"15min":0.28100000000000003},"max":{"1min":9.6479999999999997,"5min":42.631999999999998,"15min":42.631999999999998},"last":4.9379999999999997},{"interface":"front","average":{"1min":2.407,"5min":1.831,"15min":1.0860000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.29699999999999999},"max":{"1min":5.9649999999999999,"5min":44.792999999999999,"15min":44.792999999999999},"last":0.56499999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575082,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1781632,"kb_used_data":1168760,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":91963520,"statfs":{"total":95995035648,"available":94170644480,"internally_reserved":0,"allocated":1196810240,"data_stored":1196176994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":627571142},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:47:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":1.171,"15min":1.758},"min":{"1min":0.215,"5min":0.19,"15min":0.182},"max":{"1min":2.6930000000000001,"5min":7.2850000000000001,"15min":114.27},"last":1.5820000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":1.1639999999999999,"15min":1.95},"min":{"1min":0.27900000000000003,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.157,"5min":7.524,"15min":119.499},"last":0.35299999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":1.04,"15min":1.1919999999999999},"min":{"1min":0.27300000000000002,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":2.4199999999999999,"5min":8.4819999999999993,"15min":8.9209999999999994},"last":1.415},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.98699999999999999,"15min":1.3169999999999999},"min":{"1min":0.27800000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.44500000000000001,"5min":8.8300000000000001,"15min":8.8300000000000001},"last":0.32500000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:44:37 2024","interfaces":[{"interface":"back","average":{"1min":1.8340000000000001,"5min":2.0939999999999999,"15min":1.46},"min":{"1min":0.30299999999999999,"5min":0.222,"15min":0.21099999999999999},"max":{"1min":8.4139999999999997,"5min":11.535,"15min":11.535},"last":1.532},{"interface":"front","average":{"1min":2.6379999999999999,"5min":1.8859999999999999,"15min":1.3979999999999999},"min":{"1min":0.59599999999999997,"5min":0.24199999999999999,"15min":0.23799999999999999},"max":{"1min":8.6289999999999996,"5min":9.516,"15min":11.698},"last":0.26700000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:48:01 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.764,"15min":2.0569999999999999},"min":{"1min":0.38,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.0199999999999996,"5min":31.908999999999999,"15min":207.45699999999999},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.446,"15min":1.2070000000000001},"min":{"1min":0.31,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":5.7999999999999998,"5min":32.201000000000001,"15min":32.201000000000001},"last":0.89300000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:48:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.8380000000000001,"15min":2.0219999999999998},"min":{"1min":0.33100000000000002,"5min":0.308,"15min":0.308},"max":{"1min":4.1509999999999998,"5min":31.824999999999999,"15min":98.703000000000003},"last":1.464},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.9179999999999999,"15min":1.9990000000000001},"min":{"1min":0.38900000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.1740000000000004,"5min":31.841999999999999,"15min":98.664000000000001},"last":0.41399999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:47:42 2024","interfaces":[{"interface":"back","average":{"1min":3.6970000000000001,"5min":1.673,"15min":1.3200000000000001},"min":{"1min":0.38,"5min":0.32800000000000001,"15min":0.308},"max":{"1min":36.808,"5min":36.808,"15min":36.808},"last":0.92300000000000004},{"interface":"front","average":{"1min":3.8050000000000002,"5min":1.9450000000000001,"15min":1.516},"min":{"1min":0.434,"5min":0.35699999999999998,"15min":0.33300000000000002},"max":{"1min":33.774999999999999,"5min":33.774999999999999,"15min":33.774999999999999},"last":2.2629999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":1.45,"15min":1.3640000000000001},"min":{"1min":0.41899999999999998,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":3.3530000000000002,"5min":8.4239999999999995,"15min":8.4239999999999995},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.29,"15min":1.26},"min":{"1min":0.38500000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":6.1260000000000003,"5min":7.7089999999999996,"15min":11.212},"last":0.94699999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738607,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3844320,"kb_used_data":3213208,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":89900832,"statfs":{"total":95995035648,"available":92058451968,"internally_reserved":0,"allocated":3290324992,"data_stored":3289724855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:47:32 2024","interfaces":[{"interface":"back","average":{"1min":2.5099999999999998,"5min":3.6930000000000001,"15min":2.323},"min":{"1min":0.187,"5min":0.182,"15min":0.182},"max":{"1min":34.661999999999999,"5min":156.16200000000001,"15min":156.16200000000001},"last":0.29799999999999999},{"interface":"front","average":{"1min":2.4300000000000002,"5min":3.8879999999999999,"15min":2.339},"min":{"1min":0.20599999999999999,"5min":0.19,"15min":0.183},"max":{"1min":34.636000000000003,"5min":156.941,"15min":156.941},"last":0.192}]},{"osd":2,"last update":"Sat Feb 10 07:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":1.1439999999999999,"15min":1.591},"min":{"1min":0.23400000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.5609999999999999,"5min":34.603000000000002,"15min":34.603000000000002},"last":0.221},{"interface":"front","average":{"1min":0.40300000000000002,"5min":1.089,"15min":1.3740000000000001},"min":{"1min":0.247,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.66000000000000003,"5min":34.718000000000004,"15min":34.718000000000004},"last":0.36099999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:44:19 2024","interfaces":[{"interface":"back","average":{"1min":9.9640000000000004,"5min":3.6070000000000002,"15min":2.2069999999999999},"min":{"1min":0.25700000000000001,"5min":0.222,"15min":0.222},"max":{"1min":97.677000000000007,"5min":97.677000000000007,"15min":97.677000000000007},"last":0.39700000000000002},{"interface":"front","average":{"1min":5.7160000000000002,"5min":2.5670000000000002,"15min":1.784},"min":{"1min":0.32400000000000001,"5min":0.246,"15min":0.246},"max":{"1min":49.344000000000001,"5min":49.344000000000001,"15min":49.344000000000001},"last":0.48699999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.4690000000000001,"15min":1.2310000000000001},"min":{"1min":0.28299999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.0569999999999999,"5min":37.286000000000001,"15min":37.286000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.4239999999999999,"15min":1.417},"min":{"1min":0.315,"5min":0.29199999999999998,"15min":0.26900000000000002},"max":{"1min":6.548,"5min":34.683,"15min":34.683},"last":3.0779999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:47:53 2024","interfaces":[{"interface":"back","average":{"1min":3.6080000000000001,"5min":2.5379999999999998,"15min":2.5339999999999998},"min":{"1min":0.35499999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":41.429000000000002,"5min":41.429000000000002,"15min":41.429000000000002},"last":0.59699999999999998},{"interface":"front","average":{"1min":3.536,"5min":2.5099999999999998,"15min":2.7839999999999998},"min":{"1min":0.34000000000000002,"5min":0.33500000000000002,"15min":0.308},"max":{"1min":41.456000000000003,"5min":41.456000000000003,"15min":41.456000000000003},"last":0.52400000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:47:32 2024","interfaces":[{"interface":"back","average":{"1min":6.8040000000000003,"5min":2.1030000000000002,"15min":1.583},"min":{"1min":0.35899999999999999,"5min":0.35399999999999998,"15min":0.29599999999999999},"max":{"1min":41.587000000000003,"5min":41.587000000000003,"15min":41.587000000000003},"last":1.1040000000000001},{"interface":"front","average":{"1min":6.9109999999999996,"5min":2.1429999999999998,"15min":1.611},"min":{"1min":0.434,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":41.256,"5min":41.256,"15min":41.256},"last":0.67100000000000004}]},{"osd":7,"last update":"Sat Feb 10 07:48:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.615,"15min":1.5629999999999999},"min":{"1min":0.35799999999999998,"5min":0.34899999999999998,"15min":0.33300000000000002},"max":{"1min":3.1970000000000001,"5min":34.710999999999999,"15min":34.710999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.546,"15min":1.3660000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.376},"max":{"1min":4.1660000000000004,"5min":34.997999999999998,"15min":34.997999999999998},"last":0.505}]}]},{"osd":2,"up_from":0,"seq":77309411556,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":347,"seq":1490353651720,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":425444,"kb_used_data":306136,"kb_used_omap":5,"kb_used_meta":119290,"kb_avail":93319708,"statfs":{"total":95995035648,"available":95559380992,"internally_reserved":0,"allocated":313483264,"data_stored":312945543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5625,"internal_metadata":122153479},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.246}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953690,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4146472,"kb_used_data":3511388,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":89598680,"statfs":{"total":95995035648,"available":91749048320,"internally_reserved":0,"allocated":3595661312,"data_stored":3595056142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,6,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:47:54 2024","interfaces":[{"interface":"back","average":{"1min":1.9039999999999999,"5min":1.137,"15min":2.0920000000000001},"min":{"1min":0.26300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":10.859999999999999,"5min":10.859999999999999,"15min":74.533000000000001},"last":2.8210000000000002},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.077,"15min":1.9870000000000001},"min":{"1min":0.245,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.6180000000000003,"5min":4.6180000000000003,"15min":74.340999999999994},"last":4.407}]},{"osd":1,"last update":"Sat Feb 10 07:47:54 2024","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":2.0800000000000001,"15min":2.0499999999999998},"min":{"1min":0.38800000000000001,"5min":0.26600000000000001,"15min":0.23200000000000001},"max":{"1min":5.7530000000000001,"5min":53.472000000000001,"15min":69.852000000000004},"last":2.851},{"interface":"front","average":{"1min":1.927,"5min":2.1909999999999998,"15min":2.113},"min":{"1min":0.28299999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":5.9909999999999997,"5min":53.536000000000001,"15min":70.692999999999998},"last":4.2910000000000004}]},{"osd":2,"last update":"Sat Feb 10 07:47:54 2024","interfaces":[{"interface":"back","average":{"1min":1.9419999999999999,"5min":1.0900000000000001,"15min":1.3640000000000001},"min":{"1min":0.33300000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":6.0179999999999998,"5min":6.0179999999999998,"15min":50.162999999999997},"last":4.3289999999999997},{"interface":"front","average":{"1min":1.772,"5min":1.052,"15min":1.2929999999999999},"min":{"1min":0.38200000000000001,"5min":0.30099999999999999,"15min":0.27700000000000002},"max":{"1min":4.7199999999999998,"5min":4.7199999999999998,"15min":50.453000000000003},"last":4.516}]},{"osd":3,"last update":"Sat Feb 10 07:44:52 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.5369999999999999,"15min":1.6519999999999999},"min":{"1min":0.311,"5min":0.30499999999999999,"15min":0.28499999999999998},"max":{"1min":3.714,"5min":40.710000000000001,"15min":69.813000000000002},"last":2.875},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.6160000000000001,"15min":1.6930000000000001},"min":{"1min":0.33400000000000002,"5min":0.30099999999999999,"15min":0.27900000000000003},"max":{"1min":4.609,"5min":40.981000000000002,"15min":69.760999999999996},"last":4.351}]},{"osd":5,"last update":"Sat Feb 10 07:47:54 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":1.264,"15min":0.81999999999999995},"min":{"1min":0.376,"5min":0.25,"15min":0.23100000000000001},"max":{"1min":10.502000000000001,"5min":10.502000000000001,"15min":10.502000000000001},"last":4.452},{"interface":"front","average":{"1min":3.157,"5min":1.4490000000000001,"15min":0.88700000000000001},"min":{"1min":0.38,"5min":0.247,"15min":0.247},"max":{"1min":9.2460000000000004,"5min":9.2460000000000004,"15min":9.2460000000000004},"last":2.9209999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:47:38 2024","interfaces":[{"interface":"back","average":{"1min":2.3959999999999999,"5min":1.097,"15min":0.755},"min":{"1min":0.308,"5min":0.28100000000000003,"15min":0.25900000000000001},"max":{"1min":6.7510000000000003,"5min":6.7510000000000003,"15min":6.7510000000000003},"last":4.5499999999999998},{"interface":"front","average":{"1min":3.2189999999999999,"5min":1.2949999999999999,"15min":0.84199999999999997},"min":{"1min":0.371,"5min":0.26600000000000001,"15min":0.24399999999999999},"max":{"1min":9.0980000000000008,"5min":9.0980000000000008,"15min":9.0980000000000008},"last":4.2610000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:48:08 2024","interfaces":[{"interface":"back","average":{"1min":2.0299999999999998,"5min":1.288,"15min":0.82399999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.29099999999999998},"max":{"1min":4.9669999999999996,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":4.4859999999999998},{"interface":"front","average":{"1min":2.3069999999999999,"5min":1.3660000000000001,"15min":0.91000000000000003},"min":{"1min":0.42799999999999999,"5min":0.30199999999999999,"15min":0.29399999999999998},"max":{"1min":10.919,"5min":10.919,"15min":10.919},"last":2.899}]}]},{"osd":5,"up_from":156,"seq":670014898319,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2982348,"kb_used_data":2384324,"kb_used_omap":4,"kb_used_meta":598011,"kb_avail":90762804,"statfs":{"total":95995035648,"available":92941111296,"internally_reserved":0,"allocated":2441547776,"data_stored":2440967493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":612363938},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":0.90500000000000003,"15min":1.6419999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20499999999999999},"max":{"1min":3.984,"5min":3.984,"15min":52.938000000000002},"last":2.923},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.577,"15min":1.927},"min":{"1min":0.255,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.2480000000000002,"5min":17.82,"15min":52.805},"last":0.97599999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.4430000000000001,"15min":2.5049999999999999},"min":{"1min":0.27200000000000002,"5min":0.26500000000000001,"15min":0.245},"max":{"1min":5.2910000000000004,"5min":8.0090000000000003,"15min":54.298000000000002},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.1870000000000001,"15min":2.4500000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":4.9349999999999996,"5min":6.3029999999999999,"15min":52.969999999999999},"last":2.476}]},{"osd":2,"last update":"Sat Feb 10 07:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.171,"15min":1.373},"min":{"1min":0.30599999999999999,"5min":0.29399999999999998,"15min":0.28999999999999998},"max":{"1min":5.3150000000000004,"5min":5.9660000000000002,"15min":11.601000000000001},"last":1.171},{"interface":"front","average":{"1min":1.6639999999999999,"5min":1.2350000000000001,"15min":1.4430000000000001},"min":{"1min":0.28199999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.0780000000000003,"5min":13.662000000000001,"15min":13.662000000000001},"last":2.4470000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:44:46 2024","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":2.7999999999999998,"15min":1.9350000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":10,"5min":53.366999999999997,"15min":53.366999999999997},"last":2.4209999999999998},{"interface":"front","average":{"1min":1.669,"5min":2.9550000000000001,"15min":2.1880000000000002},"min":{"1min":0.34999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":6.4039999999999999,"5min":52.731000000000002,"15min":52.731000000000002},"last":2.9500000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:47:48 2024","interfaces":[{"interface":"back","average":{"1min":3.8239999999999998,"5min":1.452,"15min":0.89900000000000002},"min":{"1min":0.51500000000000001,"5min":0.252,"15min":0.246},"max":{"1min":8.952,"5min":8.952,"15min":8.952},"last":2.5219999999999998},{"interface":"front","average":{"1min":2.9729999999999999,"5min":1.284,"15min":0.83399999999999996},"min":{"1min":0.39800000000000002,"5min":0.26800000000000002,"15min":0.25900000000000001},"max":{"1min":10.180999999999999,"5min":10.180999999999999,"15min":10.180999999999999},"last":2.9729999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:47:48 2024","interfaces":[{"interface":"back","average":{"1min":2.5600000000000001,"5min":1.212,"15min":0.82399999999999995},"min":{"1min":0.41299999999999998,"5min":0.27900000000000003,"15min":0.23999999999999999},"max":{"1min":9.0229999999999997,"5min":9.0229999999999997,"15min":9.0229999999999997},"last":2.3889999999999998},{"interface":"front","average":{"1min":2.004,"5min":1.0569999999999999,"15min":0.84299999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.6150000000000002,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":2.8809999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.9399999999999999,"5min":1.522,"15min":0.97499999999999998},"min":{"1min":0.35499999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.9640000000000004,"5min":8.8059999999999992,"15min":8.8059999999999992},"last":4.0140000000000002},{"interface":"front","average":{"1min":1.873,"5min":1.218,"15min":0.86799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.944,"5min":4.944,"15min":4.944},"last":2.5}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3237679104,"data_stored":3237675032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1178009600,"data_stored":1178009600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":171507712,"data_stored":171507712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":309002240,"data_stored":309002240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3568046080,"data_stored":3568042008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2406547456,"data_stored":2406547456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2086277120,"data_stored":2086273048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1483669504,"data_stored":1483669504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:48:29.099 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-02-10T07:48:29.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:29.141 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:29.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:29.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:29.243 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:29.273 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:29.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:29.375 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:29.413 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:29.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:29.515 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:29.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:29 smithi170 ceph-mon[33394]: pgmap v760: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 592 B/s rd, 0 B/s wr, 672 op/s 2024-02-10T07:48:29.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:29 smithi170 ceph-mon[33394]: 6.12 scrub starts 2024-02-10T07:48:29.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:29 smithi170 ceph-mon[33394]: 6.12 scrub ok 2024-02-10T07:48:29.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:29.543 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:29.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:29.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:29 smithi008 ceph-mon[29816]: pgmap v760: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 592 B/s rd, 0 B/s wr, 672 op/s 2024-02-10T07:48:29.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:29 smithi008 ceph-mon[29816]: 6.12 scrub starts 2024-02-10T07:48:29.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:29 smithi008 ceph-mon[29816]: 6.12 scrub ok 2024-02-10T07:48:29.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:29 smithi008 ceph-mon[40743]: pgmap v760: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 592 B/s rd, 0 B/s wr, 672 op/s 2024-02-10T07:48:29.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:29 smithi008 ceph-mon[40743]: 6.12 scrub starts 2024-02-10T07:48:29.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:29 smithi008 ceph-mon[40743]: 6.12 scrub ok 2024-02-10T07:48:29.644 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:29.656 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:29.676 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:29.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:29.778 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:29.808 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:29.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:29.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:29.910 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:29.948 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:29.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:30.049 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:30.085 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:30.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:30.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:30.187 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:30.222 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:30.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:30.324 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:30.350 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:30.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:30.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:30.409 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:48:30.452 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:30.488 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:30.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:30.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:30.589 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:30.617 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:30.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:30.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:30.634 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:48:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=sqlstore.transactions t=2024-02-10T07:48:30.232892621Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-10T07:48:30.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:30 smithi170 ceph-mon[33394]: 5.1c scrub starts 2024-02-10T07:48:30.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:30 smithi170 ceph-mon[33394]: 5.1c scrub ok 2024-02-10T07:48:30.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:30 smithi170 ceph-mon[33394]: 3.17 scrub starts 2024-02-10T07:48:30.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:30 smithi170 ceph-mon[33394]: 3.17 scrub ok 2024-02-10T07:48:30.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:30 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/302657341' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:48:30.720 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:30.755 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:30 smithi008 ceph-mon[29816]: 5.1c scrub starts 2024-02-10T07:48:30.755 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:30 smithi008 ceph-mon[29816]: 5.1c scrub ok 2024-02-10T07:48:30.755 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:30 smithi008 ceph-mon[29816]: 3.17 scrub starts 2024-02-10T07:48:30.755 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:30 smithi008 ceph-mon[29816]: 3.17 scrub ok 2024-02-10T07:48:30.755 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:30 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/302657341' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:48:30.756 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:30 smithi008 ceph-mon[40743]: 5.1c scrub starts 2024-02-10T07:48:30.756 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:30 smithi008 ceph-mon[40743]: 5.1c scrub ok 2024-02-10T07:48:30.756 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:30 smithi008 ceph-mon[40743]: 3.17 scrub starts 2024-02-10T07:48:30.756 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:30 smithi008 ceph-mon[40743]: 3.17 scrub ok 2024-02-10T07:48:30.756 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:30 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/302657341' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:48:30.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:30.757 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:30.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:30.858 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:30.895 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:30.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:30.997 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:31.032 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:31.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:31.134 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:31.176 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:31.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:31.277 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:31.317 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:31.317 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:48:31.317 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:48:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:31.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:31.419 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:31.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:31.449 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:31.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:31.550 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:31.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:31 smithi008 ceph-mon[29816]: pgmap v761: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:48:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:31 smithi008 ceph-mon[29816]: 3.19 scrub starts 2024-02-10T07:48:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:31 smithi008 ceph-mon[29816]: 3.19 scrub ok 2024-02-10T07:48:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:31 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:48:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:31 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/302657341' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:48:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:31 smithi008 ceph-mon[29816]: osdmap e362: 8 total, 8 up, 7 in 2024-02-10T07:48:31.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:31 smithi008 ceph-mon[40743]: pgmap v761: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:48:31.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:31 smithi008 ceph-mon[40743]: 3.19 scrub starts 2024-02-10T07:48:31.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:31 smithi008 ceph-mon[40743]: 3.19 scrub ok 2024-02-10T07:48:31.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:31 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:48:31.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:31 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/302657341' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:48:31.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:31 smithi008 ceph-mon[40743]: osdmap e362: 8 total, 8 up, 7 in 2024-02-10T07:48:31.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:31.587 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:31.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:31.688 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:31.716 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:31.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:31.720 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:31 smithi170 ceph-mon[33394]: pgmap v761: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-10T07:48:31.720 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:31 smithi170 ceph-mon[33394]: 3.19 scrub starts 2024-02-10T07:48:31.720 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:31 smithi170 ceph-mon[33394]: 3.19 scrub ok 2024-02-10T07:48:31.721 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:31 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:48:31.721 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:31 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/302657341' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:48:31.721 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:31 smithi170 ceph-mon[33394]: osdmap e362: 8 total, 8 up, 7 in 2024-02-10T07:48:31.818 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:31.857 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:31.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:31.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:31.959 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:31.996 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:31.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:31.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:32.098 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:32.129 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:32.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:32.230 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:32.256 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T07:48:32.276 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:32.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:32.378 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:32.409 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:48:32.433 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:48:32.439 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:32.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:32.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:32.459 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:48:32.486 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:48:32.525 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:48:32.541 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:32.566 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:48:32.570 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:32.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:32.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:32.600 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:48:32.632 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:48:32.633 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:48:32.633 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:48:32.633 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:48:32.633 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:48:32.633 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:48:32.634 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:48:32.634 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:48:32.634 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:48:32.672 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:32.698 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:48:32.699 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:32.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:32.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:32.801 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:32.851 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:32.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:32.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:32.953 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:32.986 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:32.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:32.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:33.087 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:33.120 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:33.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:33.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:32 smithi170 ceph-mon[33394]: osdmap e363: 8 total, 8 up, 7 in 2024-02-10T07:48:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:32 smithi170 ceph-mon[33394]: pgmap v764: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 977 op/s 2024-02-10T07:48:33.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:32 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2751708004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:33.221 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:33.249 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:33.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:33.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:33.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:32 smithi008 ceph-mon[29816]: osdmap e363: 8 total, 8 up, 7 in 2024-02-10T07:48:33.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:32 smithi008 ceph-mon[29816]: pgmap v764: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 977 op/s 2024-02-10T07:48:33.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:32 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2751708004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:32 smithi008 ceph-mon[40743]: osdmap e363: 8 total, 8 up, 7 in 2024-02-10T07:48:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:32 smithi008 ceph-mon[40743]: pgmap v764: 148 pgs: 148 active+clean; 4.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 977 op/s 2024-02-10T07:48:33.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:32 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2751708004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:33.350 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:33.388 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:33.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:33.490 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:33.516 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:33.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:33.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:33.618 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:33.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:33.646 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:33.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:33.747 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:33.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:33.788 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:33.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:33.844 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:48:33.889 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:33.929 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:33.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:33.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:33.974 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:48:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:48:34.031 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:34.079 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:34.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:34.100 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 4, 6, 7, 5, 3] 2024-02-10T07:48:34.100 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 0 1 2024-02-10T07:48:34.133 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:48:34.133 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:48:34.180 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:34.186 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:34.221 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:34.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:34.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:34.277 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:48:34.323 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:34.359 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:34.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:34.461 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:34.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:34.508 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:34.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:34.609 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:34.650 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:34.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:34.752 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:34.796 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:34.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:34.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:34.817 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:48:34.817 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:48:34.817 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.818 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15630.0:157986 10.9 e364/355 10:96f8d3d3:::benchmark_data_smithi008_161452_object49525:head v 364'9831, mlcod=364'9828)", 2024-02-10T07:48:34.818 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:34.791187+0000", 2024-02-10T07:48:34.818 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.024595428999999999, 2024-02-10T07:48:34.818 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.024629997000000001, 2024-02-10T07:48:34.818 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:48:34.818 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:48:34.818 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:48:34.818 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.818 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:48:34.818 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.791187+0000", 2024-02-10T07:48:34.819 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.819 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.819 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.819 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:48:34.819 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.791187+0000", 2024-02-10T07:48:34.819 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.819 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.819 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.819 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:48:34.819 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.791187+0000", 2024-02-10T07:48:34.819 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2200000000000003e-07 2024-02-10T07:48:34.819 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.819 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.819 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:48:34.820 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.791193+0000", 2024-02-10T07:48:34.820 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9039999999999997e-06 2024-02-10T07:48:34.820 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.820 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.820 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:48:34.820 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.791206+0000", 2024-02-10T07:48:34.820 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2398999999999999e-05 2024-02-10T07:48:34.820 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.820 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.820 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:48:34.820 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.791213+0000", 2024-02-10T07:48:34.820 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6340000000000001e-06 2024-02-10T07:48:34.820 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.820 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.792878+0000", 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0016644699999999999 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.792895+0000", 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7428999999999999e-05 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.815790+0000", 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.022894804000000001 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.821 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15630.0:157989 10.9 e364/355 10:942fa63e:::benchmark_data_smithi008_161452_object49528:head v 364'9832, mlcod=364'9828)", 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:34.791235+0000", 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.024547446000000001, 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.024669686, 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.791235+0000", 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.822 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:48:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.791235+0000", 2024-02-10T07:48:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:48:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.791235+0000", 2024-02-10T07:48:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7099999999999997e-07 2024-02-10T07:48:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:48:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.791239+0000", 2024-02-10T07:48:34.823 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8789999999999996e-06 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.791246+0000", 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8070000000000002e-06 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.791251+0000", 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2689999999999999e-06 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.801922+0000", 2024-02-10T07:48:34.824 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.010670821 2024-02-10T07:48:34.825 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.825 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.825 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:48:34.825 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.801950+0000", 2024-02-10T07:48:34.825 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7763000000000002e-05 2024-02-10T07:48:34.825 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.825 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.825 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:48:34.825 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.815897+0000", 2024-02-10T07:48:34.825 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.013947553 2024-02-10T07:48:34.825 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.825 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:48:34.825 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.825 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15630.0:157997 10.9 e364/355 10:9089b420:::benchmark_data_smithi008_161452_object49536:head v 364'9833, mlcod=364'9828)", 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:34.797100+0000", 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.018681631000000001, 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.018888764999999998, 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.797100+0000", 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.797100+0000", 2024-02-10T07:48:34.826 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.827 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.827 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.827 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:48:34.827 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.797101+0000", 2024-02-10T07:48:34.827 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1399999999999995e-07 2024-02-10T07:48:34.827 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.827 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.827 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:48:34.827 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.797109+0000", 2024-02-10T07:48:34.827 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.8439999999999998e-06 2024-02-10T07:48:34.827 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.828 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.828 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:48:34.828 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.797121+0000", 2024-02-10T07:48:34.828 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2088e-05 2024-02-10T07:48:34.828 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.828 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.828 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:48:34.828 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.797129+0000", 2024-02-10T07:48:34.828 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0520000000000002e-06 2024-02-10T07:48:34.828 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.828 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.828 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:48:34.828 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.804290+0000", 2024-02-10T07:48:34.829 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0071605230000000002 2024-02-10T07:48:34.829 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.829 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.829 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:48:34.829 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.804310+0000", 2024-02-10T07:48:34.829 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9930000000000001e-05 2024-02-10T07:48:34.829 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.829 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:48:34.829 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.829 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.829 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.829 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15630.0:158001 10.9 e364/355 10:90847d9d:::benchmark_data_smithi008_161452_object49540:head v 364'9834, mlcod=364'9828)", 2024-02-10T07:48:34.829 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:34.803614+0000", 2024-02-10T07:48:34.829 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.012167987, 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012453896000000001, 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.803614+0000", 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.803614+0000", 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.830 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.831 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:48:34.831 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.803615+0000", 2024-02-10T07:48:34.831 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1300000000000001e-07 2024-02-10T07:48:34.831 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.831 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.831 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:48:34.831 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.803619+0000", 2024-02-10T07:48:34.831 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7319999999999998e-06 2024-02-10T07:48:34.831 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.831 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.832 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:48:34.832 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.803627+0000", 2024-02-10T07:48:34.832 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9489999999999996e-06 2024-02-10T07:48:34.832 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.832 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.832 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:48:34.832 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.803632+0000", 2024-02-10T07:48:34.832 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9350000000000002e-06 2024-02-10T07:48:34.832 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.832 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.832 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:48:34.832 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.810932+0000", 2024-02-10T07:48:34.832 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0072995150000000003 2024-02-10T07:48:34.832 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.810987+0000", 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5596000000000001e-05 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15630.0:158004 10.e e364/312 10:79805562:::benchmark_data_smithi008_161452_object49543:head v 364'9866, mlcod=364'9862)", 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:34.807717+0000", 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.0080654439999999997, 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0084235509999999996, 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:48:34.833 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.807717+0000", 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.807717+0000", 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.807717+0000", 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8400000000000004e-07 2024-02-10T07:48:34.834 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.807724+0000", 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2439999999999998e-06 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.807746+0000", 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1988e-05 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.807753+0000", 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.934e-06 2024-02-10T07:48:34.835 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.836 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.836 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:48:34.836 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.810198+0000", 2024-02-10T07:48:34.836 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002445029 2024-02-10T07:48:34.836 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.836 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.836 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:48:34.836 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.810240+0000", 2024-02-10T07:48:34.836 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2729999999999999e-05 2024-02-10T07:48:34.836 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.836 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:48:34.836 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.836 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.836 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15630.0:158002 10.9 e364/355 10:9624cae3:::benchmark_data_smithi008_161452_object49541:head v 364'9835, mlcod=364'9829)", 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:34.807776+0000", 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.0080056329999999998, 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0084384060000000007, 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.807776+0000", 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:48:34.837 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.807776+0000", 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.807777+0000", 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9799999999999999e-07 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.807781+0000", 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7129999999999999e-06 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:48:34.838 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.807790+0000", 2024-02-10T07:48:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4290000000000007e-06 2024-02-10T07:48:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:48:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.807795+0000", 2024-02-10T07:48:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7559999999999999e-06 2024-02-10T07:48:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:48:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.813471+0000", 2024-02-10T07:48:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0056765499999999998 2024-02-10T07:48:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.839 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:48:34.840 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.813492+0000", 2024-02-10T07:48:34.840 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0344999999999999e-05 2024-02-10T07:48:34.840 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.840 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:48:34.840 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.840 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.840 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.840 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15630.0:158007 10.9 e364/355 10:91621988:::benchmark_data_smithi008_161452_object49546:head v 364'9836, mlcod=364'9829)", 2024-02-10T07:48:34.840 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:34.812237+0000", 2024-02-10T07:48:34.840 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.0035455130000000001, 2024-02-10T07:48:34.840 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0040463909999999999, 2024-02-10T07:48:34.840 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:48:34.840 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:48:34.840 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.812237+0000", 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.812237+0000", 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.812237+0000", 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2999999999999999e-07 2024-02-10T07:48:34.841 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.812244+0000", 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4010000000000002e-06 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.812259+0000", 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4813e-05 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.812266+0000", 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6459999999999993e-06 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.842 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.816163+0000", 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0038971549999999998 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.816184+0000", 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0548999999999999e-05 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15630.0:158010 10.a e364/309 10:5a63ac2c:::benchmark_data_smithi008_161452_object49549:head v 364'9976, mlcod=364'9974)", 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:34.812673+0000", 2024-02-10T07:48:34.843 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.003109264, 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0036726440000000001, 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.812673+0000", 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.812673+0000", 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.844 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.812674+0000", 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0200000000000001e-07 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.812679+0000", 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8769999999999998e-06 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.812689+0000", 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.8349999999999999e-06 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:48:34.845 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.812696+0000", 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3709999999999996e-06 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.812772+0000", 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6602999999999998e-05 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.812790+0000", 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7561999999999999e-05 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.846 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15630.0:158011 10.6 e364/355)", 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:34.813212+0000", 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.0025697609999999998, 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0031961170000000001, 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "queued for pg", 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.813212+0000", 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:48:34.847 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.813212+0000", 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.813213+0000", 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5199999999999997e-07 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.813219+0000", 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.614e-06 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:48:34.848 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.813227+0000", 2024-02-10T07:48:34.849 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.9050000000000003e-06 2024-02-10T07:48:34.849 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.849 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.849 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:48:34.849 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.813233+0000", 2024-02-10T07:48:34.849 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9089999999999997e-06 2024-02-10T07:48:34.849 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.849 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:48:34.849 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.849 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.849 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.849 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop(client.15630.0:158016 10.9 e364/355)", 2024-02-10T07:48:34.849 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:34.815983+0000", 2024-02-10T07:48:34.849 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 4294967295.9997988, 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00047283900000000002, 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "queued for pg", 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.815983+0000", 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.815983+0000", 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.815984+0000", 2024-02-10T07:48:34.850 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.09e-07 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.815992+0000", 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7950000000000008e-06 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.816001+0000", 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.755e-06 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:34.816008+0000", 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.029e-06 2024-02-10T07:48:34.851 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.852 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:48:34.852 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.852 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:48:34.852 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2024-02-10T07:48:34.852 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 10 2024-02-10T07:48:34.852 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:48:34.899 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:34.929 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:34.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:35.031 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:35.070 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:35.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:35.172 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:35.202 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:35.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:35.304 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 7]}]: dispatch 2024-02-10T07:48:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 0]}]: dispatch 2024-02-10T07:48:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-02-10T07:48:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T07:48:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.2", "id": [0, 3]}]: dispatch 2024-02-10T07:48:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.e", "id": [4, 6]}]: dispatch 2024-02-10T07:48:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]: dispatch 2024-02-10T07:48:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 7]}]': finished 2024-02-10T07:48:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 0]}]': finished 2024-02-10T07:48:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-02-10T07:48:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-02-10T07:48:35.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.2", "id": [0, 3]}]': finished 2024-02-10T07:48:35.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.e", "id": [4, 6]}]': finished 2024-02-10T07:48:35.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]': finished 2024-02-10T07:48:35.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: osdmap e364: 8 total, 8 up, 7 in 2024-02-10T07:48:35.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:48:35.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:48:35.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[40743]: pgmap v766: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-02-10T07:48:35.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:35.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 7]}]: dispatch 2024-02-10T07:48:35.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 0]}]: dispatch 2024-02-10T07:48:35.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-02-10T07:48:35.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T07:48:35.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.2", "id": [0, 3]}]: dispatch 2024-02-10T07:48:35.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.e", "id": [4, 6]}]: dispatch 2024-02-10T07:48:35.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]: dispatch 2024-02-10T07:48:35.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 7]}]': finished 2024-02-10T07:48:35.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 0]}]': finished 2024-02-10T07:48:35.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-02-10T07:48:35.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-02-10T07:48:35.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.2", "id": [0, 3]}]': finished 2024-02-10T07:48:35.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.e", "id": [4, 6]}]': finished 2024-02-10T07:48:35.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]': finished 2024-02-10T07:48:35.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: osdmap e364: 8 total, 8 up, 7 in 2024-02-10T07:48:35.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:48:35.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:48:35.334 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:34 smithi008 ceph-mon[29816]: pgmap v766: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-02-10T07:48:35.341 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:35.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:35.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:35.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:35.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 7]}]: dispatch 2024-02-10T07:48:35.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 0]}]: dispatch 2024-02-10T07:48:35.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-02-10T07:48:35.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-02-10T07:48:35.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.2", "id": [0, 3]}]: dispatch 2024-02-10T07:48:35.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.e", "id": [4, 6]}]: dispatch 2024-02-10T07:48:35.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]: dispatch 2024-02-10T07:48:35.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 7]}]': finished 2024-02-10T07:48:35.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 0]}]': finished 2024-02-10T07:48:35.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-02-10T07:48:35.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-02-10T07:48:35.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.2", "id": [0, 3]}]': finished 2024-02-10T07:48:35.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.e", "id": [4, 6]}]': finished 2024-02-10T07:48:35.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]': finished 2024-02-10T07:48:35.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: osdmap e364: 8 total, 8 up, 7 in 2024-02-10T07:48:35.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:48:35.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:48:35.385 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:34 smithi170 ceph-mon[33394]: pgmap v766: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-02-10T07:48:35.442 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:35.469 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:35.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:35.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:35.571 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:35.588 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:48:35.600 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:35.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:35.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:35.702 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:35.732 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:35.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:35.834 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:35.872 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:35.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:35.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:35.937 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:35.973 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:36.002 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:36.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:36.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:36.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:36 smithi008 ceph-mon[29816]: osdmap e365: 8 total, 8 up, 7 in 2024-02-10T07:48:36.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:36 smithi008 ceph-mon[40743]: osdmap e365: 8 total, 8 up, 7 in 2024-02-10T07:48:36.104 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:36.140 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:36.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:36.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:36.242 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:36.273 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:36.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:36.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:36.375 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:36.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:36 smithi170 ceph-mon[33394]: osdmap e365: 8 total, 8 up, 7 in 2024-02-10T07:48:36.405 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:36.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:36.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:36.506 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:36.549 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:36.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:36.651 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:36.682 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:36.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:36.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:36.694 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:36.785 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:36.813 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:36.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:36.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:36.915 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:36.952 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:36.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:37.053 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:37.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:37 smithi008 ceph-mon[29816]: pgmap v768: 148 pgs: 148 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 903 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:48:37.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:37 smithi008 ceph-mon[40743]: pgmap v768: 148 pgs: 148 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 903 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:48:37.091 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:37.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:37.193 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:37.221 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:37.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:37.323 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:37.349 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:37.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:37.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:37 smithi170 ceph-mon[33394]: pgmap v768: 148 pgs: 148 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 903 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:48:37.451 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:37.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:37.491 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:37.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:37.537 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-10T07:48:37.592 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:37.618 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:37.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:37.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:37.720 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:37.765 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:37.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:37.867 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:37.895 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:37.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:37.975 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:48:37.997 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:38.027 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:38.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:38.065 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:48:38.129 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:38.134 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:38 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/51285674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:48:38.134 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:38 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/51285674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:48:38.171 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:38.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:38.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:38.273 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:38.310 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:38.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:38.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:38.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:38 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/51285674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:48:38.411 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:38.434 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:38.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:38.536 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:38.566 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:38.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:38.668 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:38.716 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:38.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:38.818 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:38.846 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:38.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:38.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:38.948 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:38.983 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:38.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:38.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:39.052 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:48:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:48:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:48:39.084 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:39.126 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:39.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:39.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:39 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:48:39.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:39 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/51285674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:48:39.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:39 smithi170 ceph-mon[33394]: osdmap e366: 8 total, 8 up, 7 in 2024-02-10T07:48:39.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:39 smithi170 ceph-mon[33394]: pgmap v770: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s 2024-02-10T07:48:39.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:39 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:48:39.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:39 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:48:39.228 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:39.265 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:39.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:39.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:39.329 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:48:39.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:39 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:48:39.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:39 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/51285674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:48:39.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:39 smithi008 ceph-mon[29816]: osdmap e366: 8 total, 8 up, 7 in 2024-02-10T07:48:39.329 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:39 smithi008 ceph-mon[29816]: pgmap v770: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s 2024-02-10T07:48:39.330 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:39 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:48:39.330 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:39 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:48:39.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:39 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:48:39.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:39 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/51285674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:48:39.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:39 smithi008 ceph-mon[40743]: osdmap e366: 8 total, 8 up, 7 in 2024-02-10T07:48:39.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:39 smithi008 ceph-mon[40743]: pgmap v770: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.00k op/s 2024-02-10T07:48:39.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:39 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:48:39.330 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:39 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:48:39.360 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:48:39.366 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:39.390 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:48:39.398 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:39.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:39.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:39.421 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:48:39.457 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:48:39.493 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:48:39.500 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:39.531 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:48:39.539 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:39.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:39.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:39.572 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:48:39.576 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:48:39.576 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:48:39.576 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:48:39.576 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:48:39.576 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:48:39.576 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:48:39.576 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:48:39.576 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:48:39.641 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:39.668 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:39.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:39.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:39.769 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:39.812 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:39.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:39.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:39.914 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:39.943 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:39.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:40.044 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:40.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:40.088 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:40.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:40.189 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:40.220 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:40.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:40.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:40.322 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:40.353 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:40 smithi170 ceph-mon[33394]: osdmap e367: 8 total, 8 up, 7 in 2024-02-10T07:48:40.353 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:40 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/727959686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:40.363 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:40.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:40.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:40.465 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:40.492 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:40.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:40.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:40.562 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:48:40.562 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:48:40.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:40 smithi008 ceph-mon[40743]: osdmap e367: 8 total, 8 up, 7 in 2024-02-10T07:48:40.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:40 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/727959686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:40.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:40 smithi008 ceph-mon[29816]: osdmap e367: 8 total, 8 up, 7 in 2024-02-10T07:48:40.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:40 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/727959686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:40.594 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:40.626 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:40.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:40.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:40.727 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:40.756 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:40.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:40.858 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:40.889 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:40.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:40.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:40.991 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:41.028 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:41.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:41.130 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:41.164 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:41.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:41.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:41.266 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:41.293 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:41.293 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:41 smithi008 ceph-mon[29816]: pgmap v772: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 1.29k op/s; 73 B/s, 1 objects/s recovering 2024-02-10T07:48:41.294 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:41 smithi008 ceph-mon[40743]: pgmap v772: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 1.29k op/s; 73 B/s, 1 objects/s recovering 2024-02-10T07:48:41.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:41.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:41.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:41 smithi170 ceph-mon[33394]: pgmap v772: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 1.29k op/s; 73 B/s, 1 objects/s recovering 2024-02-10T07:48:41.395 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:41.423 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:41.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:41.525 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:41.560 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:41.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:41.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:41.662 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:41.692 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:41.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:41.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:41.794 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:41.822 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:41.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:41.858 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-10T07:48:41.924 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:41.975 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:41.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:41.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:42.077 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:42.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:42.130 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:42.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:42.231 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:42.293 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:42.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:42.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:42.310 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:48:42.395 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:42.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:42.445 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:42.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:42.546 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:42.574 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:42.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:42.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:42.675 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:42.704 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:42.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:42.806 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:42.833 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:42.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:42.935 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:42.962 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:43.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:43.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:42 smithi008 ceph-mon[29816]: pgmap v773: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 582 B/s rd, 0 B/s wr, 850 op/s; 62 B/s, 1 objects/s recovering 2024-02-10T07:48:43.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:42 smithi008 ceph-mon[40743]: pgmap v773: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 582 B/s rd, 0 B/s wr, 850 op/s; 62 B/s, 1 objects/s recovering 2024-02-10T07:48:43.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:42 smithi170 ceph-mon[33394]: pgmap v773: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 582 B/s rd, 0 B/s wr, 850 op/s; 62 B/s, 1 objects/s recovering 2024-02-10T07:48:43.153 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:43.185 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:43.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:43.286 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:43.314 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:43.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:43.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:43.416 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:43.459 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:43.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:43.560 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:43.595 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:43.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:43.696 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:43.730 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:43.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:43.832 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:43.861 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:43.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:43.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:43.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:48:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:48:43.963 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:43.992 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:43.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:44.025 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.0 to 1 (10000) 2024-02-10T07:48:44.093 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:44.123 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:44.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:44.225 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:44.266 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:44.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:44.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:44.285 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:43 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3969949216' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:48:44.285 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:43 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:48:44.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:43 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3969949216' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:48:44.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:44 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:48:44.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:43 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3969949216' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:48:44.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:44 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:48:44.368 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:44.395 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:44.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:44.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:44.491 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:48:44.491 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:48:44.496 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:44.534 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:44.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:44.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:44.636 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:44.666 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:44.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:44.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:44.807 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:44.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:44.909 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:44.943 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:44.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:44.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:45.045 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:45.074 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:45.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:45.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:45 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:45.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:45 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:48:45.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:45 smithi170 ceph-mon[33394]: osdmap e368: 8 total, 8 up, 7 in 2024-02-10T07:48:45.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:45 smithi170 ceph-mon[33394]: pgmap v775: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.69k op/s; 91 B/s, 3 objects/s recovering 2024-02-10T07:48:45.177 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:45.189 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:48:45.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:45.216 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:45.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:45.239 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-10T07:48:45.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:45 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:45.270 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:45 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:48:45.271 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:45 smithi008 ceph-mon[29816]: osdmap e368: 8 total, 8 up, 7 in 2024-02-10T07:48:45.271 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:45 smithi008 ceph-mon[29816]: pgmap v775: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.69k op/s; 91 B/s, 3 objects/s recovering 2024-02-10T07:48:45.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:45 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:45.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:45 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:48:45.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:45 smithi008 ceph-mon[40743]: osdmap e368: 8 total, 8 up, 7 in 2024-02-10T07:48:45.271 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:45 smithi008 ceph-mon[40743]: pgmap v775: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.69k op/s; 91 B/s, 3 objects/s recovering 2024-02-10T07:48:45.317 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:45.351 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:45.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:45.453 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:45.488 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:45.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:45.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:45.590 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:45.604 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 1 1 2024-02-10T07:48:45.627 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:45.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:45.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:45.718 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:45.722 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:45.729 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:45.765 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:45.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:45.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:45.867 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:45.900 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:45.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:46.002 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:46.040 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:46.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:46.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:46.142 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:46.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:46.182 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:46.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:46.283 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:46.320 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:46.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:46.421 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:46.454 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:46.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:46.556 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:46.586 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:46.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:46.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:46.688 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:46.714 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:48:46.725 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:46.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:46.827 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:46.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:46.864 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:46.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:46.965 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:46.997 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:46.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:46.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:47.099 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:47.132 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:47.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:47.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:47.235 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:47.282 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:47.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:47.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:47.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:47 smithi008 ceph-mon[29816]: pgmap v776: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.52k op/s; 68 B/s, 2 objects/s recovering 2024-02-10T07:48:47.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:47 smithi008 ceph-mon[29816]: osdmap e369: 8 total, 8 up, 7 in 2024-02-10T07:48:47.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:47 smithi008 ceph-mon[40743]: pgmap v776: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.52k op/s; 68 B/s, 2 objects/s recovering 2024-02-10T07:48:47.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:47 smithi008 ceph-mon[40743]: osdmap e369: 8 total, 8 up, 7 in 2024-02-10T07:48:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:47 smithi170 ceph-mon[33394]: pgmap v776: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.52k op/s; 68 B/s, 2 objects/s recovering 2024-02-10T07:48:47.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:47 smithi170 ceph-mon[33394]: osdmap e369: 8 total, 8 up, 7 in 2024-02-10T07:48:47.383 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:47.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:47.425 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:47.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:47.526 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:47.563 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:47.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:47.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:47.665 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:47.695 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:47.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:47.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:47.796 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:47.833 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:47.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:47.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:47.935 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:47.977 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:47.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:48.079 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:48.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:48.108 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:48.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:48.210 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:48.255 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:48.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:48.356 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:48.386 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:48.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:48.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:48.407 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:48 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:48:48.407 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:48 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:48:48.407 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:48 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:48:48.407 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:48 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:48:48.488 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:48.527 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:48.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:48.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:48.628 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:48.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:48 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:48:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:48 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:48:48.663 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:48.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:48.765 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:48.800 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:48.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:48.902 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:48.938 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:48.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:49.039 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:49.070 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:49.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:49.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:49.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:48:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:48:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:48:49.172 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:49.204 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:49.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:49.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:49.306 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:49.335 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:49.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:49.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:49.408 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:48:49.408 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:48:49.408 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:49 smithi008 ceph-mon[29816]: pgmap v778: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 769 op/s 2024-02-10T07:48:49.408 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:48:49.408 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:49 smithi008 ceph-mon[40743]: pgmap v778: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 769 op/s 2024-02-10T07:48:49.409 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:48:49.437 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:49.475 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:49.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:49.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:49.577 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:49.606 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:49.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:49 smithi170 ceph-mon[33394]: pgmap v778: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 769 op/s 2024-02-10T07:48:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:48:49.707 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:49.747 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:49.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:49.848 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:49.878 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:49.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:49.980 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:50.014 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:50.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:50.113 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:50.116 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:50.121 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:50.149 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:50.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:50.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:50.251 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:50.285 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:50.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:50.387 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:50.425 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:50.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:50.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:50.527 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:50.566 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:50.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:50.668 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:50.699 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:50.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:50.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:50.801 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:50.832 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:50.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:50.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:50.934 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:50.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:50.971 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:50.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:51.072 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:51.112 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:51.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:51.214 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:51.249 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:51.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:51.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:51.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:51 smithi008 ceph-mon[29816]: pgmap v779: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-02-10T07:48:51.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:51 smithi008 ceph-mon[40743]: pgmap v779: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-02-10T07:48:51.351 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:51.389 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:51.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:51.490 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:51.522 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:51.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:51.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:51.624 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:51 smithi170 ceph-mon[33394]: pgmap v779: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.59k op/s 2024-02-10T07:48:51.663 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:51.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:51.765 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:51.793 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:51.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:51.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:51.938 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:51.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:51.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:52.040 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:52.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:52.070 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:52.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:52.169 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:48:52.171 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:52.218 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:52.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:52.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:52.320 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:52.353 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:52.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:52.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:52.455 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:52.484 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:52.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:52.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:52.498 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:52 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2679228334' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:48:52.498 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:52 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:48:52.499 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:52 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2679228334' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:48:52.499 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:52 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:48:52.586 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:52.617 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:52.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:52.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:52 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2679228334' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:48:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:52 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:48:52.719 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:52.920 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:52.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:53.022 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:53.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:53.060 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:53.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:53.161 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:53.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:53.192 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:53.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:53.294 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:53.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:53.324 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:53.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:53.425 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:53.461 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:53.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:53.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:53.563 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:53.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:53 smithi008 ceph-mon[29816]: pgmap v780: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 635 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:48:53.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:53 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:48:53.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:53 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:48:53.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:53 smithi008 ceph-mon[29816]: osdmap e370: 8 total, 8 up, 7 in 2024-02-10T07:48:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:53 smithi008 ceph-mon[40743]: pgmap v780: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 635 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:48:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:53 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:48:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:53 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:48:53.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:53 smithi008 ceph-mon[40743]: osdmap e370: 8 total, 8 up, 7 in 2024-02-10T07:48:53.590 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:53.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:53.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:53 smithi170 ceph-mon[33394]: pgmap v780: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 635 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:48:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:53 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:48:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:53 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:48:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:53 smithi170 ceph-mon[33394]: osdmap e370: 8 total, 8 up, 7 in 2024-02-10T07:48:53.693 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:53.729 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:53.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:53.831 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:53.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:53.862 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:53.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:53.963 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:53.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:53.990 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:53.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:54.091 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:54.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:54.122 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:54.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:54.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:48:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:48:54.223 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:54.254 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:54.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:54.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:54.356 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:54.387 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:54.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:54.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:54.489 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:54.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:54.527 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:54.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:54.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:54 smithi008 ceph-mon[29816]: 5.9 scrub starts 2024-02-10T07:48:54.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:54 smithi008 ceph-mon[29816]: 5.9 scrub ok 2024-02-10T07:48:54.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:54 smithi008 ceph-mon[40743]: 5.9 scrub starts 2024-02-10T07:48:54.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:54 smithi008 ceph-mon[40743]: 5.9 scrub ok 2024-02-10T07:48:54.628 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:54.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:54 smithi170 ceph-mon[33394]: 5.9 scrub starts 2024-02-10T07:48:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:54 smithi170 ceph-mon[33394]: 5.9 scrub ok 2024-02-10T07:48:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:54.667 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:54.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:54.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:54.805 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:54.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:54.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:54.907 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:54.949 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:54.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:55.050 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:55.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:55.084 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:55.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:55.185 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:55.214 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:55.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:55.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:55.317 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:55.351 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:55.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:55.453 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:55.483 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:55.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:55.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:55.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:55 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:55.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:55 smithi008 ceph-mon[29816]: pgmap v782: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-02-10T07:48:55.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:55 smithi008 ceph-mon[29816]: 10.9 scrub starts 2024-02-10T07:48:55.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:55 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:55.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:55 smithi008 ceph-mon[40743]: pgmap v782: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-02-10T07:48:55.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:55 smithi008 ceph-mon[40743]: 10.9 scrub starts 2024-02-10T07:48:55.585 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:55.615 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:55.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:55.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:55.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:55 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:48:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:55 smithi170 ceph-mon[33394]: pgmap v782: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-02-10T07:48:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:55 smithi170 ceph-mon[33394]: 10.9 scrub starts 2024-02-10T07:48:55.655 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-10T07:48:55.716 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:55.750 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:55.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:55.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:55.852 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:55.895 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:55.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:55.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:55.996 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:56.043 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:56.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:56.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:56.145 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:56.183 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:56.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:56.285 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:56.316 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:56.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:56.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:56 smithi008 ceph-mon[29816]: 10.9 scrub ok 2024-02-10T07:48:56.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:56 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:48:56.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:56 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:48:56.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:56 smithi008 ceph-mon[29816]: 5.16 scrub starts 2024-02-10T07:48:56.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:56 smithi008 ceph-mon[29816]: 5.16 scrub ok 2024-02-10T07:48:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:56 smithi008 ceph-mon[40743]: 10.9 scrub ok 2024-02-10T07:48:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:56 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:48:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:56 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:48:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:56 smithi008 ceph-mon[40743]: 5.16 scrub starts 2024-02-10T07:48:56.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:56 smithi008 ceph-mon[40743]: 5.16 scrub ok 2024-02-10T07:48:56.418 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:56.456 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:56.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:56.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:56.558 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:56.593 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:56.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:56 smithi170 ceph-mon[33394]: 10.9 scrub ok 2024-02-10T07:48:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:56 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:48:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:56 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:48:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:56 smithi170 ceph-mon[33394]: 5.16 scrub starts 2024-02-10T07:48:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:56 smithi170 ceph-mon[33394]: 5.16 scrub ok 2024-02-10T07:48:56.694 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:56.731 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:56.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:56.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:56.833 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:56.868 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:56.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:56.970 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:57.006 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:57.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:57.011 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:48:57.072 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:48:57.100 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:48:57.107 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:57.127 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:48:57.143 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:57.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:57.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:57.152 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:48:57.176 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.1 to 1 (10000) 2024-02-10T07:48:57.197 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:48:57.240 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:48:57.245 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:57.275 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:57.275 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:48:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:57.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:57.316 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:48:57.318 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:48:57.318 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:48:57.318 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:48:57.318 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:48:57.318 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:48:57.319 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:48:57.319 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:48:57.319 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:48:57.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[29816]: pgmap v783: 148 pgs: 148 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-02-10T07:48:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[29816]: osdmap e371: 8 total, 8 up, 7 in 2024-02-10T07:48:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[29816]: 5.1e scrub starts 2024-02-10T07:48:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[29816]: 5.1e scrub ok 2024-02-10T07:48:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2424045955' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:48:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[29816]: 3.e scrub starts 2024-02-10T07:48:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[29816]: 3.e scrub ok 2024-02-10T07:48:57.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/822902320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:57.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[40743]: pgmap v783: 148 pgs: 148 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-02-10T07:48:57.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[40743]: osdmap e371: 8 total, 8 up, 7 in 2024-02-10T07:48:57.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[40743]: 5.1e scrub starts 2024-02-10T07:48:57.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[40743]: 5.1e scrub ok 2024-02-10T07:48:57.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2424045955' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:48:57.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[40743]: 3.e scrub starts 2024-02-10T07:48:57.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[40743]: 3.e scrub ok 2024-02-10T07:48:57.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:57 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/822902320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:57.377 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:57.413 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:57.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:57.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:57.515 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:57.550 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:57.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:57.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:57 smithi170 ceph-mon[33394]: pgmap v783: 148 pgs: 148 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-02-10T07:48:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:57 smithi170 ceph-mon[33394]: osdmap e371: 8 total, 8 up, 7 in 2024-02-10T07:48:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:57 smithi170 ceph-mon[33394]: 5.1e scrub starts 2024-02-10T07:48:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:57 smithi170 ceph-mon[33394]: 5.1e scrub ok 2024-02-10T07:48:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:57 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2424045955' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:48:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:57 smithi170 ceph-mon[33394]: 3.e scrub starts 2024-02-10T07:48:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:57 smithi170 ceph-mon[33394]: 3.e scrub ok 2024-02-10T07:48:57.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:57 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/822902320' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:48:57.652 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:57.690 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:57.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:57.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:57.792 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:57.836 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:57.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:57.938 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:57.976 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:57.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:57.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:58.079 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:48:58.080 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 3 1 2024-02-10T07:48:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:58.123 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:58.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:48:58.224 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:58.257 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:58.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:58.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:58.359 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:58.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:58.393 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:58.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:58.463 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:58 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2424045955' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:48:58.464 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:58 smithi008 ceph-mon[29816]: osdmap e372: 8 total, 8 up, 7 in 2024-02-10T07:48:58.464 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:58 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2424045955' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:48:58.464 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:58 smithi008 ceph-mon[40743]: osdmap e372: 8 total, 8 up, 7 in 2024-02-10T07:48:58.494 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:58.495 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:48:58.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:58.531 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:58.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:58.632 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:58 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2424045955' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:48:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:58 smithi170 ceph-mon[33394]: osdmap e372: 8 total, 8 up, 7 in 2024-02-10T07:48:58.665 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:58.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:58.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:58.767 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:48:58.803 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:58.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:58.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:48:58.905 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:58.939 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:58.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:58.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:59.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:59.078 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:59.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:59.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:48:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:48:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:48:59.180 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:48:59.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:59.209 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:59.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:48:59.311 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:48:59.338 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:59.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:59.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:48:59.440 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:48:59.468 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:59.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:59.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:48:59.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:59 smithi170 ceph-mon[33394]: pgmap v786: 148 pgs: 148 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-02-10T07:48:59.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:59 smithi170 ceph-mon[33394]: 5.0 scrub starts 2024-02-10T07:48:59.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:59 smithi170 ceph-mon[33394]: 5.0 scrub ok 2024-02-10T07:48:59.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:59 smithi170 ceph-mon[33394]: 6.15 scrub starts 2024-02-10T07:48:59.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:48:59 smithi170 ceph-mon[33394]: 6.15 scrub ok 2024-02-10T07:48:59.570 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:48:59.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:59 smithi008 ceph-mon[29816]: pgmap v786: 148 pgs: 148 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-02-10T07:48:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:59 smithi008 ceph-mon[29816]: 5.0 scrub starts 2024-02-10T07:48:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:59 smithi008 ceph-mon[29816]: 5.0 scrub ok 2024-02-10T07:48:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:59 smithi008 ceph-mon[29816]: 6.15 scrub starts 2024-02-10T07:48:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:48:59 smithi008 ceph-mon[29816]: 6.15 scrub ok 2024-02-10T07:48:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:59 smithi008 ceph-mon[40743]: pgmap v786: 148 pgs: 148 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-02-10T07:48:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:59 smithi008 ceph-mon[40743]: 5.0 scrub starts 2024-02-10T07:48:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:59 smithi008 ceph-mon[40743]: 5.0 scrub ok 2024-02-10T07:48:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:59 smithi008 ceph-mon[40743]: 6.15 scrub starts 2024-02-10T07:48:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:48:59 smithi008 ceph-mon[40743]: 6.15 scrub ok 2024-02-10T07:48:59.599 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:48:59.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:59.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:48:59.701 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:48:59.739 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:59.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:48:59.841 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:48:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:48:59.879 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:48:59.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:48:59.980 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:00.010 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:00.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:00.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:00.112 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:00.144 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:00.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:00.245 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:00.281 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:00.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:00.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:00.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:00 smithi008 ceph-mon[29816]: 10.7 scrub starts 2024-02-10T07:49:00.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:00 smithi008 ceph-mon[29816]: 4.d scrub starts 2024-02-10T07:49:00.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:00 smithi008 ceph-mon[29816]: 4.d scrub ok 2024-02-10T07:49:00.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:00 smithi008 ceph-mon[29816]: 4.b scrub starts 2024-02-10T07:49:00.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:00 smithi008 ceph-mon[29816]: 4.b scrub ok 2024-02-10T07:49:00.309 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:00 smithi008 ceph-mon[40743]: 10.7 scrub starts 2024-02-10T07:49:00.309 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:00 smithi008 ceph-mon[40743]: 4.d scrub starts 2024-02-10T07:49:00.309 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:00 smithi008 ceph-mon[40743]: 4.d scrub ok 2024-02-10T07:49:00.309 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:00 smithi008 ceph-mon[40743]: 4.b scrub starts 2024-02-10T07:49:00.310 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:00 smithi008 ceph-mon[40743]: 4.b scrub ok 2024-02-10T07:49:00.332 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:00.382 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:00.384 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:49:00.387 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:49:00.387 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-02-10T07:49:00.387 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-02-10T07:49:00.387 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:49:00.421 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:00.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:00.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:00.523 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:00.555 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:00.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:00 smithi170 ceph-mon[33394]: 10.7 scrub starts 2024-02-10T07:49:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:00 smithi170 ceph-mon[33394]: 4.d scrub starts 2024-02-10T07:49:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:00 smithi170 ceph-mon[33394]: 4.d scrub ok 2024-02-10T07:49:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:00 smithi170 ceph-mon[33394]: 4.b scrub starts 2024-02-10T07:49:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:00 smithi170 ceph-mon[33394]: 4.b scrub ok 2024-02-10T07:49:00.658 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:00.685 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:00.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:00.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:00.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:00.827 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:00.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:00.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:00.958 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:00.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:00.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:01.060 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:01.096 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:01.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:01.198 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:01.233 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:01.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:01.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:01.264 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:49:01.335 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:01.374 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:01.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:01.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:01.476 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:01.509 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:01.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:01.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:01.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[29816]: 10.7 scrub ok 2024-02-10T07:49:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:49:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:49:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[29816]: pgmap v787: 148 pgs: 148 active+clean; 1.8 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-02-10T07:49:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[29816]: osdmap e373: 8 total, 8 up, 7 in 2024-02-10T07:49:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[29816]: 6.9 scrub starts 2024-02-10T07:49:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[29816]: 6.9 scrub ok 2024-02-10T07:49:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[29816]: 5.14 scrub starts 2024-02-10T07:49:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[29816]: 5.14 scrub ok 2024-02-10T07:49:01.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[40743]: 10.7 scrub ok 2024-02-10T07:49:01.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:49:01.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:49:01.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[40743]: pgmap v787: 148 pgs: 148 active+clean; 1.8 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-02-10T07:49:01.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[40743]: osdmap e373: 8 total, 8 up, 7 in 2024-02-10T07:49:01.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[40743]: 6.9 scrub starts 2024-02-10T07:49:01.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[40743]: 6.9 scrub ok 2024-02-10T07:49:01.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[40743]: 5.14 scrub starts 2024-02-10T07:49:01.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:01 smithi008 ceph-mon[40743]: 5.14 scrub ok 2024-02-10T07:49:01.610 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:01 smithi170 ceph-mon[33394]: 10.7 scrub ok 2024-02-10T07:49:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:01 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:49:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:01 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:49:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:01 smithi170 ceph-mon[33394]: pgmap v787: 148 pgs: 148 active+clean; 1.8 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-02-10T07:49:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:01 smithi170 ceph-mon[33394]: osdmap e373: 8 total, 8 up, 7 in 2024-02-10T07:49:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:01 smithi170 ceph-mon[33394]: 6.9 scrub starts 2024-02-10T07:49:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:01 smithi170 ceph-mon[33394]: 6.9 scrub ok 2024-02-10T07:49:01.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:01 smithi170 ceph-mon[33394]: 5.14 scrub starts 2024-02-10T07:49:01.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:01 smithi170 ceph-mon[33394]: 5.14 scrub ok 2024-02-10T07:49:01.648 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:01.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:01.750 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:01.783 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:01.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:01.847 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:01.885 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:01.912 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:01.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:01.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:02.013 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:02.042 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:02.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:02.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:02.144 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:02.184 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:02.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:02.286 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:02.290 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:02 smithi008 ceph-mon[29816]: 4.1c scrub starts 2024-02-10T07:49:02.291 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:02 smithi008 ceph-mon[29816]: 4.1c scrub ok 2024-02-10T07:49:02.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:02 smithi008 ceph-mon[40743]: 4.1c scrub starts 2024-02-10T07:49:02.291 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:02 smithi008 ceph-mon[40743]: 4.1c scrub ok 2024-02-10T07:49:02.317 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:02.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:02.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:02.331 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-10T07:49:02.420 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:02.452 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:02.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:02.553 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:02.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:02.588 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:02.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:02 smithi170 ceph-mon[33394]: 4.1c scrub starts 2024-02-10T07:49:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:02 smithi170 ceph-mon[33394]: 4.1c scrub ok 2024-02-10T07:49:02.689 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:02.718 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:02.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:02.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:02.820 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:02.847 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:02.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:02.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:02.949 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:02.978 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:02.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:02.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:03.080 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:03.125 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:03.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:03.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:03.227 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:03.254 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:03.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:03.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:03.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:03 smithi008 ceph-mon[29816]: 10.a scrub starts 2024-02-10T07:49:03.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:03 smithi008 ceph-mon[29816]: pgmap v789: 148 pgs: 148 active+clean; 1.8 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:49:03.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:49:03.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:03 smithi008 ceph-mon[40743]: 10.a scrub starts 2024-02-10T07:49:03.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:03 smithi008 ceph-mon[40743]: pgmap v789: 148 pgs: 148 active+clean; 1.8 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:49:03.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:49:03.356 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:03.387 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:03.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:03.489 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:03.530 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:03.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:03.632 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:03 smithi170 ceph-mon[33394]: 10.a scrub starts 2024-02-10T07:49:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:03 smithi170 ceph-mon[33394]: pgmap v789: 148 pgs: 148 active+clean; 1.8 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-10T07:49:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:49:03.660 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:03.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:03.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:03.762 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:03.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:03.797 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:03.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:03.898 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:03.933 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:03.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:03.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:04.012 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:04.035 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:04.073 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:04.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:04.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:04.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:49:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:49:04.174 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:04.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:04.211 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:04.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:04.312 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:04.341 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:04.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:04.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:04.442 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:04.472 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:04.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:04.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:04.477 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:04 smithi008 ceph-mon[29816]: 10.a scrub ok 2024-02-10T07:49:04.477 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:04 smithi008 ceph-mon[29816]: 5.2 scrub starts 2024-02-10T07:49:04.477 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:04 smithi008 ceph-mon[29816]: 5.2 scrub ok 2024-02-10T07:49:04.478 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:04 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:49:04.478 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:04 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:49:04.478 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:04 smithi008 ceph-mon[40743]: 10.a scrub ok 2024-02-10T07:49:04.478 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:04 smithi008 ceph-mon[40743]: 5.2 scrub starts 2024-02-10T07:49:04.478 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:04 smithi008 ceph-mon[40743]: 5.2 scrub ok 2024-02-10T07:49:04.478 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:04 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:49:04.478 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:04 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:49:04.518 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:04.574 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:04.610 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:04.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:04.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:04 smithi170 ceph-mon[33394]: 10.a scrub ok 2024-02-10T07:49:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:04 smithi170 ceph-mon[33394]: 5.2 scrub starts 2024-02-10T07:49:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:04 smithi170 ceph-mon[33394]: 5.2 scrub ok 2024-02-10T07:49:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:04 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:49:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:04 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:49:04.712 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:04.746 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:04.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:04.848 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:04.884 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:04.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:04.986 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:05.027 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:05.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:05.128 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:05.160 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:05.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:05.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:05.228 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.3 to 1 (10000) 2024-02-10T07:49:05.262 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:05.294 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:05.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[29816]: pgmap v790: 148 pgs: 148 active+clean; 1.5 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-02-10T07:49:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[29816]: osdmap e374: 8 total, 8 up, 7 in 2024-02-10T07:49:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[29816]: 5.8 scrub starts 2024-02-10T07:49:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[29816]: 5.8 scrub ok 2024-02-10T07:49:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[29816]: 4.7 scrub starts 2024-02-10T07:49:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[29816]: 4.7 scrub ok 2024-02-10T07:49:05.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/881763706' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:49:05.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:05.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[40743]: pgmap v790: 148 pgs: 148 active+clean; 1.5 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-02-10T07:49:05.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[40743]: osdmap e374: 8 total, 8 up, 7 in 2024-02-10T07:49:05.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[40743]: 5.8 scrub starts 2024-02-10T07:49:05.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[40743]: 5.8 scrub ok 2024-02-10T07:49:05.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[40743]: 4.7 scrub starts 2024-02-10T07:49:05.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[40743]: 4.7 scrub ok 2024-02-10T07:49:05.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:05 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/881763706' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:49:05.396 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:05.420 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:05.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:05.522 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:05.545 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:05.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:05.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:05 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:05 smithi170 ceph-mon[33394]: pgmap v790: 148 pgs: 148 active+clean; 1.5 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-02-10T07:49:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:05 smithi170 ceph-mon[33394]: osdmap e374: 8 total, 8 up, 7 in 2024-02-10T07:49:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:05 smithi170 ceph-mon[33394]: 5.8 scrub starts 2024-02-10T07:49:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:05 smithi170 ceph-mon[33394]: 5.8 scrub ok 2024-02-10T07:49:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:05 smithi170 ceph-mon[33394]: 4.7 scrub starts 2024-02-10T07:49:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:05 smithi170 ceph-mon[33394]: 4.7 scrub ok 2024-02-10T07:49:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:05 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/881763706' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:49:05.648 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:05.672 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:05.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:05.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:05.774 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:05.798 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:05.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:05.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:05.900 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:05.924 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:05.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:06.026 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:06.049 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:06.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:06.151 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:06.174 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:06.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:06.236 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:49:06.277 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:06.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:06.311 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:06.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:06.412 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:06.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:06.437 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:06.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:06.538 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:06.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:06.563 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:06.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[29816]: 10.d scrub starts 2024-02-10T07:49:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[29816]: 3.4 scrub starts 2024-02-10T07:49:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[29816]: 3.4 scrub ok 2024-02-10T07:49:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/881763706' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:49:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[29816]: osdmap e375: 8 total, 8 up, 7 in 2024-02-10T07:49:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[29816]: 10.d scrub ok 2024-02-10T07:49:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1177269114' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:49:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:49:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:49:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1177269114' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:49:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[29816]: osdmap e376: 8 total, 8 up, 7 in 2024-02-10T07:49:06.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[40743]: 10.d scrub starts 2024-02-10T07:49:06.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[40743]: 3.4 scrub starts 2024-02-10T07:49:06.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[40743]: 3.4 scrub ok 2024-02-10T07:49:06.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/881763706' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:49:06.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[40743]: osdmap e375: 8 total, 8 up, 7 in 2024-02-10T07:49:06.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[40743]: 10.d scrub ok 2024-02-10T07:49:06.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1177269114' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:49:06.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:49:06.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:49:06.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1177269114' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:49:06.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:06 smithi008 ceph-mon[40743]: osdmap e376: 8 total, 8 up, 7 in 2024-02-10T07:49:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:06 smithi170 ceph-mon[33394]: 10.d scrub starts 2024-02-10T07:49:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:06 smithi170 ceph-mon[33394]: 3.4 scrub starts 2024-02-10T07:49:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:06 smithi170 ceph-mon[33394]: 3.4 scrub ok 2024-02-10T07:49:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:06 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/881763706' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:49:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:06 smithi170 ceph-mon[33394]: osdmap e375: 8 total, 8 up, 7 in 2024-02-10T07:49:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:06 smithi170 ceph-mon[33394]: 10.d scrub ok 2024-02-10T07:49:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:06 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1177269114' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:49:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:06 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:49:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:06 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:49:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:06 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1177269114' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:49:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:06 smithi170 ceph-mon[33394]: osdmap e376: 8 total, 8 up, 7 in 2024-02-10T07:49:06.664 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:06.687 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:06.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:06.789 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:06.813 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:06.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:06.913 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 4 1 2024-02-10T07:49:06.914 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:06.937 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:06.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:07.039 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:07.068 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:07.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:07.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:07.170 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:07.212 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:07.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:07.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:07.313 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:07.347 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:07.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:07.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:07.449 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:07.480 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:07.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:07.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:07.544 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:07 smithi008 ceph-mon[29816]: pgmap v793: 148 pgs: 148 active+clean; 1.4 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s 2024-02-10T07:49:07.544 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:07 smithi008 ceph-mon[29816]: 3.1c scrub starts 2024-02-10T07:49:07.545 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:07 smithi008 ceph-mon[29816]: 3.1c scrub ok 2024-02-10T07:49:07.545 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:07 smithi008 ceph-mon[29816]: 6.d scrub starts 2024-02-10T07:49:07.545 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:07 smithi008 ceph-mon[29816]: 6.d scrub ok 2024-02-10T07:49:07.545 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:07 smithi008 ceph-mon[29816]: osdmap e377: 8 total, 8 up, 7 in 2024-02-10T07:49:07.545 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:07 smithi008 ceph-mon[29816]: 10.5 starting backfill to osd.3 from (0'0,0'0] MAX to 374'12370 2024-02-10T07:49:07.545 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:07 smithi008 ceph-mon[40743]: pgmap v793: 148 pgs: 148 active+clean; 1.4 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s 2024-02-10T07:49:07.545 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:07 smithi008 ceph-mon[40743]: 3.1c scrub starts 2024-02-10T07:49:07.545 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:07 smithi008 ceph-mon[40743]: 3.1c scrub ok 2024-02-10T07:49:07.545 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:07 smithi008 ceph-mon[40743]: 6.d scrub starts 2024-02-10T07:49:07.546 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:07 smithi008 ceph-mon[40743]: 6.d scrub ok 2024-02-10T07:49:07.546 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:07 smithi008 ceph-mon[40743]: osdmap e377: 8 total, 8 up, 7 in 2024-02-10T07:49:07.546 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:07 smithi008 ceph-mon[40743]: 10.5 starting backfill to osd.3 from (0'0,0'0] MAX to 374'12370 2024-02-10T07:49:07.582 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:07.618 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:07.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:07.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:07 smithi170 ceph-mon[33394]: pgmap v793: 148 pgs: 148 active+clean; 1.4 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s 2024-02-10T07:49:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:07 smithi170 ceph-mon[33394]: 3.1c scrub starts 2024-02-10T07:49:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:07 smithi170 ceph-mon[33394]: 3.1c scrub ok 2024-02-10T07:49:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:07 smithi170 ceph-mon[33394]: 6.d scrub starts 2024-02-10T07:49:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:07 smithi170 ceph-mon[33394]: 6.d scrub ok 2024-02-10T07:49:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:07 smithi170 ceph-mon[33394]: osdmap e377: 8 total, 8 up, 7 in 2024-02-10T07:49:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:07 smithi170 ceph-mon[33394]: 10.5 starting backfill to osd.3 from (0'0,0'0] MAX to 374'12370 2024-02-10T07:49:07.720 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:07.757 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:07.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:07.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:07.859 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:07.895 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:07.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:07.996 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:08.030 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:08.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:08.132 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:08.166 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:08.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:08.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:08.268 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:08.308 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:08.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:08.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:08.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-mon[29816]: 10.3 starting backfill to osd.3 from (0'0,0'0] MAX to 374'12227 2024-02-10T07:49:08.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-mon[29816]: 10.7 starting backfill to osd.3 from (0'0,0'0] MAX to 374'12326 2024-02-10T07:49:08.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-mon[29816]: 10.f starting backfill to osd.3 from (0'0,0'0] MAX to 374'12306 2024-02-10T07:49:08.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-mon[29816]: 10.b scrub starts 2024-02-10T07:49:08.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-mon[29816]: 5.c scrub starts 2024-02-10T07:49:08.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-mon[29816]: 5.c scrub ok 2024-02-10T07:49:08.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-mon[29816]: 10.b scrub ok 2024-02-10T07:49:08.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-mon[40743]: 10.3 starting backfill to osd.3 from (0'0,0'0] MAX to 374'12227 2024-02-10T07:49:08.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-mon[40743]: 10.7 starting backfill to osd.3 from (0'0,0'0] MAX to 374'12326 2024-02-10T07:49:08.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-mon[40743]: 10.f starting backfill to osd.3 from (0'0,0'0] MAX to 374'12306 2024-02-10T07:49:08.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-mon[40743]: 10.b scrub starts 2024-02-10T07:49:08.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-mon[40743]: 5.c scrub starts 2024-02-10T07:49:08.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-mon[40743]: 5.c scrub ok 2024-02-10T07:49:08.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-mon[40743]: 10.b scrub ok 2024-02-10T07:49:08.410 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:08.445 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:08.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:08.546 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:08.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:08.605 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:08.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:08 smithi170 ceph-mon[33394]: 10.3 starting backfill to osd.3 from (0'0,0'0] MAX to 374'12227 2024-02-10T07:49:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:08 smithi170 ceph-mon[33394]: 10.7 starting backfill to osd.3 from (0'0,0'0] MAX to 374'12326 2024-02-10T07:49:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:08 smithi170 ceph-mon[33394]: 10.f starting backfill to osd.3 from (0'0,0'0] MAX to 374'12306 2024-02-10T07:49:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:08 smithi170 ceph-mon[33394]: 10.b scrub starts 2024-02-10T07:49:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:08 smithi170 ceph-mon[33394]: 5.c scrub starts 2024-02-10T07:49:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:08 smithi170 ceph-mon[33394]: 5.c scrub ok 2024-02-10T07:49:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:08 smithi170 ceph-mon[33394]: 10.b scrub ok 2024-02-10T07:49:08.707 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:08.741 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:08.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:08.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:08.843 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:08.882 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:08.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:08.984 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:09.024 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:09.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:09.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:49:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:49:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:49:09.098 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:49:09.125 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:09.135 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:49:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:09.164 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:09.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:09.170 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:49:09.207 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:49:09.255 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:49:09.265 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:09.300 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:49:09.305 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:09.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:09.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:09.333 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:49:09.370 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:49:09.370 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[29816]: pgmap v796: 148 pgs: 1 active+recovering, 147 active+clean; 1.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 487 op/s; 1/68382 objects misplaced (0.001%); 27 B/s, 1 objects/s recovering 2024-02-10T07:49:09.370 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[29816]: 5.19 scrub starts 2024-02-10T07:49:09.370 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[29816]: 5.19 scrub ok 2024-02-10T07:49:09.370 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[29816]: 3.f deep-scrub starts 2024-02-10T07:49:09.371 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[29816]: 3.f deep-scrub ok 2024-02-10T07:49:09.371 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:49:09.371 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:49:09.371 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2279536723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:49:09.371 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[40743]: pgmap v796: 148 pgs: 1 active+recovering, 147 active+clean; 1.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 487 op/s; 1/68382 objects misplaced (0.001%); 27 B/s, 1 objects/s recovering 2024-02-10T07:49:09.371 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[40743]: 5.19 scrub starts 2024-02-10T07:49:09.371 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[40743]: 5.19 scrub ok 2024-02-10T07:49:09.371 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[40743]: 3.f deep-scrub starts 2024-02-10T07:49:09.371 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[40743]: 3.f deep-scrub ok 2024-02-10T07:49:09.371 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:49:09.372 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:49:09.372 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:09 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2279536723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:49:09.372 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:49:09.372 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:49:09.372 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:49:09.372 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:49:09.372 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:49:09.373 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:49:09.373 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:49:09.373 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:49:09.406 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:09.444 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:09.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:09.545 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:09.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:09.580 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:09.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:09 smithi170 ceph-mon[33394]: pgmap v796: 148 pgs: 1 active+recovering, 147 active+clean; 1.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 487 op/s; 1/68382 objects misplaced (0.001%); 27 B/s, 1 objects/s recovering 2024-02-10T07:49:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:09 smithi170 ceph-mon[33394]: 5.19 scrub starts 2024-02-10T07:49:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:09 smithi170 ceph-mon[33394]: 5.19 scrub ok 2024-02-10T07:49:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:09 smithi170 ceph-mon[33394]: 3.f deep-scrub starts 2024-02-10T07:49:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:09 smithi170 ceph-mon[33394]: 3.f deep-scrub ok 2024-02-10T07:49:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:09 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:49:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:09 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:49:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:09 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2279536723' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:49:09.681 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:09.718 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:09.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:09.820 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:09.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:09.860 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:09.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:09.961 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:09.997 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:09.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:09.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:10.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:10.136 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:10.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:10.196 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:10.238 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:10.278 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:10.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:10.380 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:10.416 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:10.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:10.518 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:10.558 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:10.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:10.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:10.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:10 smithi008 ceph-mon[29816]: 4.12 scrub starts 2024-02-10T07:49:10.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:10 smithi008 ceph-mon[29816]: 4.12 scrub ok 2024-02-10T07:49:10.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:10 smithi008 ceph-mon[29816]: osdmap e378: 8 total, 8 up, 7 in 2024-02-10T07:49:10.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:10 smithi008 ceph-mon[40743]: 4.12 scrub starts 2024-02-10T07:49:10.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:10 smithi008 ceph-mon[40743]: 4.12 scrub ok 2024-02-10T07:49:10.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:10 smithi008 ceph-mon[40743]: osdmap e378: 8 total, 8 up, 7 in 2024-02-10T07:49:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:10 smithi170 ceph-mon[33394]: 4.12 scrub starts 2024-02-10T07:49:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:10 smithi170 ceph-mon[33394]: 4.12 scrub ok 2024-02-10T07:49:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:10 smithi170 ceph-mon[33394]: osdmap e378: 8 total, 8 up, 7 in 2024-02-10T07:49:10.660 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:10.691 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:10.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:10.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:10.716 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:49:10.793 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:10.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:10.834 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:10.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:10.935 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:10.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:10.975 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:10.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:11.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:11.114 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:11.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:11.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:11.216 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:11.256 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:11.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:11.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:11 smithi008 ceph-mon[29816]: pgmap v798: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering, 143 active+clean; 1.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 783 op/s; 3770/56970 objects misplaced (6.618%); 13 MiB/s, 217 objects/s recovering 2024-02-10T07:49:11.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:11 smithi008 ceph-mon[29816]: 2.2 scrub starts 2024-02-10T07:49:11.309 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:11 smithi008 ceph-mon[29816]: 2.2 scrub ok 2024-02-10T07:49:11.309 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:11 smithi008 ceph-mon[40743]: pgmap v798: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering, 143 active+clean; 1.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 783 op/s; 3770/56970 objects misplaced (6.618%); 13 MiB/s, 217 objects/s recovering 2024-02-10T07:49:11.309 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:11 smithi008 ceph-mon[40743]: 2.2 scrub starts 2024-02-10T07:49:11.309 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:11 smithi008 ceph-mon[40743]: 2.2 scrub ok 2024-02-10T07:49:11.358 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:11.400 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:11.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:11.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:11.501 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:11.531 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:11.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:11 smithi170 ceph-mon[33394]: pgmap v798: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering, 143 active+clean; 1.1 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 783 op/s; 3770/56970 objects misplaced (6.618%); 13 MiB/s, 217 objects/s recovering 2024-02-10T07:49:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:11 smithi170 ceph-mon[33394]: 2.2 scrub starts 2024-02-10T07:49:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:11 smithi170 ceph-mon[33394]: 2.2 scrub ok 2024-02-10T07:49:11.633 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:11.661 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:11.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:11.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:11.763 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:11.792 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:11.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:11.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:11.894 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:11.921 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:11.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:11.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:12.000 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:12.023 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:12.051 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:12.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:12.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:12.152 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:12.178 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:12.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:12.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:12.280 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:12.286 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:49:12.286 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:49:12.307 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:12.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:12.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:12.409 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:12.435 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:12.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:12.537 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:12.562 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:12.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:12.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:12 smithi008 ceph-mon[40743]: 6.1d scrub starts 2024-02-10T07:49:12.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:12 smithi008 ceph-mon[40743]: 6.1d scrub ok 2024-02-10T07:49:12.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:12 smithi008 ceph-mon[40743]: osdmap e379: 8 total, 8 up, 7 in 2024-02-10T07:49:12.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:12 smithi008 ceph-mon[40743]: 6.8 deep-scrub starts 2024-02-10T07:49:12.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:12 smithi008 ceph-mon[40743]: 6.8 deep-scrub ok 2024-02-10T07:49:12.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:12 smithi008 ceph-mon[29816]: 6.1d scrub starts 2024-02-10T07:49:12.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:12 smithi008 ceph-mon[29816]: 6.1d scrub ok 2024-02-10T07:49:12.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:12 smithi008 ceph-mon[29816]: osdmap e379: 8 total, 8 up, 7 in 2024-02-10T07:49:12.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:12 smithi008 ceph-mon[29816]: 6.8 deep-scrub starts 2024-02-10T07:49:12.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:12 smithi008 ceph-mon[29816]: 6.8 deep-scrub ok 2024-02-10T07:49:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:12 smithi170 ceph-mon[33394]: 6.1d scrub starts 2024-02-10T07:49:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:12 smithi170 ceph-mon[33394]: 6.1d scrub ok 2024-02-10T07:49:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:12 smithi170 ceph-mon[33394]: osdmap e379: 8 total, 8 up, 7 in 2024-02-10T07:49:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:12 smithi170 ceph-mon[33394]: 6.8 deep-scrub starts 2024-02-10T07:49:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:12 smithi170 ceph-mon[33394]: 6.8 deep-scrub ok 2024-02-10T07:49:12.664 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:12.688 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:12.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:12.790 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:12.814 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:12.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:12.916 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:12.939 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:12.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:13.040 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:49:13.041 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:13.064 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:13.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:13.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:13.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:13.168 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-10T07:49:13.190 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:13.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:13.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:13.292 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:13.319 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:13.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:13.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:13 smithi008 ceph-mon[29816]: pgmap v800: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 144 active+clean; 1.1 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 873 B/s rd, 0 B/s wr, 650 op/s; 3769/56970 objects misplaced (6.616%); 11 MiB/s, 180 objects/s recovering 2024-02-10T07:49:13.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:13 smithi008 ceph-mon[29816]: 4.14 scrub starts 2024-02-10T07:49:13.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:13 smithi008 ceph-mon[29816]: 4.14 scrub ok 2024-02-10T07:49:13.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:13 smithi008 ceph-mon[29816]: osdmap e380: 8 total, 8 up, 7 in 2024-02-10T07:49:13.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:13 smithi008 ceph-mon[40743]: pgmap v800: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 144 active+clean; 1.1 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 873 B/s rd, 0 B/s wr, 650 op/s; 3769/56970 objects misplaced (6.616%); 11 MiB/s, 180 objects/s recovering 2024-02-10T07:49:13.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:13 smithi008 ceph-mon[40743]: 4.14 scrub starts 2024-02-10T07:49:13.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:13 smithi008 ceph-mon[40743]: 4.14 scrub ok 2024-02-10T07:49:13.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:13 smithi008 ceph-mon[40743]: osdmap e380: 8 total, 8 up, 7 in 2024-02-10T07:49:13.421 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:13.453 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:13.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:13.555 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:13.578 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:13.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:13 smithi170 ceph-mon[33394]: pgmap v800: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_wait, 144 active+clean; 1.1 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 873 B/s rd, 0 B/s wr, 650 op/s; 3769/56970 objects misplaced (6.616%); 11 MiB/s, 180 objects/s recovering 2024-02-10T07:49:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:13 smithi170 ceph-mon[33394]: 4.14 scrub starts 2024-02-10T07:49:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:13 smithi170 ceph-mon[33394]: 4.14 scrub ok 2024-02-10T07:49:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:13 smithi170 ceph-mon[33394]: osdmap e380: 8 total, 8 up, 7 in 2024-02-10T07:49:13.681 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:13.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:13.705 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:13.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:13.806 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:13.830 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:13.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:13.931 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:13.954 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:13.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:14.056 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:14.081 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:14.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:14.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:14.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:49:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:49:14.182 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:14.208 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:14.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:14.309 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:14.334 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:14.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:14.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:14.436 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:14.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:14.473 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:14.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:14.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:14 smithi008 ceph-mon[29816]: 3.1d deep-scrub starts 2024-02-10T07:49:14.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:14 smithi008 ceph-mon[29816]: 3.1d deep-scrub ok 2024-02-10T07:49:14.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:14 smithi008 ceph-mon[29816]: osdmap e381: 8 total, 8 up, 7 in 2024-02-10T07:49:14.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:14 smithi008 ceph-mon[29816]: 5.4 scrub starts 2024-02-10T07:49:14.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:14 smithi008 ceph-mon[29816]: 5.4 scrub ok 2024-02-10T07:49:14.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:14 smithi008 ceph-mon[29816]: osdmap e382: 8 total, 8 up, 7 in 2024-02-10T07:49:14.548 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:14 smithi008 ceph-mon[40743]: 3.1d deep-scrub starts 2024-02-10T07:49:14.548 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:14 smithi008 ceph-mon[40743]: 3.1d deep-scrub ok 2024-02-10T07:49:14.548 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:14 smithi008 ceph-mon[40743]: osdmap e381: 8 total, 8 up, 7 in 2024-02-10T07:49:14.548 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:14 smithi008 ceph-mon[40743]: 5.4 scrub starts 2024-02-10T07:49:14.549 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:14 smithi008 ceph-mon[40743]: 5.4 scrub ok 2024-02-10T07:49:14.549 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:14 smithi008 ceph-mon[40743]: osdmap e382: 8 total, 8 up, 7 in 2024-02-10T07:49:14.574 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:14.610 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:14.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:14 smithi170 ceph-mon[33394]: 3.1d deep-scrub starts 2024-02-10T07:49:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:14 smithi170 ceph-mon[33394]: 3.1d deep-scrub ok 2024-02-10T07:49:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:14 smithi170 ceph-mon[33394]: osdmap e381: 8 total, 8 up, 7 in 2024-02-10T07:49:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:14 smithi170 ceph-mon[33394]: 5.4 scrub starts 2024-02-10T07:49:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:14 smithi170 ceph-mon[33394]: 5.4 scrub ok 2024-02-10T07:49:14.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:14 smithi170 ceph-mon[33394]: osdmap e382: 8 total, 8 up, 7 in 2024-02-10T07:49:14.712 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:14.752 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:14.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:14.853 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:14.891 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:14.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:14.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:14.993 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:15.022 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:15.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:15.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:15.124 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:15.161 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:15.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:15.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:15.263 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:15.294 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:15.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:15.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:15.396 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:15.432 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:15.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:15.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:15.534 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:15.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:15.558 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:15.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:15.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:15.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[29816]: 10.0 scrub starts 2024-02-10T07:49:15.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[29816]: 10.0 scrub ok 2024-02-10T07:49:15.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[29816]: pgmap v803: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 1010 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 547 op/s; 507/49062 objects misplaced (1.033%); 35 MiB/s, 562 objects/s recovering 2024-02-10T07:49:15.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[29816]: 10.2 scrub starts 2024-02-10T07:49:15.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[29816]: 10.2 scrub ok 2024-02-10T07:49:15.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[29816]: 5.d scrub starts 2024-02-10T07:49:15.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[29816]: 5.d scrub ok 2024-02-10T07:49:15.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:15.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[40743]: 10.0 scrub starts 2024-02-10T07:49:15.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[40743]: 10.0 scrub ok 2024-02-10T07:49:15.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[40743]: pgmap v803: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 1010 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 547 op/s; 507/49062 objects misplaced (1.033%); 35 MiB/s, 562 objects/s recovering 2024-02-10T07:49:15.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[40743]: 10.2 scrub starts 2024-02-10T07:49:15.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[40743]: 10.2 scrub ok 2024-02-10T07:49:15.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[40743]: 5.d scrub starts 2024-02-10T07:49:15.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:15 smithi008 ceph-mon[40743]: 5.d scrub ok 2024-02-10T07:49:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:15 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:15 smithi170 ceph-mon[33394]: 10.0 scrub starts 2024-02-10T07:49:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:15 smithi170 ceph-mon[33394]: 10.0 scrub ok 2024-02-10T07:49:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:15 smithi170 ceph-mon[33394]: pgmap v803: 148 pgs: 1 peering, 1 active+remapped+backfilling, 146 active+clean; 1010 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 547 op/s; 507/49062 objects misplaced (1.033%); 35 MiB/s, 562 objects/s recovering 2024-02-10T07:49:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:15 smithi170 ceph-mon[33394]: 10.2 scrub starts 2024-02-10T07:49:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:15 smithi170 ceph-mon[33394]: 10.2 scrub ok 2024-02-10T07:49:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:15 smithi170 ceph-mon[33394]: 5.d scrub starts 2024-02-10T07:49:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:15 smithi170 ceph-mon[33394]: 5.d scrub ok 2024-02-10T07:49:15.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:15.695 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:15.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:15.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:15.797 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:15.824 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:15.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:15.926 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:15.958 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:15.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:16.058 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:16.092 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:16.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:16.194 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:16.223 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:16.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:16.324 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:16.351 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:16.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:16.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:16.453 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:16.490 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:16.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:16.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:16.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:16 smithi008 ceph-mon[29816]: 6.1b deep-scrub starts 2024-02-10T07:49:16.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:16 smithi008 ceph-mon[29816]: 6.1b deep-scrub ok 2024-02-10T07:49:16.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:16 smithi008 ceph-mon[40743]: 6.1b deep-scrub starts 2024-02-10T07:49:16.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:16 smithi008 ceph-mon[40743]: 6.1b deep-scrub ok 2024-02-10T07:49:16.592 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:16.622 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:16.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:16.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:16.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:16 smithi170 ceph-mon[33394]: 6.1b deep-scrub starts 2024-02-10T07:49:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:16 smithi170 ceph-mon[33394]: 6.1b deep-scrub ok 2024-02-10T07:49:16.723 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:16.751 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:16.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:16.853 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:16.883 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:16.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:16.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:16.985 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:17.016 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:17.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:17.117 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:17.154 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:17.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:17.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:17.255 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:17.282 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:17.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:17.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:17.384 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:17.411 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:17.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:17.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:17.512 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:17.541 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:17.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:17.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:17.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:17 smithi008 ceph-mon[29816]: 4.10 scrub starts 2024-02-10T07:49:17.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:17 smithi008 ceph-mon[29816]: 4.10 scrub ok 2024-02-10T07:49:17.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:17 smithi008 ceph-mon[29816]: pgmap v805: 148 pgs: 1 peering, 147 active+clean; 954 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 736 op/s; 42 MiB/s, 667 objects/s recovering 2024-02-10T07:49:17.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:17 smithi008 ceph-mon[29816]: 4.e scrub starts 2024-02-10T07:49:17.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:17 smithi008 ceph-mon[29816]: 4.e scrub ok 2024-02-10T07:49:17.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:17 smithi008 ceph-mon[29816]: 3.1b scrub starts 2024-02-10T07:49:17.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:17 smithi008 ceph-mon[29816]: 3.1b scrub ok 2024-02-10T07:49:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:17 smithi008 ceph-mon[40743]: 4.10 scrub starts 2024-02-10T07:49:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:17 smithi008 ceph-mon[40743]: 4.10 scrub ok 2024-02-10T07:49:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:17 smithi008 ceph-mon[40743]: pgmap v805: 148 pgs: 1 peering, 147 active+clean; 954 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 736 op/s; 42 MiB/s, 667 objects/s recovering 2024-02-10T07:49:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:17 smithi008 ceph-mon[40743]: 4.e scrub starts 2024-02-10T07:49:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:17 smithi008 ceph-mon[40743]: 4.e scrub ok 2024-02-10T07:49:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:17 smithi008 ceph-mon[40743]: 3.1b scrub starts 2024-02-10T07:49:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:17 smithi008 ceph-mon[40743]: 3.1b scrub ok 2024-02-10T07:49:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:17 smithi170 ceph-mon[33394]: 4.10 scrub starts 2024-02-10T07:49:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:17 smithi170 ceph-mon[33394]: 4.10 scrub ok 2024-02-10T07:49:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:17 smithi170 ceph-mon[33394]: pgmap v805: 148 pgs: 1 peering, 147 active+clean; 954 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 736 op/s; 42 MiB/s, 667 objects/s recovering 2024-02-10T07:49:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:17 smithi170 ceph-mon[33394]: 4.e scrub starts 2024-02-10T07:49:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:17 smithi170 ceph-mon[33394]: 4.e scrub ok 2024-02-10T07:49:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:17 smithi170 ceph-mon[33394]: 3.1b scrub starts 2024-02-10T07:49:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:17 smithi170 ceph-mon[33394]: 3.1b scrub ok 2024-02-10T07:49:17.643 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:17.679 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:17.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:17.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:17.781 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:17.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:17.806 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:17.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:17.907 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:17.946 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:17.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:17.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:18.047 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:18.072 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:18.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:18.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:18.173 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:18.207 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:18.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:18.309 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:18.339 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:18.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:18.441 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:18.470 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:18.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:18.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:18.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[29816]: 6.f scrub starts 2024-02-10T07:49:18.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[29816]: 6.f scrub ok 2024-02-10T07:49:18.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[29816]: 3.13 scrub starts 2024-02-10T07:49:18.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[29816]: 3.13 scrub ok 2024-02-10T07:49:18.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[29816]: 5.5 scrub starts 2024-02-10T07:49:18.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[29816]: 5.5 scrub ok 2024-02-10T07:49:18.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:18.512 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:49:18.512 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[40743]: 6.f scrub starts 2024-02-10T07:49:18.512 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[40743]: 6.f scrub ok 2024-02-10T07:49:18.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[40743]: 3.13 scrub starts 2024-02-10T07:49:18.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[40743]: 3.13 scrub ok 2024-02-10T07:49:18.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[40743]: 5.5 scrub starts 2024-02-10T07:49:18.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[40743]: 5.5 scrub ok 2024-02-10T07:49:18.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:18.513 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:49:18.572 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:18.613 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:18.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:18 smithi170 ceph-mon[33394]: 6.f scrub starts 2024-02-10T07:49:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:18 smithi170 ceph-mon[33394]: 6.f scrub ok 2024-02-10T07:49:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:18 smithi170 ceph-mon[33394]: 3.13 scrub starts 2024-02-10T07:49:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:18 smithi170 ceph-mon[33394]: 3.13 scrub ok 2024-02-10T07:49:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:18 smithi170 ceph-mon[33394]: 5.5 scrub starts 2024-02-10T07:49:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:18 smithi170 ceph-mon[33394]: 5.5 scrub ok 2024-02-10T07:49:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:49:18.715 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:18.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:18.748 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:18.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:18.849 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:18.886 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:18.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:18.987 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:19.018 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:19.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:19.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:19.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:49:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:49:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:49:19.120 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:19.156 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:19.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:19.161 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:19.258 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:19.287 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:19.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:19.332 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.4 to 1 (10000) 2024-02-10T07:49:19.389 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:19.421 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:19.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:19.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:19.523 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:19.552 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:19.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:19.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:19.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:19 smithi008 ceph-mon[29816]: pgmap v806: 148 pgs: 1 peering, 147 active+clean; 954 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 589 op/s; 33 MiB/s, 534 objects/s recovering 2024-02-10T07:49:19.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:19 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2303428778' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:49:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:19 smithi008 ceph-mon[40743]: pgmap v806: 148 pgs: 1 peering, 147 active+clean; 954 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 589 op/s; 33 MiB/s, 534 objects/s recovering 2024-02-10T07:49:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:19 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2303428778' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:49:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:19 smithi170 ceph-mon[33394]: pgmap v806: 148 pgs: 1 peering, 147 active+clean; 954 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 589 op/s; 33 MiB/s, 534 objects/s recovering 2024-02-10T07:49:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:19 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2303428778' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:49:19.653 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:19.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:19.688 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:19.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:19.789 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:19.825 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:19.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:19.926 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:19.957 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:19.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:19.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:20.059 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:20.089 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:20.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:20.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:20.179 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:20.191 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:20.236 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:20.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:20.338 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:20.339 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:49:20.373 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:20.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:20.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:20.475 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:20.515 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:20.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:20.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:20.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[29816]: 4.2 deep-scrub starts 2024-02-10T07:49:20.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[29816]: 4.2 deep-scrub ok 2024-02-10T07:49:20.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[29816]: 3.3 scrub starts 2024-02-10T07:49:20.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[29816]: 3.3 scrub ok 2024-02-10T07:49:20.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2303428778' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:49:20.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[29816]: osdmap e383: 8 total, 8 up, 7 in 2024-02-10T07:49:20.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2844219310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:49:20.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:49:20.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[40743]: 4.2 deep-scrub starts 2024-02-10T07:49:20.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[40743]: 4.2 deep-scrub ok 2024-02-10T07:49:20.585 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[40743]: 3.3 scrub starts 2024-02-10T07:49:20.585 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[40743]: 3.3 scrub ok 2024-02-10T07:49:20.585 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2303428778' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:49:20.585 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[40743]: osdmap e383: 8 total, 8 up, 7 in 2024-02-10T07:49:20.585 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2844219310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:49:20.585 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:20 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:49:20.616 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:20 smithi170 ceph-mon[33394]: 4.2 deep-scrub starts 2024-02-10T07:49:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:20 smithi170 ceph-mon[33394]: 4.2 deep-scrub ok 2024-02-10T07:49:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:20 smithi170 ceph-mon[33394]: 3.3 scrub starts 2024-02-10T07:49:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:20 smithi170 ceph-mon[33394]: 3.3 scrub ok 2024-02-10T07:49:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:20 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2303428778' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:49:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:20 smithi170 ceph-mon[33394]: osdmap e383: 8 total, 8 up, 7 in 2024-02-10T07:49:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:20 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2844219310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:49:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:20 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:49:20.646 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:20.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:20.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:20.748 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:20.783 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:20.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:20.885 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:20.922 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:20.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:20.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:21.024 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:21.057 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:21.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:21.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:21.159 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:21.190 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:21.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:21.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:21.292 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:21.335 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:21.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:21.436 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:21.469 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:21.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:21.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:21.571 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:21.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:21.601 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:21.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:21 smithi170 ceph-mon[33394]: pgmap v808: 148 pgs: 148 active+clean; 623 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.30k op/s; 29 MiB/s, 471 objects/s recovering 2024-02-10T07:49:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:21 smithi170 ceph-mon[33394]: 4.1 scrub starts 2024-02-10T07:49:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:21 smithi170 ceph-mon[33394]: 4.1 scrub ok 2024-02-10T07:49:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:21 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:49:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:21 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:49:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:21 smithi170 ceph-mon[33394]: osdmap e384: 8 total, 8 up, 7 in 2024-02-10T07:49:21.702 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:21.741 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:21.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:21.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:21.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:21 smithi008 ceph-mon[29816]: pgmap v808: 148 pgs: 148 active+clean; 623 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.30k op/s; 29 MiB/s, 471 objects/s recovering 2024-02-10T07:49:21.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:21 smithi008 ceph-mon[29816]: 4.1 scrub starts 2024-02-10T07:49:21.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:21 smithi008 ceph-mon[29816]: 4.1 scrub ok 2024-02-10T07:49:21.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:21 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:49:21.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:21 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:49:21.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:21 smithi008 ceph-mon[29816]: osdmap e384: 8 total, 8 up, 7 in 2024-02-10T07:49:21.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:21 smithi008 ceph-mon[40743]: pgmap v808: 148 pgs: 148 active+clean; 623 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.30k op/s; 29 MiB/s, 471 objects/s recovering 2024-02-10T07:49:21.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:21 smithi008 ceph-mon[40743]: 4.1 scrub starts 2024-02-10T07:49:21.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:21 smithi008 ceph-mon[40743]: 4.1 scrub ok 2024-02-10T07:49:21.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:21 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:49:21.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:21 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:49:21.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:21 smithi008 ceph-mon[40743]: osdmap e384: 8 total, 8 up, 7 in 2024-02-10T07:49:21.843 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:21.872 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:21.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:21.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:21.974 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:22.020 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:22.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:22.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:22.122 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:22.160 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:22.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:22.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:22.262 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:22.304 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:22.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:22.405 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:22.438 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:22.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:22.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:22.540 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:22.577 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:22.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:22.678 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:22.703 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:22.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:22.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:22.808 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:22.845 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:22.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:22.946 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:22.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:22.985 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:22.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:23.086 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:23.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:23.125 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:23.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:23.227 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:23.260 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:23.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:23.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:23.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:22 smithi008 ceph-mon[29816]: pgmap v810: 148 pgs: 148 active+clean; 623 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 656 B/s rd, 0 B/s wr, 680 op/s 2024-02-10T07:49:23.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:22 smithi008 ceph-mon[40743]: pgmap v810: 148 pgs: 148 active+clean; 623 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 656 B/s rd, 0 B/s wr, 680 op/s 2024-02-10T07:49:23.362 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:22 smithi170 ceph-mon[33394]: pgmap v810: 148 pgs: 148 active+clean; 623 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 656 B/s rd, 0 B/s wr, 680 op/s 2024-02-10T07:49:23.392 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:23.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:23.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:23.494 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:23.524 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:23.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:23.625 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:23.662 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:23.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:23.762 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:23.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:23.803 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:23.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:23.905 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:23.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:23.944 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:23.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:24.045 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:24.082 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:24.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:24.107 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:49:24.136 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:49:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:49:24.184 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:24.220 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:24.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:24.322 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:24.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:24.372 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:24.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:24.473 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:24.509 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:24.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:24.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:24.611 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:24.651 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:24.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:24.753 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:24.793 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:24.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:24.795 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 5 1 2024-02-10T07:49:24.894 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:24.938 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:24.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:24.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:25.040 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:25.075 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:25.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:25.177 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:25.224 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:25.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:25.325 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:25.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:25.358 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:25.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:25.400 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:25 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:25.400 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:25 smithi008 ceph-mon[29816]: pgmap v811: 148 pgs: 148 active+clean; 352 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-02-10T07:49:25.401 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:25 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:25.401 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:25 smithi008 ceph-mon[40743]: pgmap v811: 148 pgs: 148 active+clean; 352 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-02-10T07:49:25.459 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:25.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:25.494 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:25.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:25.595 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:25.632 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:25 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:25 smithi170 ceph-mon[33394]: pgmap v811: 148 pgs: 148 active+clean; 352 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-02-10T07:49:25.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:25.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:25.734 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:25.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:25.769 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:25.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:25.870 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:25.903 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:25.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:25.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:26.005 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:26.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:26.046 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:26.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:26.147 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:26.184 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:26.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:26.285 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:26.321 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:26.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:26.423 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:26.456 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:26.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:26.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:26.559 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:26.597 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:26.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:26.657 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Removed 108458 objects 2024-02-10T07:49:26.658 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Clean up completed and total clean up time :95.6989 2024-02-10T07:49:26.698 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:26.722 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:26.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:26.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:26.791 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:49:26.791 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:49:26.823 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:26.846 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:26.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:26.948 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:26.974 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:26.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:27.075 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:27.098 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:27.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:27.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:27.200 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:27.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:27.224 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:27.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:27.325 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:27.350 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:27.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:27.452 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:27.476 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:27.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:27.550 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:27.578 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:27.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:27 smithi008 ceph-mon[29816]: pgmap v812: 148 pgs: 148 active+clean; 232 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-02-10T07:49:27.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:27 smithi008 ceph-mon[40743]: pgmap v812: 148 pgs: 148 active+clean; 232 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-02-10T07:49:27.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:27.604 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:27.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:27 smithi170 ceph-mon[33394]: pgmap v812: 148 pgs: 148 active+clean; 232 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-02-10T07:49:27.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:27.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:27.732 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:27.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:27.833 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:27.858 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:27.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:27.959 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:27.984 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:27.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:28.086 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:28.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:28.111 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:28.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:28.212 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:28.237 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:28.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:28.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:28.260 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:49:28.260 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:28 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:49:28.260 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:28 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:49:28.281 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:49:28.302 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:49:28.322 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:49:28.338 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:28.341 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:49:28.363 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:49:28.363 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:28.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:28.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:28.385 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:49:28.406 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:49:28.407 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:49:28.407 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:49:28.407 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:49:28.407 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:49:28.407 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:49:28.408 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:49:28.408 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:49:28.408 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:49:28.465 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:28.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:28.489 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:28.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:28.590 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:28.613 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:28.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:28.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:28.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:28 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:49:28.715 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:28.738 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:28.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:28.840 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:28.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:28.864 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:28.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:28.965 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:28.989 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:28.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:28.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:29.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:49:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:49:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:49:29.091 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:29.115 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:29.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:29.216 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:29.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:29.242 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:29.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:29.343 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:29.368 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:29.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:29.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:29 smithi008 ceph-mon[29816]: pgmap v813: 148 pgs: 148 active+clean; 232 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 584 B/s rd, 0 B/s wr, 715 op/s 2024-02-10T07:49:29.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:29 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2935271391' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:49:29.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:29 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:29.453 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:29 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:29.453 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:29 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:49:29.453 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:29 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:49:29.453 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:29 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:29.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:29 smithi008 ceph-mon[40743]: pgmap v813: 148 pgs: 148 active+clean; 232 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 584 B/s rd, 0 B/s wr, 715 op/s 2024-02-10T07:49:29.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:29 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2935271391' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:49:29.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:29 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:29.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:29 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:29.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:29 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:49:29.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:29 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:49:29.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:29 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:29.469 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:29.492 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:29.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:29.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:29 smithi170 ceph-mon[33394]: pgmap v813: 148 pgs: 148 active+clean; 232 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 584 B/s rd, 0 B/s wr, 715 op/s 2024-02-10T07:49:29.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:29 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2935271391' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:49:29.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:29 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:29.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:29 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:29.534 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:29 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:49:29.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:29 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:49:29.535 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:29 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:29.594 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:29.619 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:29.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:29.720 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:29.745 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:29.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:29.830 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-02-10T07:49:29.830 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-02-10T07:49:29.846 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:29.897 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:29.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:29.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:29.999 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:30.023 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:30.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:30.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:30.124 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:30.148 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:30.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:30.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:30.226 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-10T07:49:30.251 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:30.292 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:30.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:30.394 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:30.419 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:30.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:30.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:30.521 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:30.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:30.572 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:30.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:30.673 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:30.696 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:30.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:30.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:30.758 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:30.798 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:30.823 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:30.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:30.925 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:30.955 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:30.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:30.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:31.057 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:31.081 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:31.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:31.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:31.153 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:49:31.182 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:31.210 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:31.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:31.312 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:31.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:31.338 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:31.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:31.437 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:31 smithi008 ceph-mon[29816]: pgmap v814: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-02-10T07:49:31.437 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:31 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2716973151' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:49:31.438 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:31 smithi008 ceph-mon[40743]: pgmap v814: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-02-10T07:49:31.438 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:31 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2716973151' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:49:31.439 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:31.464 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:31.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:31.566 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:31.590 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:31.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:31 smithi170 ceph-mon[33394]: pgmap v814: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-02-10T07:49:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:31 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2716973151' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:49:31.691 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:31.714 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:31.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:31.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:31.816 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:31.840 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:31.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:31.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:31.942 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:31.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:31.971 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:31.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:32.067 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:32.067 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:32.072 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:32.098 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:32.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:32.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:32.200 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:32.228 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:32.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:32.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:32.304 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:32 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:49:32.304 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:32 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2716973151' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:49:32.304 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:32 smithi008 ceph-mon[29816]: osdmap e385: 8 total, 8 up, 7 in 2024-02-10T07:49:32.305 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:32 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:49:32.305 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:32 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2716973151' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:49:32.305 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:32 smithi008 ceph-mon[40743]: osdmap e385: 8 total, 8 up, 7 in 2024-02-10T07:49:32.330 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:32.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:32.355 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:32.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:32.456 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:32.480 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:32.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:32.581 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:32.610 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:32.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:32.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:32 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:49:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:32 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2716973151' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:49:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:32 smithi170 ceph-mon[33394]: osdmap e385: 8 total, 8 up, 7 in 2024-02-10T07:49:32.713 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:32.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:32.737 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:32.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:32.838 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:32.867 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:32.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:32.969 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:32.997 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:32.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:32.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:33.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:33.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:33.129 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:33.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:33.230 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:33.254 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:33.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:33.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:33.356 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:33.384 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:33.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:33.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:33.444 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:33 smithi008 ceph-mon[29816]: pgmap v816: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-02-10T07:49:33.444 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:33 smithi008 ceph-mon[40743]: pgmap v816: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-02-10T07:49:33.467 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:49:33.485 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:33.532 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:33.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:33.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:33.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:33 smithi170 ceph-mon[33394]: pgmap v816: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-02-10T07:49:33.634 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:33.662 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:33.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:33.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:33.764 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:33.793 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:33.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:33.894 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:33.922 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:33.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:33.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:34.025 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:34.049 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:34.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:34.081 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:49:34.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:49:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:49:34.150 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:34.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:34.250 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:34.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:34.351 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:34.375 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:34.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:34.478 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:34.506 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:34.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:34.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:34.559 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:34.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:49:34.560 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:34.560 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:49:34.608 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:34.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:49:34.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:49:34.635 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:34.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:34.737 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:34.763 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:34.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:34.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:34.865 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:34.889 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:34.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:34.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:34.991 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:35.019 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:35.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:35.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:35.121 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:35.149 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:35.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:35.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:35.251 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:35.277 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:35.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:35.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:35.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:35 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:35.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:35 smithi008 ceph-mon[29816]: pgmap v817: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 562 op/s 2024-02-10T07:49:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:35 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:35.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:35 smithi008 ceph-mon[40743]: pgmap v817: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 562 op/s 2024-02-10T07:49:35.379 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:35.406 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:35.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:35.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:35.508 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:35.536 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:35.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:35.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:35 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:35.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:35 smithi170 ceph-mon[33394]: pgmap v817: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 562 op/s 2024-02-10T07:49:35.638 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:35.661 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:35.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:35.763 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:35.787 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:35.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:35.888 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:35.917 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:35.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:36.018 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:36.046 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:36.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:36.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:36.147 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:36.174 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:36.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:36.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:36.276 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:36.302 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:36.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:36.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:36.310 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:49:36.311 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:49:36.311 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-02-10T07:49:36.311 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-02-10T07:49:36.311 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:49:36.403 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:36.428 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:36.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:36.530 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:36.554 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:36.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:36.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:36.656 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:36.680 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:36.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:36.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:36.782 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:36.810 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:36.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:36.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:36.856 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:36.857 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:36.911 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:36.940 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:36.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:36.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:37.042 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:37.068 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:37.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:37.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:37.170 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:37.193 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:37.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:37.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:37.223 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.5 to 1 (10000) 2024-02-10T07:49:37.296 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:37.320 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:37.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:37.421 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:37.450 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:37.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:37.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:37.552 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:37.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:37.577 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:37.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:37.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:37 smithi008 ceph-mon[29816]: pgmap v818: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 370 op/s 2024-02-10T07:49:37.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:37 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:49:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:37 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:49:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:37 smithi008 ceph-mon[29816]: osdmap e386: 8 total, 8 up, 7 in 2024-02-10T07:49:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:37 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/241694469' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:49:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:37 smithi008 ceph-mon[40743]: pgmap v818: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 370 op/s 2024-02-10T07:49:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:37 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:49:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:37 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:49:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:37 smithi008 ceph-mon[40743]: osdmap e386: 8 total, 8 up, 7 in 2024-02-10T07:49:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:37 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/241694469' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:49:37.629 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-10T07:49:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:37 smithi170 ceph-mon[33394]: pgmap v818: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 370 op/s 2024-02-10T07:49:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:37 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:49:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:37 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:49:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:37 smithi170 ceph-mon[33394]: osdmap e386: 8 total, 8 up, 7 in 2024-02-10T07:49:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:37 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/241694469' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:49:37.678 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:37.735 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:37.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:37.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:37.837 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:37.864 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:37.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:37.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:37.966 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:37.990 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:37.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:38.092 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:38.116 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:38.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:38.219 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:38.235 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'unique_pool_3' removed 2024-02-10T07:49:38.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:38.256 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:38.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:38.358 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:38.390 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:38.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:38.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:38.492 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:38.516 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:38.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:38.533 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:38 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/241694469' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:49:38.533 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:38 smithi008 ceph-mon[29816]: osdmap e387: 8 total, 8 up, 7 in 2024-02-10T07:49:38.533 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:38 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3347317228' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:49:38.533 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:38 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/241694469' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:49:38.533 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:38 smithi008 ceph-mon[40743]: osdmap e387: 8 total, 8 up, 7 in 2024-02-10T07:49:38.533 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:38 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3347317228' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:49:38.618 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:38.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:38 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/241694469' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:49:38.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:38 smithi170 ceph-mon[33394]: osdmap e387: 8 total, 8 up, 7 in 2024-02-10T07:49:38.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:38 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3347317228' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:49:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:38.649 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:38.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:38.750 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:38.775 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:38.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:38.877 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:38.900 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:38.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:38.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:39.002 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:39.026 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:39.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:39.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:39.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:49:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:49:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:49:39.128 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:39.152 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:39.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:39.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:39.254 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:39.278 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:39.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:39.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:39.380 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:39.416 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:39.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:39.514 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:39 smithi008 ceph-mon[29816]: pgmap v821: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 737 B/s rd, 0 op/s 2024-02-10T07:49:39.514 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:39 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3347317228' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:49:39.514 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:39 smithi008 ceph-mon[29816]: osdmap e388: 8 total, 8 up, 7 in 2024-02-10T07:49:39.515 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:39 smithi008 ceph-mon[40743]: pgmap v821: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 737 B/s rd, 0 op/s 2024-02-10T07:49:39.515 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:39 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3347317228' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:49:39.515 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:39 smithi008 ceph-mon[40743]: osdmap e388: 8 total, 8 up, 7 in 2024-02-10T07:49:39.517 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:39.543 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:39.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:39 smithi170 ceph-mon[33394]: pgmap v821: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 737 B/s rd, 0 op/s 2024-02-10T07:49:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:39 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3347317228' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:49:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:39 smithi170 ceph-mon[33394]: osdmap e388: 8 total, 8 up, 7 in 2024-02-10T07:49:39.644 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:39.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:39.669 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:39.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:39.770 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:39.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:39.796 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:39.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:39.897 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:39.921 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:39.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:39.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:40.023 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:40.046 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:40.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:40.148 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:40.174 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:40.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:40.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:40.276 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:40.303 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:40.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:40.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:40.404 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 6 1 2024-02-10T07:49:40.406 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:40.433 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:40.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:40.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:40.535 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:40.561 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:40.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:40.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:40.663 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:40.689 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:40.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:40.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:40.791 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:40.816 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:40.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:40.917 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:40.940 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:40.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:41.042 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:41.065 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:41.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:41.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:41.134 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-10T07:49:41.134 INFO:tasks.radosbench:Beginning radosbench... 2024-02-10T07:49:41.135 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-02-10T07:49:41.135 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-02-10T07:49:41.167 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:41.202 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:41.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:41.304 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:41.332 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:41.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:41.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:41.360 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:41 smithi008 ceph-mon[29816]: pgmap v823: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:49:41.360 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:41 smithi008 ceph-mon[40743]: pgmap v823: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:49:41.434 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:41.458 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:41.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:41.560 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:41.583 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:41.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:41.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:41.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:41 smithi170 ceph-mon[33394]: pgmap v823: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:49:41.685 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:41.714 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:41.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:41.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:41.816 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:41.839 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:41.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:41.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:41.941 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:41.965 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:41.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:41.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:42.067 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:42.091 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:42.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:42.193 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:42.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:42.222 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:42.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:42.246 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:49:42.323 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:42.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:42.349 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:42.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:42.437 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:42 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1392932947' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:49:42.437 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:42 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:49:42.438 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:42 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:49:42.438 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:42 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:49:42.438 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:42 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1392932947' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:49:42.438 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:42 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:49:42.438 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:42 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:49:42.438 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:42 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:49:42.450 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:42.478 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:42.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:42.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:42.580 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:42.605 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:42.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:42 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1392932947' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:49:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:42 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:49:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:42 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:49:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:42 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:49:42.706 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:42.734 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:42.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:42.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:42.836 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:42.864 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:42.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:42.965 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:42.990 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:42.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:42.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:43.092 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:43.117 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:43.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:43.125 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:43.143 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:43.158 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:43.219 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:43.243 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:43.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:43.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:43.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:43 smithi008 ceph-mon[29816]: pgmap v824: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:49:43.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:43 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:49:43.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:43 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:49:43.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:43 smithi008 ceph-mon[29816]: osdmap e389: 8 total, 8 up, 7 in 2024-02-10T07:49:43.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:43 smithi008 ceph-mon[40743]: pgmap v824: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:49:43.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:43 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:49:43.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:43 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:49:43.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:43 smithi008 ceph-mon[40743]: osdmap e389: 8 total, 8 up, 7 in 2024-02-10T07:49:43.346 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:43.370 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:43.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:43.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:43.472 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:43.497 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:43.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:43.599 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:43.622 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:43.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:43.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:43 smithi170 ceph-mon[33394]: pgmap v824: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:49:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:43 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:49:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:43 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:49:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:43 smithi170 ceph-mon[33394]: osdmap e389: 8 total, 8 up, 7 in 2024-02-10T07:49:43.723 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:43.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:43.753 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:43.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:43.761 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:49:43.780 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:49:43.799 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:49:43.818 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:49:43.838 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:49:43.854 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:43.860 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:49:43.879 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:43.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:43.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:43.882 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:49:43.902 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:49:43.903 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:49:43.903 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:49:43.904 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:49:43.904 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:49:43.904 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:49:43.904 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:49:43.904 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:49:43.904 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:49:43.981 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:44.008 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:44.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:44.110 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:44.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:49:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:49:44.136 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:44.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:44.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:44.238 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:44.262 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:44.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:44.364 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:44.389 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:44.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:44.490 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:44.513 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:44.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:44.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:44.543 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:44 smithi008 ceph-mon[29816]: 6.b scrub starts 2024-02-10T07:49:44.543 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:44 smithi008 ceph-mon[29816]: 6.b scrub ok 2024-02-10T07:49:44.543 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:44 smithi008 ceph-mon[29816]: 4.9 scrub starts 2024-02-10T07:49:44.543 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:44 smithi008 ceph-mon[29816]: 4.9 scrub ok 2024-02-10T07:49:44.543 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:44 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1574726884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:49:44.543 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:44 smithi008 ceph-mon[40743]: 6.b scrub starts 2024-02-10T07:49:44.543 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:44 smithi008 ceph-mon[40743]: 6.b scrub ok 2024-02-10T07:49:44.543 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:44 smithi008 ceph-mon[40743]: 4.9 scrub starts 2024-02-10T07:49:44.544 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:44 smithi008 ceph-mon[40743]: 4.9 scrub ok 2024-02-10T07:49:44.544 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:44 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1574726884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:49:44.615 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:44 smithi170 ceph-mon[33394]: 6.b scrub starts 2024-02-10T07:49:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:44 smithi170 ceph-mon[33394]: 6.b scrub ok 2024-02-10T07:49:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:44 smithi170 ceph-mon[33394]: 4.9 scrub starts 2024-02-10T07:49:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:44 smithi170 ceph-mon[33394]: 4.9 scrub ok 2024-02-10T07:49:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:44 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1574726884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:49:44.638 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:44.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:44.740 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:44.764 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:44.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:44.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:44.866 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:44.890 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:44.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:44.991 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:45.015 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:45.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:45.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:45.078 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:49:45.117 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:45.145 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:45.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:45.247 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:45.270 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:45.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:45.372 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:45.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:45.397 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:45.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:45.498 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:45.521 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:45.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:45.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:45 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:45 smithi008 ceph-mon[40743]: 4.17 scrub starts 2024-02-10T07:49:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:45 smithi008 ceph-mon[40743]: 4.17 scrub ok 2024-02-10T07:49:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:45 smithi008 ceph-mon[40743]: pgmap v826: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:49:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:45 smithi008 ceph-mon[40743]: 6.14 scrub starts 2024-02-10T07:49:45.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:45 smithi008 ceph-mon[40743]: 6.14 scrub ok 2024-02-10T07:49:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:45 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:45 smithi008 ceph-mon[29816]: 4.17 scrub starts 2024-02-10T07:49:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:45 smithi008 ceph-mon[29816]: 4.17 scrub ok 2024-02-10T07:49:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:45 smithi008 ceph-mon[29816]: pgmap v826: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:49:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:45 smithi008 ceph-mon[29816]: 6.14 scrub starts 2024-02-10T07:49:45.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:45 smithi008 ceph-mon[29816]: 6.14 scrub ok 2024-02-10T07:49:45.623 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:45 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:45 smithi170 ceph-mon[33394]: 4.17 scrub starts 2024-02-10T07:49:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:45 smithi170 ceph-mon[33394]: 4.17 scrub ok 2024-02-10T07:49:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:45 smithi170 ceph-mon[33394]: pgmap v826: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:49:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:45 smithi170 ceph-mon[33394]: 6.14 scrub starts 2024-02-10T07:49:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:45 smithi170 ceph-mon[33394]: 6.14 scrub ok 2024-02-10T07:49:45.651 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:45.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:45.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:45.753 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:45.781 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:45.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:45.883 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:45.908 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:45.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:45.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:46.010 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:46.033 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:46.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:46.135 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:46.164 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:46.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:46.265 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:46.290 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:46.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:46.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:46.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:46 smithi008 ceph-mon[29816]: 3.8 scrub starts 2024-02-10T07:49:46.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:46 smithi008 ceph-mon[29816]: 3.8 scrub ok 2024-02-10T07:49:46.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:46 smithi008 ceph-mon[40743]: 3.8 scrub starts 2024-02-10T07:49:46.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:46 smithi008 ceph-mon[40743]: 3.8 scrub ok 2024-02-10T07:49:46.392 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:46.418 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:46.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:46.520 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:46.545 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:46.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:46 smithi170 ceph-mon[33394]: 3.8 scrub starts 2024-02-10T07:49:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:46 smithi170 ceph-mon[33394]: 3.8 scrub ok 2024-02-10T07:49:46.645 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:46.674 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:46.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:46.775 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:46.800 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:46.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:46.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:46.902 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:46.930 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:46.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:46.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:47.032 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:47.085 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:47.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:47.186 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:47.210 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:47.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:47.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:47.312 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:47.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:47 smithi008 ceph-mon[29816]: pgmap v827: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:49:47.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:47 smithi008 ceph-mon[40743]: pgmap v827: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:49:47.340 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:47.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:47.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:47.442 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:47.467 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:47.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:47.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:47.569 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:47.593 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:47.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:47.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:47 smithi170 ceph-mon[33394]: pgmap v827: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:49:47.694 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:47.717 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:47.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:47.819 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:47.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:47.849 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:47.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:47.951 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:47.974 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:47.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:47.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:48.076 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:48.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:48.104 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:48.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:48.205 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:48.233 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:48.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:48.257 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.6 to 1 (10000) 2024-02-10T07:49:48.334 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:48.361 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:48.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:48.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:48.420 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-mon[29816]: 6.e scrub starts 2024-02-10T07:49:48.421 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-mon[29816]: 6.e scrub ok 2024-02-10T07:49:48.421 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-mon[29816]: 4.6 scrub starts 2024-02-10T07:49:48.421 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-mon[29816]: 4.6 scrub ok 2024-02-10T07:49:48.421 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3619012835' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:49:48.421 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:49:48.421 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:49:48.421 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-mon[40743]: 6.e scrub starts 2024-02-10T07:49:48.421 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-mon[40743]: 6.e scrub ok 2024-02-10T07:49:48.421 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-mon[40743]: 4.6 scrub starts 2024-02-10T07:49:48.422 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-mon[40743]: 4.6 scrub ok 2024-02-10T07:49:48.422 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3619012835' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:49:48.422 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:49:48.422 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:49:48.463 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:48.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:48.490 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:48.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:48.591 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:48.616 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:48.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:48 smithi170 ceph-mon[33394]: 6.e scrub starts 2024-02-10T07:49:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:48 smithi170 ceph-mon[33394]: 6.e scrub ok 2024-02-10T07:49:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:48 smithi170 ceph-mon[33394]: 4.6 scrub starts 2024-02-10T07:49:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:48 smithi170 ceph-mon[33394]: 4.6 scrub ok 2024-02-10T07:49:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:48 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3619012835' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:49:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:48 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:49:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:48 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:49:48.718 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:48.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:48.743 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:48.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:48.792 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:49:48.845 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:48.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:48.873 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:48.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:48.973 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:49.000 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:49.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:49.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:49.034 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 7 1 2024-02-10T07:49:49.059 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:49:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:49:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:49:49.101 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:49.126 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:49.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:49.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:49.228 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:49.251 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:49.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:49.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:49.322 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:49:49.322 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:49:49.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:49 smithi008 ceph-mon[40743]: 3.1e scrub starts 2024-02-10T07:49:49.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:49 smithi008 ceph-mon[40743]: 3.1e scrub ok 2024-02-10T07:49:49.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:49 smithi008 ceph-mon[40743]: pgmap v828: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:49:49.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:49 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:49:49.322 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:49 smithi008 ceph-mon[40743]: osdmap e390: 8 total, 8 up, 7 in 2024-02-10T07:49:49.323 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:49 smithi008 ceph-mon[29816]: 3.1e scrub starts 2024-02-10T07:49:49.323 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:49 smithi008 ceph-mon[29816]: 3.1e scrub ok 2024-02-10T07:49:49.323 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:49 smithi008 ceph-mon[29816]: pgmap v828: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:49:49.323 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:49 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:49:49.323 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:49 smithi008 ceph-mon[29816]: osdmap e390: 8 total, 8 up, 7 in 2024-02-10T07:49:49.354 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:49.383 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:49.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:49.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:49.485 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:49.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:49.509 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:49.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:49.610 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:49 smithi170 ceph-mon[33394]: 3.1e scrub starts 2024-02-10T07:49:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:49 smithi170 ceph-mon[33394]: 3.1e scrub ok 2024-02-10T07:49:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:49 smithi170 ceph-mon[33394]: pgmap v828: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:49:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:49 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:49:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:49 smithi170 ceph-mon[33394]: osdmap e390: 8 total, 8 up, 7 in 2024-02-10T07:49:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:49.634 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:49.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:49.648 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:49.736 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:49.764 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:49.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:49.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:49.866 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:49.896 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:49.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:49.997 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:50.023 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:50.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:50.125 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:50.151 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:50.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:50.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:50.252 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:50.281 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:50.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:50.382 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:50.387 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:50 smithi008 ceph-mon[40743]: 3.5 scrub starts 2024-02-10T07:49:50.387 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:50 smithi008 ceph-mon[40743]: 3.5 scrub ok 2024-02-10T07:49:50.387 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:50 smithi008 ceph-mon[29816]: 3.5 scrub starts 2024-02-10T07:49:50.387 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:50 smithi008 ceph-mon[29816]: 3.5 scrub ok 2024-02-10T07:49:50.410 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:50.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:50.413 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-10T07:49:50.512 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:50.536 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:50.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:50 smithi170 ceph-mon[33394]: 3.5 scrub starts 2024-02-10T07:49:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:50 smithi170 ceph-mon[33394]: 3.5 scrub ok 2024-02-10T07:49:50.638 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:50.666 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:50.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:50.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:50.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:50.794 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:50.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:50.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:50.896 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:50.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:50.923 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:50.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:51.024 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:51.050 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:51.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:51.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:51.152 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:51.178 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:51.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:51.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:51.280 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:51.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:51.305 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:51.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:51.405 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:51.434 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:51.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:51.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:51.536 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:51.563 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:51.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:51.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:51.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[29816]: 4.a scrub starts 2024-02-10T07:49:51.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[29816]: 4.a scrub ok 2024-02-10T07:49:51.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[29816]: 3.0 scrub starts 2024-02-10T07:49:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[29816]: 3.0 scrub ok 2024-02-10T07:49:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[29816]: 6.6 scrub starts 2024-02-10T07:49:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[29816]: 6.6 scrub ok 2024-02-10T07:49:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[29816]: pgmap v830: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:49:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[29816]: 5.6 scrub starts 2024-02-10T07:49:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[29816]: 5.6 scrub ok 2024-02-10T07:49:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[29816]: 5.3 scrub starts 2024-02-10T07:49:51.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[29816]: 5.3 scrub ok 2024-02-10T07:49:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[40743]: 4.a scrub starts 2024-02-10T07:49:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[40743]: 4.a scrub ok 2024-02-10T07:49:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[40743]: 3.0 scrub starts 2024-02-10T07:49:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[40743]: 3.0 scrub ok 2024-02-10T07:49:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[40743]: 6.6 scrub starts 2024-02-10T07:49:51.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[40743]: 6.6 scrub ok 2024-02-10T07:49:51.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[40743]: pgmap v830: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:49:51.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[40743]: 5.6 scrub starts 2024-02-10T07:49:51.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[40743]: 5.6 scrub ok 2024-02-10T07:49:51.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[40743]: 5.3 scrub starts 2024-02-10T07:49:51.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:51 smithi008 ceph-mon[40743]: 5.3 scrub ok 2024-02-10T07:49:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:51 smithi170 ceph-mon[33394]: 4.a scrub starts 2024-02-10T07:49:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:51 smithi170 ceph-mon[33394]: 4.a scrub ok 2024-02-10T07:49:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:51 smithi170 ceph-mon[33394]: 3.0 scrub starts 2024-02-10T07:49:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:51 smithi170 ceph-mon[33394]: 3.0 scrub ok 2024-02-10T07:49:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:51 smithi170 ceph-mon[33394]: 6.6 scrub starts 2024-02-10T07:49:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:51 smithi170 ceph-mon[33394]: 6.6 scrub ok 2024-02-10T07:49:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:51 smithi170 ceph-mon[33394]: pgmap v830: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:49:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:51 smithi170 ceph-mon[33394]: 5.6 scrub starts 2024-02-10T07:49:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:51 smithi170 ceph-mon[33394]: 5.6 scrub ok 2024-02-10T07:49:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:51 smithi170 ceph-mon[33394]: 5.3 scrub starts 2024-02-10T07:49:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:51 smithi170 ceph-mon[33394]: 5.3 scrub ok 2024-02-10T07:49:51.661 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:51.665 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:51.695 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:51.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:51.797 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:51.820 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:51.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:51.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:51.922 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:51.948 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:51.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:51.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:52.050 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:52.073 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:52.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:52.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:52.175 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:52.199 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:52.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:52.271 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'unique_pool_4' created 2024-02-10T07:49:52.301 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:52.324 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:52.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:52.426 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:52.449 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:52.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:52.551 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:52.575 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:52.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:52.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:52 smithi008 ceph-mon[29816]: 4.1f scrub starts 2024-02-10T07:49:52.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:52 smithi008 ceph-mon[29816]: 4.1f scrub ok 2024-02-10T07:49:52.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:52 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/184696234' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-10T07:49:52.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:52 smithi008 ceph-mon[40743]: 4.1f scrub starts 2024-02-10T07:49:52.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:52 smithi008 ceph-mon[40743]: 4.1f scrub ok 2024-02-10T07:49:52.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:52 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/184696234' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-10T07:49:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:52 smithi170 ceph-mon[33394]: 4.1f scrub starts 2024-02-10T07:49:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:52 smithi170 ceph-mon[33394]: 4.1f scrub ok 2024-02-10T07:49:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:52 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/184696234' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-10T07:49:52.676 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:52.700 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:52.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:52.803 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:52.826 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:52.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:52.928 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:52.952 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:52.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:52.958 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:52.977 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:52.988 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-02-10T07:49:53.055 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:53.096 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:53.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:53.197 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:53.222 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:53.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:53.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:53.324 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:53.349 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:53.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:53.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:53.451 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:53.475 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:53.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:53.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:53.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[29816]: 5.1d scrub starts 2024-02-10T07:49:53.489 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[29816]: 5.1d scrub ok 2024-02-10T07:49:53.489 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[29816]: 6.1f scrub starts 2024-02-10T07:49:53.489 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[29816]: 6.1f scrub ok 2024-02-10T07:49:53.489 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[29816]: pgmap v831: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:49:53.489 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/184696234' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-10T07:49:53.489 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[29816]: osdmap e391: 8 total, 8 up, 7 in 2024-02-10T07:49:53.489 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[29816]: 6.10 scrub starts 2024-02-10T07:49:53.489 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[29816]: 6.10 scrub ok 2024-02-10T07:49:53.489 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[40743]: 5.1d scrub starts 2024-02-10T07:49:53.489 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[40743]: 5.1d scrub ok 2024-02-10T07:49:53.490 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[40743]: 6.1f scrub starts 2024-02-10T07:49:53.490 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[40743]: 6.1f scrub ok 2024-02-10T07:49:53.490 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[40743]: pgmap v831: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:49:53.490 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/184696234' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-10T07:49:53.490 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[40743]: osdmap e391: 8 total, 8 up, 7 in 2024-02-10T07:49:53.490 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[40743]: 6.10 scrub starts 2024-02-10T07:49:53.490 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:53 smithi008 ceph-mon[40743]: 6.10 scrub ok 2024-02-10T07:49:53.578 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:53.603 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:53.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:53 smithi170 ceph-mon[33394]: 5.1d scrub starts 2024-02-10T07:49:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:53 smithi170 ceph-mon[33394]: 5.1d scrub ok 2024-02-10T07:49:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:53 smithi170 ceph-mon[33394]: 6.1f scrub starts 2024-02-10T07:49:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:53 smithi170 ceph-mon[33394]: 6.1f scrub ok 2024-02-10T07:49:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:53 smithi170 ceph-mon[33394]: pgmap v831: 132 pgs: 132 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:49:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:53 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/184696234' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-10T07:49:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:53 smithi170 ceph-mon[33394]: osdmap e391: 8 total, 8 up, 7 in 2024-02-10T07:49:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:53 smithi170 ceph-mon[33394]: 6.10 scrub starts 2024-02-10T07:49:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:53 smithi170 ceph-mon[33394]: 6.10 scrub ok 2024-02-10T07:49:53.704 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:53.729 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:53.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:53.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:53.831 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:53.856 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:53.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:53.957 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:53.985 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:53.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:53.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:54.087 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:54.112 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:54.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:54.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:49:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:49:54.213 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:54.237 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:54.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:54.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:54.285 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:49:54.339 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:54.365 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:54.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:54.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:54.374 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[40743]: 5.7 scrub starts 2024-02-10T07:49:54.374 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[40743]: 5.7 scrub ok 2024-02-10T07:49:54.374 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[40743]: 3.10 scrub starts 2024-02-10T07:49:54.374 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[40743]: 3.10 scrub ok 2024-02-10T07:49:54.374 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[40743]: 3.c scrub starts 2024-02-10T07:49:54.374 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[40743]: 3.c scrub ok 2024-02-10T07:49:54.374 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[40743]: 4.11 scrub starts 2024-02-10T07:49:54.374 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[40743]: 4.11 scrub ok 2024-02-10T07:49:54.374 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[40743]: osdmap e392: 8 total, 8 up, 7 in 2024-02-10T07:49:54.374 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2159451192' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:49:54.375 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:49:54.375 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[29816]: 5.7 scrub starts 2024-02-10T07:49:54.375 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[29816]: 5.7 scrub ok 2024-02-10T07:49:54.375 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[29816]: 3.10 scrub starts 2024-02-10T07:49:54.375 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[29816]: 3.10 scrub ok 2024-02-10T07:49:54.375 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[29816]: 3.c scrub starts 2024-02-10T07:49:54.375 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[29816]: 3.c scrub ok 2024-02-10T07:49:54.375 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[29816]: 4.11 scrub starts 2024-02-10T07:49:54.375 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[29816]: 4.11 scrub ok 2024-02-10T07:49:54.375 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[29816]: osdmap e392: 8 total, 8 up, 7 in 2024-02-10T07:49:54.375 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2159451192' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:49:54.375 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:54 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:49:54.467 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:54.491 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:54.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:54.593 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:54.620 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:54.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:54 smithi170 ceph-mon[33394]: 5.7 scrub starts 2024-02-10T07:49:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:54 smithi170 ceph-mon[33394]: 5.7 scrub ok 2024-02-10T07:49:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:54 smithi170 ceph-mon[33394]: 3.10 scrub starts 2024-02-10T07:49:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:54 smithi170 ceph-mon[33394]: 3.10 scrub ok 2024-02-10T07:49:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:54 smithi170 ceph-mon[33394]: 3.c scrub starts 2024-02-10T07:49:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:54 smithi170 ceph-mon[33394]: 3.c scrub ok 2024-02-10T07:49:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:54 smithi170 ceph-mon[33394]: 4.11 scrub starts 2024-02-10T07:49:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:54 smithi170 ceph-mon[33394]: 4.11 scrub ok 2024-02-10T07:49:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:54 smithi170 ceph-mon[33394]: osdmap e392: 8 total, 8 up, 7 in 2024-02-10T07:49:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:54 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2159451192' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:49:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:54 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:49:54.722 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:54.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:54.747 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:54.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:54.848 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:54.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:54.873 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:54.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:54.975 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:55.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:55.005 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:55.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:55.105 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:55.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:55.134 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:55.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:55.235 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:55.262 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:55.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:55.364 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:55.391 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:55.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:55.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:55.493 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:55.518 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:55.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:55.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[29816]: 3.16 scrub starts 2024-02-10T07:49:55.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[29816]: 3.16 scrub ok 2024-02-10T07:49:55.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:55.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[29816]: pgmap v834: 148 pgs: 1 creating+activating, 6 unknown, 141 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:49:55.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[29816]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:49:55.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:49:55.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:49:55.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:49:55.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[29816]: osdmap e393: 8 total, 8 up, 7 in 2024-02-10T07:49:55.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[40743]: 3.16 scrub starts 2024-02-10T07:49:55.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[40743]: 3.16 scrub ok 2024-02-10T07:49:55.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:55.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[40743]: pgmap v834: 148 pgs: 1 creating+activating, 6 unknown, 141 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:49:55.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[40743]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:49:55.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:49:55.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:49:55.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:49:55.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:55 smithi008 ceph-mon[40743]: osdmap e393: 8 total, 8 up, 7 in 2024-02-10T07:49:55.621 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:55 smithi170 ceph-mon[33394]: 3.16 scrub starts 2024-02-10T07:49:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:55 smithi170 ceph-mon[33394]: 3.16 scrub ok 2024-02-10T07:49:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:55 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:49:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:55 smithi170 ceph-mon[33394]: pgmap v834: 148 pgs: 1 creating+activating, 6 unknown, 141 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:49:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:55 smithi170 ceph-mon[33394]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T07:49:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:55 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:49:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:55 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:49:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:55 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:49:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:55 smithi170 ceph-mon[33394]: osdmap e393: 8 total, 8 up, 7 in 2024-02-10T07:49:55.649 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:55.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:55.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:55.750 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:55.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:55.779 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:55.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:55.880 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:55.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:55.906 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:55.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:56.007 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:56.035 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:56.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:56.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:56.137 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:56.164 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:56.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:56.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:56.266 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:56.298 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:56.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:56.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:56.398 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:56 smithi008 ceph-mon[29816]: 5.11 scrub starts 2024-02-10T07:49:56.398 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:56 smithi008 ceph-mon[29816]: 5.11 scrub ok 2024-02-10T07:49:56.398 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:56 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:49:56.399 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:56 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:49:56.399 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:56 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:49:56.399 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:56 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:49:56.399 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:56 smithi008 ceph-mon[40743]: 5.11 scrub starts 2024-02-10T07:49:56.399 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:56 smithi008 ceph-mon[40743]: 5.11 scrub ok 2024-02-10T07:49:56.399 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:56 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:49:56.399 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:56 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:49:56.399 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:56 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:49:56.399 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:56 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:49:56.399 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:56.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:56.431 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:56.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:56.532 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:56.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:56.558 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:56.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:56 smithi170 ceph-mon[33394]: 5.11 scrub starts 2024-02-10T07:49:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:56 smithi170 ceph-mon[33394]: 5.11 scrub ok 2024-02-10T07:49:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:56 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:49:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:56 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:49:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:56 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:49:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:56 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:49:56.659 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:56.683 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:56.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:56.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:56.785 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:56.808 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:56.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:56.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:56.910 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:56.933 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:56.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:56.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:57.015 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:49:57.035 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:57.063 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:57.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:57.165 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:57.172 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:49:57.187 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:49:57.193 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:57.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:57.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:57.201 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:49:57.228 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:49:57.247 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:49:57.268 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:49:57.291 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:49:57.295 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:57.312 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:49:57.319 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:57.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:57.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:57.333 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:49:57.335 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:49:57.335 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:49:57.335 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:49:57.335 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:49:57.335 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:49:57.335 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:49:57.335 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:49:57.335 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:49:57.421 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:57.444 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:57.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:57.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:57.546 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:49:57.569 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:57.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:57.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:49:57.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:57 smithi008 ceph-mon[29816]: 4.4 scrub starts 2024-02-10T07:49:57.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:57 smithi008 ceph-mon[29816]: 4.4 scrub ok 2024-02-10T07:49:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:57 smithi008 ceph-mon[29816]: pgmap v836: 148 pgs: 1 creating+activating, 1 unknown, 146 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:49:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:57 smithi008 ceph-mon[29816]: osdmap e394: 8 total, 8 up, 7 in 2024-02-10T07:49:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:57 smithi008 ceph-mon[29816]: 6.1a scrub starts 2024-02-10T07:49:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:57 smithi008 ceph-mon[29816]: 6.1a scrub ok 2024-02-10T07:49:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:57 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1595828428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:49:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:57 smithi008 ceph-mon[40743]: 4.4 scrub starts 2024-02-10T07:49:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:57 smithi008 ceph-mon[40743]: 4.4 scrub ok 2024-02-10T07:49:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:57 smithi008 ceph-mon[40743]: pgmap v836: 148 pgs: 1 creating+activating, 1 unknown, 146 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:49:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:57 smithi008 ceph-mon[40743]: osdmap e394: 8 total, 8 up, 7 in 2024-02-10T07:49:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:57 smithi008 ceph-mon[40743]: 6.1a scrub starts 2024-02-10T07:49:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:57 smithi008 ceph-mon[40743]: 6.1a scrub ok 2024-02-10T07:49:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:57 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1595828428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:49:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:57 smithi170 ceph-mon[33394]: 4.4 scrub starts 2024-02-10T07:49:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:57 smithi170 ceph-mon[33394]: 4.4 scrub ok 2024-02-10T07:49:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:57 smithi170 ceph-mon[33394]: pgmap v836: 148 pgs: 1 creating+activating, 1 unknown, 146 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:49:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:57 smithi170 ceph-mon[33394]: osdmap e394: 8 total, 8 up, 7 in 2024-02-10T07:49:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:57 smithi170 ceph-mon[33394]: 6.1a scrub starts 2024-02-10T07:49:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:57 smithi170 ceph-mon[33394]: 6.1a scrub ok 2024-02-10T07:49:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:57 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1595828428' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:49:57.672 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:57.699 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:57.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:57.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:49:57.801 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:57.825 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:57.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:57.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:57.927 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:57.952 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:57.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:58.053 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:58.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:58.081 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:58.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:58.182 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:58.208 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:58.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:58.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:58.311 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:58.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:58.338 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:58.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:58.439 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:58.464 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:58.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:58.566 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:58.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[29816]: 6.19 scrub starts 2024-02-10T07:49:58.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[29816]: 6.19 scrub ok 2024-02-10T07:49:58.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[29816]: 5.1f scrub starts 2024-02-10T07:49:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[29816]: 5.1f scrub ok 2024-02-10T07:49:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[29816]: 4.0 scrub starts 2024-02-10T07:49:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[29816]: 4.0 scrub ok 2024-02-10T07:49:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[29816]: 5.f scrub starts 2024-02-10T07:49:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[29816]: 5.f scrub ok 2024-02-10T07:49:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[29816]: 4.3 scrub starts 2024-02-10T07:49:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[29816]: 4.3 scrub ok 2024-02-10T07:49:58.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[40743]: 6.19 scrub starts 2024-02-10T07:49:58.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[40743]: 6.19 scrub ok 2024-02-10T07:49:58.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[40743]: 5.1f scrub starts 2024-02-10T07:49:58.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[40743]: 5.1f scrub ok 2024-02-10T07:49:58.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[40743]: 4.0 scrub starts 2024-02-10T07:49:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[40743]: 4.0 scrub ok 2024-02-10T07:49:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[40743]: 5.f scrub starts 2024-02-10T07:49:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[40743]: 5.f scrub ok 2024-02-10T07:49:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[40743]: 4.3 scrub starts 2024-02-10T07:49:58.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-mon[40743]: 4.3 scrub ok 2024-02-10T07:49:58.591 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:58.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:58.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:58 smithi170 ceph-mon[33394]: 6.19 scrub starts 2024-02-10T07:49:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:58 smithi170 ceph-mon[33394]: 6.19 scrub ok 2024-02-10T07:49:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:58 smithi170 ceph-mon[33394]: 5.1f scrub starts 2024-02-10T07:49:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:58 smithi170 ceph-mon[33394]: 5.1f scrub ok 2024-02-10T07:49:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:58 smithi170 ceph-mon[33394]: 4.0 scrub starts 2024-02-10T07:49:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:58 smithi170 ceph-mon[33394]: 4.0 scrub ok 2024-02-10T07:49:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:58 smithi170 ceph-mon[33394]: 5.f scrub starts 2024-02-10T07:49:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:58 smithi170 ceph-mon[33394]: 5.f scrub ok 2024-02-10T07:49:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:58 smithi170 ceph-mon[33394]: 4.3 scrub starts 2024-02-10T07:49:58.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:58 smithi170 ceph-mon[33394]: 4.3 scrub ok 2024-02-10T07:49:58.693 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:58.719 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:58.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:58.821 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:58.847 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:58.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:58.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:58.948 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:49:58.973 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:58.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:58.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:49:59.075 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:59.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:49:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:49:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:49:59.099 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:59.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:59.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:59.200 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:49:59.228 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:59.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:59.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:49:59.330 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:49:59.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:59.359 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:59.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:49:59.460 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:49:59.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:59.484 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:59.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:49:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:59 smithi008 ceph-mon[29816]: 5.1b scrub starts 2024-02-10T07:49:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:59 smithi008 ceph-mon[29816]: 5.1b scrub ok 2024-02-10T07:49:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:59 smithi008 ceph-mon[29816]: 6.0 scrub starts 2024-02-10T07:49:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:59 smithi008 ceph-mon[29816]: 6.0 scrub ok 2024-02-10T07:49:59.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:49:59 smithi008 ceph-mon[29816]: pgmap v838: 148 pgs: 1 creating+activating, 147 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 878 B/s rd, 0 op/s 2024-02-10T07:49:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:59 smithi008 ceph-mon[40743]: 5.1b scrub starts 2024-02-10T07:49:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:59 smithi008 ceph-mon[40743]: 5.1b scrub ok 2024-02-10T07:49:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:59 smithi008 ceph-mon[40743]: 6.0 scrub starts 2024-02-10T07:49:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:59 smithi008 ceph-mon[40743]: 6.0 scrub ok 2024-02-10T07:49:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:49:59 smithi008 ceph-mon[40743]: pgmap v838: 148 pgs: 1 creating+activating, 147 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 878 B/s rd, 0 op/s 2024-02-10T07:49:59.586 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:49:59.614 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:59.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:59.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:49:59.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:59 smithi170 ceph-mon[33394]: 5.1b scrub starts 2024-02-10T07:49:59.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:59 smithi170 ceph-mon[33394]: 5.1b scrub ok 2024-02-10T07:49:59.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:59 smithi170 ceph-mon[33394]: 6.0 scrub starts 2024-02-10T07:49:59.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:59 smithi170 ceph-mon[33394]: 6.0 scrub ok 2024-02-10T07:49:59.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:49:59 smithi170 ceph-mon[33394]: pgmap v838: 148 pgs: 1 creating+activating, 147 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 878 B/s rd, 0 op/s 2024-02-10T07:49:59.717 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:59.741 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:59.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:59.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:59.843 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:49:59.867 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:49:59.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:59.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:49:59.969 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:49:59.983 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:49:59.983 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:49:59.983 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-02-10T07:49:59.983 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:49:59.997 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:49:59.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:49:59.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:00.099 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:00.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:00.126 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:00.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:00.227 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:00.251 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:00.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:00.297 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.7 to 1 (10000) 2024-02-10T07:50:00.352 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:00.376 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:00.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:00.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:00.478 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:00.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:00.502 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:00.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[29816]: 4.1d scrub starts 2024-02-10T07:50:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[29816]: 4.1d scrub ok 2024-02-10T07:50:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[29816]: 4.5 scrub starts 2024-02-10T07:50:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[29816]: 4.5 scrub ok 2024-02-10T07:50:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[29816]: 3.11 scrub starts 2024-02-10T07:50:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[29816]: 3.11 scrub ok 2024-02-10T07:50:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/769722205' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:50:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[29816]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-02-10T07:50:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[29816]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-10T07:50:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[29816]: application not enabled on pool 'unique_pool_4' 2024-02-10T07:50:00.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[29816]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T07:50:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[40743]: 4.1d scrub starts 2024-02-10T07:50:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[40743]: 4.1d scrub ok 2024-02-10T07:50:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[40743]: 4.5 scrub starts 2024-02-10T07:50:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[40743]: 4.5 scrub ok 2024-02-10T07:50:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[40743]: 3.11 scrub starts 2024-02-10T07:50:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[40743]: 3.11 scrub ok 2024-02-10T07:50:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/769722205' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:50:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[40743]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-02-10T07:50:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[40743]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-10T07:50:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[40743]: application not enabled on pool 'unique_pool_4' 2024-02-10T07:50:00.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:00 smithi008 ceph-mon[40743]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T07:50:00.603 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:00.627 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:00.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:00 smithi170 ceph-mon[33394]: 4.1d scrub starts 2024-02-10T07:50:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:00 smithi170 ceph-mon[33394]: 4.1d scrub ok 2024-02-10T07:50:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:00 smithi170 ceph-mon[33394]: 4.5 scrub starts 2024-02-10T07:50:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:00 smithi170 ceph-mon[33394]: 4.5 scrub ok 2024-02-10T07:50:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:00 smithi170 ceph-mon[33394]: 3.11 scrub starts 2024-02-10T07:50:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:00 smithi170 ceph-mon[33394]: 3.11 scrub ok 2024-02-10T07:50:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:00 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/769722205' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:50:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:00 smithi170 ceph-mon[33394]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-02-10T07:50:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:00 smithi170 ceph-mon[33394]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-10T07:50:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:00 smithi170 ceph-mon[33394]: application not enabled on pool 'unique_pool_4' 2024-02-10T07:50:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:00 smithi170 ceph-mon[33394]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-10T07:50:00.729 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:00.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:00.754 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:00.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:00.855 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:00.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:00.879 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:00.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:00.980 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:01.008 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:01.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:01.110 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:01.138 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:01.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:01.240 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:01.269 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:01.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:01.370 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:01.395 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:01.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:01.497 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:01.520 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:01.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:01.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:01.551 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[29816]: 6.1 scrub starts 2024-02-10T07:50:01.551 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[29816]: 6.1 scrub ok 2024-02-10T07:50:01.551 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[29816]: pgmap v839: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T07:50:01.551 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[29816]: 3.15 scrub starts 2024-02-10T07:50:01.551 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[29816]: 3.15 scrub ok 2024-02-10T07:50:01.552 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/769722205' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:50:01.552 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[29816]: osdmap e395: 8 total, 8 up, 7 in 2024-02-10T07:50:01.552 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[29816]: 3.14 scrub starts 2024-02-10T07:50:01.552 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[29816]: 3.14 scrub ok 2024-02-10T07:50:01.552 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[40743]: 6.1 scrub starts 2024-02-10T07:50:01.552 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[40743]: 6.1 scrub ok 2024-02-10T07:50:01.552 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[40743]: pgmap v839: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T07:50:01.552 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[40743]: 3.15 scrub starts 2024-02-10T07:50:01.552 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[40743]: 3.15 scrub ok 2024-02-10T07:50:01.552 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/769722205' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:50:01.552 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[40743]: osdmap e395: 8 total, 8 up, 7 in 2024-02-10T07:50:01.552 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[40743]: 3.14 scrub starts 2024-02-10T07:50:01.553 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:01 smithi008 ceph-mon[40743]: 3.14 scrub ok 2024-02-10T07:50:01.578 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-10T07:50:01.622 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:01.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:01 smithi170 ceph-mon[33394]: 6.1 scrub starts 2024-02-10T07:50:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:01 smithi170 ceph-mon[33394]: 6.1 scrub ok 2024-02-10T07:50:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:01 smithi170 ceph-mon[33394]: pgmap v839: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T07:50:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:01 smithi170 ceph-mon[33394]: 3.15 scrub starts 2024-02-10T07:50:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:01 smithi170 ceph-mon[33394]: 3.15 scrub ok 2024-02-10T07:50:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:01 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/769722205' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:50:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:01 smithi170 ceph-mon[33394]: osdmap e395: 8 total, 8 up, 7 in 2024-02-10T07:50:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:01 smithi170 ceph-mon[33394]: 3.14 scrub starts 2024-02-10T07:50:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:01 smithi170 ceph-mon[33394]: 3.14 scrub ok 2024-02-10T07:50:01.646 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:01.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:01.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:01.748 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:01.773 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:01.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:01.874 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:01.898 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:01.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:01.999 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:02.024 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:02.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:02.126 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:02.152 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:02.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:02.253 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:02.278 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:02.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:02.379 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:02.404 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:02.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:02.505 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:02.529 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:02.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:02.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:02.532 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[29816]: 4.f scrub starts 2024-02-10T07:50:02.532 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[29816]: 4.f scrub ok 2024-02-10T07:50:02.532 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[29816]: 6.c scrub starts 2024-02-10T07:50:02.533 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[29816]: 6.c scrub ok 2024-02-10T07:50:02.533 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[29816]: 6.17 scrub starts 2024-02-10T07:50:02.533 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[29816]: 6.17 scrub ok 2024-02-10T07:50:02.533 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/699705363' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:50:02.533 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:50:02.533 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[40743]: 4.f scrub starts 2024-02-10T07:50:02.533 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[40743]: 4.f scrub ok 2024-02-10T07:50:02.533 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[40743]: 6.c scrub starts 2024-02-10T07:50:02.533 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[40743]: 6.c scrub ok 2024-02-10T07:50:02.533 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[40743]: 6.17 scrub starts 2024-02-10T07:50:02.533 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[40743]: 6.17 scrub ok 2024-02-10T07:50:02.533 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/699705363' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:50:02.534 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:02 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:50:02.558 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-10T07:50:02.558 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:50:02.622 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:50:02.624 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:02.627 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:02.631 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:02.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:02 smithi170 ceph-mon[33394]: 4.f scrub starts 2024-02-10T07:50:02.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:02 smithi170 ceph-mon[33394]: 4.f scrub ok 2024-02-10T07:50:02.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:02 smithi170 ceph-mon[33394]: 6.c scrub starts 2024-02-10T07:50:02.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:02 smithi170 ceph-mon[33394]: 6.c scrub ok 2024-02-10T07:50:02.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:02 smithi170 ceph-mon[33394]: 6.17 scrub starts 2024-02-10T07:50:02.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:02 smithi170 ceph-mon[33394]: 6.17 scrub ok 2024-02-10T07:50:02.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:02 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/699705363' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:50:02.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:02 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T07:50:02.655 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:02.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:02.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:02.757 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:02.781 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:02.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:02.882 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:02.907 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:02.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:03.001 INFO:teuthology.orchestra.run.smithi008.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-02-10T07:50:03.008 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:03.033 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:03.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:03.135 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:03.158 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:03.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:03.260 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:03.285 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:03.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:03.386 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:03.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:03.410 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:03.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:03.511 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:03.535 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:03.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:03.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:03 smithi008 ceph-mon[29816]: pgmap v841: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 654 B/s rd, 0 op/s 2024-02-10T07:50:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:03 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:50:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:03 smithi008 ceph-mon[29816]: osdmap e396: 8 total, 8 up, 7 in 2024-02-10T07:50:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:50:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:03 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:50:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:03 smithi008 ceph-mon[40743]: pgmap v841: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 654 B/s rd, 0 op/s 2024-02-10T07:50:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:03 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:50:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:03 smithi008 ceph-mon[40743]: osdmap e396: 8 total, 8 up, 7 in 2024-02-10T07:50:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:50:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:03 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:50:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:03 smithi170 ceph-mon[33394]: pgmap v841: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 654 B/s rd, 0 op/s 2024-02-10T07:50:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:03 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-10T07:50:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:03 smithi170 ceph-mon[33394]: osdmap e396: 8 total, 8 up, 7 in 2024-02-10T07:50:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:50:03.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:03 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:50:03.637 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:03.661 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:03.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:03.763 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:03.787 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:03.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:03.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:03.889 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:03.914 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:03.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:04.015 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:04.038 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:04.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:04.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:04.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:50:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:50:04.140 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:04.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:04.166 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:04.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:04.267 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:04.292 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:04.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:04.395 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:04.420 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:04.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:04.521 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:04.545 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:04.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:04.555 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:04 smithi008 ceph-mon[29816]: 5.12 deep-scrub starts 2024-02-10T07:50:04.555 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:04 smithi008 ceph-mon[29816]: 5.12 deep-scrub ok 2024-02-10T07:50:04.555 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:04 smithi008 ceph-mon[29816]: 5.e deep-scrub starts 2024-02-10T07:50:04.555 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:04 smithi008 ceph-mon[29816]: 5.e deep-scrub ok 2024-02-10T07:50:04.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:04 smithi008 ceph-mon[29816]: 3.6 scrub starts 2024-02-10T07:50:04.556 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:04 smithi008 ceph-mon[29816]: 3.6 scrub ok 2024-02-10T07:50:04.556 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:04 smithi008 ceph-mon[40743]: 5.12 deep-scrub starts 2024-02-10T07:50:04.556 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:04 smithi008 ceph-mon[40743]: 5.12 deep-scrub ok 2024-02-10T07:50:04.556 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:04 smithi008 ceph-mon[40743]: 5.e deep-scrub starts 2024-02-10T07:50:04.556 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:04 smithi008 ceph-mon[40743]: 5.e deep-scrub ok 2024-02-10T07:50:04.556 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:04 smithi008 ceph-mon[40743]: 3.6 scrub starts 2024-02-10T07:50:04.556 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:04 smithi008 ceph-mon[40743]: 3.6 scrub ok 2024-02-10T07:50:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:04 smithi170 ceph-mon[33394]: 5.12 deep-scrub starts 2024-02-10T07:50:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:04 smithi170 ceph-mon[33394]: 5.12 deep-scrub ok 2024-02-10T07:50:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:04 smithi170 ceph-mon[33394]: 5.e deep-scrub starts 2024-02-10T07:50:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:04 smithi170 ceph-mon[33394]: 5.e deep-scrub ok 2024-02-10T07:50:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:04 smithi170 ceph-mon[33394]: 3.6 scrub starts 2024-02-10T07:50:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:04 smithi170 ceph-mon[33394]: 3.6 scrub ok 2024-02-10T07:50:04.647 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:04.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:04.673 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:04.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:04.774 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:04.799 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:04.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:04.901 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:04.929 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:04.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:05.031 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:05.056 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:05.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:05.157 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:05.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:05.184 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:05.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:05.285 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:05.310 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:05.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:05.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:05.411 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:05.435 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:05.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:05.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:05.537 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:05.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:05.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[29816]: 3.a deep-scrub starts 2024-02-10T07:50:05.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[29816]: 3.a deep-scrub ok 2024-02-10T07:50:05.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[29816]: pgmap v843: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:50:05.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[29816]: 2.0 scrub starts 2024-02-10T07:50:05.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[29816]: 2.0 scrub ok 2024-02-10T07:50:05.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[29816]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:50:05.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:50:05.562 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[29816]: 3.1a scrub starts 2024-02-10T07:50:05.563 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[29816]: 3.1a scrub ok 2024-02-10T07:50:05.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:05.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[40743]: 3.a deep-scrub starts 2024-02-10T07:50:05.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[40743]: 3.a deep-scrub ok 2024-02-10T07:50:05.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[40743]: pgmap v843: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:50:05.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[40743]: 2.0 scrub starts 2024-02-10T07:50:05.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[40743]: 2.0 scrub ok 2024-02-10T07:50:05.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[40743]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:50:05.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:50:05.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[40743]: 3.1a scrub starts 2024-02-10T07:50:05.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:05 smithi008 ceph-mon[40743]: 3.1a scrub ok 2024-02-10T07:50:05.565 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:05.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:05.582 DEBUG:teuthology.orchestra.run.smithi008:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-02-10T07:50:05.585 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-10T07:50:05.585 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-10T07:50:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:05 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:05 smithi170 ceph-mon[33394]: 3.a deep-scrub starts 2024-02-10T07:50:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:05 smithi170 ceph-mon[33394]: 3.a deep-scrub ok 2024-02-10T07:50:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:05 smithi170 ceph-mon[33394]: pgmap v843: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:50:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:05 smithi170 ceph-mon[33394]: 2.0 scrub starts 2024-02-10T07:50:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:05 smithi170 ceph-mon[33394]: 2.0 scrub ok 2024-02-10T07:50:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:05 smithi170 ceph-mon[33394]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T07:50:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:05 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:50:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:05 smithi170 ceph-mon[33394]: 3.1a scrub starts 2024-02-10T07:50:05.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:05 smithi170 ceph-mon[33394]: 3.1a scrub ok 2024-02-10T07:50:05.643 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:hints = 1 2024-02-10T07:50:05.643 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-10T07:50:05.643 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Object prefix: benchmark_data_smithi008_186804 2024-02-10T07:50:05.644 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:50:05.644 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 0 0 0 0 0 0 - 0 2024-02-10T07:50:05.667 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:05.682 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:05.684 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:05.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:05.730 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:05.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:05.734 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:50:05.734 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:50:05.734 INFO:teuthology.orchestra.run.smithi008.stdout: "complaint_time": 30, 2024-02-10T07:50:05.735 INFO:teuthology.orchestra.run.smithi008.stdout: "num_blocked_ops": 0 2024-02-10T07:50:05.735 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:50:05.832 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:05.868 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:05.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:05.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:05.971 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:06.008 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:06.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:06.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:06.109 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:06.149 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:06.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:06.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:06.251 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:06.288 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:06.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:06.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:06.390 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:06.428 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:06.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:06.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:06.530 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:06.568 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:06.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:06.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:06.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:06 smithi008 ceph-mon[29816]: 5.16 scrub starts 2024-02-10T07:50:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:06 smithi008 ceph-mon[29816]: 5.16 scrub ok 2024-02-10T07:50:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:06 smithi008 ceph-mon[29816]: 6.18 scrub starts 2024-02-10T07:50:06.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:06 smithi008 ceph-mon[29816]: 6.18 scrub ok 2024-02-10T07:50:06.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:06 smithi008 ceph-mon[40743]: 5.16 scrub starts 2024-02-10T07:50:06.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:06 smithi008 ceph-mon[40743]: 5.16 scrub ok 2024-02-10T07:50:06.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:06 smithi008 ceph-mon[40743]: 6.18 scrub starts 2024-02-10T07:50:06.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:06 smithi008 ceph-mon[40743]: 6.18 scrub ok 2024-02-10T07:50:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:06 smithi170 ceph-mon[33394]: 5.16 scrub starts 2024-02-10T07:50:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:06 smithi170 ceph-mon[33394]: 5.16 scrub ok 2024-02-10T07:50:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:06 smithi170 ceph-mon[33394]: 6.18 scrub starts 2024-02-10T07:50:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:06 smithi170 ceph-mon[33394]: 6.18 scrub ok 2024-02-10T07:50:06.644 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 1 16 3646 3630 226.851 226.875 0.00688159 0.00439358 2024-02-10T07:50:06.669 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:06.710 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:06.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:06.811 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:06.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:06.854 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:06.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:06.955 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:06.996 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:06.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:07.098 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:07.135 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:07.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:07.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:07.237 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:07.275 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:07.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:07.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[29816]: 6.a scrub starts 2024-02-10T07:50:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[29816]: 6.a scrub ok 2024-02-10T07:50:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:50:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:50:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[29816]: pgmap v844: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:50:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[29816]: osdmap e397: 8 total, 8 up, 7 in 2024-02-10T07:50:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[29816]: 5.18 scrub starts 2024-02-10T07:50:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[29816]: 5.18 scrub ok 2024-02-10T07:50:07.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[40743]: 6.a scrub starts 2024-02-10T07:50:07.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[40743]: 6.a scrub ok 2024-02-10T07:50:07.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:50:07.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:50:07.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[40743]: pgmap v844: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:50:07.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[40743]: osdmap e397: 8 total, 8 up, 7 in 2024-02-10T07:50:07.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[40743]: 5.18 scrub starts 2024-02-10T07:50:07.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:07 smithi008 ceph-mon[40743]: 5.18 scrub ok 2024-02-10T07:50:07.378 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:07.406 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:07.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:07.508 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:07.545 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:07.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:07 smithi170 ceph-mon[33394]: 6.a scrub starts 2024-02-10T07:50:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:07 smithi170 ceph-mon[33394]: 6.a scrub ok 2024-02-10T07:50:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:07 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:50:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:07 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:50:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:07 smithi170 ceph-mon[33394]: pgmap v844: 148 pgs: 148 active+clean; 582 KiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:50:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:07 smithi170 ceph-mon[33394]: osdmap e397: 8 total, 8 up, 7 in 2024-02-10T07:50:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:07 smithi170 ceph-mon[33394]: 5.18 scrub starts 2024-02-10T07:50:07.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:07 smithi170 ceph-mon[33394]: 5.18 scrub ok 2024-02-10T07:50:07.644 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 2 16 6936 6920 216.224 205.625 0.00749467 0.00461435 2024-02-10T07:50:07.647 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:07.687 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:07.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:07.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:07.790 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:07.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:07.829 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:07.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:07.930 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:07.967 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:07.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:08.069 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:08.108 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:08.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:08.210 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:08.246 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:08.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:08.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:08.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:08 smithi008 ceph-mon[29816]: 3.d scrub starts 2024-02-10T07:50:08.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:08 smithi008 ceph-mon[29816]: 3.d scrub ok 2024-02-10T07:50:08.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:08 smithi008 ceph-mon[29816]: 3.12 deep-scrub starts 2024-02-10T07:50:08.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:08 smithi008 ceph-mon[29816]: 3.12 deep-scrub ok 2024-02-10T07:50:08.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:08 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2226489124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:50:08.333 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:08 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:50:08.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:08 smithi008 ceph-mon[40743]: 3.d scrub starts 2024-02-10T07:50:08.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:08 smithi008 ceph-mon[40743]: 3.d scrub ok 2024-02-10T07:50:08.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:08 smithi008 ceph-mon[40743]: 3.12 deep-scrub starts 2024-02-10T07:50:08.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:08 smithi008 ceph-mon[40743]: 3.12 deep-scrub ok 2024-02-10T07:50:08.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:08 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2226489124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:50:08.334 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:08 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:50:08.337 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:50:08.348 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:08.387 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:08.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:08.490 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:08.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:08.530 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:08.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:08.631 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:08.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:08 smithi170 ceph-mon[33394]: 3.d scrub starts 2024-02-10T07:50:08.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:08 smithi170 ceph-mon[33394]: 3.d scrub ok 2024-02-10T07:50:08.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:08 smithi170 ceph-mon[33394]: 3.12 deep-scrub starts 2024-02-10T07:50:08.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:08 smithi170 ceph-mon[33394]: 3.12 deep-scrub ok 2024-02-10T07:50:08.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:08 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2226489124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:50:08.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:08 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:50:08.644 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 3 16 10074 10058 209.518 196.125 0.00346476 0.00476478 2024-02-10T07:50:08.669 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:08.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:08.771 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:08.809 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:08.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:08.869 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:50:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:50:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:50:08.910 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:08.947 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:08.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:08.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:09.049 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:09.089 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:09.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:09.191 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:09.227 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:09.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:09.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:09.329 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:09.367 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:09.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:09.469 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:09.507 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:09.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:09.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:09 smithi008 ceph-mon[29816]: 6.4 scrub starts 2024-02-10T07:50:09.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:09 smithi008 ceph-mon[29816]: 6.4 scrub ok 2024-02-10T07:50:09.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:09 smithi008 ceph-mon[29816]: pgmap v846: 148 pgs: 148 active+clean; 10 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 655 B/s rd, 1.3 MiB/s wr, 20 op/s 2024-02-10T07:50:09.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:09 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:50:09.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:09 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:50:09.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:09 smithi008 ceph-mon[29816]: osdmap e398: 8 total, 8 up, 7 in 2024-02-10T07:50:09.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:09 smithi008 ceph-mon[40743]: 6.4 scrub starts 2024-02-10T07:50:09.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:09 smithi008 ceph-mon[40743]: 6.4 scrub ok 2024-02-10T07:50:09.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:09 smithi008 ceph-mon[40743]: pgmap v846: 148 pgs: 148 active+clean; 10 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 655 B/s rd, 1.3 MiB/s wr, 20 op/s 2024-02-10T07:50:09.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:09 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:50:09.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:09 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:50:09.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:09 smithi008 ceph-mon[40743]: osdmap e398: 8 total, 8 up, 7 in 2024-02-10T07:50:09.609 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:09 smithi170 ceph-mon[33394]: 6.4 scrub starts 2024-02-10T07:50:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:09 smithi170 ceph-mon[33394]: 6.4 scrub ok 2024-02-10T07:50:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:09 smithi170 ceph-mon[33394]: pgmap v846: 148 pgs: 148 active+clean; 10 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 655 B/s rd, 1.3 MiB/s wr, 20 op/s 2024-02-10T07:50:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:09 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:50:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:09 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:50:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:09 smithi170 ceph-mon[33394]: osdmap e398: 8 total, 8 up, 7 in 2024-02-10T07:50:09.644 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 4 16 13056 13040 203.731 186.375 0.00419643 0.00490177 2024-02-10T07:50:09.646 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:09.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:09.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:09.748 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:09.785 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:09.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:09.813 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-10T07:50:09.885 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:09.933 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:09.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:10.035 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:10.089 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:10.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:10.145 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:50:10.185 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:50:10.189 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:10.224 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:50:10.234 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:10.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:10.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:10.278 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:50:10.315 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:50:10.336 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:10.357 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:50:10.384 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:10.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:10.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:10.397 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:50:10.438 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:50:10.439 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:50:10.439 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:50:10.439 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:50:10.440 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:50:10.440 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:50:10.440 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:50:10.440 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:50:10.440 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:50:10.485 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:10.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:10.516 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:10.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:10.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:10 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2923654123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:50:10.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:10 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2923654123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:50:10.617 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:10 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2923654123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:50:10.644 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 5 16 15617 15601 194.995 160.062 0.00291825 0.00512099 2024-02-10T07:50:10.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:10.654 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:10.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:10.755 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:10.796 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:10.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:10.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:10.897 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:10.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:10.930 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:10.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:11.031 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:11.072 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:11.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:11.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:11.174 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:11.213 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:11.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:11.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:11.315 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:11.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:11.356 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:11.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:11.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:11.496 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:11.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:11.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:11.598 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:11 smithi170 ceph-mon[33394]: pgmap v848: 148 pgs: 148 active+clean; 600 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-10T07:50:11.634 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:11.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:11.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:11.644 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 6 16 18188 18172 189.275 160.688 0.00369096 0.00527617 2024-02-10T07:50:11.736 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:11.774 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:11.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:11.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:11 smithi008 ceph-mon[29816]: pgmap v848: 148 pgs: 148 active+clean; 600 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-10T07:50:11.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:11 smithi008 ceph-mon[40743]: pgmap v848: 148 pgs: 148 active+clean; 600 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-10T07:50:11.877 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:11.916 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:11.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:11.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:12.018 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:12.055 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:12.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:12.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:12.156 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:12.195 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:12.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:12.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:12.297 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:12.336 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:12.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:12.438 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:12.475 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:12.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:12.577 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:12.622 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:12.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:12.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:12.644 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 7 16 20873 20857 186.207 167.812 0.00371926 0.00536471 2024-02-10T07:50:12.724 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:12.762 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:12.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:12.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:12.864 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:12.899 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:12.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:12.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:13.001 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:13.037 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:13.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:13.138 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:13.175 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:13.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:13.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:13.277 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:13.316 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:13.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:13.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:13 smithi008 ceph-mon[29816]: pgmap v849: 148 pgs: 148 active+clean; 659 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-02-10T07:50:13.331 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:13 smithi008 ceph-mon[40743]: pgmap v849: 148 pgs: 148 active+clean; 659 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-02-10T07:50:13.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:13 smithi170 ceph-mon[33394]: pgmap v849: 148 pgs: 148 active+clean; 659 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-02-10T07:50:13.418 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:13.455 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:13.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:13.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:13.557 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:13.593 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:13.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:13.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:13.644 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 8 16 23288 23272 181.796 150.938 0.00700434 0.00549491 2024-02-10T07:50:13.696 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:13.751 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:13.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:13.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:13.853 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:13.894 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:13.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:13.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:13.996 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:14.032 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:14.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:14.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:14.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:50:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:50:14.134 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:14.172 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:14.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:14.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:14.274 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:14.311 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:14.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:14.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:14.398 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:14.416 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:14.474 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:14.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:14.576 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:14.615 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:14.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:14.637 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:50:14.644 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 9 16 25695 25679 178.31 150.438 0.00594374 0.00560294 2024-02-10T07:50:14.717 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:14.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:14.761 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:14.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:14.862 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:14.899 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:14.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:14.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:15.001 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:15.028 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:15.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:15.133 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:15.170 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:15.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:15.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:15.272 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:15.277 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:15 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:15.277 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:15 smithi008 ceph-mon[29816]: pgmap v850: 148 pgs: 148 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.36k op/s 2024-02-10T07:50:15.277 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:15 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:15.277 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:15 smithi008 ceph-mon[40743]: pgmap v850: 148 pgs: 148 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.36k op/s 2024-02-10T07:50:15.305 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:15.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:15.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:15.407 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:15.447 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:15.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:15.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:15.548 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:15.584 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:15.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:15 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:15 smithi170 ceph-mon[33394]: pgmap v850: 148 pgs: 148 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 148 MiB/s wr, 2.36k op/s 2024-02-10T07:50:15.647 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 10 16 27913 27897 174.341 138.625 0.00389223 0.00572977 2024-02-10T07:50:15.687 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:15.724 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:15.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:15.805 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:50:15.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:15.858 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:15.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:15.960 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:16.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:16.027 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:16.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:16.128 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:16.161 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:16.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:16.263 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:16.304 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:16.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:16.405 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:16.441 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:16.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:16.477 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:50:16.483 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:50:16.542 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:16.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:16.578 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:16.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:16.644 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 11 16 29960 29944 170.121 127.938 0.00673224 0.00587205 2024-02-10T07:50:16.679 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:16.714 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:16.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:16.816 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:16.853 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:16.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:16.954 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:16.991 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:16.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:16.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:17.093 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:17.129 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:17.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:17.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:17.231 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:17.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:17.269 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:17.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:17.370 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:17.409 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:17.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:17.510 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:17.540 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:17.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:17.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:17.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:17 smithi008 ceph-mon[29816]: pgmap v851: 148 pgs: 148 active+clean; 1.4 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 146 MiB/s wr, 2.33k op/s 2024-02-10T07:50:17.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:17 smithi008 ceph-mon[29816]: from='client.15963 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:50:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:17 smithi008 ceph-mon[40743]: pgmap v851: 148 pgs: 148 active+clean; 1.4 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 146 MiB/s wr, 2.33k op/s 2024-02-10T07:50:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:17 smithi008 ceph-mon[40743]: from='client.15963 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:50:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:17 smithi170 ceph-mon[33394]: pgmap v851: 148 pgs: 148 active+clean; 1.4 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 146 MiB/s wr, 2.33k op/s 2024-02-10T07:50:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:17 smithi170 ceph-mon[33394]: from='client.15963 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:50:17.642 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:17.644 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 12 16 31943 31927 166.272 123.938 0.00749199 0.00600885 2024-02-10T07:50:17.678 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:17.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:17.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:17.780 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:17.817 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:17.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:17.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:17.919 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:17.955 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:17.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:17.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:18.057 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:18.094 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:18.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:18.196 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:18.233 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:18.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:18.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:18.336 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:18.370 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:18.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:18.471 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:18.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:18.510 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:18.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:18.611 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:18.645 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 13 16 33807 33791 162.436 116.5 0.0129598 0.00615066 2024-02-10T07:50:18.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:18.648 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:18.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:18.749 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:18.787 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:18.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:18.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:18.889 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:18.920 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:18.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:19.021 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:19.057 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:19.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:19.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:19.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:50:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:50:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:50:19.161 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:19.197 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:19.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:19.299 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:19.338 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:19.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:19.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:19.440 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:19.479 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:19.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:19.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:19.519 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:19 smithi008 ceph-mon[29816]: pgmap v852: 148 pgs: 148 active+clean; 1.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-02-10T07:50:19.519 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:19 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:50:19.519 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:19 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:50:19.519 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:19 smithi008 ceph-mon[40743]: pgmap v852: 148 pgs: 148 active+clean; 1.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-02-10T07:50:19.519 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:19 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:50:19.519 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:19 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:50:19.576 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":851,"stamp":"2024-02-10T07:50:16.101265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1504123228,"num_objects":23140,"num_object_clones":0,"num_object_copies":69420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23140,"num_whiteouts":0,"num_read":2442,"num_read_kb":2288,"num_write":23843,"num_write_kb":1470749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":1778520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23949,"ondisk_log_size":23949,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7993452,"kb_used_data":4088180,"kb_used_omap":17,"kb_used_meta":3905198,"kb_avail":648222612,"statfs":{"total":671965249536,"available":663779954688,"internally_reserved":0,"allocated":4186296320,"data_stored":4183932693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18039,"internal_metadata":3998923145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1493172224,"num_objects":22784,"num_object_clones":0,"num_object_copies":68352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22784,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":22784,"num_write_kb":1458176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.783144"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":732,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336448+0000","last_change":"2024-02-10T07:49:56.315126+0000","last_active":"2024-02-10T07:50:08.336448+0000","last_peered":"2024-02-10T07:50:08.336448+0000","last_clean":"2024-02-10T07:50:08.336448+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:50:08.336448+0000","last_undegraded":"2024-02-10T07:50:08.336448+0000","last_fullsized":"2024-02-10T07:50:08.336448+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.315029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.315029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:19.186486+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":742,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335860+0000","last_change":"2024-02-10T07:49:08.138553+0000","last_active":"2024-02-10T07:50:08.335860+0000","last_peered":"2024-02-10T07:50:08.335860+0000","last_clean":"2024-02-10T07:50:08.335860+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:50:08.335860+0000","last_undegraded":"2024-02-10T07:50:08.335860+0000","last_fullsized":"2024-02-10T07:50:08.335860+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:08.138442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.138442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:35.396509+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":761,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.331688+0000","last_change":"2024-02-10T07:48:17.198789+0000","last_active":"2024-02-10T07:50:08.331688+0000","last_peered":"2024-02-10T07:50:08.331688+0000","last_clean":"2024-02-10T07:50:08.331688+0000","last_became_active":"2024-02-10T07:46:10.718945+0000","last_became_peered":"2024-02-10T07:46:10.718945+0000","last_unstale":"2024-02-10T07:50:08.331688+0000","last_undegraded":"2024-02-10T07:50:08.331688+0000","last_fullsized":"2024-02-10T07:50:08.331688+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:17.198701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:17.198701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"287'19","reported_seq":792,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335482+0000","last_change":"2024-02-10T07:46:37.008978+0000","last_active":"2024-02-10T07:50:08.335482+0000","last_peered":"2024-02-10T07:50:08.335482+0000","last_clean":"2024-02-10T07:50:08.335482+0000","last_became_active":"2024-02-10T07:46:10.717475+0000","last_became_peered":"2024-02-10T07:46:10.717475+0000","last_unstale":"2024-02-10T07:50:08.335482+0000","last_undegraded":"2024-02-10T07:50:08.335482+0000","last_fullsized":"2024-02-10T07:50:08.335482+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'19","last_scrub_stamp":"2024-02-10T07:46:37.008841+0000","last_deep_scrub":"287'19","last_deep_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.008841+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":63,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.334975+0000","last_change":"2024-02-10T07:49:15.079645+0000","last_active":"2024-02-10T07:50:08.334975+0000","last_peered":"2024-02-10T07:50:08.334975+0000","last_clean":"2024-02-10T07:50:08.334975+0000","last_became_active":"2024-02-10T07:49:06.236235+0000","last_became_peered":"2024-02-10T07:49:06.236235+0000","last_unstale":"2024-02-10T07:50:08.334975+0000","last_undegraded":"2024-02-10T07:50:08.334975+0000","last_fullsized":"2024-02-10T07:50:08.334975+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:15.079428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:15.079428+0000","last_clean_scrub_stamp":"2024-02-10T07:49:15.079428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:19.677052+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":735,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335387+0000","last_change":"2024-02-10T07:50:06.335390+0000","last_active":"2024-02-10T07:50:08.335387+0000","last_peered":"2024-02-10T07:50:08.335387+0000","last_clean":"2024-02-10T07:50:08.335387+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:50:08.335387+0000","last_undegraded":"2024-02-10T07:50:08.335387+0000","last_fullsized":"2024-02-10T07:50:08.335387+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:06.335262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:50:06.335262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:25.829837+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":244,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330241+0000","last_change":"2024-02-10T07:49:47.812212+0000","last_active":"2024-02-10T07:50:08.330241+0000","last_peered":"2024-02-10T07:50:08.330241+0000","last_clean":"2024-02-10T07:50:08.330241+0000","last_became_active":"2024-02-10T07:49:06.262984+0000","last_became_peered":"2024-02-10T07:49:06.262984+0000","last_unstale":"2024-02-10T07:50:08.330241+0000","last_undegraded":"2024-02-10T07:50:08.330241+0000","last_fullsized":"2024-02-10T07:50:08.330241+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:47.812065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.812065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:07.921649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"250'27","reported_seq":699,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330255+0000","last_change":"2024-02-10T07:49:06.320471+0000","last_active":"2024-02-10T07:50:08.330255+0000","last_peered":"2024-02-10T07:50:08.330255+0000","last_clean":"2024-02-10T07:50:08.330255+0000","last_became_active":"2024-02-10T07:49:06.266213+0000","last_became_peered":"2024-02-10T07:49:06.266213+0000","last_unstale":"2024-02-10T07:50:08.330255+0000","last_undegraded":"2024-02-10T07:50:08.330255+0000","last_fullsized":"2024-02-10T07:50:08.330255+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:48:13.230908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.230908+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":731,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330534+0000","last_change":"2024-02-10T07:50:05.620136+0000","last_active":"2024-02-10T07:50:08.330534+0000","last_peered":"2024-02-10T07:50:08.330534+0000","last_clean":"2024-02-10T07:50:08.330534+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:50:08.330534+0000","last_undegraded":"2024-02-10T07:50:08.330534+0000","last_fullsized":"2024-02-10T07:50:08.330534+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:05.619958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:05.619958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:09.870665+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":704,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.332775+0000","last_change":"2024-02-10T07:49:57.669438+0000","last_active":"2024-02-10T07:50:08.332775+0000","last_peered":"2024-02-10T07:50:08.332775+0000","last_clean":"2024-02-10T07:50:08.332775+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:50:08.332775+0000","last_undegraded":"2024-02-10T07:50:08.332775+0000","last_fullsized":"2024-02-10T07:50:08.332775+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:57.669314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.669314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:17.355126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":272,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336262+0000","last_change":"2024-02-10T07:49:13.218634+0000","last_active":"2024-02-10T07:50:08.336262+0000","last_peered":"2024-02-10T07:50:08.336262+0000","last_clean":"2024-02-10T07:50:08.336262+0000","last_became_active":"2024-02-10T07:48:02.332960+0000","last_became_peered":"2024-02-10T07:48:02.332960+0000","last_unstale":"2024-02-10T07:50:08.336262+0000","last_undegraded":"2024-02-10T07:50:08.336262+0000","last_fullsized":"2024-02-10T07:50:08.336262+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.218549+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:42.382557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"250'15","reported_seq":757,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336670+0000","last_change":"2024-02-10T07:47:49.553645+0000","last_active":"2024-02-10T07:50:08.336670+0000","last_peered":"2024-02-10T07:50:08.336670+0000","last_clean":"2024-02-10T07:50:08.336670+0000","last_became_active":"2024-02-10T07:47:47.965930+0000","last_became_peered":"2024-02-10T07:47:47.965930+0000","last_unstale":"2024-02-10T07:50:08.336670+0000","last_undegraded":"2024-02-10T07:50:08.336670+0000","last_fullsized":"2024-02-10T07:50:08.336670+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:49.553517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.553517+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":643,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336043+0000","last_change":"2024-02-10T07:49:56.718482+0000","last_active":"2024-02-10T07:50:08.336043+0000","last_peered":"2024-02-10T07:50:08.336043+0000","last_clean":"2024-02-10T07:50:08.336043+0000","last_became_active":"2024-02-10T07:49:06.266689+0000","last_became_peered":"2024-02-10T07:49:06.266689+0000","last_unstale":"2024-02-10T07:50:08.336043+0000","last_undegraded":"2024-02-10T07:50:08.336043+0000","last_fullsized":"2024-02-10T07:50:08.336043+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.718382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.718382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:12.254870+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":735,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329081+0000","last_change":"2024-02-10T07:48:18.069773+0000","last_active":"2024-02-10T07:50:08.329081+0000","last_peered":"2024-02-10T07:50:08.329081+0000","last_clean":"2024-02-10T07:50:08.329081+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:50:08.329081+0000","last_undegraded":"2024-02-10T07:50:08.329081+0000","last_fullsized":"2024-02-10T07:50:08.329081+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:18.069641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:18.069641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":260,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335842+0000","last_change":"2024-02-10T07:49:06.115547+0000","last_active":"2024-02-10T07:50:08.335842+0000","last_peered":"2024-02-10T07:50:08.335842+0000","last_clean":"2024-02-10T07:50:08.335842+0000","last_became_active":"2024-02-10T07:48:02.329859+0000","last_became_peered":"2024-02-10T07:48:02.329859+0000","last_unstale":"2024-02-10T07:50:08.335842+0000","last_undegraded":"2024-02-10T07:50:08.335842+0000","last_fullsized":"2024-02-10T07:50:08.335842+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.115466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.473686+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.115466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:32.640728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"250'7","reported_seq":745,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.337394+0000","last_change":"2024-02-10T07:48:11.611397+0000","last_active":"2024-02-10T07:50:08.337394+0000","last_peered":"2024-02-10T07:50:08.337394+0000","last_clean":"2024-02-10T07:50:08.337394+0000","last_became_active":"2024-02-10T07:47:47.966093+0000","last_became_peered":"2024-02-10T07:47:47.966093+0000","last_unstale":"2024-02-10T07:50:08.337394+0000","last_undegraded":"2024-02-10T07:50:08.337394+0000","last_fullsized":"2024-02-10T07:50:08.337394+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7","last_scrub_stamp":"2024-02-10T07:48:11.608499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.608499+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":734,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336465+0000","last_change":"2024-02-10T07:48:13.621135+0000","last_active":"2024-02-10T07:50:08.336465+0000","last_peered":"2024-02-10T07:50:08.336465+0000","last_clean":"2024-02-10T07:50:08.336465+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:50:08.336465+0000","last_undegraded":"2024-02-10T07:50:08.336465+0000","last_fullsized":"2024-02-10T07:50:08.336465+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:13.620938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.620938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":741,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336634+0000","last_change":"2024-02-10T07:49:51.810158+0000","last_active":"2024-02-10T07:50:08.336634+0000","last_peered":"2024-02-10T07:50:08.336634+0000","last_clean":"2024-02-10T07:50:08.336634+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:50:08.336634+0000","last_undegraded":"2024-02-10T07:50:08.336634+0000","last_fullsized":"2024-02-10T07:50:08.336634+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:51.810062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:51.810062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:59.448781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":743,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330686+0000","last_change":"2024-02-10T07:49:16.313655+0000","last_active":"2024-02-10T07:50:08.330686+0000","last_peered":"2024-02-10T07:50:08.330686+0000","last_clean":"2024-02-10T07:50:08.330686+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:50:08.330686+0000","last_undegraded":"2024-02-10T07:50:08.330686+0000","last_fullsized":"2024-02-10T07:50:08.330686+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:16.313573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.313573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:21.959857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"250'34","reported_seq":123,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.333025+0000","last_change":"2024-02-10T07:49:06.320822+0000","last_active":"2024-02-10T07:50:08.333025+0000","last_peered":"2024-02-10T07:50:08.333025+0000","last_clean":"2024-02-10T07:50:08.333025+0000","last_became_active":"2024-02-10T07:49:06.268427+0000","last_became_peered":"2024-02-10T07:49:06.268427+0000","last_unstale":"2024-02-10T07:50:08.333025+0000","last_undegraded":"2024-02-10T07:50:08.333025+0000","last_fullsized":"2024-02-10T07:50:08.333025+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:49:00.551605+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.551605+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:12.633041+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":429,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330430+0000","last_change":"2024-02-10T07:49:51.990345+0000","last_active":"2024-02-10T07:50:08.330430+0000","last_peered":"2024-02-10T07:50:08.330430+0000","last_clean":"2024-02-10T07:50:08.330430+0000","last_became_active":"2024-02-10T07:49:06.263790+0000","last_became_peered":"2024-02-10T07:49:06.263790+0000","last_unstale":"2024-02-10T07:50:08.330430+0000","last_undegraded":"2024-02-10T07:50:08.330430+0000","last_fullsized":"2024-02-10T07:50:08.330430+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:51.990237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:51.990237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:13.825668+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":746,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336162+0000","last_change":"2024-02-10T07:49:06.266778+0000","last_active":"2024-02-10T07:50:08.336162+0000","last_peered":"2024-02-10T07:50:08.336162+0000","last_clean":"2024-02-10T07:50:08.336162+0000","last_became_active":"2024-02-10T07:49:06.266390+0000","last_became_peered":"2024-02-10T07:49:06.266390+0000","last_unstale":"2024-02-10T07:50:08.336162+0000","last_undegraded":"2024-02-10T07:50:08.336162+0000","last_fullsized":"2024-02-10T07:50:08.336162+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.680331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.680331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":749,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336768+0000","last_change":"2024-02-10T07:50:04.396854+0000","last_active":"2024-02-10T07:50:08.336768+0000","last_peered":"2024-02-10T07:50:08.336768+0000","last_clean":"2024-02-10T07:50:08.336768+0000","last_became_active":"2024-02-10T07:46:10.717041+0000","last_became_peered":"2024-02-10T07:46:10.717041+0000","last_unstale":"2024-02-10T07:50:08.336768+0000","last_undegraded":"2024-02-10T07:50:08.336768+0000","last_fullsized":"2024-02-10T07:50:08.336768+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:04.396771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.396771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:32.209449+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"250'18","reported_seq":702,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336509+0000","last_change":"2024-02-10T07:49:58.618564+0000","last_active":"2024-02-10T07:50:08.336509+0000","last_peered":"2024-02-10T07:50:08.336509+0000","last_clean":"2024-02-10T07:50:08.336509+0000","last_became_active":"2024-02-10T07:49:06.263261+0000","last_became_peered":"2024-02-10T07:49:06.263261+0000","last_unstale":"2024-02-10T07:50:08.336509+0000","last_undegraded":"2024-02-10T07:50:08.336509+0000","last_fullsized":"2024-02-10T07:50:08.336509+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'18","last_scrub_stamp":"2024-02-10T07:49:58.618464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:58.618464+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:59.071088+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":757,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329955+0000","last_change":"2024-02-10T07:49:06.317985+0000","last_active":"2024-02-10T07:50:08.329955+0000","last_peered":"2024-02-10T07:50:08.329955+0000","last_clean":"2024-02-10T07:50:08.329955+0000","last_became_active":"2024-02-10T07:49:06.267590+0000","last_became_peered":"2024-02-10T07:49:06.267590+0000","last_unstale":"2024-02-10T07:50:08.329955+0000","last_undegraded":"2024-02-10T07:50:08.329955+0000","last_fullsized":"2024-02-10T07:50:08.329955+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:47:34.074746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:34.074746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":991,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330596+0000","last_change":"2024-02-10T07:49:56.824743+0000","last_active":"2024-02-10T07:50:08.330596+0000","last_peered":"2024-02-10T07:50:08.330596+0000","last_clean":"2024-02-10T07:50:08.330596+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:50:08.330596+0000","last_undegraded":"2024-02-10T07:50:08.330596+0000","last_fullsized":"2024-02-10T07:50:08.330596+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:49:56.824601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.824601+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:04.771314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":751,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335763+0000","last_change":"2024-02-10T07:48:30.384635+0000","last_active":"2024-02-10T07:50:08.335763+0000","last_peered":"2024-02-10T07:50:08.335763+0000","last_clean":"2024-02-10T07:50:08.335763+0000","last_became_active":"2024-02-10T07:47:47.966127+0000","last_became_peered":"2024-02-10T07:47:47.966127+0000","last_unstale":"2024-02-10T07:50:08.335763+0000","last_undegraded":"2024-02-10T07:50:08.335763+0000","last_fullsized":"2024-02-10T07:50:08.335763+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:30.384546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:30.384546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"250'25","reported_seq":815,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.331593+0000","last_change":"2024-02-10T07:49:06.308245+0000","last_active":"2024-02-10T07:50:08.331593+0000","last_peered":"2024-02-10T07:50:08.331593+0000","last_clean":"2024-02-10T07:50:08.331593+0000","last_became_active":"2024-02-10T07:49:06.261487+0000","last_became_peered":"2024-02-10T07:49:06.261487+0000","last_unstale":"2024-02-10T07:50:08.331593+0000","last_undegraded":"2024-02-10T07:50:08.331593+0000","last_fullsized":"2024-02-10T07:50:08.331593+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:48:12.278982+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-02-10T07:42:17.820959+0000","last_clean_scrub_stamp":"2024-02-10T07:48:12.278982+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":738,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336830+0000","last_change":"2024-02-10T07:49:11.231819+0000","last_active":"2024-02-10T07:50:08.336830+0000","last_peered":"2024-02-10T07:50:08.336830+0000","last_clean":"2024-02-10T07:50:08.336830+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:50:08.336830+0000","last_undegraded":"2024-02-10T07:50:08.336830+0000","last_fullsized":"2024-02-10T07:50:08.336830+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.231704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.231704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:31.609046+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":735,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.332026+0000","last_change":"2024-02-10T07:48:56.208535+0000","last_active":"2024-02-10T07:50:08.332026+0000","last_peered":"2024-02-10T07:50:08.332026+0000","last_clean":"2024-02-10T07:50:08.332026+0000","last_became_active":"2024-02-10T07:46:10.724252+0000","last_became_peered":"2024-02-10T07:46:10.724252+0000","last_unstale":"2024-02-10T07:50:08.332026+0000","last_undegraded":"2024-02-10T07:50:08.332026+0000","last_fullsized":"2024-02-10T07:50:08.332026+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.208451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:06.200009+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.208451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":239,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330407+0000","last_change":"2024-02-10T07:49:06.259104+0000","last_active":"2024-02-10T07:50:08.330407+0000","last_peered":"2024-02-10T07:50:08.330407+0000","last_clean":"2024-02-10T07:50:08.330407+0000","last_became_active":"2024-02-10T07:49:06.258789+0000","last_became_peered":"2024-02-10T07:49:06.258789+0000","last_unstale":"2024-02-10T07:50:08.330407+0000","last_undegraded":"2024-02-10T07:50:08.330407+0000","last_fullsized":"2024-02-10T07:50:08.330407+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:45.224482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:47:45.224482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"250'19","reported_seq":787,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330263+0000","last_change":"2024-02-10T07:49:50.808829+0000","last_active":"2024-02-10T07:50:08.330263+0000","last_peered":"2024-02-10T07:50:08.330263+0000","last_clean":"2024-02-10T07:50:08.330263+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:50:08.330263+0000","last_undegraded":"2024-02-10T07:50:08.330263+0000","last_fullsized":"2024-02-10T07:50:08.330263+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:50.808710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.808710+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:14.055156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":747,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.328989+0000","last_change":"2024-02-10T07:49:06.249490+0000","last_active":"2024-02-10T07:50:08.328989+0000","last_peered":"2024-02-10T07:50:08.328989+0000","last_clean":"2024-02-10T07:50:08.328989+0000","last_became_active":"2024-02-10T07:49:06.249162+0000","last_became_peered":"2024-02-10T07:49:06.249162+0000","last_unstale":"2024-02-10T07:50:08.328989+0000","last_undegraded":"2024-02-10T07:50:08.328989+0000","last_fullsized":"2024-02-10T07:50:08.328989+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.087614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.087614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":750,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.328781+0000","last_change":"2024-02-10T07:49:54.769358+0000","last_active":"2024-02-10T07:50:08.328781+0000","last_peered":"2024-02-10T07:50:08.328781+0000","last_clean":"2024-02-10T07:50:08.328781+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:50:08.328781+0000","last_undegraded":"2024-02-10T07:50:08.328781+0000","last_fullsized":"2024-02-10T07:50:08.328781+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:54.769226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:54.769226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:24.449065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":767,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.331959+0000","last_change":"2024-02-10T07:49:06.259100+0000","last_active":"2024-02-10T07:50:08.331959+0000","last_peered":"2024-02-10T07:50:08.331959+0000","last_clean":"2024-02-10T07:50:08.331959+0000","last_became_active":"2024-02-10T07:49:06.258763+0000","last_became_peered":"2024-02-10T07:49:06.258763+0000","last_unstale":"2024-02-10T07:50:08.331959+0000","last_undegraded":"2024-02-10T07:50:08.331959+0000","last_fullsized":"2024-02-10T07:50:08.331959+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:29.188832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:29.188832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"250'10","reported_seq":69,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.334780+0000","last_change":"2024-02-10T07:49:16.017097+0000","last_active":"2024-02-10T07:50:08.334780+0000","last_peered":"2024-02-10T07:50:08.334780+0000","last_clean":"2024-02-10T07:50:08.334780+0000","last_became_active":"2024-02-10T07:49:06.250577+0000","last_became_peered":"2024-02-10T07:49:06.250577+0000","last_unstale":"2024-02-10T07:50:08.334780+0000","last_undegraded":"2024-02-10T07:50:08.334780+0000","last_fullsized":"2024-02-10T07:50:08.334780+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'10","last_scrub_stamp":"2024-02-10T07:49:16.017003+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.017003+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:23.056219+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":233,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330351+0000","last_change":"2024-02-10T07:49:06.259895+0000","last_active":"2024-02-10T07:50:08.330351+0000","last_peered":"2024-02-10T07:50:08.330351+0000","last_clean":"2024-02-10T07:50:08.330351+0000","last_became_active":"2024-02-10T07:49:06.259341+0000","last_became_peered":"2024-02-10T07:49:06.259341+0000","last_unstale":"2024-02-10T07:50:08.330351+0000","last_undegraded":"2024-02-10T07:50:08.330351+0000","last_fullsized":"2024-02-10T07:50:08.330351+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:32.258321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:32.258321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":735,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329000+0000","last_change":"2024-02-10T07:48:24.171978+0000","last_active":"2024-02-10T07:50:08.329000+0000","last_peered":"2024-02-10T07:50:08.329000+0000","last_clean":"2024-02-10T07:50:08.329000+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:50:08.329000+0000","last_undegraded":"2024-02-10T07:50:08.329000+0000","last_fullsized":"2024-02-10T07:50:08.329000+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:24.171859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:24.171859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":676,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.333494+0000","last_change":"2024-02-10T07:49:53.578357+0000","last_active":"2024-02-10T07:50:08.333494+0000","last_peered":"2024-02-10T07:50:08.333494+0000","last_clean":"2024-02-10T07:50:08.333494+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:50:08.333494+0000","last_undegraded":"2024-02-10T07:50:08.333494+0000","last_fullsized":"2024-02-10T07:50:08.333494+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:53.578274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:49:53.578274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.901374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"250'19","reported_seq":702,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.328595+0000","last_change":"2024-02-10T07:49:53.245490+0000","last_active":"2024-02-10T07:50:08.328595+0000","last_peered":"2024-02-10T07:50:08.328595+0000","last_clean":"2024-02-10T07:50:08.328595+0000","last_became_active":"2024-02-10T07:49:06.239701+0000","last_became_peered":"2024-02-10T07:49:06.239701+0000","last_unstale":"2024-02-10T07:50:08.328595+0000","last_undegraded":"2024-02-10T07:50:08.328595+0000","last_fullsized":"2024-02-10T07:50:08.328595+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:53.245410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:53.245410+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:19.652831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":732,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330230+0000","last_change":"2024-02-10T07:49:52.530010+0000","last_active":"2024-02-10T07:50:08.330230+0000","last_peered":"2024-02-10T07:50:08.330230+0000","last_clean":"2024-02-10T07:50:08.330230+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:50:08.330230+0000","last_undegraded":"2024-02-10T07:50:08.330230+0000","last_fullsized":"2024-02-10T07:50:08.330230+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.529908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:59.179342+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.529908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.765757+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":233,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330536+0000","last_change":"2024-02-10T07:49:06.265551+0000","last_active":"2024-02-10T07:50:08.330536+0000","last_peered":"2024-02-10T07:50:08.330536+0000","last_clean":"2024-02-10T07:50:08.330536+0000","last_became_active":"2024-02-10T07:49:06.265267+0000","last_became_peered":"2024-02-10T07:49:06.265267+0000","last_unstale":"2024-02-10T07:50:08.330536+0000","last_undegraded":"2024-02-10T07:50:08.330536+0000","last_fullsized":"2024-02-10T07:50:08.330536+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:40.140557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:40.140557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":752,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329562+0000","last_change":"2024-02-10T07:50:00.265308+0000","last_active":"2024-02-10T07:50:08.329562+0000","last_peered":"2024-02-10T07:50:08.329562+0000","last_clean":"2024-02-10T07:50:08.329562+0000","last_became_active":"2024-02-10T07:47:47.966472+0000","last_became_peered":"2024-02-10T07:47:47.966472+0000","last_unstale":"2024-02-10T07:50:08.329562+0000","last_undegraded":"2024-02-10T07:50:08.329562+0000","last_fullsized":"2024-02-10T07:50:08.329562+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.265208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.265208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:23.837007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"250'19","reported_seq":560,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335771+0000","last_change":"2024-02-10T07:49:09.117006+0000","last_active":"2024-02-10T07:50:08.335771+0000","last_peered":"2024-02-10T07:50:08.335771+0000","last_clean":"2024-02-10T07:50:08.335771+0000","last_became_active":"2024-02-10T07:46:10.726359+0000","last_became_peered":"2024-02-10T07:46:10.726359+0000","last_unstale":"2024-02-10T07:50:08.335771+0000","last_undegraded":"2024-02-10T07:50:08.335771+0000","last_fullsized":"2024-02-10T07:50:08.335771+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:09.116615+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-02-10T07:39:32.934269+0000","last_clean_scrub_stamp":"2024-02-10T07:49:09.116615+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:35.109389+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":421,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330156+0000","last_change":"2024-02-10T07:49:06.260883+0000","last_active":"2024-02-10T07:50:08.330156+0000","last_peered":"2024-02-10T07:50:08.330156+0000","last_clean":"2024-02-10T07:50:08.330156+0000","last_became_active":"2024-02-10T07:49:06.260569+0000","last_became_peered":"2024-02-10T07:49:06.260569+0000","last_unstale":"2024-02-10T07:50:08.330156+0000","last_undegraded":"2024-02-10T07:50:08.330156+0000","last_fullsized":"2024-02-10T07:50:08.330156+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:31.117976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:31.117976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":765,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336240+0000","last_change":"2024-02-10T07:50:02.685941+0000","last_active":"2024-02-10T07:50:08.336240+0000","last_peered":"2024-02-10T07:50:08.336240+0000","last_clean":"2024-02-10T07:50:08.336240+0000","last_became_active":"2024-02-10T07:49:06.252380+0000","last_became_peered":"2024-02-10T07:49:06.252380+0000","last_unstale":"2024-02-10T07:50:08.336240+0000","last_undegraded":"2024-02-10T07:50:08.336240+0000","last_fullsized":"2024-02-10T07:50:08.336240+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_clean_scrub_stamp":"2024-02-10T07:50:02.685831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:09.446762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":743,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335926+0000","last_change":"2024-02-10T07:50:00.345087+0000","last_active":"2024-02-10T07:50:08.335926+0000","last_peered":"2024-02-10T07:50:08.335926+0000","last_clean":"2024-02-10T07:50:08.335926+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:50:08.335926+0000","last_undegraded":"2024-02-10T07:50:08.335926+0000","last_fullsized":"2024-02-10T07:50:08.335926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.344993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.344993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:26.742023+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"250'24","reported_seq":803,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.337509+0000","last_change":"2024-02-10T07:47:43.494877+0000","last_active":"2024-02-10T07:50:08.337509+0000","last_peered":"2024-02-10T07:50:08.337509+0000","last_clean":"2024-02-10T07:50:08.337509+0000","last_became_active":"2024-02-10T07:46:10.721336+0000","last_became_peered":"2024-02-10T07:46:10.721336+0000","last_unstale":"2024-02-10T07:50:08.337509+0000","last_undegraded":"2024-02-10T07:50:08.337509+0000","last_fullsized":"2024-02-10T07:50:08.337509+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:47:43.494350+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.494350+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":738,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.331333+0000","last_change":"2024-02-10T07:49:06.262127+0000","last_active":"2024-02-10T07:50:08.331333+0000","last_peered":"2024-02-10T07:50:08.331333+0000","last_clean":"2024-02-10T07:50:08.331333+0000","last_became_active":"2024-02-10T07:49:06.261570+0000","last_became_peered":"2024-02-10T07:49:06.261570+0000","last_unstale":"2024-02-10T07:50:08.331333+0000","last_undegraded":"2024-02-10T07:50:08.331333+0000","last_fullsized":"2024-02-10T07:50:08.331333+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:43.186809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.186809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"356'11","reported_seq":200,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336670+0000","last_change":"2024-02-10T07:48:25.339974+0000","last_active":"2024-02-10T07:50:08.336670+0000","last_peered":"2024-02-10T07:50:08.336670+0000","last_clean":"2024-02-10T07:50:08.336670+0000","last_became_active":"2024-02-10T07:48:02.371888+0000","last_became_peered":"2024-02-10T07:48:02.371888+0000","last_unstale":"2024-02-10T07:50:08.336670+0000","last_undegraded":"2024-02-10T07:50:08.336670+0000","last_fullsized":"2024-02-10T07:50:08.336670+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'11","last_scrub_stamp":"2024-02-10T07:48:25.339855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:25.339855+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":757,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329349+0000","last_change":"2024-02-10T07:49:17.335559+0000","last_active":"2024-02-10T07:50:08.329349+0000","last_peered":"2024-02-10T07:50:08.329349+0000","last_clean":"2024-02-10T07:50:08.329349+0000","last_became_active":"2024-02-10T07:46:10.716647+0000","last_became_peered":"2024-02-10T07:46:10.716647+0000","last_unstale":"2024-02-10T07:50:08.329349+0000","last_undegraded":"2024-02-10T07:50:08.329349+0000","last_fullsized":"2024-02-10T07:50:08.329349+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.335455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.335455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:45.237183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"250'25","reported_seq":364,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336355+0000","last_change":"2024-02-10T07:49:12.150442+0000","last_active":"2024-02-10T07:50:08.336355+0000","last_peered":"2024-02-10T07:50:08.336355+0000","last_clean":"2024-02-10T07:50:08.336355+0000","last_became_active":"2024-02-10T07:48:02.329187+0000","last_became_peered":"2024-02-10T07:48:02.329187+0000","last_unstale":"2024-02-10T07:50:08.336355+0000","last_undegraded":"2024-02-10T07:50:08.336355+0000","last_fullsized":"2024-02-10T07:50:08.336355+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:49:12.150298+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:49:12.150298+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:17.442636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":738,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.337882+0000","last_change":"2024-02-10T07:50:01.514066+0000","last_active":"2024-02-10T07:50:08.337882+0000","last_peered":"2024-02-10T07:50:08.337882+0000","last_clean":"2024-02-10T07:50:08.337882+0000","last_became_active":"2024-02-10T07:46:10.720680+0000","last_became_peered":"2024-02-10T07:46:10.720680+0000","last_unstale":"2024-02-10T07:50:08.337882+0000","last_undegraded":"2024-02-10T07:50:08.337882+0000","last_fullsized":"2024-02-10T07:50:08.337882+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:01.513972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:01.513972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.583346+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"376'14","reported_seq":346,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330274+0000","last_change":"2024-02-10T07:49:06.314956+0000","last_active":"2024-02-10T07:50:08.330274+0000","last_peered":"2024-02-10T07:50:08.330274+0000","last_clean":"2024-02-10T07:50:08.330274+0000","last_became_active":"2024-02-10T07:49:06.267009+0000","last_became_peered":"2024-02-10T07:49:06.267009+0000","last_unstale":"2024-02-10T07:50:08.330274+0000","last_undegraded":"2024-02-10T07:50:08.330274+0000","last_fullsized":"2024-02-10T07:50:08.330274+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'13","last_scrub_stamp":"2024-02-10T07:49:00.366610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.366610+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:24.646255+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":758,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.331301+0000","last_change":"2024-02-10T07:50:07.501014+0000","last_active":"2024-02-10T07:50:08.331301+0000","last_peered":"2024-02-10T07:50:08.331301+0000","last_clean":"2024-02-10T07:50:08.331301+0000","last_became_active":"2024-02-10T07:49:06.257043+0000","last_became_peered":"2024-02-10T07:49:06.257043+0000","last_unstale":"2024-02-10T07:50:08.331301+0000","last_undegraded":"2024-02-10T07:50:08.331301+0000","last_fullsized":"2024-02-10T07:50:08.331301+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.500890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:33.615213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"250'15","reported_seq":496,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329848+0000","last_change":"2024-02-10T07:49:06.277154+0000","last_active":"2024-02-10T07:50:08.329848+0000","last_peered":"2024-02-10T07:50:08.329848+0000","last_clean":"2024-02-10T07:50:08.329848+0000","last_became_active":"2024-02-10T07:49:06.255239+0000","last_became_peered":"2024-02-10T07:49:06.255239+0000","last_unstale":"2024-02-10T07:50:08.329848+0000","last_undegraded":"2024-02-10T07:50:08.329848+0000","last_fullsized":"2024-02-10T07:50:08.329848+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:39.610017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:39.610017+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":198,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336278+0000","last_change":"2024-02-10T07:49:44.324178+0000","last_active":"2024-02-10T07:50:08.336278+0000","last_peered":"2024-02-10T07:50:08.336278+0000","last_clean":"2024-02-10T07:50:08.336278+0000","last_became_active":"2024-02-10T07:46:10.716869+0000","last_became_peered":"2024-02-10T07:46:10.716869+0000","last_unstale":"2024-02-10T07:50:08.336278+0000","last_undegraded":"2024-02-10T07:50:08.336278+0000","last_fullsized":"2024-02-10T07:50:08.336278+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:44.324046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:49:44.324046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:46.499909+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":83,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330042+0000","last_change":"2024-02-10T07:49:06.263378+0000","last_active":"2024-02-10T07:50:08.330042+0000","last_peered":"2024-02-10T07:50:08.330042+0000","last_clean":"2024-02-10T07:50:08.330042+0000","last_became_active":"2024-02-10T07:49:06.262933+0000","last_became_peered":"2024-02-10T07:49:06.262933+0000","last_unstale":"2024-02-10T07:50:08.330042+0000","last_undegraded":"2024-02-10T07:50:08.330042+0000","last_fullsized":"2024-02-10T07:50:08.330042+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:17.294700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:48:17.294700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":751,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.328450+0000","last_change":"2024-02-10T07:49:59.278666+0000","last_active":"2024-02-10T07:50:08.328450+0000","last_peered":"2024-02-10T07:50:08.328450+0000","last_clean":"2024-02-10T07:50:08.328450+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:50:08.328450+0000","last_undegraded":"2024-02-10T07:50:08.328450+0000","last_fullsized":"2024-02-10T07:50:08.328450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.278574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.278574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.042755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"250'19","reported_seq":803,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.331876+0000","last_change":"2024-02-10T07:49:06.306965+0000","last_active":"2024-02-10T07:50:08.331876+0000","last_peered":"2024-02-10T07:50:08.331876+0000","last_clean":"2024-02-10T07:50:08.331876+0000","last_became_active":"2024-02-10T07:49:06.260962+0000","last_became_peered":"2024-02-10T07:49:06.260962+0000","last_unstale":"2024-02-10T07:50:08.331876+0000","last_undegraded":"2024-02-10T07:50:08.331876+0000","last_fullsized":"2024-02-10T07:50:08.331876+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:47:42.150605+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.150605+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":731,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329478+0000","last_change":"2024-02-10T07:48:58.511630+0000","last_active":"2024-02-10T07:50:08.329478+0000","last_peered":"2024-02-10T07:50:08.329478+0000","last_clean":"2024-02-10T07:50:08.329478+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:50:08.329478+0000","last_undegraded":"2024-02-10T07:50:08.329478+0000","last_fullsized":"2024-02-10T07:50:08.329478+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:58.511251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.511251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:24.287877+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":63,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336301+0000","last_change":"2024-02-10T07:50:04.978177+0000","last_active":"2024-02-10T07:50:08.336301+0000","last_peered":"2024-02-10T07:50:08.336301+0000","last_clean":"2024-02-10T07:50:08.336301+0000","last_became_active":"2024-02-10T07:49:06.250832+0000","last_became_peered":"2024-02-10T07:49:06.250832+0000","last_unstale":"2024-02-10T07:50:08.336301+0000","last_undegraded":"2024-02-10T07:50:08.336301+0000","last_fullsized":"2024-02-10T07:50:08.336301+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:04.978097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.978097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:19.962282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":775,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329029+0000","last_change":"2024-02-10T07:49:52.832935+0000","last_active":"2024-02-10T07:50:08.329029+0000","last_peered":"2024-02-10T07:50:08.329029+0000","last_clean":"2024-02-10T07:50:08.329029+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:50:08.329029+0000","last_undegraded":"2024-02-10T07:50:08.329029+0000","last_fullsized":"2024-02-10T07:50:08.329029+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.832862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.832862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:16.134029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"250'12","reported_seq":71,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335987+0000","last_change":"2024-02-10T07:49:43.919297+0000","last_active":"2024-02-10T07:50:08.335987+0000","last_peered":"2024-02-10T07:50:08.335987+0000","last_clean":"2024-02-10T07:50:08.335987+0000","last_became_active":"2024-02-10T07:49:06.258281+0000","last_became_peered":"2024-02-10T07:49:06.258281+0000","last_unstale":"2024-02-10T07:50:08.335987+0000","last_undegraded":"2024-02-10T07:50:08.335987+0000","last_fullsized":"2024-02-10T07:50:08.335987+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'12","last_scrub_stamp":"2024-02-10T07:49:43.919178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:43.919178+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:52.611584+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"398'1547","reported_seq":1571,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:14.125280+0000","last_change":"2024-02-10T07:49:53.374457+0000","last_active":"2024-02-10T07:50:14.125280+0000","last_peered":"2024-02-10T07:50:14.125280+0000","last_clean":"2024-02-10T07:50:14.125280+0000","last_became_active":"2024-02-10T07:49:53.374185+0000","last_became_peered":"2024-02-10T07:49:53.374185+0000","last_unstale":"2024-02-10T07:50:14.125280+0000","last_undegraded":"2024-02-10T07:50:14.125280+0000","last_fullsized":"2024-02-10T07:50:14.125280+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1547,"log_dups_size":0,"ondisk_log_size":1547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:58.752412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101384192,"num_objects":1547,"num_object_clones":0,"num_object_copies":4641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1547,"num_write_kb":99008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":265,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.328814+0000","last_change":"2024-02-10T07:50:05.847991+0000","last_active":"2024-02-10T07:50:08.328814+0000","last_peered":"2024-02-10T07:50:08.328814+0000","last_clean":"2024-02-10T07:50:08.328814+0000","last_became_active":"2024-02-10T07:48:02.366002+0000","last_became_peered":"2024-02-10T07:48:02.366002+0000","last_unstale":"2024-02-10T07:50:08.328814+0000","last_undegraded":"2024-02-10T07:50:08.328814+0000","last_fullsized":"2024-02-10T07:50:08.328814+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:05.847866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:05.847866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:23.653860+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"245'4","reported_seq":766,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330500+0000","last_change":"2024-02-10T07:49:08.311056+0000","last_active":"2024-02-10T07:50:08.330500+0000","last_peered":"2024-02-10T07:50:08.330500+0000","last_clean":"2024-02-10T07:50:08.330500+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:50:08.330500+0000","last_undegraded":"2024-02-10T07:50:08.330500+0000","last_fullsized":"2024-02-10T07:50:08.330500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'4","last_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_deep_scrub":"245'4","last_deep_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.310917+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:22.674123+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"250'34","reported_seq":492,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.328861+0000","last_change":"2024-02-10T07:48:11.967227+0000","last_active":"2024-02-10T07:50:08.328861+0000","last_peered":"2024-02-10T07:50:08.328861+0000","last_clean":"2024-02-10T07:50:08.328861+0000","last_became_active":"2024-02-10T07:48:02.330130+0000","last_became_peered":"2024-02-10T07:48:02.330130+0000","last_unstale":"2024-02-10T07:50:08.328861+0000","last_undegraded":"2024-02-10T07:50:08.328861+0000","last_fullsized":"2024-02-10T07:50:08.328861+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:48:11.966957+0000","last_deep_scrub":"246'22","last_deep_scrub_stamp":"2024-02-10T07:41:26.771112+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.966957+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":973,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.328727+0000","last_change":"2024-02-10T07:48:53.423840+0000","last_active":"2024-02-10T07:50:08.328727+0000","last_peered":"2024-02-10T07:50:08.328727+0000","last_clean":"2024-02-10T07:50:08.328727+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:50:08.328727+0000","last_undegraded":"2024-02-10T07:50:08.328727+0000","last_fullsized":"2024-02-10T07:50:08.328727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:48:53.423613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:53.423613+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"398'1539","reported_seq":1563,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:14.200671+0000","last_change":"2024-02-10T07:49:53.282437+0000","last_active":"2024-02-10T07:50:14.200671+0000","last_peered":"2024-02-10T07:50:14.200671+0000","last_clean":"2024-02-10T07:50:14.200671+0000","last_became_active":"2024-02-10T07:49:53.282163+0000","last_became_peered":"2024-02-10T07:49:53.282163+0000","last_unstale":"2024-02-10T07:50:14.200671+0000","last_undegraded":"2024-02-10T07:50:14.200671+0000","last_fullsized":"2024-02-10T07:50:14.200671+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1539,"log_dups_size":0,"ondisk_log_size":1539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:16.220098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100859904,"num_objects":1539,"num_object_clones":0,"num_object_copies":4617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1539,"num_write_kb":98496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":570,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335234+0000","last_change":"2024-02-10T07:49:42.990229+0000","last_active":"2024-02-10T07:50:08.335234+0000","last_peered":"2024-02-10T07:50:08.335234+0000","last_clean":"2024-02-10T07:50:08.335234+0000","last_became_active":"2024-02-10T07:47:47.966183+0000","last_became_peered":"2024-02-10T07:47:47.966183+0000","last_unstale":"2024-02-10T07:50:08.335234+0000","last_undegraded":"2024-02-10T07:50:08.335234+0000","last_fullsized":"2024-02-10T07:50:08.335234+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:42.990098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:42.990098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.708032+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":744,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329795+0000","last_change":"2024-02-10T07:48:56.416159+0000","last_active":"2024-02-10T07:50:08.329795+0000","last_peered":"2024-02-10T07:50:08.329795+0000","last_clean":"2024-02-10T07:50:08.329795+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:50:08.329795+0000","last_undegraded":"2024-02-10T07:50:08.329795+0000","last_fullsized":"2024-02-10T07:50:08.329795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.416037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.416037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"250'23","reported_seq":850,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336276+0000","last_change":"2024-02-10T07:49:43.339027+0000","last_active":"2024-02-10T07:50:08.336276+0000","last_peered":"2024-02-10T07:50:08.336276+0000","last_clean":"2024-02-10T07:50:08.336276+0000","last_became_active":"2024-02-10T07:49:06.243569+0000","last_became_peered":"2024-02-10T07:49:06.243569+0000","last_unstale":"2024-02-10T07:50:08.336276+0000","last_undegraded":"2024-02-10T07:50:08.336276+0000","last_fullsized":"2024-02-10T07:50:08.336276+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-02-10T07:49:43.338928+0000","last_deep_scrub":"245'17","last_deep_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_clean_scrub_stamp":"2024-02-10T07:49:43.338928+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:49.402156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":191,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330494+0000","last_change":"2024-02-10T07:49:04.308934+0000","last_active":"2024-02-10T07:50:08.330494+0000","last_peered":"2024-02-10T07:50:08.330494+0000","last_clean":"2024-02-10T07:50:08.330494+0000","last_became_active":"2024-02-10T07:46:10.723006+0000","last_became_peered":"2024-02-10T07:46:10.723006+0000","last_unstale":"2024-02-10T07:50:08.330494+0000","last_undegraded":"2024-02-10T07:50:08.330494+0000","last_fullsized":"2024-02-10T07:50:08.330494+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:04.308807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.308807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.812692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"398'1441","reported_seq":1465,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:13.327611+0000","last_change":"2024-02-10T07:49:53.374953+0000","last_active":"2024-02-10T07:50:13.327611+0000","last_peered":"2024-02-10T07:50:13.327611+0000","last_clean":"2024-02-10T07:50:13.327611+0000","last_became_active":"2024-02-10T07:49:53.374631+0000","last_became_peered":"2024-02-10T07:49:53.374631+0000","last_unstale":"2024-02-10T07:50:13.327611+0000","last_undegraded":"2024-02-10T07:50:13.327611+0000","last_fullsized":"2024-02-10T07:50:13.327611+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1441,"log_dups_size":0,"ondisk_log_size":1441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:08.897366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94437376,"num_objects":1441,"num_object_clones":0,"num_object_copies":4323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1441,"num_write_kb":92224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":759,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.328163+0000","last_change":"2024-02-10T07:49:11.361783+0000","last_active":"2024-02-10T07:50:08.328163+0000","last_peered":"2024-02-10T07:50:08.328163+0000","last_clean":"2024-02-10T07:50:08.328163+0000","last_became_active":"2024-02-10T07:49:06.249148+0000","last_became_peered":"2024-02-10T07:49:06.249148+0000","last_unstale":"2024-02-10T07:50:08.328163+0000","last_undegraded":"2024-02-10T07:50:08.328163+0000","last_fullsized":"2024-02-10T07:50:08.328163+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.361139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:35.884283+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":391,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330453+0000","last_change":"2024-02-10T07:48:21.172677+0000","last_active":"2024-02-10T07:50:08.330453+0000","last_peered":"2024-02-10T07:50:08.330453+0000","last_clean":"2024-02-10T07:50:08.330453+0000","last_became_active":"2024-02-10T07:46:10.719610+0000","last_became_peered":"2024-02-10T07:46:10.719610+0000","last_unstale":"2024-02-10T07:50:08.330453+0000","last_undegraded":"2024-02-10T07:50:08.330453+0000","last_fullsized":"2024-02-10T07:50:08.330453+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:21.172601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:21.172601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":761,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329824+0000","last_change":"2024-02-10T07:50:07.252577+0000","last_active":"2024-02-10T07:50:08.329824+0000","last_peered":"2024-02-10T07:50:08.329824+0000","last_clean":"2024-02-10T07:50:08.329824+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:50:08.329824+0000","last_undegraded":"2024-02-10T07:50:08.329824+0000","last_fullsized":"2024-02-10T07:50:08.329824+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.252458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.252458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:09.491987+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"250'35","reported_seq":810,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329575+0000","last_change":"2024-02-10T07:49:49.772965+0000","last_active":"2024-02-10T07:50:08.329575+0000","last_peered":"2024-02-10T07:50:08.329575+0000","last_clean":"2024-02-10T07:50:08.329575+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:50:08.329575+0000","last_undegraded":"2024-02-10T07:50:08.329575+0000","last_fullsized":"2024-02-10T07:50:08.329575+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'35","last_scrub_stamp":"2024-02-10T07:49:49.772852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:49.772852+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.199744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"398'1468","reported_seq":1492,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:13.345030+0000","last_change":"2024-02-10T07:49:53.280069+0000","last_active":"2024-02-10T07:50:13.345030+0000","last_peered":"2024-02-10T07:50:13.345030+0000","last_clean":"2024-02-10T07:50:13.345030+0000","last_became_active":"2024-02-10T07:49:53.279758+0000","last_became_peered":"2024-02-10T07:49:53.279758+0000","last_unstale":"2024-02-10T07:50:13.345030+0000","last_undegraded":"2024-02-10T07:50:13.345030+0000","last_fullsized":"2024-02-10T07:50:13.345030+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1468,"log_dups_size":0,"ondisk_log_size":1468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:19.618017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96206848,"num_objects":1468,"num_object_clones":0,"num_object_copies":4404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1468,"num_write_kb":93952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":748,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330349+0000","last_change":"2024-02-10T07:49:06.249190+0000","last_active":"2024-02-10T07:50:08.330349+0000","last_peered":"2024-02-10T07:50:08.330349+0000","last_clean":"2024-02-10T07:50:08.330349+0000","last_became_active":"2024-02-10T07:49:06.248040+0000","last_became_peered":"2024-02-10T07:49:06.248040+0000","last_unstale":"2024-02-10T07:50:08.330349+0000","last_undegraded":"2024-02-10T07:50:08.330349+0000","last_fullsized":"2024-02-10T07:50:08.330349+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:00.336175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.336175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":720,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336913+0000","last_change":"2024-02-10T07:49:06.257180+0000","last_active":"2024-02-10T07:50:08.336913+0000","last_peered":"2024-02-10T07:50:08.336913+0000","last_clean":"2024-02-10T07:50:08.336913+0000","last_became_active":"2024-02-10T07:49:06.256303+0000","last_became_peered":"2024-02-10T07:49:06.256303+0000","last_unstale":"2024-02-10T07:50:08.336913+0000","last_undegraded":"2024-02-10T07:50:08.336913+0000","last_fullsized":"2024-02-10T07:50:08.336913+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:28.822700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:28.822700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":63,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335424+0000","last_change":"2024-02-10T07:49:52.968676+0000","last_active":"2024-02-10T07:50:08.335424+0000","last_peered":"2024-02-10T07:50:08.335424+0000","last_clean":"2024-02-10T07:50:08.335424+0000","last_became_active":"2024-02-10T07:49:06.249431+0000","last_became_peered":"2024-02-10T07:49:06.249431+0000","last_unstale":"2024-02-10T07:50:08.335424+0000","last_undegraded":"2024-02-10T07:50:08.335424+0000","last_fullsized":"2024-02-10T07:50:08.335424+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.968567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.968567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:06.997404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"250'15","reported_seq":765,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330864+0000","last_change":"2024-02-10T07:48:59.304277+0000","last_active":"2024-02-10T07:50:08.330864+0000","last_peered":"2024-02-10T07:50:08.330864+0000","last_clean":"2024-02-10T07:50:08.330864+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:50:08.330864+0000","last_undegraded":"2024-02-10T07:50:08.330864+0000","last_fullsized":"2024-02-10T07:50:08.330864+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:48:59.304199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.304199+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.764234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"398'1099","reported_seq":1121,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:11.311992+0000","last_change":"2024-02-10T07:49:53.284745+0000","last_active":"2024-02-10T07:50:11.311992+0000","last_peered":"2024-02-10T07:50:11.311992+0000","last_clean":"2024-02-10T07:50:11.311992+0000","last_became_active":"2024-02-10T07:49:53.283642+0000","last_became_peered":"2024-02-10T07:49:53.283642+0000","last_unstale":"2024-02-10T07:50:11.311992+0000","last_undegraded":"2024-02-10T07:50:11.311992+0000","last_fullsized":"2024-02-10T07:50:11.311992+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:18.016780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72024064,"num_objects":1099,"num_object_clones":0,"num_object_copies":3297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1099,"num_write_kb":70336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":63,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336387+0000","last_change":"2024-02-10T07:49:50.032577+0000","last_active":"2024-02-10T07:50:08.336387+0000","last_peered":"2024-02-10T07:50:08.336387+0000","last_clean":"2024-02-10T07:50:08.336387+0000","last_became_active":"2024-02-10T07:49:06.238367+0000","last_became_peered":"2024-02-10T07:49:06.238367+0000","last_unstale":"2024-02-10T07:50:08.336387+0000","last_undegraded":"2024-02-10T07:50:08.336387+0000","last_fullsized":"2024-02-10T07:50:08.336387+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:50.032505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.032505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:15.102164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":164,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335972+0000","last_change":"2024-02-10T07:49:10.087939+0000","last_active":"2024-02-10T07:50:08.335972+0000","last_peered":"2024-02-10T07:50:08.335972+0000","last_clean":"2024-02-10T07:50:08.335972+0000","last_became_active":"2024-02-10T07:48:02.328316+0000","last_became_peered":"2024-02-10T07:48:02.328316+0000","last_unstale":"2024-02-10T07:50:08.335972+0000","last_undegraded":"2024-02-10T07:50:08.335972+0000","last_fullsized":"2024-02-10T07:50:08.335972+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:49:10.087795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:49:10.087795+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:16.190923+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":736,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.337137+0000","last_change":"2024-02-10T07:49:19.198419+0000","last_active":"2024-02-10T07:50:08.337137+0000","last_peered":"2024-02-10T07:50:08.337137+0000","last_clean":"2024-02-10T07:50:08.337137+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:50:08.337137+0000","last_undegraded":"2024-02-10T07:50:08.337137+0000","last_fullsized":"2024-02-10T07:50:08.337137+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:19.198312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:04.220328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.198312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:32.194353+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"376'46","reported_seq":1114,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.337031+0000","last_change":"2024-02-10T07:49:55.714434+0000","last_active":"2024-02-10T07:50:08.337031+0000","last_peered":"2024-02-10T07:50:08.337031+0000","last_clean":"2024-02-10T07:50:08.337031+0000","last_became_active":"2024-02-10T07:49:06.271470+0000","last_became_peered":"2024-02-10T07:49:06.271470+0000","last_unstale":"2024-02-10T07:50:08.337031+0000","last_undegraded":"2024-02-10T07:50:08.337031+0000","last_fullsized":"2024-02-10T07:50:08.337031+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'46","last_scrub_stamp":"2024-02-10T07:49:55.714323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:55.714323+0000","objects_scrubbed":10,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:17.817850+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":744,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.331419+0000","last_change":"2024-02-10T07:49:17.357664+0000","last_active":"2024-02-10T07:50:08.331419+0000","last_peered":"2024-02-10T07:50:08.331419+0000","last_clean":"2024-02-10T07:50:08.331419+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:50:08.331419+0000","last_undegraded":"2024-02-10T07:50:08.331419+0000","last_fullsized":"2024-02-10T07:50:08.331419+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.357533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:22.786949+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.357533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:31.046922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"398'1425","reported_seq":1449,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:13.452245+0000","last_change":"2024-02-10T07:49:53.374436+0000","last_active":"2024-02-10T07:50:13.452245+0000","last_peered":"2024-02-10T07:50:13.452245+0000","last_clean":"2024-02-10T07:50:13.452245+0000","last_became_active":"2024-02-10T07:49:53.374133+0000","last_became_peered":"2024-02-10T07:49:53.374133+0000","last_unstale":"2024-02-10T07:50:13.452245+0000","last_undegraded":"2024-02-10T07:50:13.452245+0000","last_fullsized":"2024-02-10T07:50:13.452245+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1425,"log_dups_size":0,"ondisk_log_size":1425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.202856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93388800,"num_objects":1425,"num_object_clones":0,"num_object_copies":4275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1425,"num_write_kb":91200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":733,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329949+0000","last_change":"2024-02-10T07:47:50.081222+0000","last_active":"2024-02-10T07:50:08.329949+0000","last_peered":"2024-02-10T07:50:08.329949+0000","last_clean":"2024-02-10T07:50:08.329949+0000","last_became_active":"2024-02-10T07:47:48.149256+0000","last_became_peered":"2024-02-10T07:47:48.149256+0000","last_unstale":"2024-02-10T07:50:08.329949+0000","last_undegraded":"2024-02-10T07:50:08.329949+0000","last_fullsized":"2024-02-10T07:50:08.329949+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.080714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:42.730875+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.080714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":757,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336207+0000","last_change":"2024-02-10T07:49:49.886431+0000","last_active":"2024-02-10T07:50:08.336207+0000","last_peered":"2024-02-10T07:50:08.336207+0000","last_clean":"2024-02-10T07:50:08.336207+0000","last_became_active":"2024-02-10T07:46:10.726265+0000","last_became_peered":"2024-02-10T07:46:10.726265+0000","last_unstale":"2024-02-10T07:50:08.336207+0000","last_undegraded":"2024-02-10T07:50:08.336207+0000","last_fullsized":"2024-02-10T07:50:08.336207+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:49.886354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:49.886354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:59.343977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":739,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329872+0000","last_change":"2024-02-10T07:49:06.261139+0000","last_active":"2024-02-10T07:50:08.329872+0000","last_peered":"2024-02-10T07:50:08.329872+0000","last_clean":"2024-02-10T07:50:08.329872+0000","last_became_active":"2024-02-10T07:49:06.260826+0000","last_became_peered":"2024-02-10T07:49:06.260826+0000","last_unstale":"2024-02-10T07:50:08.329872+0000","last_undegraded":"2024-02-10T07:50:08.329872+0000","last_fullsized":"2024-02-10T07:50:08.329872+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:15.234112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.234112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"250'17","reported_seq":97,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330425+0000","last_change":"2024-02-10T07:49:04.340445+0000","last_active":"2024-02-10T07:50:08.330425+0000","last_peered":"2024-02-10T07:50:08.330425+0000","last_clean":"2024-02-10T07:50:08.330425+0000","last_became_active":"2024-02-10T07:48:35.057662+0000","last_became_peered":"2024-02-10T07:48:35.057662+0000","last_unstale":"2024-02-10T07:50:08.330425+0000","last_undegraded":"2024-02-10T07:50:08.330425+0000","last_fullsized":"2024-02-10T07:50:08.330425+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'17","last_scrub_stamp":"2024-02-10T07:49:04.340366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.340366+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:15.372195+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":23,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":307,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329999+0000","last_change":"2024-02-10T07:49:50.232630+0000","last_active":"2024-02-10T07:50:08.329999+0000","last_peered":"2024-02-10T07:50:08.329999+0000","last_clean":"2024-02-10T07:50:08.329999+0000","last_became_active":"2024-02-10T07:48:02.331496+0000","last_became_peered":"2024-02-10T07:48:02.331496+0000","last_unstale":"2024-02-10T07:50:08.329999+0000","last_undegraded":"2024-02-10T07:50:08.329999+0000","last_fullsized":"2024-02-10T07:50:08.329999+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:50.232544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.232544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:05.339508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"398'1381","reported_seq":1405,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:13.437217+0000","last_change":"2024-02-10T07:49:53.374470+0000","last_active":"2024-02-10T07:50:13.437217+0000","last_peered":"2024-02-10T07:50:13.437217+0000","last_clean":"2024-02-10T07:50:13.437217+0000","last_became_active":"2024-02-10T07:49:53.374185+0000","last_became_peered":"2024-02-10T07:49:53.374185+0000","last_unstale":"2024-02-10T07:50:13.437217+0000","last_undegraded":"2024-02-10T07:50:13.437217+0000","last_fullsized":"2024-02-10T07:50:13.437217+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1381,"log_dups_size":0,"ondisk_log_size":1381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:53.548261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90505216,"num_objects":1381,"num_object_clones":0,"num_object_copies":4143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1381,"num_write_kb":88384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":153,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335743+0000","last_change":"2024-02-10T07:48:02.330253+0000","last_active":"2024-02-10T07:50:08.335743+0000","last_peered":"2024-02-10T07:50:08.335743+0000","last_clean":"2024-02-10T07:50:08.335743+0000","last_became_active":"2024-02-10T07:48:02.329687+0000","last_became_peered":"2024-02-10T07:48:02.329687+0000","last_unstale":"2024-02-10T07:50:08.335743+0000","last_undegraded":"2024-02-10T07:50:08.335743+0000","last_fullsized":"2024-02-10T07:50:08.335743+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:47:49.750508+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.750508+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:14.252035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":990,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.328256+0000","last_change":"2024-02-10T07:48:15.033077+0000","last_active":"2024-02-10T07:50:08.328256+0000","last_peered":"2024-02-10T07:50:08.328256+0000","last_clean":"2024-02-10T07:50:08.328256+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:50:08.328256+0000","last_undegraded":"2024-02-10T07:50:08.328256+0000","last_fullsized":"2024-02-10T07:50:08.328256+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:48:15.032944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.032944+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":374,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330398+0000","last_change":"2024-02-10T07:49:06.257190+0000","last_active":"2024-02-10T07:50:08.330398+0000","last_peered":"2024-02-10T07:50:08.330398+0000","last_clean":"2024-02-10T07:50:08.330398+0000","last_became_active":"2024-02-10T07:49:06.256223+0000","last_became_peered":"2024-02-10T07:49:06.256223+0000","last_unstale":"2024-02-10T07:50:08.330398+0000","last_undegraded":"2024-02-10T07:50:08.330398+0000","last_fullsized":"2024-02-10T07:50:08.330398+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:17.198213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:17.198213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"250'36","reported_seq":1024,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330533+0000","last_change":"2024-02-10T07:49:58.841289+0000","last_active":"2024-02-10T07:50:08.330533+0000","last_peered":"2024-02-10T07:50:08.330533+0000","last_clean":"2024-02-10T07:50:08.330533+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:50:08.330533+0000","last_undegraded":"2024-02-10T07:50:08.330533+0000","last_fullsized":"2024-02-10T07:50:08.330533+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'36","last_scrub_stamp":"2024-02-10T07:49:58.841172+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:49:58.841172+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:07.505517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":759,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330049+0000","last_change":"2024-02-10T07:49:13.369870+0000","last_active":"2024-02-10T07:50:08.330049+0000","last_peered":"2024-02-10T07:50:08.330049+0000","last_clean":"2024-02-10T07:50:08.330049+0000","last_became_active":"2024-02-10T07:46:10.714440+0000","last_became_peered":"2024-02-10T07:46:10.714440+0000","last_unstale":"2024-02-10T07:50:08.330049+0000","last_undegraded":"2024-02-10T07:50:08.330049+0000","last_fullsized":"2024-02-10T07:50:08.330049+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:13.369607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.736901+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.369607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:29.365328+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"398'1373","reported_seq":1397,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:13.443794+0000","last_change":"2024-02-10T07:49:53.283728+0000","last_active":"2024-02-10T07:50:13.443794+0000","last_peered":"2024-02-10T07:50:13.443794+0000","last_clean":"2024-02-10T07:50:13.443794+0000","last_became_active":"2024-02-10T07:49:53.283492+0000","last_became_peered":"2024-02-10T07:49:53.283492+0000","last_unstale":"2024-02-10T07:50:13.443794+0000","last_undegraded":"2024-02-10T07:50:13.443794+0000","last_fullsized":"2024-02-10T07:50:13.443794+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1373,"log_dups_size":0,"ondisk_log_size":1373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.523341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89980928,"num_objects":1373,"num_object_clones":0,"num_object_copies":4119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1373,"num_write_kb":87872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":537,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336246+0000","last_change":"2024-02-10T07:50:07.589020+0000","last_active":"2024-02-10T07:50:08.336246+0000","last_peered":"2024-02-10T07:50:08.336246+0000","last_clean":"2024-02-10T07:50:08.336246+0000","last_became_active":"2024-02-10T07:49:06.270457+0000","last_became_peered":"2024-02-10T07:49:06.270457+0000","last_unstale":"2024-02-10T07:50:08.336246+0000","last_undegraded":"2024-02-10T07:50:08.336246+0000","last_fullsized":"2024-02-10T07:50:08.336246+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.588888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.588888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:26.055454+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":751,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330061+0000","last_change":"2024-02-10T07:49:06.267039+0000","last_active":"2024-02-10T07:50:08.330061+0000","last_peered":"2024-02-10T07:50:08.330061+0000","last_clean":"2024-02-10T07:50:08.330061+0000","last_became_active":"2024-02-10T07:49:06.266625+0000","last_became_peered":"2024-02-10T07:49:06.266625+0000","last_unstale":"2024-02-10T07:50:08.330061+0000","last_undegraded":"2024-02-10T07:50:08.330061+0000","last_fullsized":"2024-02-10T07:50:08.330061+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_clean_scrub_stamp":"2024-02-10T07:47:41.181770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1860,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:12.034664+0000","last_change":"2024-02-10T07:50:04.159154+0000","last_active":"2024-02-10T07:50:12.034664+0000","last_peered":"2024-02-10T07:50:12.034664+0000","last_clean":"2024-02-10T07:50:12.034664+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:50:12.034664+0000","last_undegraded":"2024-02-10T07:50:12.034664+0000","last_fullsized":"2024-02-10T07:50:12.034664+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:50:04.159046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.159046+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:14.041650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1095,"num_read_kb":1090,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"250'30","reported_seq":802,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.331432+0000","last_change":"2024-02-10T07:49:47.417190+0000","last_active":"2024-02-10T07:50:08.331432+0000","last_peered":"2024-02-10T07:50:08.331432+0000","last_clean":"2024-02-10T07:50:08.331432+0000","last_became_active":"2024-02-10T07:46:10.724393+0000","last_became_peered":"2024-02-10T07:46:10.724393+0000","last_unstale":"2024-02-10T07:50:08.331432+0000","last_undegraded":"2024-02-10T07:50:08.331432+0000","last_fullsized":"2024-02-10T07:50:08.331432+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:47.417096+0000","last_deep_scrub":"237'18","last_deep_scrub_stamp":"2024-02-10T07:40:31.005376+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.417096+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:05.253527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":691,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.333699+0000","last_change":"2024-02-10T07:49:52.615217+0000","last_active":"2024-02-10T07:50:08.333699+0000","last_peered":"2024-02-10T07:50:08.333699+0000","last_clean":"2024-02-10T07:50:08.333699+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:50:08.333699+0000","last_undegraded":"2024-02-10T07:50:08.333699+0000","last_fullsized":"2024-02-10T07:50:08.333699+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.615130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.615130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.239753+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"398'1495","reported_seq":1519,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:14.133795+0000","last_change":"2024-02-10T07:49:53.280610+0000","last_active":"2024-02-10T07:50:14.133795+0000","last_peered":"2024-02-10T07:50:14.133795+0000","last_clean":"2024-02-10T07:50:14.133795+0000","last_became_active":"2024-02-10T07:49:53.280272+0000","last_became_peered":"2024-02-10T07:49:53.280272+0000","last_unstale":"2024-02-10T07:50:14.133795+0000","last_undegraded":"2024-02-10T07:50:14.133795+0000","last_fullsized":"2024-02-10T07:50:14.133795+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1495,"log_dups_size":0,"ondisk_log_size":1495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:55.982947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97976320,"num_objects":1495,"num_object_clones":0,"num_object_copies":4485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1495,"num_write_kb":95680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":752,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335946+0000","last_change":"2024-02-10T07:49:59.630159+0000","last_active":"2024-02-10T07:50:08.335946+0000","last_peered":"2024-02-10T07:50:08.335946+0000","last_clean":"2024-02-10T07:50:08.335946+0000","last_became_active":"2024-02-10T07:46:10.716852+0000","last_became_peered":"2024-02-10T07:46:10.716852+0000","last_unstale":"2024-02-10T07:50:08.335946+0000","last_undegraded":"2024-02-10T07:50:08.335946+0000","last_fullsized":"2024-02-10T07:50:08.335946+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.630083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.630083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:11.435808+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":757,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336791+0000","last_change":"2024-02-10T07:49:05.193351+0000","last_active":"2024-02-10T07:50:08.336791+0000","last_peered":"2024-02-10T07:50:08.336791+0000","last_clean":"2024-02-10T07:50:08.336791+0000","last_became_active":"2024-02-10T07:46:10.718340+0000","last_became_peered":"2024-02-10T07:46:10.718340+0000","last_unstale":"2024-02-10T07:50:08.336791+0000","last_undegraded":"2024-02-10T07:50:08.336791+0000","last_fullsized":"2024-02-10T07:50:08.336791+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:05.193158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:49:05.193158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:31.087181+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"271'37","reported_seq":808,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330636+0000","last_change":"2024-02-10T07:49:57.522503+0000","last_active":"2024-02-10T07:50:08.330636+0000","last_peered":"2024-02-10T07:50:08.330636+0000","last_clean":"2024-02-10T07:50:08.330636+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:50:08.330636+0000","last_undegraded":"2024-02-10T07:50:08.330636+0000","last_fullsized":"2024-02-10T07:50:08.330636+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'37","last_scrub_stamp":"2024-02-10T07:49:57.522391+0000","last_deep_scrub":"271'37","last_deep_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.522391+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:18.289907+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":766,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329669+0000","last_change":"2024-02-10T07:49:03.013730+0000","last_active":"2024-02-10T07:50:08.329669+0000","last_peered":"2024-02-10T07:50:08.329669+0000","last_clean":"2024-02-10T07:50:08.329669+0000","last_became_active":"2024-02-10T07:48:35.062118+0000","last_became_peered":"2024-02-10T07:48:35.062118+0000","last_unstale":"2024-02-10T07:50:08.329669+0000","last_undegraded":"2024-02-10T07:50:08.329669+0000","last_fullsized":"2024-02-10T07:50:08.329669+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:03.013625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:00.292320+0000","last_clean_scrub_stamp":"2024-02-10T07:49:03.013625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:23.398722+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"398'1433","reported_seq":1457,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:13.459373+0000","last_change":"2024-02-10T07:49:53.280094+0000","last_active":"2024-02-10T07:50:13.459373+0000","last_peered":"2024-02-10T07:50:13.459373+0000","last_clean":"2024-02-10T07:50:13.459373+0000","last_became_active":"2024-02-10T07:49:53.279765+0000","last_became_peered":"2024-02-10T07:49:53.279765+0000","last_unstale":"2024-02-10T07:50:13.459373+0000","last_undegraded":"2024-02-10T07:50:13.459373+0000","last_fullsized":"2024-02-10T07:50:13.459373+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1433,"log_dups_size":0,"ondisk_log_size":1433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:54.175208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93913088,"num_objects":1433,"num_object_clones":0,"num_object_copies":4299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1433,"num_write_kb":91712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":627,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.328745+0000","last_change":"2024-02-10T07:49:57.848098+0000","last_active":"2024-02-10T07:50:08.328745+0000","last_peered":"2024-02-10T07:50:08.328745+0000","last_clean":"2024-02-10T07:50:08.328745+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:50:08.328745+0000","last_undegraded":"2024-02-10T07:50:08.328745+0000","last_fullsized":"2024-02-10T07:50:08.328745+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:57.847989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.847989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:26.850826+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":666,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330064+0000","last_change":"2024-02-10T07:49:48.809609+0000","last_active":"2024-02-10T07:50:08.330064+0000","last_peered":"2024-02-10T07:50:08.330064+0000","last_clean":"2024-02-10T07:50:08.330064+0000","last_became_active":"2024-02-10T07:49:06.256488+0000","last_became_peered":"2024-02-10T07:49:06.256488+0000","last_unstale":"2024-02-10T07:50:08.330064+0000","last_undegraded":"2024-02-10T07:50:08.330064+0000","last_fullsized":"2024-02-10T07:50:08.330064+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:48.809513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:48.809513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:03.569894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"382'25","reported_seq":554,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336218+0000","last_change":"2024-02-10T07:49:19.084010+0000","last_active":"2024-02-10T07:50:08.336218+0000","last_peered":"2024-02-10T07:50:08.336218+0000","last_clean":"2024-02-10T07:50:08.336218+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:50:08.336218+0000","last_undegraded":"2024-02-10T07:50:08.336218+0000","last_fullsized":"2024-02-10T07:50:08.336218+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'25","last_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_deep_scrub":"382'25","last_deep_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.083856+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:34.153113+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"158'10","reported_seq":975,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330859+0000","last_change":"2024-02-10T07:49:50.396822+0000","last_active":"2024-02-10T07:50:08.330859+0000","last_peered":"2024-02-10T07:50:08.330859+0000","last_clean":"2024-02-10T07:50:08.330859+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:50:08.330859+0000","last_undegraded":"2024-02-10T07:50:08.330859+0000","last_fullsized":"2024-02-10T07:50:08.330859+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:49:50.396730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.396730+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:52.711726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.e","version":"398'1399","reported_seq":1423,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:13.457047+0000","last_change":"2024-02-10T07:49:53.281229+0000","last_active":"2024-02-10T07:50:13.457047+0000","last_peered":"2024-02-10T07:50:13.457047+0000","last_clean":"2024-02-10T07:50:13.457047+0000","last_became_active":"2024-02-10T07:49:53.280920+0000","last_became_peered":"2024-02-10T07:49:53.280920+0000","last_unstale":"2024-02-10T07:50:13.457047+0000","last_undegraded":"2024-02-10T07:50:13.457047+0000","last_fullsized":"2024-02-10T07:50:13.457047+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1399,"log_dups_size":0,"ondisk_log_size":1399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:05.553881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91684864,"num_objects":1399,"num_object_clones":0,"num_object_copies":4197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1399,"num_write_kb":89536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":738,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329815+0000","last_change":"2024-02-10T07:48:23.271762+0000","last_active":"2024-02-10T07:50:08.329815+0000","last_peered":"2024-02-10T07:50:08.329815+0000","last_clean":"2024-02-10T07:50:08.329815+0000","last_became_active":"2024-02-10T07:46:10.719534+0000","last_became_peered":"2024-02-10T07:46:10.719534+0000","last_unstale":"2024-02-10T07:50:08.329815+0000","last_undegraded":"2024-02-10T07:50:08.329815+0000","last_fullsized":"2024-02-10T07:50:08.329815+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_clean_scrub_stamp":"2024-02-10T07:48:23.271454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"296'2","reported_seq":754,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.331499+0000","last_change":"2024-02-10T07:50:03.643473+0000","last_active":"2024-02-10T07:50:08.331499+0000","last_peered":"2024-02-10T07:50:08.331499+0000","last_clean":"2024-02-10T07:50:08.331499+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:50:08.331499+0000","last_undegraded":"2024-02-10T07:50:08.331499+0000","last_fullsized":"2024-02-10T07:50:08.331499+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'2","last_scrub_stamp":"2024-02-10T07:50:03.643367+0000","last_deep_scrub":"296'2","last_deep_scrub_stamp":"2024-02-10T07:45:03.233375+0000","last_clean_scrub_stamp":"2024-02-10T07:50:03.643367+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:15.296940+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"250'28","reported_seq":783,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336078+0000","last_change":"2024-02-10T07:49:20.216865+0000","last_active":"2024-02-10T07:50:08.336078+0000","last_peered":"2024-02-10T07:50:08.336078+0000","last_clean":"2024-02-10T07:50:08.336078+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:50:08.336078+0000","last_undegraded":"2024-02-10T07:50:08.336078+0000","last_fullsized":"2024-02-10T07:50:08.336078+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-02-10T07:49:20.216750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:20.216750+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:24.176703+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"376'12","reported_seq":762,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336040+0000","last_change":"2024-02-10T07:49:06.282456+0000","last_active":"2024-02-10T07:50:08.336040+0000","last_peered":"2024-02-10T07:50:08.336040+0000","last_clean":"2024-02-10T07:50:08.336040+0000","last_became_active":"2024-02-10T07:49:06.255324+0000","last_became_peered":"2024-02-10T07:49:06.255324+0000","last_unstale":"2024-02-10T07:50:08.336040+0000","last_undegraded":"2024-02-10T07:50:08.336040+0000","last_fullsized":"2024-02-10T07:50:08.336040+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'11","last_scrub_stamp":"2024-02-10T07:48:58.368663+0000","last_deep_scrub":"227'9","last_deep_scrub_stamp":"2024-02-10T07:39:53.440221+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.368663+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.583316+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"398'1391","reported_seq":1415,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:13.332665+0000","last_change":"2024-02-10T07:49:53.283453+0000","last_active":"2024-02-10T07:50:13.332665+0000","last_peered":"2024-02-10T07:50:13.332665+0000","last_clean":"2024-02-10T07:50:13.332665+0000","last_became_active":"2024-02-10T07:49:53.283144+0000","last_became_peered":"2024-02-10T07:49:53.283144+0000","last_unstale":"2024-02-10T07:50:13.332665+0000","last_undegraded":"2024-02-10T07:50:13.332665+0000","last_fullsized":"2024-02-10T07:50:13.332665+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1391,"log_dups_size":0,"ondisk_log_size":1391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:09.376120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91160576,"num_objects":1391,"num_object_clones":0,"num_object_copies":4173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1391,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":735,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336534+0000","last_change":"2024-02-10T07:49:06.252967+0000","last_active":"2024-02-10T07:50:08.336534+0000","last_peered":"2024-02-10T07:50:08.336534+0000","last_clean":"2024-02-10T07:50:08.336534+0000","last_became_active":"2024-02-10T07:49:06.251750+0000","last_became_peered":"2024-02-10T07:49:06.251750+0000","last_unstale":"2024-02-10T07:50:08.336534+0000","last_undegraded":"2024-02-10T07:50:08.336534+0000","last_fullsized":"2024-02-10T07:50:08.336534+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:27.819612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:46:27.819612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":226,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330374+0000","last_change":"2024-02-10T07:47:50.948710+0000","last_active":"2024-02-10T07:50:08.330374+0000","last_peered":"2024-02-10T07:50:08.330374+0000","last_clean":"2024-02-10T07:50:08.330374+0000","last_became_active":"2024-02-10T07:47:47.966443+0000","last_became_peered":"2024-02-10T07:47:47.966443+0000","last_unstale":"2024-02-10T07:50:08.330374+0000","last_undegraded":"2024-02-10T07:50:08.330374+0000","last_fullsized":"2024-02-10T07:50:08.330374+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.948624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.948624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"250'30","reported_seq":725,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.328906+0000","last_change":"2024-02-10T07:49:57.303310+0000","last_active":"2024-02-10T07:50:08.328906+0000","last_peered":"2024-02-10T07:50:08.328906+0000","last_clean":"2024-02-10T07:50:08.328906+0000","last_became_active":"2024-02-10T07:49:06.252935+0000","last_became_peered":"2024-02-10T07:49:06.252935+0000","last_unstale":"2024-02-10T07:50:08.328906+0000","last_undegraded":"2024-02-10T07:50:08.328906+0000","last_fullsized":"2024-02-10T07:50:08.328906+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:57.303200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.303200+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:15.856547+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":743,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335830+0000","last_change":"2024-02-10T07:49:06.239655+0000","last_active":"2024-02-10T07:50:08.335830+0000","last_peered":"2024-02-10T07:50:08.335830+0000","last_clean":"2024-02-10T07:50:08.335830+0000","last_became_active":"2024-02-10T07:49:06.239190+0000","last_became_peered":"2024-02-10T07:49:06.239190+0000","last_unstale":"2024-02-10T07:50:08.335830+0000","last_undegraded":"2024-02-10T07:50:08.335830+0000","last_fullsized":"2024-02-10T07:50:08.335830+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:42.549018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.549018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"398'1523","reported_seq":1547,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:14.131485+0000","last_change":"2024-02-10T07:49:53.282174+0000","last_active":"2024-02-10T07:50:14.131485+0000","last_peered":"2024-02-10T07:50:14.131485+0000","last_clean":"2024-02-10T07:50:14.131485+0000","last_became_active":"2024-02-10T07:49:53.281832+0000","last_became_peered":"2024-02-10T07:49:53.281832+0000","last_unstale":"2024-02-10T07:50:14.131485+0000","last_undegraded":"2024-02-10T07:50:14.131485+0000","last_fullsized":"2024-02-10T07:50:14.131485+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1523,"log_dups_size":0,"ondisk_log_size":1523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:08.492345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99811328,"num_objects":1523,"num_object_clones":0,"num_object_copies":4569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1523,"num_write_kb":97472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":250,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336419+0000","last_change":"2024-02-10T07:49:06.274374+0000","last_active":"2024-02-10T07:50:08.336419+0000","last_peered":"2024-02-10T07:50:08.336419+0000","last_clean":"2024-02-10T07:50:08.336419+0000","last_became_active":"2024-02-10T07:48:02.371926+0000","last_became_peered":"2024-02-10T07:48:02.371926+0000","last_unstale":"2024-02-10T07:50:08.336419+0000","last_undegraded":"2024-02-10T07:50:08.336419+0000","last_fullsized":"2024-02-10T07:50:08.336419+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.274263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:49.704024+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.274263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:09.103161+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"395'12","reported_seq":110,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330081+0000","last_change":"2024-02-10T07:50:02.810307+0000","last_active":"2024-02-10T07:50:08.330081+0000","last_peered":"2024-02-10T07:50:08.330081+0000","last_clean":"2024-02-10T07:50:08.330081+0000","last_became_active":"2024-02-10T07:48:35.071916+0000","last_became_peered":"2024-02-10T07:48:35.071916+0000","last_unstale":"2024-02-10T07:50:08.330081+0000","last_undegraded":"2024-02-10T07:50:08.330081+0000","last_fullsized":"2024-02-10T07:50:08.330081+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'12","last_scrub_stamp":"2024-02-10T07:50:02.810229+0000","last_deep_scrub":"395'12","last_deep_scrub_stamp":"2024-02-10T07:50:02.810229+0000","last_clean_scrub_stamp":"2024-02-10T07:50:02.810229+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:28.369227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":63,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.334942+0000","last_change":"2024-02-10T07:49:44.962126+0000","last_active":"2024-02-10T07:50:08.334942+0000","last_peered":"2024-02-10T07:50:08.334942+0000","last_clean":"2024-02-10T07:50:08.334942+0000","last_became_active":"2024-02-10T07:49:06.251038+0000","last_became_peered":"2024-02-10T07:49:06.251038+0000","last_unstale":"2024-02-10T07:50:08.334942+0000","last_undegraded":"2024-02-10T07:50:08.334942+0000","last_fullsized":"2024-02-10T07:50:08.334942+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:44.962041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:44.962041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:12.429908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"250'27","reported_seq":798,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.339314+0000","last_change":"2024-02-10T07:50:00.655466+0000","last_active":"2024-02-10T07:50:08.339314+0000","last_peered":"2024-02-10T07:50:08.339314+0000","last_clean":"2024-02-10T07:50:08.339314+0000","last_became_active":"2024-02-10T07:47:47.967427+0000","last_became_peered":"2024-02-10T07:47:47.967427+0000","last_unstale":"2024-02-10T07:50:08.339314+0000","last_undegraded":"2024-02-10T07:50:08.339314+0000","last_fullsized":"2024-02-10T07:50:08.339314+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:50:00.655377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.655377+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:26.934415+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"398'1505","reported_seq":1529,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:13.444141+0000","last_change":"2024-02-10T07:49:53.283708+0000","last_active":"2024-02-10T07:50:13.444141+0000","last_peered":"2024-02-10T07:50:13.444141+0000","last_clean":"2024-02-10T07:50:13.444141+0000","last_became_active":"2024-02-10T07:49:53.283415+0000","last_became_peered":"2024-02-10T07:49:53.283415+0000","last_unstale":"2024-02-10T07:50:13.444141+0000","last_undegraded":"2024-02-10T07:50:13.444141+0000","last_fullsized":"2024-02-10T07:50:13.444141+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.247094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98631680,"num_objects":1505,"num_object_clones":0,"num_object_copies":4515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1505,"num_write_kb":96320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":437,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.328599+0000","last_change":"2024-02-10T07:50:00.799049+0000","last_active":"2024-02-10T07:50:08.328599+0000","last_peered":"2024-02-10T07:50:08.328599+0000","last_clean":"2024-02-10T07:50:08.328599+0000","last_became_active":"2024-02-10T07:49:06.268874+0000","last_became_peered":"2024-02-10T07:49:06.268874+0000","last_unstale":"2024-02-10T07:50:08.328599+0000","last_undegraded":"2024-02-10T07:50:08.328599+0000","last_fullsized":"2024-02-10T07:50:08.328599+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.798928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.798928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:06.248341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":224,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.337834+0000","last_change":"2024-02-10T07:49:57.310447+0000","last_active":"2024-02-10T07:50:08.337834+0000","last_peered":"2024-02-10T07:50:08.337834+0000","last_clean":"2024-02-10T07:50:08.337834+0000","last_became_active":"2024-02-10T07:48:02.333689+0000","last_became_peered":"2024-02-10T07:48:02.333689+0000","last_unstale":"2024-02-10T07:50:08.337834+0000","last_undegraded":"2024-02-10T07:50:08.337834+0000","last_fullsized":"2024-02-10T07:50:08.337834+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:57.310333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.310333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:22.249646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":742,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.335927+0000","last_change":"2024-02-10T07:46:37.995480+0000","last_active":"2024-02-10T07:50:08.335927+0000","last_peered":"2024-02-10T07:50:08.335927+0000","last_clean":"2024-02-10T07:50:08.335927+0000","last_became_active":"2024-02-10T07:46:10.721886+0000","last_became_peered":"2024-02-10T07:46:10.721886+0000","last_unstale":"2024-02-10T07:50:08.335927+0000","last_undegraded":"2024-02-10T07:50:08.335927+0000","last_fullsized":"2024-02-10T07:50:08.335927+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:37.995362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.995362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"250'24","reported_seq":822,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336377+0000","last_change":"2024-02-10T07:49:16.249257+0000","last_active":"2024-02-10T07:50:08.336377+0000","last_peered":"2024-02-10T07:50:08.336377+0000","last_clean":"2024-02-10T07:50:08.336377+0000","last_became_active":"2024-02-10T07:48:35.062573+0000","last_became_peered":"2024-02-10T07:48:35.062573+0000","last_unstale":"2024-02-10T07:50:08.336377+0000","last_undegraded":"2024-02-10T07:50:08.336377+0000","last_fullsized":"2024-02-10T07:50:08.336377+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:49:16.248936+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.248936+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:25.863763+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"398'1518","reported_seq":1542,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:14.205145+0000","last_change":"2024-02-10T07:49:53.285110+0000","last_active":"2024-02-10T07:50:14.205145+0000","last_peered":"2024-02-10T07:50:14.205145+0000","last_clean":"2024-02-10T07:50:14.205145+0000","last_became_active":"2024-02-10T07:49:53.284807+0000","last_became_peered":"2024-02-10T07:49:53.284807+0000","last_unstale":"2024-02-10T07:50:14.205145+0000","last_undegraded":"2024-02-10T07:50:14.205145+0000","last_fullsized":"2024-02-10T07:50:14.205145+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:05.874090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99483648,"num_objects":1518,"num_object_clones":0,"num_object_copies":4554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1518,"num_write_kb":97152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":63,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336099+0000","last_change":"2024-02-10T07:49:17.153443+0000","last_active":"2024-02-10T07:50:08.336099+0000","last_peered":"2024-02-10T07:50:08.336099+0000","last_clean":"2024-02-10T07:50:08.336099+0000","last_became_active":"2024-02-10T07:49:06.256104+0000","last_became_peered":"2024-02-10T07:49:06.256104+0000","last_unstale":"2024-02-10T07:50:08.336099+0000","last_undegraded":"2024-02-10T07:50:08.336099+0000","last_fullsized":"2024-02-10T07:50:08.336099+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.153368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.153368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:27.413097+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":734,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.337843+0000","last_change":"2024-02-10T07:49:07.170105+0000","last_active":"2024-02-10T07:50:08.337843+0000","last_peered":"2024-02-10T07:50:08.337843+0000","last_clean":"2024-02-10T07:50:08.337843+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:50:08.337843+0000","last_undegraded":"2024-02-10T07:50:08.337843+0000","last_fullsized":"2024-02-10T07:50:08.337843+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:07.169962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:07.169962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:27.005819+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":767,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329760+0000","last_change":"2024-02-10T07:50:03.803447+0000","last_active":"2024-02-10T07:50:08.329760+0000","last_peered":"2024-02-10T07:50:08.329760+0000","last_clean":"2024-02-10T07:50:08.329760+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:50:08.329760+0000","last_undegraded":"2024-02-10T07:50:08.329760+0000","last_fullsized":"2024-02-10T07:50:08.329760+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:03.803361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:03.803361+0000","last_clean_scrub_stamp":"2024-02-10T07:50:03.803361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:28.611872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"250'39","reported_seq":837,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336355+0000","last_change":"2024-02-10T07:48:59.066379+0000","last_active":"2024-02-10T07:50:08.336355+0000","last_peered":"2024-02-10T07:50:08.336355+0000","last_clean":"2024-02-10T07:50:08.336355+0000","last_became_active":"2024-02-10T07:48:35.062134+0000","last_became_peered":"2024-02-10T07:48:35.062134+0000","last_unstale":"2024-02-10T07:50:08.336355+0000","last_undegraded":"2024-02-10T07:50:08.336355+0000","last_fullsized":"2024-02-10T07:50:08.336355+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T07:48:59.066256+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.066256+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"398'1405","reported_seq":1429,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:13.429424+0000","last_change":"2024-02-10T07:49:53.281316+0000","last_active":"2024-02-10T07:50:13.429424+0000","last_peered":"2024-02-10T07:50:13.429424+0000","last_clean":"2024-02-10T07:50:13.429424+0000","last_became_active":"2024-02-10T07:49:53.280975+0000","last_became_peered":"2024-02-10T07:49:53.280975+0000","last_unstale":"2024-02-10T07:50:13.429424+0000","last_undegraded":"2024-02-10T07:50:13.429424+0000","last_fullsized":"2024-02-10T07:50:13.429424+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":1405,"log_dups_size":0,"ondisk_log_size":1405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:54.358771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92078080,"num_objects":1405,"num_object_clones":0,"num_object_copies":4215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1405,"num_write_kb":89920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":730,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336551+0000","last_change":"2024-02-10T07:49:47.275981+0000","last_active":"2024-02-10T07:50:08.336551+0000","last_peered":"2024-02-10T07:50:08.336551+0000","last_clean":"2024-02-10T07:50:08.336551+0000","last_became_active":"2024-02-10T07:46:10.717081+0000","last_became_peered":"2024-02-10T07:46:10.717081+0000","last_unstale":"2024-02-10T07:50:08.336551+0000","last_undegraded":"2024-02-10T07:50:08.336551+0000","last_fullsized":"2024-02-10T07:50:08.336551+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:47.275882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.275882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:47.745153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"313'11","reported_seq":279,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.329691+0000","last_change":"2024-02-10T07:49:14.335191+0000","last_active":"2024-02-10T07:50:08.329691+0000","last_peered":"2024-02-10T07:50:08.329691+0000","last_clean":"2024-02-10T07:50:08.329691+0000","last_became_active":"2024-02-10T07:46:10.719380+0000","last_became_peered":"2024-02-10T07:46:10.719380+0000","last_unstale":"2024-02-10T07:50:08.329691+0000","last_undegraded":"2024-02-10T07:50:08.329691+0000","last_fullsized":"2024-02-10T07:50:08.329691+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'11","last_scrub_stamp":"2024-02-10T07:49:14.335064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:14.335064+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:32.784071+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":233,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.330729+0000","last_change":"2024-02-10T07:49:06.259646+0000","last_active":"2024-02-10T07:50:08.330729+0000","last_peered":"2024-02-10T07:50:08.330729+0000","last_clean":"2024-02-10T07:50:08.330729+0000","last_became_active":"2024-02-10T07:49:06.258926+0000","last_became_peered":"2024-02-10T07:49:06.258926+0000","last_unstale":"2024-02-10T07:50:08.330729+0000","last_undegraded":"2024-02-10T07:50:08.330729+0000","last_fullsized":"2024-02-10T07:50:08.330729+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:24.220575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_clean_scrub_stamp":"2024-02-10T07:47:24.220575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"250'20","reported_seq":801,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-02-10T07:50:08.336061+0000","last_change":"2024-02-10T07:49:06.289632+0000","last_active":"2024-02-10T07:50:08.336061+0000","last_peered":"2024-02-10T07:50:08.336061+0000","last_clean":"2024-02-10T07:50:08.336061+0000","last_became_active":"2024-02-10T07:49:06.238858+0000","last_became_peered":"2024-02-10T07:49:06.238858+0000","last_unstale":"2024-02-10T07:50:08.336061+0000","last_undegraded":"2024-02-10T07:50:08.336061+0000","last_fullsized":"2024-02-10T07:50:08.336061+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:48:15.588199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.588199+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1503526912,"num_objects":22942,"num_object_clones":0,"num_object_copies":68826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22942,"num_write_kb":1468288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4397072384,"data_stored":4397072384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22942,"ondisk_log_size":22942,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1095,"num_read_kb":1090,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":5804,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":11904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":773,"ondisk_log_size":773,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037985,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1356860,"kb_used_data":716212,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":92388292,"statfs":{"total":95995035648,"available":94605611008,"internally_reserved":0,"allocated":733401088,"data_stored":733064793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":656013770},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:49:17 2024","interfaces":[{"interface":"back","average":{"1min":2.7280000000000002,"5min":1.4690000000000001,"15min":1.9770000000000001},"min":{"1min":0.254,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":39.494,"5min":39.494,"15min":52.591999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":2.6549999999999998,"5min":1.395,"15min":1.839},"min":{"1min":0.26100000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":38.524000000000001,"5min":38.524000000000001,"15min":52.494},"last":0.97699999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:49:17 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":1.2549999999999999,"15min":1.6870000000000001},"min":{"1min":0.314,"5min":0.25700000000000001,"15min":0.24399999999999999},"max":{"1min":2.7869999999999999,"5min":17.765000000000001,"15min":35.459000000000003},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.1990000000000001,"15min":1.6759999999999999},"min":{"1min":0.29899999999999999,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":4.0750000000000002,"5min":21.829999999999998,"15min":35.225000000000001},"last":1.1739999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:49:17 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.93500000000000005,"15min":1.228},"min":{"1min":0.27700000000000002,"5min":0.255,"15min":0.255},"max":{"1min":3.9140000000000001,"5min":5.4980000000000002,"15min":19.710000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.876,"5min":0.97799999999999998,"15min":1.3640000000000001},"min":{"1min":0.34799999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.8340000000000001,"5min":4.2320000000000002,"15min":19.643000000000001},"last":0.76300000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:49:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.103,"15min":1.069},"min":{"1min":0.33000000000000002,"5min":0.307,"15min":0.307},"max":{"1min":4.0990000000000002,"5min":4.0990000000000002,"15min":4.0990000000000002},"last":0.88},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.92300000000000004,"15min":0.92200000000000004},"min":{"1min":0.33400000000000002,"5min":0.313,"15min":0.313},"max":{"1min":3.9580000000000002,"5min":3.9580000000000002,"15min":3.9580000000000002},"last":0.44700000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:49:17 2024","interfaces":[{"interface":"back","average":{"1min":3.4910000000000001,"5min":1.9039999999999999,"15min":1.0860000000000001},"min":{"1min":0.41899999999999998,"5min":0.22,"15min":0.22},"max":{"1min":26.001000000000001,"5min":26.001000000000001,"15min":26.001000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":3.895,"5min":2.0289999999999999,"15min":1.107},"min":{"1min":0.313,"5min":0.26800000000000002,"15min":0.24099999999999999},"max":{"1min":27.09,"5min":27.09,"15min":27.09},"last":0.39300000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.47,"15min":1.052},"min":{"1min":0.47299999999999998,"5min":0.26600000000000001,"15min":0.245},"max":{"1min":3.5489999999999999,"5min":18.126999999999999,"15min":18.126999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.7929999999999999,"5min":1.6339999999999999,"15min":1.149},"min":{"1min":0.52900000000000003,"5min":0.318,"15min":0.29999999999999999},"max":{"1min":5.8490000000000002,"5min":19.548999999999999,"15min":19.548999999999999},"last":0.93100000000000005}]},{"osd":6,"last update":"Sat Feb 10 07:49:17 2024","interfaces":[{"interface":"back","average":{"1min":1.956,"5min":1.5720000000000001,"15min":1.0609999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.28199999999999997},"max":{"1min":6.9770000000000003,"5min":17.832000000000001,"15min":17.832000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":2.161,"5min":1.891,"15min":1.1379999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.27400000000000002},"max":{"1min":6.101,"5min":20.449999999999999,"15min":20.449999999999999},"last":0.42199999999999999}]}]},{"osd":6,"up_from":50,"seq":214748365030,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1227576,"kb_used_data":686896,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92517576,"statfs":{"total":95995035648,"available":94737997824,"internally_reserved":0,"allocated":703381504,"data_stored":703049664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":553646534},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:49:43 2024","interfaces":[{"interface":"back","average":{"1min":2.6349999999999998,"5min":1.865,"15min":1.8069999999999999},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":19.484999999999999,"5min":24.727,"15min":63.405999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":1.3240000000000001,"15min":1.77},"min":{"1min":0.25600000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.3929999999999998,"5min":29.084,"15min":67.748000000000005},"last":1.0089999999999999}]},{"osd":1,"last update":"Sat Feb 10 07:49:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":1.1719999999999999,"15min":1.631},"min":{"1min":0.307,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.4910000000000001,"5min":11.406000000000001,"15min":37.659999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.9470000000000001,"5min":1.6200000000000001,"15min":1.79},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.24399999999999999},"max":{"1min":19.524999999999999,"5min":19.524999999999999,"15min":37.75},"last":0.875}]},{"osd":2,"last update":"Sat Feb 10 07:49:43 2024","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":1.4570000000000001,"15min":1.6779999999999999},"min":{"1min":0.34899999999999998,"5min":0.307,"15min":0.253},"max":{"1min":15.768000000000001,"5min":15.768000000000001,"15min":65.278999999999996},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.629,"5min":1.2190000000000001,"15min":1.645},"min":{"1min":0.36899999999999999,"5min":0.30199999999999999,"15min":0.25800000000000001},"max":{"1min":15.728999999999999,"5min":15.728999999999999,"15min":65.265000000000001},"last":0.72799999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.5600000000000001,"15min":1.6899999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.4489999999999998,"5min":13.279,"15min":13.279},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.1339999999999999,"15min":1.208},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.5169999999999999,"5min":5.6470000000000002,"15min":5.6470000000000002},"last":0.43099999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:49:43 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.3660000000000001,"15min":0.877},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.23400000000000001},"max":{"1min":2.8039999999999998,"5min":7.6509999999999998,"15min":7.7839999999999998},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":1.3340000000000001,"15min":0.84999999999999998},"min":{"1min":0.34799999999999998,"5min":0.28000000000000003,"15min":0.26000000000000001},"max":{"1min":2.1930000000000001,"5min":7.0300000000000002,"15min":7.0300000000000002},"last":1.145}]},{"osd":5,"last update":"Sat Feb 10 07:49:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.879,"15min":1.117},"min":{"1min":0.317,"5min":0.26000000000000001,"15min":0.24099999999999999},"max":{"1min":2.5680000000000001,"5min":44.689,"15min":44.689},"last":1.2569999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":2.0609999999999999,"15min":1.181},"min":{"1min":0.309,"5min":0.309,"15min":0.29199999999999998},"max":{"1min":3.5699999999999998,"5min":44.465000000000003,"15min":44.465000000000003},"last":0.86399999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:50:09 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":2.2730000000000001,"15min":1.2869999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.28100000000000003},"max":{"1min":4.7750000000000004,"5min":42.631999999999998,"15min":42.631999999999998},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":1.9950000000000001,"15min":1.1859999999999999},"min":{"1min":0.43099999999999999,"5min":0.32800000000000001,"15min":0.29699999999999999},"max":{"1min":1.6819999999999999,"5min":44.792999999999999,"15min":44.792999999999999},"last":0.51700000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575104,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1575872,"kb_used_data":926904,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92169280,"statfs":{"total":95995035648,"available":94381342720,"internally_reserved":0,"allocated":949149696,"data_stored":948808505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":664533446},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:49:45 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.57599999999999996,"15min":1.7649999999999999},"min":{"1min":0.24199999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.88700000000000001,"5min":5.3799999999999999,"15min":114.27},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.67700000000000005,"15min":1.9670000000000001},"min":{"1min":0.26800000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.823,"5min":7.3470000000000004,"15min":119.499},"last":0.64100000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:50:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.46400000000000002,"15min":1.1639999999999999},"min":{"1min":0.253,"5min":0.219,"15min":0.19600000000000001},"max":{"1min":0.93400000000000005,"5min":5.9909999999999997,"15min":8.9209999999999994},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.46300000000000002,"15min":1.2909999999999999},"min":{"1min":0.28999999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.53600000000000003,"5min":7.3879999999999999,"15min":8.8300000000000001},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Feb 10 07:49:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":0.67400000000000004,"15min":0.55000000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.367999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.4710000000000001,"5min":0.60299999999999998,"15min":0.45900000000000002},"min":{"1min":0.29399999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":22.199999999999999,"5min":22.199999999999999,"15min":22.199999999999999},"last":0.57599999999999996}]},{"osd":4,"last update":"Sat Feb 10 07:50:03 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.4399999999999999,"15min":2.0939999999999999},"min":{"1min":0.34499999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.294,"5min":31.908999999999999,"15min":207.45699999999999},"last":0.379},{"interface":"front","average":{"1min":0.78400000000000003,"5min":1.3959999999999999,"15min":1.294},"min":{"1min":0.34699999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":2.528,"5min":32.201000000000001,"15min":32.201000000000001},"last":0.68300000000000005}]},{"osd":5,"last update":"Sat Feb 10 07:50:03 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.4530000000000001,"15min":1.9450000000000001},"min":{"1min":0.34200000000000003,"5min":0.308,"15min":0.308},"max":{"1min":2.4910000000000001,"5min":31.824999999999999,"15min":98.703000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.79200000000000004,"5min":1.5089999999999999,"15min":1.923},"min":{"1min":0.39700000000000002,"5min":0.38200000000000001,"15min":0.30199999999999999},"max":{"1min":2.4740000000000002,"5min":31.841999999999999,"15min":98.664000000000001},"last":0.60799999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.6419999999999999,"15min":1.4139999999999999},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.308},"max":{"1min":4.9669999999999996,"5min":36.808,"15min":36.808},"last":1.381},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.6459999999999999,"15min":1.581},"min":{"1min":0.40300000000000002,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":2.4929999999999999,"5min":33.774999999999999,"15min":33.774999999999999},"last":1.8919999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:50:13 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.95799999999999996,"15min":1.381},"min":{"1min":0.435,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":1.8,"5min":7.444,"15min":8.4239999999999995},"last":1.244},{"interface":"front","average":{"1min":0.749,"5min":1.016,"15min":1.3009999999999999},"min":{"1min":0.41899999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.3490000000000002,"5min":6.2089999999999996,"15min":11.212},"last":0.82999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738629,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1216708,"kb_used_data":553852,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92528444,"statfs":{"total":95995035648,"available":94749126656,"internally_reserved":0,"allocated":567144448,"data_stored":566800578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:49:36 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":1.1850000000000001,"15min":2.3399999999999999},"min":{"1min":0.22800000000000001,"5min":0.187,"15min":0.182},"max":{"1min":2.2229999999999999,"5min":34.661999999999999,"15min":156.16200000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.46500000000000002,"5min":1.202,"15min":2.3359999999999999},"min":{"1min":0.22,"5min":0.192,"15min":0.183},"max":{"1min":1.202,"5min":34.636000000000003,"15min":156.941},"last":0.76800000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:49:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.97099999999999997,"15min":1.609},"min":{"1min":0.221,"5min":0.21299999999999999,"15min":0.20000000000000001},"max":{"1min":3.117,"5min":34.603000000000002,"15min":34.603000000000002},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.97599999999999998,"15min":1.3939999999999999},"min":{"1min":0.31,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.476,"5min":34.718000000000004,"15min":34.718000000000004},"last":0.71899999999999997}]},{"osd":3,"last update":"Sat Feb 10 07:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.61899999999999999,"15min":0.628},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.2589999999999999,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.60399999999999998,"15min":0.61699999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3089999999999999,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":0.85399999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.6619999999999999,"15min":1.3420000000000001},"min":{"1min":0.44700000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.4329999999999998,"5min":37.286000000000001,"15min":37.286000000000001},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.444,"5min":1.6479999999999999,"15min":1.4730000000000001},"min":{"1min":0.38100000000000001,"5min":0.315,"15min":0.26900000000000002},"max":{"1min":5.1609999999999996,"5min":34.683,"15min":34.683},"last":1.1739999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":2.5489999999999999,"15min":2.25},"min":{"1min":0.46100000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":8.0239999999999991,"5min":41.429000000000002,"15min":41.429000000000002},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.48,"5min":2.5230000000000001,"15min":2.387},"min":{"1min":0.442,"5min":0.33500000000000002,"15min":0.308},"max":{"1min":8.0060000000000002,"5min":41.456000000000003,"15min":41.456000000000003},"last":1.099}]},{"osd":6,"last update":"Sat Feb 10 07:49:36 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":2.202,"15min":1.5660000000000001},"min":{"1min":0.40899999999999997,"5min":0.35399999999999998,"15min":0.29599999999999999},"max":{"1min":2.585,"5min":41.587000000000003,"15min":41.587000000000003},"last":2.8359999999999999},{"interface":"front","average":{"1min":1.337,"5min":2.282,"15min":1.7150000000000001},"min":{"1min":0.48099999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":4.6509999999999998,"5min":41.256,"15min":41.256},"last":1.9079999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:49:16 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.7350000000000001,"15min":1.6359999999999999},"min":{"1min":0.51300000000000001,"5min":0.34899999999999998,"15min":0.33300000000000002},"max":{"1min":5.0549999999999997,"5min":34.710999999999999,"15min":34.710999999999999},"last":1.145},{"interface":"front","average":{"1min":1.512,"5min":1.611,"15min":1.4259999999999999},"min":{"1min":0.45800000000000002,"5min":0.38500000000000001,"15min":0.376},"max":{"1min":3.5670000000000002,"5min":34.997999999999998,"15min":34.997999999999998},"last":0.98299999999999998}]}]},{"osd":2,"up_from":0,"seq":77309411578,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":347,"seq":1490353651742,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":594756,"kb_used_data":461564,"kb_used_omap":5,"kb_used_meta":133178,"kb_avail":93150396,"statfs":{"total":95995035648,"available":95386005504,"internally_reserved":0,"allocated":472641536,"data_stored":472296397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5625,"internal_metadata":136374791},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.58099999999999996,"15min":0.54400000000000004},"min":{"1min":0.245,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":0.76700000000000002,"15min":0.67300000000000004},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.939,"5min":16.939,"15min":16.939},"last":0.63100000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.443,"15min":0.41099999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":1.996},{"interface":"front","average":{"1min":1.327,"5min":0.60699999999999998,"15min":0.48699999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":16.716000000000001,"5min":16.716000000000001,"15min":16.716000000000001},"last":0.85899999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.49099999999999999,"15min":0.47099999999999997},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.56699999999999995,"15min":0.505},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":10.122,"5min":10.122,"15min":10.122},"last":0.69799999999999995}]},{"osd":4,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":2.4910000000000001,"15min":2.6680000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":10.170999999999999,"5min":28.707999999999998,"15min":28.707999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.1080000000000001,"5min":2.6800000000000002,"15min":2.9420000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":3.3860000000000001,"5min":28.760999999999999,"15min":28.760999999999999},"last":0.45900000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.97999999999999998,"15min":0.89100000000000001},"min":{"1min":0.34100000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":10.083,"5min":10.083,"15min":10.083},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.4330000000000001,"5min":0.88500000000000001,"15min":0.79300000000000004},"min":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":10.298,"5min":10.298,"15min":10.298},"last":0.60799999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":2.363,"15min":2.5030000000000001},"min":{"1min":0.41799999999999998,"5min":0.372,"15min":0.372},"max":{"1min":10.039,"5min":28.661999999999999,"15min":28.661999999999999},"last":0.752},{"interface":"front","average":{"1min":1.778,"5min":2.9390000000000001,"15min":3.133},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":10.217000000000001,"5min":28.785,"15min":28.785},"last":1.0920000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.1879999999999999,"15min":1.139},"min":{"1min":0.45900000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":10.26,"5min":10.26,"15min":10.26},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.3029999999999999,"15min":1.28},"min":{"1min":0.48499999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":4.2830000000000004,"5min":4.2830000000000004,"15min":4.2830000000000004},"last":0.65700000000000003}]}]},{"osd":4,"up_from":32,"seq":137438953712,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1112700,"kb_used_data":445684,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":92632452,"statfs":{"total":95995035648,"available":94855630848,"internally_reserved":0,"allocated":456380416,"data_stored":456043212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":1.0569999999999999,"15min":2.089},"min":{"1min":0.26400000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.504,"5min":10.859999999999999,"15min":74.533000000000001},"last":1.869},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.2390000000000001,"15min":2.1030000000000002},"min":{"1min":0.32600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.8499999999999996,"5min":7.298,"15min":74.340999999999994},"last":0.626}]},{"osd":1,"last update":"Sat Feb 10 07:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":1.794,"15min":2.3170000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.23200000000000001},"max":{"1min":2.3999999999999999,"5min":66.343999999999994,"15min":69.852000000000004},"last":2.472},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.331,"15min":2.2040000000000002},"min":{"1min":0.32200000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.847,"5min":5.9909999999999997,"15min":70.692999999999998},"last":0.36799999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.3460000000000001,"15min":1.4470000000000001},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.26600000000000001},"max":{"1min":5.0700000000000003,"5min":6.0179999999999998,"15min":50.162999999999997},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.212,"15min":1.3939999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":5.9359999999999999,"5min":7.335,"15min":50.453000000000003},"last":0.86499999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":3.9710000000000001,"15min":4.4859999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.5640000000000001,"5min":66.379999999999995,"15min":66.379999999999995},"last":2.0339999999999998},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.6299999999999999,"15min":1.726},"min":{"1min":0.39600000000000002,"5min":0.309,"15min":0.309},"max":{"1min":5.0350000000000001,"5min":5.3780000000000001,"15min":5.3780000000000001},"last":0.442}]},{"osd":5,"last update":"Sat Feb 10 07:49:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.5609999999999999,"15min":0.95999999999999996},"min":{"1min":0.28799999999999998,"5min":0.28599999999999998,"15min":0.23100000000000001},"max":{"1min":5.9720000000000004,"5min":10.502000000000001,"15min":10.502000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.79300000000000004,"5min":1.524,"15min":0.96299999999999997},"min":{"1min":0.33300000000000002,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":2.9820000000000002,"5min":9.2460000000000004,"15min":9.2460000000000004},"last":2.1339999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":2.306,"15min":1.1759999999999999},"min":{"1min":0.35999999999999999,"5min":0.28100000000000003,"15min":0.25900000000000001},"max":{"1min":6.0039999999999996,"5min":67.433999999999997,"15min":67.433999999999997},"last":1.665},{"interface":"front","average":{"1min":1.3009999999999999,"5min":2.468,"15min":1.246},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.24399999999999999},"max":{"1min":5.0090000000000003,"5min":66.876000000000005,"15min":66.876000000000005},"last":1.302}]},{"osd":7,"last update":"Sat Feb 10 07:50:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":1.514,"15min":0.94299999999999995},"min":{"1min":0.42099999999999999,"5min":0.307,"15min":0.29099999999999998},"max":{"1min":1.5269999999999999,"5min":7.4930000000000003,"15min":7.4930000000000003},"last":1.204},{"interface":"front","average":{"1min":0.81799999999999995,"5min":1.474,"15min":0.996},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.29399999999999998},"max":{"1min":3.298,"5min":10.919,"15min":10.919},"last":3.298}]}]},{"osd":5,"up_from":156,"seq":670014898341,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":908980,"kb_used_data":297068,"kb_used_omap":4,"kb_used_meta":611899,"kb_avail":92836172,"statfs":{"total":95995035648,"available":95064240128,"internally_reserved":0,"allocated":304197632,"data_stored":303869544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":626585250},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.952,"5min":1.5209999999999999,"15min":1.7370000000000001},"min":{"1min":0.22600000000000001,"5min":0.20899999999999999,"15min":0.20499999999999999},"max":{"1min":14.978,"5min":21.817,"15min":52.938000000000002},"last":0.629},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.7330000000000001,"15min":1.974},"min":{"1min":0.23899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.643000000000001,"5min":21.347999999999999,"15min":52.805},"last":0.41299999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":1.3560000000000001,"15min":2.198},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.245},"max":{"1min":3.3130000000000002,"5min":8.0090000000000003,"15min":54.298000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.2290000000000001,"15min":2.1499999999999999},"min":{"1min":0.248,"5min":0.246,"15min":0.246},"max":{"1min":3.391,"5min":7.7649999999999997,"15min":52.969999999999999},"last":0.78100000000000003}]},{"osd":2,"last update":"Sat Feb 10 07:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":1.282,"15min":1.2929999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.2879999999999998,"5min":6.3970000000000002,"15min":11.601000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.995,"5min":1.4630000000000001,"15min":1.3720000000000001},"min":{"1min":0.43099999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.3209999999999997,"5min":13.662000000000001,"15min":13.662000000000001},"last":0.81799999999999995}]},{"osd":3,"last update":"Sat Feb 10 07:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.5760000000000001,"15min":1.6779999999999999},"min":{"1min":0.40699999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.4340000000000002,"5min":8.1530000000000005,"15min":8.1530000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.4139999999999999,"15min":1.454},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.8470000000000004,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":1.0609999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.998,"15min":1.1100000000000001},"min":{"1min":0.317,"5min":0.252,"15min":0.246},"max":{"1min":6.7759999999999998,"5min":8.952,"15min":8.952},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.655,"5min":1.7170000000000001,"15min":1.0209999999999999},"min":{"1min":0.32900000000000001,"5min":0.26800000000000002,"15min":0.25900000000000001},"max":{"1min":6.202,"5min":10.180999999999999,"15min":10.180999999999999},"last":0.503}]},{"osd":6,"last update":"Sat Feb 10 07:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.659,"15min":1.0209999999999999},"min":{"1min":0.34399999999999997,"5min":0.28699999999999998,"15min":0.23999999999999999},"max":{"1min":6.7110000000000003,"5min":9.0229999999999997,"15min":9.0229999999999997},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.7,"5min":1.5269999999999999,"15min":1.0009999999999999},"min":{"1min":0.29599999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":6.1699999999999999,"5min":6.1699999999999999,"15min":6.1699999999999999},"last":0.55000000000000004}]},{"osd":7,"last update":"Sat Feb 10 07:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.7250000000000001,"15min":1.1359999999999999},"min":{"1min":0.30499999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":9.0199999999999996,"5min":9.0199999999999996,"15min":9.0199999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.538,"15min":1.0229999999999999},"min":{"1min":0.35799999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":6.75,"5min":7.8109999999999999,"15min":7.8109999999999999},"last":0.85499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":607059968,"data_stored":607059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1001521152,"data_stored":1001521152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":496304128,"data_stored":496304128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":464781312,"data_stored":464781312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":336789504,"data_stored":336789504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":752091136,"data_stored":752091136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":738525184,"data_stored":738525184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:50:19.584 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:50:19.585 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:19 smithi170 ceph-mon[33394]: pgmap v852: 148 pgs: 148 active+clean; 1.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 147 MiB/s wr, 2.35k op/s 2024-02-10T07:50:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:19 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:50:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:19 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:50:19.645 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 14 16 35811 35795 159.779 125.25 0.00351342 0.00625194 2024-02-10T07:50:19.653 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:19.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:19.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:19.718 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:19.756 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:19.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:19.795 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:19.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:19.896 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:19.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:19.938 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:19.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:20.039 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:20.077 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:20.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:20.178 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:20.223 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:20.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:20.324 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:20.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:20.362 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:20.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:20.463 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:20.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:20.504 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:20.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:20.605 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:20.642 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:20.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:20.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:20.645 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 15 16 37778 37762 157.322 122.938 0.00706637 0.00635151 2024-02-10T07:50:20.743 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:20.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:20.775 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:20.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:20.876 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:20.916 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:20.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:20.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:21.018 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:21.061 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:21.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:21.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:21.163 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:21.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:21.197 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:21.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:21.298 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:21.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:21 smithi008 ceph-mon[29816]: pgmap v853: 148 pgs: 148 active+clean; 2.1 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 178 MiB/s wr, 2.84k op/s 2024-02-10T07:50:21.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:21 smithi008 ceph-mon[40743]: pgmap v853: 148 pgs: 148 active+clean; 2.1 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 178 MiB/s wr, 2.84k op/s 2024-02-10T07:50:21.338 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:21.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:21.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:21.354 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:50:21.354 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:50:21.441 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:21.478 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:21.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:21.580 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:21.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:21.617 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:21.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:21 smithi170 ceph-mon[33394]: pgmap v853: 148 pgs: 148 active+clean; 2.1 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 178 MiB/s wr, 2.84k op/s 2024-02-10T07:50:21.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 16 16 39634 39618 154.739 116 0.00777049 0.0064569 2024-02-10T07:50:21.718 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:21.753 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:21.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:21.855 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:21.892 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:21.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:21.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:21.994 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:22.028 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:22.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:22.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:22.130 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:22.166 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:22.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:22.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:22.268 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:22.305 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:22.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:22.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:22.407 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:22.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:22.442 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:22.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:22.543 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:22.577 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:22.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:22.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 17 16 41558 41542 152.71 120.25 0.00886108 0.00654314 2024-02-10T07:50:22.679 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:22.715 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:22.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:22.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:22.817 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:22.853 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:22.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:22.897 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:22.954 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:22.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:22.990 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:22.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:23.092 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:23.130 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:23.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:23.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:23.232 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:23.264 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:23.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:23.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:23.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:23 smithi008 ceph-mon[29816]: pgmap v854: 148 pgs: 148 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-02-10T07:50:23.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:23 smithi008 ceph-mon[40743]: pgmap v854: 148 pgs: 148 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-02-10T07:50:23.366 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:23.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:23 smithi170 ceph-mon[33394]: pgmap v854: 148 pgs: 148 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-02-10T07:50:23.407 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:23.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:23.486 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-10T07:50:23.509 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:23.544 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:23.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:23.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 18 16 43393 43377 150.597 114.688 0.0122463 0.00663523 2024-02-10T07:50:23.646 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:23.679 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:23.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:23.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:23.781 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:23.851 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:23.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:23.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:23.987 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:23.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:23.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:24.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:24 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:50:24.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:24 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:50:24.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:24 smithi008 ceph-mon[29816]: osdmap e399: 8 total, 8 up, 7 in 2024-02-10T07:50:24.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:24 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:50:24.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:24 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:50:24.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:24 smithi008 ceph-mon[40743]: osdmap e399: 8 total, 8 up, 7 in 2024-02-10T07:50:24.088 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:24.122 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:24.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:24.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:50:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:50:24.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:24 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:50:24.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:24 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:50:24.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:24 smithi170 ceph-mon[33394]: osdmap e399: 8 total, 8 up, 7 in 2024-02-10T07:50:24.145 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:24.202 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:50:24.224 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:24.244 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:50:24.266 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:24.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:24.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:24.284 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:50:24.329 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:50:24.367 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:24.371 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:50:24.413 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:50:24.421 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:24.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:24.460 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:50:24.502 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:50:24.509 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:50:24.510 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:50:24.510 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:50:24.510 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:50:24.510 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:50:24.510 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:50:24.510 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:50:24.510 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:50:24.522 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:24.563 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:24.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:24.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:24.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 19 16 45005 44989 147.973 100.75 0.00903266 0.00675214 2024-02-10T07:50:24.665 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:24.704 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:24.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:24.807 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:24.845 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:24.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:24.947 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:24.986 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:24.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:25.088 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:25.123 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:25.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:25.226 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:25.261 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:25.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:25 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:25 smithi008 ceph-mon[29816]: pgmap v856: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 133 MiB/s wr, 2.14k op/s 2024-02-10T07:50:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:25 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2840437919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:50:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:25 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:25 smithi008 ceph-mon[40743]: pgmap v856: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 133 MiB/s wr, 2.14k op/s 2024-02-10T07:50:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:25 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2840437919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:50:25.364 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:25 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:25 smithi170 ceph-mon[33394]: pgmap v856: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 133 MiB/s wr, 2.14k op/s 2024-02-10T07:50:25.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:25 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2840437919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:50:25.399 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:25.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:25.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:25.501 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:25.539 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:25.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:25.641 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:25.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:50:25.646912+0000 min lat: 0.00135892 max lat: 0.0806554 avg lat: 0.0068315 2024-02-10T07:50:25.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:50:25.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 20 16 46828 46812 146.271 113.938 0.00586314 0.0068315 2024-02-10T07:50:25.679 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:25.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:25.781 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:25.818 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:25.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:25.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:25.920 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:25.961 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:25.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:26.063 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:26.105 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:26.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:26.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:26.206 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:26.243 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:26.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:26.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:26 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:50:26.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:26 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:50:26.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:26 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:50:26.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:26 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:50:26.346 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:26.381 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:26.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:26.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:26 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:50:26.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:26 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:50:26.483 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:26.521 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:26.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:26.622 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:26.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 21 16 48608 48592 144.603 111.25 0.00528196 0.00691101 2024-02-10T07:50:26.660 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:26.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:26.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:26.762 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:26.801 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:26.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:26.902 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:26.938 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:26.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:26.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:27.040 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:27.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:27.079 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:27.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:27.180 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:27.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:27.234 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:27.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:27.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:27 smithi008 ceph-mon[29816]: osdmap e400: 8 total, 8 up, 7 in 2024-02-10T07:50:27.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:27 smithi008 ceph-mon[29816]: pgmap v858: 148 pgs: 148 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.43k op/s 2024-02-10T07:50:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:27 smithi008 ceph-mon[40743]: osdmap e400: 8 total, 8 up, 7 in 2024-02-10T07:50:27.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:27 smithi008 ceph-mon[40743]: pgmap v858: 148 pgs: 148 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.43k op/s 2024-02-10T07:50:27.334 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:27.371 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:27.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:27 smithi170 ceph-mon[33394]: osdmap e400: 8 total, 8 up, 7 in 2024-02-10T07:50:27.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:27 smithi170 ceph-mon[33394]: pgmap v858: 148 pgs: 148 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 152 MiB/s wr, 2.43k op/s 2024-02-10T07:50:27.443 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:27.473 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:27.508 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:27.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:27.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:27.610 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:27.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 22 16 50341 50325 142.952 108.312 0.0102707 0.00699102 2024-02-10T07:50:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:27.649 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:27.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:27.750 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:27.783 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:27.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:27.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:27.884 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:27.921 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:27.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:27.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:28.023 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:28.062 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:28.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:28.164 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:28.187 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:28.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:28.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:28.289 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:28.323 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:28.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:28.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:28.425 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:28.463 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:28.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:28.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:28.501 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:50:28.565 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:28.591 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:28.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:28.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:28.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 23 16 51640 51624 140.267 81.1875 0.0103699 0.00712453 2024-02-10T07:50:28.692 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:28.738 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:28.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:28.840 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:28.881 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:28.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:28.982 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:29.020 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:29.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:29.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:50:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:50:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:50:29.121 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:29.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:29.161 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:29.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:29.262 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:29.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:29.300 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:29.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:29.401 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:29.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:29.447 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:29.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:29.548 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:29.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:29 smithi008 ceph-mon[29816]: pgmap v859: 148 pgs: 148 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-02-10T07:50:29.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:29 smithi008 ceph-mon[40743]: pgmap v859: 148 pgs: 148 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-02-10T07:50:29.593 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:29.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:29.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:29.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:29 smithi170 ceph-mon[33394]: pgmap v859: 148 pgs: 148 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-02-10T07:50:29.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 24 16 53013 52997 137.997 85.8125 0.00800852 0.0072413 2024-02-10T07:50:29.695 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:29.733 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:29.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:29.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:29.835 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:29.875 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:29.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:29.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:29.977 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:30.014 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:30.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:30.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:30.117 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:30.157 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:30.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:30.258 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:30.297 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:30.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:30.398 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:30.435 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:30.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:30.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:30.439 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:50:30.440 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:30 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:50:30.441 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:30 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1688431104' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:50:30.441 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:30 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:50:30.441 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:30 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:50:30.441 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:30 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1688431104' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:50:30.441 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:30 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:50:30.445 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:50:30.537 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:30.574 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:30.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:30.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:30 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:50:30.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:30 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1688431104' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:50:30.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:30 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:50:30.646 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 25 16 54471 54455 136.122 91.125 0.00572156 0.00734154 2024-02-10T07:50:30.676 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:30.713 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:30.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:30.762 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:50:30.815 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:30.855 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:30.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:30.956 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:30.996 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:30.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:31.099 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:31.148 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:31.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:31.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:31.250 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:31.286 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:31.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:31.388 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:31.422 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:31.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:31.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:31.524 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:31.549 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:31.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:31.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:31.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:31 smithi008 ceph-mon[40743]: pgmap v860: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-02-10T07:50:31.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:31 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:50:31.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:31 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:50:31.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:31 smithi008 ceph-mon[40743]: osdmap e401: 8 total, 8 up, 7 in 2024-02-10T07:50:31.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:31 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:50:31.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:31 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:50:31.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:31 smithi008 ceph-mon[40743]: from='client.15972 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:50:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:31 smithi008 ceph-mon[29816]: pgmap v860: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-02-10T07:50:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:31 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:50:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:31 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:50:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:31 smithi008 ceph-mon[29816]: osdmap e401: 8 total, 8 up, 7 in 2024-02-10T07:50:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:31 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:50:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:31 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:50:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:31 smithi008 ceph-mon[29816]: from='client.15972 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:50:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:31 smithi170 ceph-mon[33394]: pgmap v860: 148 pgs: 148 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-02-10T07:50:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:31 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:50:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:31 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:50:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:31 smithi170 ceph-mon[33394]: osdmap e401: 8 total, 8 up, 7 in 2024-02-10T07:50:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:31 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:50:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:31 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:50:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:31 smithi170 ceph-mon[33394]: from='client.15972 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:50:31.647 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 26 16 55855 55839 134.213 86.5 0.0126286 0.00744583 2024-02-10T07:50:31.651 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:31.686 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:31.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:31.787 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:31.825 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:31.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:31.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:31.927 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:31.961 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:31.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:31.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:32.063 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:32.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:32.104 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:32.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:32.204 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:32.243 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:32.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:32.343 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:32.386 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:32.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:32.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:32 smithi008 ceph-mon[29816]: osdmap e402: 8 total, 8 up, 7 in 2024-02-10T07:50:32.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:50:32.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:50:32.470 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:32 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:50:32.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:32 smithi008 ceph-mon[40743]: osdmap e402: 8 total, 8 up, 7 in 2024-02-10T07:50:32.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:50:32.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:50:32.471 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:32 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:50:32.487 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:32.524 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:32.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:32.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:32.626 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:32 smithi170 ceph-mon[33394]: osdmap e402: 8 total, 8 up, 7 in 2024-02-10T07:50:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:50:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:50:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:32 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:50:32.647 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 27 16 57265 57249 132.506 88.125 0.00462295 0.00754029 2024-02-10T07:50:32.665 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:32.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:32.767 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:32.810 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:32.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:32.911 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:32.949 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:32.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:33.050 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:33.088 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:33.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:33.189 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:33.230 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":861,"stamp":"2024-02-10T07:50:30.179742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3377469788,"num_objects":51725,"num_object_clones":0,"num_object_copies":155175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51725,"num_whiteouts":0,"num_read":2457,"num_read_kb":2303,"num_write":52428,"num_write_kb":3300189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":1778520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52534,"ondisk_log_size":52534,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13907616,"kb_used_data":9687196,"kb_used_omap":17,"kb_used_meta":4220334,"kb_avail":642308448,"statfs":{"total":671965249536,"available":657723850752,"internally_reserved":0,"allocated":9919688704,"data_stored":9917269707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18039,"internal_metadata":4321622409},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,4,5,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1133903872,"num_objects":17302,"num_object_clones":0,"num_object_copies":51906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17302,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":17302,"num_write_kb":1107328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.170393"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":736,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.038029+0000","last_change":"2024-02-10T07:49:56.315126+0000","last_active":"2024-02-10T07:50:26.038029+0000","last_peered":"2024-02-10T07:50:26.038029+0000","last_clean":"2024-02-10T07:50:26.038029+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:50:26.038029+0000","last_undegraded":"2024-02-10T07:50:26.038029+0000","last_fullsized":"2024-02-10T07:50:26.038029+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.315029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.315029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:19.186486+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":746,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.041160+0000","last_change":"2024-02-10T07:49:08.138553+0000","last_active":"2024-02-10T07:50:26.041160+0000","last_peered":"2024-02-10T07:50:26.041160+0000","last_clean":"2024-02-10T07:50:26.041160+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:50:26.041160+0000","last_undegraded":"2024-02-10T07:50:26.041160+0000","last_fullsized":"2024-02-10T07:50:26.041160+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:08.138442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.138442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:35.396509+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":765,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.047614+0000","last_change":"2024-02-10T07:48:17.198789+0000","last_active":"2024-02-10T07:50:26.047614+0000","last_peered":"2024-02-10T07:50:26.047614+0000","last_clean":"2024-02-10T07:50:26.047614+0000","last_became_active":"2024-02-10T07:46:10.718945+0000","last_became_peered":"2024-02-10T07:46:10.718945+0000","last_unstale":"2024-02-10T07:50:26.047614+0000","last_undegraded":"2024-02-10T07:50:26.047614+0000","last_fullsized":"2024-02-10T07:50:26.047614+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:17.198701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:17.198701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377238+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"287'19","reported_seq":796,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.037665+0000","last_change":"2024-02-10T07:46:37.008978+0000","last_active":"2024-02-10T07:50:26.037665+0000","last_peered":"2024-02-10T07:50:26.037665+0000","last_clean":"2024-02-10T07:50:26.037665+0000","last_became_active":"2024-02-10T07:46:10.717475+0000","last_became_peered":"2024-02-10T07:46:10.717475+0000","last_unstale":"2024-02-10T07:50:26.037665+0000","last_undegraded":"2024-02-10T07:50:26.037665+0000","last_fullsized":"2024-02-10T07:50:26.037665+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'19","last_scrub_stamp":"2024-02-10T07:46:37.008841+0000","last_deep_scrub":"287'19","last_deep_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.008841+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.422584+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":67,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043138+0000","last_change":"2024-02-10T07:49:15.079645+0000","last_active":"2024-02-10T07:50:26.043138+0000","last_peered":"2024-02-10T07:50:26.043138+0000","last_clean":"2024-02-10T07:50:26.043138+0000","last_became_active":"2024-02-10T07:49:06.236235+0000","last_became_peered":"2024-02-10T07:49:06.236235+0000","last_unstale":"2024-02-10T07:50:26.043138+0000","last_undegraded":"2024-02-10T07:50:26.043138+0000","last_fullsized":"2024-02-10T07:50:26.043138+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:15.079428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:15.079428+0000","last_clean_scrub_stamp":"2024-02-10T07:49:15.079428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":739,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.037576+0000","last_change":"2024-02-10T07:50:06.335390+0000","last_active":"2024-02-10T07:50:26.037576+0000","last_peered":"2024-02-10T07:50:26.037576+0000","last_clean":"2024-02-10T07:50:26.037576+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:50:26.037576+0000","last_undegraded":"2024-02-10T07:50:26.037576+0000","last_fullsized":"2024-02-10T07:50:26.037576+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:06.335262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:50:06.335262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:25.829837+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":248,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043768+0000","last_change":"2024-02-10T07:49:47.812212+0000","last_active":"2024-02-10T07:50:26.043768+0000","last_peered":"2024-02-10T07:50:26.043768+0000","last_clean":"2024-02-10T07:50:26.043768+0000","last_became_active":"2024-02-10T07:49:06.262984+0000","last_became_peered":"2024-02-10T07:49:06.262984+0000","last_unstale":"2024-02-10T07:50:26.043768+0000","last_undegraded":"2024-02-10T07:50:26.043768+0000","last_fullsized":"2024-02-10T07:50:26.043768+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:47.812065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.812065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:07.921649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"250'27","reported_seq":703,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.047199+0000","last_change":"2024-02-10T07:49:06.320471+0000","last_active":"2024-02-10T07:50:26.047199+0000","last_peered":"2024-02-10T07:50:26.047199+0000","last_clean":"2024-02-10T07:50:26.047199+0000","last_became_active":"2024-02-10T07:49:06.266213+0000","last_became_peered":"2024-02-10T07:49:06.266213+0000","last_unstale":"2024-02-10T07:50:26.047199+0000","last_undegraded":"2024-02-10T07:50:26.047199+0000","last_fullsized":"2024-02-10T07:50:26.047199+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:48:13.230908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.230908+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":735,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045564+0000","last_change":"2024-02-10T07:50:05.620136+0000","last_active":"2024-02-10T07:50:26.045564+0000","last_peered":"2024-02-10T07:50:26.045564+0000","last_clean":"2024-02-10T07:50:26.045564+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:50:26.045564+0000","last_undegraded":"2024-02-10T07:50:26.045564+0000","last_fullsized":"2024-02-10T07:50:26.045564+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:05.619958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:05.619958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:09.870665+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":708,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.048763+0000","last_change":"2024-02-10T07:49:57.669438+0000","last_active":"2024-02-10T07:50:26.048763+0000","last_peered":"2024-02-10T07:50:26.048763+0000","last_clean":"2024-02-10T07:50:26.048763+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:50:26.048763+0000","last_undegraded":"2024-02-10T07:50:26.048763+0000","last_fullsized":"2024-02-10T07:50:26.048763+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:57.669314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.669314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:17.355126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":276,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.039570+0000","last_change":"2024-02-10T07:49:13.218634+0000","last_active":"2024-02-10T07:50:26.039570+0000","last_peered":"2024-02-10T07:50:26.039570+0000","last_clean":"2024-02-10T07:50:26.039570+0000","last_became_active":"2024-02-10T07:48:02.332960+0000","last_became_peered":"2024-02-10T07:48:02.332960+0000","last_unstale":"2024-02-10T07:50:26.039570+0000","last_undegraded":"2024-02-10T07:50:26.039570+0000","last_fullsized":"2024-02-10T07:50:26.039570+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.218549+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:42.382557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"250'15","reported_seq":761,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.038247+0000","last_change":"2024-02-10T07:47:49.553645+0000","last_active":"2024-02-10T07:50:26.038247+0000","last_peered":"2024-02-10T07:50:26.038247+0000","last_clean":"2024-02-10T07:50:26.038247+0000","last_became_active":"2024-02-10T07:47:47.965930+0000","last_became_peered":"2024-02-10T07:47:47.965930+0000","last_unstale":"2024-02-10T07:50:26.038247+0000","last_undegraded":"2024-02-10T07:50:26.038247+0000","last_fullsized":"2024-02-10T07:50:26.038247+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:49.553517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.553517+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.422714+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":648,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.041447+0000","last_change":"2024-02-10T07:49:56.718482+0000","last_active":"2024-02-10T07:50:26.041447+0000","last_peered":"2024-02-10T07:50:26.041447+0000","last_clean":"2024-02-10T07:50:26.041447+0000","last_became_active":"2024-02-10T07:49:06.266689+0000","last_became_peered":"2024-02-10T07:49:06.266689+0000","last_unstale":"2024-02-10T07:50:26.041447+0000","last_undegraded":"2024-02-10T07:50:26.041447+0000","last_fullsized":"2024-02-10T07:50:26.041447+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.718382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.718382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:12.254870+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":739,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044249+0000","last_change":"2024-02-10T07:48:18.069773+0000","last_active":"2024-02-10T07:50:26.044249+0000","last_peered":"2024-02-10T07:50:26.044249+0000","last_clean":"2024-02-10T07:50:26.044249+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:50:26.044249+0000","last_undegraded":"2024-02-10T07:50:26.044249+0000","last_fullsized":"2024-02-10T07:50:26.044249+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:18.069641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:18.069641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":264,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.039156+0000","last_change":"2024-02-10T07:49:06.115547+0000","last_active":"2024-02-10T07:50:26.039156+0000","last_peered":"2024-02-10T07:50:26.039156+0000","last_clean":"2024-02-10T07:50:26.039156+0000","last_became_active":"2024-02-10T07:48:02.329859+0000","last_became_peered":"2024-02-10T07:48:02.329859+0000","last_unstale":"2024-02-10T07:50:26.039156+0000","last_undegraded":"2024-02-10T07:50:26.039156+0000","last_fullsized":"2024-02-10T07:50:26.039156+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.115466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.473686+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.115466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:32.640728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"250'7","reported_seq":749,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.038982+0000","last_change":"2024-02-10T07:48:11.611397+0000","last_active":"2024-02-10T07:50:26.038982+0000","last_peered":"2024-02-10T07:50:26.038982+0000","last_clean":"2024-02-10T07:50:26.038982+0000","last_became_active":"2024-02-10T07:47:47.966093+0000","last_became_peered":"2024-02-10T07:47:47.966093+0000","last_unstale":"2024-02-10T07:50:26.038982+0000","last_undegraded":"2024-02-10T07:50:26.038982+0000","last_fullsized":"2024-02-10T07:50:26.038982+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7","last_scrub_stamp":"2024-02-10T07:48:11.608499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.608499+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.422819+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":738,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.040323+0000","last_change":"2024-02-10T07:48:13.621135+0000","last_active":"2024-02-10T07:50:26.040323+0000","last_peered":"2024-02-10T07:50:26.040323+0000","last_clean":"2024-02-10T07:50:26.040323+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:50:26.040323+0000","last_undegraded":"2024-02-10T07:50:26.040323+0000","last_fullsized":"2024-02-10T07:50:26.040323+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:13.620938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.620938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.422752+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":745,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043873+0000","last_change":"2024-02-10T07:49:51.810158+0000","last_active":"2024-02-10T07:50:26.043873+0000","last_peered":"2024-02-10T07:50:26.043873+0000","last_clean":"2024-02-10T07:50:26.043873+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:50:26.043873+0000","last_undegraded":"2024-02-10T07:50:26.043873+0000","last_fullsized":"2024-02-10T07:50:26.043873+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:51.810062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:51.810062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:59.448781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":747,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.046810+0000","last_change":"2024-02-10T07:49:16.313655+0000","last_active":"2024-02-10T07:50:26.046810+0000","last_peered":"2024-02-10T07:50:26.046810+0000","last_clean":"2024-02-10T07:50:26.046810+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:50:26.046810+0000","last_undegraded":"2024-02-10T07:50:26.046810+0000","last_fullsized":"2024-02-10T07:50:26.046810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:16.313573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.313573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.445556+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"250'34","reported_seq":127,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.048705+0000","last_change":"2024-02-10T07:49:06.320822+0000","last_active":"2024-02-10T07:50:26.048705+0000","last_peered":"2024-02-10T07:50:26.048705+0000","last_clean":"2024-02-10T07:50:26.048705+0000","last_became_active":"2024-02-10T07:49:06.268427+0000","last_became_peered":"2024-02-10T07:49:06.268427+0000","last_unstale":"2024-02-10T07:50:26.048705+0000","last_undegraded":"2024-02-10T07:50:26.048705+0000","last_fullsized":"2024-02-10T07:50:26.048705+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:49:00.551605+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.551605+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":433,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043725+0000","last_change":"2024-02-10T07:49:51.990345+0000","last_active":"2024-02-10T07:50:26.043725+0000","last_peered":"2024-02-10T07:50:26.043725+0000","last_clean":"2024-02-10T07:50:26.043725+0000","last_became_active":"2024-02-10T07:49:06.263790+0000","last_became_peered":"2024-02-10T07:49:06.263790+0000","last_unstale":"2024-02-10T07:50:26.043725+0000","last_undegraded":"2024-02-10T07:50:26.043725+0000","last_fullsized":"2024-02-10T07:50:26.043725+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:51.990237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:51.990237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:13.825668+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":750,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.039375+0000","last_change":"2024-02-10T07:49:06.266778+0000","last_active":"2024-02-10T07:50:26.039375+0000","last_peered":"2024-02-10T07:50:26.039375+0000","last_clean":"2024-02-10T07:50:26.039375+0000","last_became_active":"2024-02-10T07:49:06.266390+0000","last_became_peered":"2024-02-10T07:49:06.266390+0000","last_unstale":"2024-02-10T07:50:26.039375+0000","last_undegraded":"2024-02-10T07:50:26.039375+0000","last_fullsized":"2024-02-10T07:50:26.039375+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.680331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.680331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.422848+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":753,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.038330+0000","last_change":"2024-02-10T07:50:04.396854+0000","last_active":"2024-02-10T07:50:26.038330+0000","last_peered":"2024-02-10T07:50:26.038330+0000","last_clean":"2024-02-10T07:50:26.038330+0000","last_became_active":"2024-02-10T07:46:10.717041+0000","last_became_peered":"2024-02-10T07:46:10.717041+0000","last_unstale":"2024-02-10T07:50:26.038330+0000","last_undegraded":"2024-02-10T07:50:26.038330+0000","last_fullsized":"2024-02-10T07:50:26.038330+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:04.396771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.396771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:32.209449+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"250'18","reported_seq":706,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043678+0000","last_change":"2024-02-10T07:49:58.618564+0000","last_active":"2024-02-10T07:50:26.043678+0000","last_peered":"2024-02-10T07:50:26.043678+0000","last_clean":"2024-02-10T07:50:26.043678+0000","last_became_active":"2024-02-10T07:49:06.263261+0000","last_became_peered":"2024-02-10T07:49:06.263261+0000","last_unstale":"2024-02-10T07:50:26.043678+0000","last_undegraded":"2024-02-10T07:50:26.043678+0000","last_fullsized":"2024-02-10T07:50:26.043678+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'18","last_scrub_stamp":"2024-02-10T07:49:58.618464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:58.618464+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:59.071088+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":761,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044880+0000","last_change":"2024-02-10T07:49:06.317985+0000","last_active":"2024-02-10T07:50:26.044880+0000","last_peered":"2024-02-10T07:50:26.044880+0000","last_clean":"2024-02-10T07:50:26.044880+0000","last_became_active":"2024-02-10T07:49:06.267590+0000","last_became_peered":"2024-02-10T07:49:06.267590+0000","last_unstale":"2024-02-10T07:50:26.044880+0000","last_undegraded":"2024-02-10T07:50:26.044880+0000","last_fullsized":"2024-02-10T07:50:26.044880+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:47:34.074746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:34.074746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":998,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043664+0000","last_change":"2024-02-10T07:49:56.824743+0000","last_active":"2024-02-10T07:50:26.043664+0000","last_peered":"2024-02-10T07:50:26.043664+0000","last_clean":"2024-02-10T07:50:26.043664+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:50:26.043664+0000","last_undegraded":"2024-02-10T07:50:26.043664+0000","last_fullsized":"2024-02-10T07:50:26.043664+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:49:56.824601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.824601+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:04.771314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":755,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.041037+0000","last_change":"2024-02-10T07:48:30.384635+0000","last_active":"2024-02-10T07:50:26.041037+0000","last_peered":"2024-02-10T07:50:26.041037+0000","last_clean":"2024-02-10T07:50:26.041037+0000","last_became_active":"2024-02-10T07:47:47.966127+0000","last_became_peered":"2024-02-10T07:47:47.966127+0000","last_unstale":"2024-02-10T07:50:26.041037+0000","last_undegraded":"2024-02-10T07:50:26.041037+0000","last_fullsized":"2024-02-10T07:50:26.041037+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:30.384546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:30.384546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.513586+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"250'25","reported_seq":819,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.049568+0000","last_change":"2024-02-10T07:49:06.308245+0000","last_active":"2024-02-10T07:50:26.049568+0000","last_peered":"2024-02-10T07:50:26.049568+0000","last_clean":"2024-02-10T07:50:26.049568+0000","last_became_active":"2024-02-10T07:49:06.261487+0000","last_became_peered":"2024-02-10T07:49:06.261487+0000","last_unstale":"2024-02-10T07:50:26.049568+0000","last_undegraded":"2024-02-10T07:50:26.049568+0000","last_fullsized":"2024-02-10T07:50:26.049568+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:48:12.278982+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-02-10T07:42:17.820959+0000","last_clean_scrub_stamp":"2024-02-10T07:48:12.278982+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377299+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":742,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044005+0000","last_change":"2024-02-10T07:49:11.231819+0000","last_active":"2024-02-10T07:50:26.044005+0000","last_peered":"2024-02-10T07:50:26.044005+0000","last_clean":"2024-02-10T07:50:26.044005+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:50:26.044005+0000","last_undegraded":"2024-02-10T07:50:26.044005+0000","last_fullsized":"2024-02-10T07:50:26.044005+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.231704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.231704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:31.609046+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":739,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.049823+0000","last_change":"2024-02-10T07:48:56.208535+0000","last_active":"2024-02-10T07:50:26.049823+0000","last_peered":"2024-02-10T07:50:26.049823+0000","last_clean":"2024-02-10T07:50:26.049823+0000","last_became_active":"2024-02-10T07:46:10.724252+0000","last_became_peered":"2024-02-10T07:46:10.724252+0000","last_unstale":"2024-02-10T07:50:26.049823+0000","last_undegraded":"2024-02-10T07:50:26.049823+0000","last_fullsized":"2024-02-10T07:50:26.049823+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.208451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:06.200009+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.208451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":243,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045574+0000","last_change":"2024-02-10T07:49:06.259104+0000","last_active":"2024-02-10T07:50:26.045574+0000","last_peered":"2024-02-10T07:50:26.045574+0000","last_clean":"2024-02-10T07:50:26.045574+0000","last_became_active":"2024-02-10T07:49:06.258789+0000","last_became_peered":"2024-02-10T07:49:06.258789+0000","last_unstale":"2024-02-10T07:50:26.045574+0000","last_undegraded":"2024-02-10T07:50:26.045574+0000","last_fullsized":"2024-02-10T07:50:26.045574+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:45.224482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:47:45.224482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377048+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"250'19","reported_seq":791,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043622+0000","last_change":"2024-02-10T07:49:50.808829+0000","last_active":"2024-02-10T07:50:26.043622+0000","last_peered":"2024-02-10T07:50:26.043622+0000","last_clean":"2024-02-10T07:50:26.043622+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:50:26.043622+0000","last_undegraded":"2024-02-10T07:50:26.043622+0000","last_fullsized":"2024-02-10T07:50:26.043622+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:50.808710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.808710+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:14.055156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":751,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044165+0000","last_change":"2024-02-10T07:49:06.249490+0000","last_active":"2024-02-10T07:50:26.044165+0000","last_peered":"2024-02-10T07:50:26.044165+0000","last_clean":"2024-02-10T07:50:26.044165+0000","last_became_active":"2024-02-10T07:49:06.249162+0000","last_became_peered":"2024-02-10T07:49:06.249162+0000","last_unstale":"2024-02-10T07:50:26.044165+0000","last_undegraded":"2024-02-10T07:50:26.044165+0000","last_fullsized":"2024-02-10T07:50:26.044165+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.087614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.087614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":754,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044003+0000","last_change":"2024-02-10T07:49:54.769358+0000","last_active":"2024-02-10T07:50:26.044003+0000","last_peered":"2024-02-10T07:50:26.044003+0000","last_clean":"2024-02-10T07:50:26.044003+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:50:26.044003+0000","last_undegraded":"2024-02-10T07:50:26.044003+0000","last_fullsized":"2024-02-10T07:50:26.044003+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:54.769226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:54.769226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:24.449065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":771,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.048378+0000","last_change":"2024-02-10T07:49:06.259100+0000","last_active":"2024-02-10T07:50:26.048378+0000","last_peered":"2024-02-10T07:50:26.048378+0000","last_clean":"2024-02-10T07:50:26.048378+0000","last_became_active":"2024-02-10T07:49:06.258763+0000","last_became_peered":"2024-02-10T07:49:06.258763+0000","last_unstale":"2024-02-10T07:50:26.048378+0000","last_undegraded":"2024-02-10T07:50:26.048378+0000","last_fullsized":"2024-02-10T07:50:26.048378+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:29.188832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:29.188832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"250'10","reported_seq":73,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043164+0000","last_change":"2024-02-10T07:49:16.017097+0000","last_active":"2024-02-10T07:50:26.043164+0000","last_peered":"2024-02-10T07:50:26.043164+0000","last_clean":"2024-02-10T07:50:26.043164+0000","last_became_active":"2024-02-10T07:49:06.250577+0000","last_became_peered":"2024-02-10T07:49:06.250577+0000","last_unstale":"2024-02-10T07:50:26.043164+0000","last_undegraded":"2024-02-10T07:50:26.043164+0000","last_fullsized":"2024-02-10T07:50:26.043164+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'10","last_scrub_stamp":"2024-02-10T07:49:16.017003+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.017003+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":237,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.046277+0000","last_change":"2024-02-10T07:49:06.259895+0000","last_active":"2024-02-10T07:50:26.046277+0000","last_peered":"2024-02-10T07:50:26.046277+0000","last_clean":"2024-02-10T07:50:26.046277+0000","last_became_active":"2024-02-10T07:49:06.259341+0000","last_became_peered":"2024-02-10T07:49:06.259341+0000","last_unstale":"2024-02-10T07:50:26.046277+0000","last_undegraded":"2024-02-10T07:50:26.046277+0000","last_fullsized":"2024-02-10T07:50:26.046277+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:32.258321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:32.258321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377086+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":739,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044759+0000","last_change":"2024-02-10T07:48:24.171978+0000","last_active":"2024-02-10T07:50:26.044759+0000","last_peered":"2024-02-10T07:50:26.044759+0000","last_clean":"2024-02-10T07:50:26.044759+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:50:26.044759+0000","last_undegraded":"2024-02-10T07:50:26.044759+0000","last_fullsized":"2024-02-10T07:50:26.044759+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:24.171859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:24.171859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":680,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.049890+0000","last_change":"2024-02-10T07:49:53.578357+0000","last_active":"2024-02-10T07:50:26.049890+0000","last_peered":"2024-02-10T07:50:26.049890+0000","last_clean":"2024-02-10T07:50:26.049890+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:50:26.049890+0000","last_undegraded":"2024-02-10T07:50:26.049890+0000","last_fullsized":"2024-02-10T07:50:26.049890+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:53.578274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:49:53.578274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.901374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"250'19","reported_seq":706,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044094+0000","last_change":"2024-02-10T07:49:53.245490+0000","last_active":"2024-02-10T07:50:26.044094+0000","last_peered":"2024-02-10T07:50:26.044094+0000","last_clean":"2024-02-10T07:50:26.044094+0000","last_became_active":"2024-02-10T07:49:06.239701+0000","last_became_peered":"2024-02-10T07:49:06.239701+0000","last_unstale":"2024-02-10T07:50:26.044094+0000","last_undegraded":"2024-02-10T07:50:26.044094+0000","last_fullsized":"2024-02-10T07:50:26.044094+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:53.245410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:53.245410+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:19.652831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":736,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045303+0000","last_change":"2024-02-10T07:49:52.530010+0000","last_active":"2024-02-10T07:50:26.045303+0000","last_peered":"2024-02-10T07:50:26.045303+0000","last_clean":"2024-02-10T07:50:26.045303+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:50:26.045303+0000","last_undegraded":"2024-02-10T07:50:26.045303+0000","last_fullsized":"2024-02-10T07:50:26.045303+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.529908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:59.179342+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.529908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.765757+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":237,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.046509+0000","last_change":"2024-02-10T07:49:06.265551+0000","last_active":"2024-02-10T07:50:26.046509+0000","last_peered":"2024-02-10T07:50:26.046509+0000","last_clean":"2024-02-10T07:50:26.046509+0000","last_became_active":"2024-02-10T07:49:06.265267+0000","last_became_peered":"2024-02-10T07:49:06.265267+0000","last_unstale":"2024-02-10T07:50:26.046509+0000","last_undegraded":"2024-02-10T07:50:26.046509+0000","last_fullsized":"2024-02-10T07:50:26.046509+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:40.140557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:40.140557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.376980+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":756,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044781+0000","last_change":"2024-02-10T07:50:00.265308+0000","last_active":"2024-02-10T07:50:26.044781+0000","last_peered":"2024-02-10T07:50:26.044781+0000","last_clean":"2024-02-10T07:50:26.044781+0000","last_became_active":"2024-02-10T07:47:47.966472+0000","last_became_peered":"2024-02-10T07:47:47.966472+0000","last_unstale":"2024-02-10T07:50:26.044781+0000","last_undegraded":"2024-02-10T07:50:26.044781+0000","last_fullsized":"2024-02-10T07:50:26.044781+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.265208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.265208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:23.837007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"250'19","reported_seq":564,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.041127+0000","last_change":"2024-02-10T07:49:09.117006+0000","last_active":"2024-02-10T07:50:26.041127+0000","last_peered":"2024-02-10T07:50:26.041127+0000","last_clean":"2024-02-10T07:50:26.041127+0000","last_became_active":"2024-02-10T07:46:10.726359+0000","last_became_peered":"2024-02-10T07:46:10.726359+0000","last_unstale":"2024-02-10T07:50:26.041127+0000","last_undegraded":"2024-02-10T07:50:26.041127+0000","last_fullsized":"2024-02-10T07:50:26.041127+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:09.116615+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-02-10T07:39:32.934269+0000","last_clean_scrub_stamp":"2024-02-10T07:49:09.116615+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:35.109389+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":425,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.047100+0000","last_change":"2024-02-10T07:49:06.260883+0000","last_active":"2024-02-10T07:50:26.047100+0000","last_peered":"2024-02-10T07:50:26.047100+0000","last_clean":"2024-02-10T07:50:26.047100+0000","last_became_active":"2024-02-10T07:49:06.260569+0000","last_became_peered":"2024-02-10T07:49:06.260569+0000","last_unstale":"2024-02-10T07:50:26.047100+0000","last_undegraded":"2024-02-10T07:50:26.047100+0000","last_fullsized":"2024-02-10T07:50:26.047100+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:31.117976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:31.117976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377120+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":769,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044305+0000","last_change":"2024-02-10T07:50:02.685941+0000","last_active":"2024-02-10T07:50:26.044305+0000","last_peered":"2024-02-10T07:50:26.044305+0000","last_clean":"2024-02-10T07:50:26.044305+0000","last_became_active":"2024-02-10T07:49:06.252380+0000","last_became_peered":"2024-02-10T07:49:06.252380+0000","last_unstale":"2024-02-10T07:50:26.044305+0000","last_undegraded":"2024-02-10T07:50:26.044305+0000","last_fullsized":"2024-02-10T07:50:26.044305+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_clean_scrub_stamp":"2024-02-10T07:50:02.685831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:09.446762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":747,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.039246+0000","last_change":"2024-02-10T07:50:00.345087+0000","last_active":"2024-02-10T07:50:26.039246+0000","last_peered":"2024-02-10T07:50:26.039246+0000","last_clean":"2024-02-10T07:50:26.039246+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:50:26.039246+0000","last_undegraded":"2024-02-10T07:50:26.039246+0000","last_fullsized":"2024-02-10T07:50:26.039246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.344993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.344993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:26.742023+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"250'24","reported_seq":807,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.038836+0000","last_change":"2024-02-10T07:47:43.494877+0000","last_active":"2024-02-10T07:50:26.038836+0000","last_peered":"2024-02-10T07:50:26.038836+0000","last_clean":"2024-02-10T07:50:26.038836+0000","last_became_active":"2024-02-10T07:46:10.721336+0000","last_became_peered":"2024-02-10T07:46:10.721336+0000","last_unstale":"2024-02-10T07:50:26.038836+0000","last_undegraded":"2024-02-10T07:50:26.038836+0000","last_fullsized":"2024-02-10T07:50:26.038836+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:47:43.494350+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.494350+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.422681+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":742,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.049263+0000","last_change":"2024-02-10T07:49:06.262127+0000","last_active":"2024-02-10T07:50:26.049263+0000","last_peered":"2024-02-10T07:50:26.049263+0000","last_clean":"2024-02-10T07:50:26.049263+0000","last_became_active":"2024-02-10T07:49:06.261570+0000","last_became_peered":"2024-02-10T07:49:06.261570+0000","last_unstale":"2024-02-10T07:50:26.049263+0000","last_undegraded":"2024-02-10T07:50:26.049263+0000","last_fullsized":"2024-02-10T07:50:26.049263+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:43.186809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.186809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377179+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"356'11","reported_seq":208,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.591887+0000","last_change":"2024-02-10T07:48:25.339974+0000","last_active":"2024-02-10T07:50:26.591887+0000","last_peered":"2024-02-10T07:50:26.591887+0000","last_clean":"2024-02-10T07:50:26.591887+0000","last_became_active":"2024-02-10T07:48:02.371888+0000","last_became_peered":"2024-02-10T07:48:02.371888+0000","last_unstale":"2024-02-10T07:50:26.591887+0000","last_undegraded":"2024-02-10T07:50:26.591887+0000","last_fullsized":"2024-02-10T07:50:26.591887+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'11","last_scrub_stamp":"2024-02-10T07:48:25.339855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:25.339855+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":761,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045017+0000","last_change":"2024-02-10T07:49:17.335559+0000","last_active":"2024-02-10T07:50:26.045017+0000","last_peered":"2024-02-10T07:50:26.045017+0000","last_clean":"2024-02-10T07:50:26.045017+0000","last_became_active":"2024-02-10T07:46:10.716647+0000","last_became_peered":"2024-02-10T07:46:10.716647+0000","last_unstale":"2024-02-10T07:50:26.045017+0000","last_undegraded":"2024-02-10T07:50:26.045017+0000","last_fullsized":"2024-02-10T07:50:26.045017+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.335455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.335455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:45.237183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"250'25","reported_seq":369,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043991+0000","last_change":"2024-02-10T07:49:12.150442+0000","last_active":"2024-02-10T07:50:26.043991+0000","last_peered":"2024-02-10T07:50:26.043991+0000","last_clean":"2024-02-10T07:50:26.043991+0000","last_became_active":"2024-02-10T07:48:02.329187+0000","last_became_peered":"2024-02-10T07:48:02.329187+0000","last_unstale":"2024-02-10T07:50:26.043991+0000","last_undegraded":"2024-02-10T07:50:26.043991+0000","last_fullsized":"2024-02-10T07:50:26.043991+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:49:12.150298+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:49:12.150298+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.513675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":742,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.040042+0000","last_change":"2024-02-10T07:50:01.514066+0000","last_active":"2024-02-10T07:50:26.040042+0000","last_peered":"2024-02-10T07:50:26.040042+0000","last_clean":"2024-02-10T07:50:26.040042+0000","last_became_active":"2024-02-10T07:46:10.720680+0000","last_became_peered":"2024-02-10T07:46:10.720680+0000","last_unstale":"2024-02-10T07:50:26.040042+0000","last_undegraded":"2024-02-10T07:50:26.040042+0000","last_fullsized":"2024-02-10T07:50:26.040042+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:01.513972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:01.513972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.583346+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"376'14","reported_seq":353,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045110+0000","last_change":"2024-02-10T07:49:06.314956+0000","last_active":"2024-02-10T07:50:26.045110+0000","last_peered":"2024-02-10T07:50:26.045110+0000","last_clean":"2024-02-10T07:50:26.045110+0000","last_became_active":"2024-02-10T07:49:06.267009+0000","last_became_peered":"2024-02-10T07:49:06.267009+0000","last_unstale":"2024-02-10T07:50:26.045110+0000","last_undegraded":"2024-02-10T07:50:26.045110+0000","last_fullsized":"2024-02-10T07:50:26.045110+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'13","last_scrub_stamp":"2024-02-10T07:49:00.366610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.366610+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":762,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.046575+0000","last_change":"2024-02-10T07:50:07.501014+0000","last_active":"2024-02-10T07:50:26.046575+0000","last_peered":"2024-02-10T07:50:26.046575+0000","last_clean":"2024-02-10T07:50:26.046575+0000","last_became_active":"2024-02-10T07:49:06.257043+0000","last_became_peered":"2024-02-10T07:49:06.257043+0000","last_unstale":"2024-02-10T07:50:26.046575+0000","last_undegraded":"2024-02-10T07:50:26.046575+0000","last_fullsized":"2024-02-10T07:50:26.046575+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.500890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:33.615213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"250'15","reported_seq":501,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045264+0000","last_change":"2024-02-10T07:49:06.277154+0000","last_active":"2024-02-10T07:50:26.045264+0000","last_peered":"2024-02-10T07:50:26.045264+0000","last_clean":"2024-02-10T07:50:26.045264+0000","last_became_active":"2024-02-10T07:49:06.255239+0000","last_became_peered":"2024-02-10T07:49:06.255239+0000","last_unstale":"2024-02-10T07:50:26.045264+0000","last_undegraded":"2024-02-10T07:50:26.045264+0000","last_fullsized":"2024-02-10T07:50:26.045264+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:39.610017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:39.610017+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:32.294540+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":202,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.039472+0000","last_change":"2024-02-10T07:49:44.324178+0000","last_active":"2024-02-10T07:50:26.039472+0000","last_peered":"2024-02-10T07:50:26.039472+0000","last_clean":"2024-02-10T07:50:26.039472+0000","last_became_active":"2024-02-10T07:46:10.716869+0000","last_became_peered":"2024-02-10T07:46:10.716869+0000","last_unstale":"2024-02-10T07:50:26.039472+0000","last_undegraded":"2024-02-10T07:50:26.039472+0000","last_fullsized":"2024-02-10T07:50:26.039472+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:44.324046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:49:44.324046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:46.499909+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":87,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043252+0000","last_change":"2024-02-10T07:49:06.263378+0000","last_active":"2024-02-10T07:50:26.043252+0000","last_peered":"2024-02-10T07:50:26.043252+0000","last_clean":"2024-02-10T07:50:26.043252+0000","last_became_active":"2024-02-10T07:49:06.262933+0000","last_became_peered":"2024-02-10T07:49:06.262933+0000","last_unstale":"2024-02-10T07:50:26.043252+0000","last_undegraded":"2024-02-10T07:50:26.043252+0000","last_fullsized":"2024-02-10T07:50:26.043252+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:17.294700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:48:17.294700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:30.915169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":755,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043951+0000","last_change":"2024-02-10T07:49:59.278666+0000","last_active":"2024-02-10T07:50:26.043951+0000","last_peered":"2024-02-10T07:50:26.043951+0000","last_clean":"2024-02-10T07:50:26.043951+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:50:26.043951+0000","last_undegraded":"2024-02-10T07:50:26.043951+0000","last_fullsized":"2024-02-10T07:50:26.043951+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.278574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.278574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.042755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"250'19","reported_seq":807,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.049668+0000","last_change":"2024-02-10T07:49:06.306965+0000","last_active":"2024-02-10T07:50:26.049668+0000","last_peered":"2024-02-10T07:50:26.049668+0000","last_clean":"2024-02-10T07:50:26.049668+0000","last_became_active":"2024-02-10T07:49:06.260962+0000","last_became_peered":"2024-02-10T07:49:06.260962+0000","last_unstale":"2024-02-10T07:50:26.049668+0000","last_undegraded":"2024-02-10T07:50:26.049668+0000","last_fullsized":"2024-02-10T07:50:26.049668+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:47:42.150605+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.150605+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377208+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":735,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044563+0000","last_change":"2024-02-10T07:48:58.511630+0000","last_active":"2024-02-10T07:50:26.044563+0000","last_peered":"2024-02-10T07:50:26.044563+0000","last_clean":"2024-02-10T07:50:26.044563+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:50:26.044563+0000","last_undegraded":"2024-02-10T07:50:26.044563+0000","last_fullsized":"2024-02-10T07:50:26.044563+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:58.511251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.511251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":67,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044142+0000","last_change":"2024-02-10T07:50:04.978177+0000","last_active":"2024-02-10T07:50:26.044142+0000","last_peered":"2024-02-10T07:50:26.044142+0000","last_clean":"2024-02-10T07:50:26.044142+0000","last_became_active":"2024-02-10T07:49:06.250832+0000","last_became_peered":"2024-02-10T07:49:06.250832+0000","last_unstale":"2024-02-10T07:50:26.044142+0000","last_undegraded":"2024-02-10T07:50:26.044142+0000","last_fullsized":"2024-02-10T07:50:26.044142+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:04.978097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.978097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:19.962282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":779,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043016+0000","last_change":"2024-02-10T07:49:52.832935+0000","last_active":"2024-02-10T07:50:26.043016+0000","last_peered":"2024-02-10T07:50:26.043016+0000","last_clean":"2024-02-10T07:50:26.043016+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:50:26.043016+0000","last_undegraded":"2024-02-10T07:50:26.043016+0000","last_fullsized":"2024-02-10T07:50:26.043016+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.832862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.832862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:16.134029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"250'12","reported_seq":75,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044442+0000","last_change":"2024-02-10T07:49:43.919297+0000","last_active":"2024-02-10T07:50:26.044442+0000","last_peered":"2024-02-10T07:50:26.044442+0000","last_clean":"2024-02-10T07:50:26.044442+0000","last_became_active":"2024-02-10T07:49:06.258281+0000","last_became_peered":"2024-02-10T07:49:06.258281+0000","last_unstale":"2024-02-10T07:50:26.044442+0000","last_undegraded":"2024-02-10T07:50:26.044442+0000","last_fullsized":"2024-02-10T07:50:26.044442+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'12","last_scrub_stamp":"2024-02-10T07:49:43.919178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:43.919178+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:52.611584+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"400'3328","reported_seq":3356,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:29.124517+0000","last_change":"2024-02-10T07:49:53.374457+0000","last_active":"2024-02-10T07:50:29.124517+0000","last_peered":"2024-02-10T07:50:29.124517+0000","last_clean":"2024-02-10T07:50:29.124517+0000","last_became_active":"2024-02-10T07:49:53.374185+0000","last_became_peered":"2024-02-10T07:49:53.374185+0000","last_unstale":"2024-02-10T07:50:29.124517+0000","last_undegraded":"2024-02-10T07:50:29.124517+0000","last_fullsized":"2024-02-10T07:50:29.124517+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3328,"log_dups_size":0,"ondisk_log_size":3328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:58.752412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103808,"num_objects":3328,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3328,"num_write_kb":212992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":269,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043128+0000","last_change":"2024-02-10T07:50:05.847991+0000","last_active":"2024-02-10T07:50:26.043128+0000","last_peered":"2024-02-10T07:50:26.043128+0000","last_clean":"2024-02-10T07:50:26.043128+0000","last_became_active":"2024-02-10T07:48:02.366002+0000","last_became_peered":"2024-02-10T07:48:02.366002+0000","last_unstale":"2024-02-10T07:50:26.043128+0000","last_undegraded":"2024-02-10T07:50:26.043128+0000","last_fullsized":"2024-02-10T07:50:26.043128+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:05.847866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:05.847866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:23.653860+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"245'4","reported_seq":770,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045216+0000","last_change":"2024-02-10T07:49:08.311056+0000","last_active":"2024-02-10T07:50:26.045216+0000","last_peered":"2024-02-10T07:50:26.045216+0000","last_clean":"2024-02-10T07:50:26.045216+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:50:26.045216+0000","last_undegraded":"2024-02-10T07:50:26.045216+0000","last_fullsized":"2024-02-10T07:50:26.045216+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'4","last_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_deep_scrub":"245'4","last_deep_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.310917+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"250'34","reported_seq":496,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044561+0000","last_change":"2024-02-10T07:48:11.967227+0000","last_active":"2024-02-10T07:50:26.044561+0000","last_peered":"2024-02-10T07:50:26.044561+0000","last_clean":"2024-02-10T07:50:26.044561+0000","last_became_active":"2024-02-10T07:48:02.330130+0000","last_became_peered":"2024-02-10T07:48:02.330130+0000","last_unstale":"2024-02-10T07:50:26.044561+0000","last_undegraded":"2024-02-10T07:50:26.044561+0000","last_fullsized":"2024-02-10T07:50:26.044561+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:48:11.966957+0000","last_deep_scrub":"246'22","last_deep_scrub_stamp":"2024-02-10T07:41:26.771112+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.966957+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":980,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044224+0000","last_change":"2024-02-10T07:48:53.423840+0000","last_active":"2024-02-10T07:50:26.044224+0000","last_peered":"2024-02-10T07:50:26.044224+0000","last_clean":"2024-02-10T07:50:26.044224+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:50:26.044224+0000","last_undegraded":"2024-02-10T07:50:26.044224+0000","last_fullsized":"2024-02-10T07:50:26.044224+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:48:53.423613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:53.423613+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"400'3235","reported_seq":3263,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:29.170336+0000","last_change":"2024-02-10T07:49:53.282437+0000","last_active":"2024-02-10T07:50:29.170336+0000","last_peered":"2024-02-10T07:50:29.170336+0000","last_clean":"2024-02-10T07:50:29.170336+0000","last_became_active":"2024-02-10T07:49:53.282163+0000","last_became_peered":"2024-02-10T07:49:53.282163+0000","last_unstale":"2024-02-10T07:50:29.170336+0000","last_undegraded":"2024-02-10T07:50:29.170336+0000","last_fullsized":"2024-02-10T07:50:29.170336+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3235,"log_dups_size":0,"ondisk_log_size":3235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:16.220098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212008960,"num_objects":3235,"num_object_clones":0,"num_object_copies":9705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3235,"num_write_kb":207040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":574,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043296+0000","last_change":"2024-02-10T07:49:42.990229+0000","last_active":"2024-02-10T07:50:26.043296+0000","last_peered":"2024-02-10T07:50:26.043296+0000","last_clean":"2024-02-10T07:50:26.043296+0000","last_became_active":"2024-02-10T07:47:47.966183+0000","last_became_peered":"2024-02-10T07:47:47.966183+0000","last_unstale":"2024-02-10T07:50:26.043296+0000","last_undegraded":"2024-02-10T07:50:26.043296+0000","last_fullsized":"2024-02-10T07:50:26.043296+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:42.990098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:42.990098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.708032+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":748,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045671+0000","last_change":"2024-02-10T07:48:56.416159+0000","last_active":"2024-02-10T07:50:26.045671+0000","last_peered":"2024-02-10T07:50:26.045671+0000","last_clean":"2024-02-10T07:50:26.045671+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:50:26.045671+0000","last_undegraded":"2024-02-10T07:50:26.045671+0000","last_fullsized":"2024-02-10T07:50:26.045671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.416037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.416037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"250'23","reported_seq":854,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.038368+0000","last_change":"2024-02-10T07:49:43.339027+0000","last_active":"2024-02-10T07:50:26.038368+0000","last_peered":"2024-02-10T07:50:26.038368+0000","last_clean":"2024-02-10T07:50:26.038368+0000","last_became_active":"2024-02-10T07:49:06.243569+0000","last_became_peered":"2024-02-10T07:49:06.243569+0000","last_unstale":"2024-02-10T07:50:26.038368+0000","last_undegraded":"2024-02-10T07:50:26.038368+0000","last_fullsized":"2024-02-10T07:50:26.038368+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-02-10T07:49:43.338928+0000","last_deep_scrub":"245'17","last_deep_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_clean_scrub_stamp":"2024-02-10T07:49:43.338928+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:49.402156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":195,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045655+0000","last_change":"2024-02-10T07:49:04.308934+0000","last_active":"2024-02-10T07:50:26.045655+0000","last_peered":"2024-02-10T07:50:26.045655+0000","last_clean":"2024-02-10T07:50:26.045655+0000","last_became_active":"2024-02-10T07:46:10.723006+0000","last_became_peered":"2024-02-10T07:46:10.723006+0000","last_unstale":"2024-02-10T07:50:26.045655+0000","last_undegraded":"2024-02-10T07:50:26.045655+0000","last_fullsized":"2024-02-10T07:50:26.045655+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:04.308807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.308807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.812692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"400'3261","reported_seq":3289,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:28.341350+0000","last_change":"2024-02-10T07:49:53.374953+0000","last_active":"2024-02-10T07:50:28.341350+0000","last_peered":"2024-02-10T07:50:28.341350+0000","last_clean":"2024-02-10T07:50:28.341350+0000","last_became_active":"2024-02-10T07:49:53.374631+0000","last_became_peered":"2024-02-10T07:49:53.374631+0000","last_unstale":"2024-02-10T07:50:28.341350+0000","last_undegraded":"2024-02-10T07:50:28.341350+0000","last_fullsized":"2024-02-10T07:50:28.341350+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3261,"log_dups_size":0,"ondisk_log_size":3261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:08.897366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213712896,"num_objects":3261,"num_object_clones":0,"num_object_copies":9783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3261,"num_write_kb":208704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":763,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043851+0000","last_change":"2024-02-10T07:49:11.361783+0000","last_active":"2024-02-10T07:50:26.043851+0000","last_peered":"2024-02-10T07:50:26.043851+0000","last_clean":"2024-02-10T07:50:26.043851+0000","last_became_active":"2024-02-10T07:49:06.249148+0000","last_became_peered":"2024-02-10T07:49:06.249148+0000","last_unstale":"2024-02-10T07:50:26.043851+0000","last_undegraded":"2024-02-10T07:50:26.043851+0000","last_fullsized":"2024-02-10T07:50:26.043851+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.361139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:35.884283+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":395,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.046372+0000","last_change":"2024-02-10T07:48:21.172677+0000","last_active":"2024-02-10T07:50:26.046372+0000","last_peered":"2024-02-10T07:50:26.046372+0000","last_clean":"2024-02-10T07:50:26.046372+0000","last_became_active":"2024-02-10T07:46:10.719610+0000","last_became_peered":"2024-02-10T07:46:10.719610+0000","last_unstale":"2024-02-10T07:50:26.046372+0000","last_undegraded":"2024-02-10T07:50:26.046372+0000","last_fullsized":"2024-02-10T07:50:26.046372+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:21.172601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:21.172601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377267+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":765,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045152+0000","last_change":"2024-02-10T07:50:07.252577+0000","last_active":"2024-02-10T07:50:26.045152+0000","last_peered":"2024-02-10T07:50:26.045152+0000","last_clean":"2024-02-10T07:50:26.045152+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:50:26.045152+0000","last_undegraded":"2024-02-10T07:50:26.045152+0000","last_fullsized":"2024-02-10T07:50:26.045152+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.252458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.252458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:09.491987+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"250'35","reported_seq":814,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043273+0000","last_change":"2024-02-10T07:49:49.772965+0000","last_active":"2024-02-10T07:50:26.043273+0000","last_peered":"2024-02-10T07:50:26.043273+0000","last_clean":"2024-02-10T07:50:26.043273+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:50:26.043273+0000","last_undegraded":"2024-02-10T07:50:26.043273+0000","last_fullsized":"2024-02-10T07:50:26.043273+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'35","last_scrub_stamp":"2024-02-10T07:49:49.772852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:49.772852+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.199744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"400'3308","reported_seq":3336,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:28.326564+0000","last_change":"2024-02-10T07:49:53.280069+0000","last_active":"2024-02-10T07:50:28.326564+0000","last_peered":"2024-02-10T07:50:28.326564+0000","last_clean":"2024-02-10T07:50:28.326564+0000","last_became_active":"2024-02-10T07:49:53.279758+0000","last_became_peered":"2024-02-10T07:49:53.279758+0000","last_unstale":"2024-02-10T07:50:28.326564+0000","last_undegraded":"2024-02-10T07:50:28.326564+0000","last_fullsized":"2024-02-10T07:50:28.326564+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3308,"log_dups_size":0,"ondisk_log_size":3308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:19.618017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216793088,"num_objects":3308,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3308,"num_write_kb":211712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":752,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.047294+0000","last_change":"2024-02-10T07:49:06.249190+0000","last_active":"2024-02-10T07:50:26.047294+0000","last_peered":"2024-02-10T07:50:26.047294+0000","last_clean":"2024-02-10T07:50:26.047294+0000","last_became_active":"2024-02-10T07:49:06.248040+0000","last_became_peered":"2024-02-10T07:49:06.248040+0000","last_unstale":"2024-02-10T07:50:26.047294+0000","last_undegraded":"2024-02-10T07:50:26.047294+0000","last_fullsized":"2024-02-10T07:50:26.047294+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:00.336175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.336175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377459+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":724,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.038465+0000","last_change":"2024-02-10T07:49:06.257180+0000","last_active":"2024-02-10T07:50:26.038465+0000","last_peered":"2024-02-10T07:50:26.038465+0000","last_clean":"2024-02-10T07:50:26.038465+0000","last_became_active":"2024-02-10T07:49:06.256303+0000","last_became_peered":"2024-02-10T07:49:06.256303+0000","last_unstale":"2024-02-10T07:50:26.038465+0000","last_undegraded":"2024-02-10T07:50:26.038465+0000","last_fullsized":"2024-02-10T07:50:26.038465+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:28.822700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:28.822700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.422480+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":67,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043516+0000","last_change":"2024-02-10T07:49:52.968676+0000","last_active":"2024-02-10T07:50:26.043516+0000","last_peered":"2024-02-10T07:50:26.043516+0000","last_clean":"2024-02-10T07:50:26.043516+0000","last_became_active":"2024-02-10T07:49:06.249431+0000","last_became_peered":"2024-02-10T07:49:06.249431+0000","last_unstale":"2024-02-10T07:50:26.043516+0000","last_undegraded":"2024-02-10T07:50:26.043516+0000","last_fullsized":"2024-02-10T07:50:26.043516+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.968567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.968567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:06.997404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"250'15","reported_seq":769,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.046962+0000","last_change":"2024-02-10T07:48:59.304277+0000","last_active":"2024-02-10T07:50:26.046962+0000","last_peered":"2024-02-10T07:50:26.046962+0000","last_clean":"2024-02-10T07:50:26.046962+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:50:26.046962+0000","last_undegraded":"2024-02-10T07:50:26.046962+0000","last_fullsized":"2024-02-10T07:50:26.046962+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:48:59.304199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.304199+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.764234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"400'3010","reported_seq":3038,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.303598+0000","last_change":"2024-02-10T07:49:53.284745+0000","last_active":"2024-02-10T07:50:26.303598+0000","last_peered":"2024-02-10T07:50:26.303598+0000","last_clean":"2024-02-10T07:50:26.303598+0000","last_became_active":"2024-02-10T07:49:53.283642+0000","last_became_peered":"2024-02-10T07:49:53.283642+0000","last_unstale":"2024-02-10T07:50:26.303598+0000","last_undegraded":"2024-02-10T07:50:26.303598+0000","last_fullsized":"2024-02-10T07:50:26.303598+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3010,"log_dups_size":0,"ondisk_log_size":3010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:18.016780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197263360,"num_objects":3010,"num_object_clones":0,"num_object_copies":9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3010,"num_write_kb":192640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":67,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044228+0000","last_change":"2024-02-10T07:49:50.032577+0000","last_active":"2024-02-10T07:50:26.044228+0000","last_peered":"2024-02-10T07:50:26.044228+0000","last_clean":"2024-02-10T07:50:26.044228+0000","last_became_active":"2024-02-10T07:49:06.238367+0000","last_became_peered":"2024-02-10T07:49:06.238367+0000","last_unstale":"2024-02-10T07:50:26.044228+0000","last_undegraded":"2024-02-10T07:50:26.044228+0000","last_fullsized":"2024-02-10T07:50:26.044228+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:50.032505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.032505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:15.102164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":168,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.041287+0000","last_change":"2024-02-10T07:49:10.087939+0000","last_active":"2024-02-10T07:50:26.041287+0000","last_peered":"2024-02-10T07:50:26.041287+0000","last_clean":"2024-02-10T07:50:26.041287+0000","last_became_active":"2024-02-10T07:48:02.328316+0000","last_became_peered":"2024-02-10T07:48:02.328316+0000","last_unstale":"2024-02-10T07:50:26.041287+0000","last_undegraded":"2024-02-10T07:50:26.041287+0000","last_fullsized":"2024-02-10T07:50:26.041287+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:49:10.087795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:49:10.087795+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.513645+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":740,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.038628+0000","last_change":"2024-02-10T07:49:19.198419+0000","last_active":"2024-02-10T07:50:26.038628+0000","last_peered":"2024-02-10T07:50:26.038628+0000","last_clean":"2024-02-10T07:50:26.038628+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:50:26.038628+0000","last_undegraded":"2024-02-10T07:50:26.038628+0000","last_fullsized":"2024-02-10T07:50:26.038628+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:19.198312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:04.220328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.198312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:32.194353+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"376'46","reported_seq":1121,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044401+0000","last_change":"2024-02-10T07:49:55.714434+0000","last_active":"2024-02-10T07:50:26.044401+0000","last_peered":"2024-02-10T07:50:26.044401+0000","last_clean":"2024-02-10T07:50:26.044401+0000","last_became_active":"2024-02-10T07:49:06.271470+0000","last_became_peered":"2024-02-10T07:49:06.271470+0000","last_unstale":"2024-02-10T07:50:26.044401+0000","last_undegraded":"2024-02-10T07:50:26.044401+0000","last_fullsized":"2024-02-10T07:50:26.044401+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'46","last_scrub_stamp":"2024-02-10T07:49:55.714323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:55.714323+0000","objects_scrubbed":10,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:17.817850+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":748,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.046759+0000","last_change":"2024-02-10T07:49:17.357664+0000","last_active":"2024-02-10T07:50:26.046759+0000","last_peered":"2024-02-10T07:50:26.046759+0000","last_clean":"2024-02-10T07:50:26.046759+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:50:26.046759+0000","last_undegraded":"2024-02-10T07:50:26.046759+0000","last_fullsized":"2024-02-10T07:50:26.046759+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.357533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:22.786949+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.357533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:31.046922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"400'3256","reported_seq":3283,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:28.457465+0000","last_change":"2024-02-10T07:49:53.374436+0000","last_active":"2024-02-10T07:50:28.457465+0000","last_peered":"2024-02-10T07:50:28.457465+0000","last_clean":"2024-02-10T07:50:28.457465+0000","last_became_active":"2024-02-10T07:49:53.374133+0000","last_became_peered":"2024-02-10T07:49:53.374133+0000","last_unstale":"2024-02-10T07:50:28.457465+0000","last_undegraded":"2024-02-10T07:50:28.457465+0000","last_fullsized":"2024-02-10T07:50:28.457465+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3256,"log_dups_size":0,"ondisk_log_size":3256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.202856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213385216,"num_objects":3256,"num_object_clones":0,"num_object_copies":9768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3256,"num_write_kb":208384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":737,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045228+0000","last_change":"2024-02-10T07:47:50.081222+0000","last_active":"2024-02-10T07:50:26.045228+0000","last_peered":"2024-02-10T07:50:26.045228+0000","last_clean":"2024-02-10T07:50:26.045228+0000","last_became_active":"2024-02-10T07:47:48.149256+0000","last_became_peered":"2024-02-10T07:47:48.149256+0000","last_unstale":"2024-02-10T07:50:26.045228+0000","last_undegraded":"2024-02-10T07:50:26.045228+0000","last_fullsized":"2024-02-10T07:50:26.045228+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.080714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:42.730875+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.080714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":761,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.041380+0000","last_change":"2024-02-10T07:49:49.886431+0000","last_active":"2024-02-10T07:50:26.041380+0000","last_peered":"2024-02-10T07:50:26.041380+0000","last_clean":"2024-02-10T07:50:26.041380+0000","last_became_active":"2024-02-10T07:46:10.726265+0000","last_became_peered":"2024-02-10T07:46:10.726265+0000","last_unstale":"2024-02-10T07:50:26.041380+0000","last_undegraded":"2024-02-10T07:50:26.041380+0000","last_fullsized":"2024-02-10T07:50:26.041380+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:49.886354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:49.886354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:59.343977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":744,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045941+0000","last_change":"2024-02-10T07:49:06.261139+0000","last_active":"2024-02-10T07:50:26.045941+0000","last_peered":"2024-02-10T07:50:26.045941+0000","last_clean":"2024-02-10T07:50:26.045941+0000","last_became_active":"2024-02-10T07:49:06.260826+0000","last_became_peered":"2024-02-10T07:49:06.260826+0000","last_unstale":"2024-02-10T07:50:26.045941+0000","last_undegraded":"2024-02-10T07:50:26.045941+0000","last_fullsized":"2024-02-10T07:50:26.045941+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:15.234112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.234112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377327+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"250'17","reported_seq":101,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044864+0000","last_change":"2024-02-10T07:49:04.340445+0000","last_active":"2024-02-10T07:50:26.044864+0000","last_peered":"2024-02-10T07:50:26.044864+0000","last_clean":"2024-02-10T07:50:26.044864+0000","last_became_active":"2024-02-10T07:48:35.057662+0000","last_became_peered":"2024-02-10T07:48:35.057662+0000","last_unstale":"2024-02-10T07:50:26.044864+0000","last_undegraded":"2024-02-10T07:50:26.044864+0000","last_fullsized":"2024-02-10T07:50:26.044864+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'17","last_scrub_stamp":"2024-02-10T07:49:04.340366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.340366+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":23,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":311,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045930+0000","last_change":"2024-02-10T07:49:50.232630+0000","last_active":"2024-02-10T07:50:26.045930+0000","last_peered":"2024-02-10T07:50:26.045930+0000","last_clean":"2024-02-10T07:50:26.045930+0000","last_became_active":"2024-02-10T07:48:02.331496+0000","last_became_peered":"2024-02-10T07:48:02.331496+0000","last_unstale":"2024-02-10T07:50:26.045930+0000","last_undegraded":"2024-02-10T07:50:26.045930+0000","last_fullsized":"2024-02-10T07:50:26.045930+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:50.232544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.232544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:05.339508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"400'3170","reported_seq":3198,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:28.445678+0000","last_change":"2024-02-10T07:49:53.374470+0000","last_active":"2024-02-10T07:50:28.445678+0000","last_peered":"2024-02-10T07:50:28.445678+0000","last_clean":"2024-02-10T07:50:28.445678+0000","last_became_active":"2024-02-10T07:49:53.374185+0000","last_became_peered":"2024-02-10T07:49:53.374185+0000","last_unstale":"2024-02-10T07:50:28.445678+0000","last_undegraded":"2024-02-10T07:50:28.445678+0000","last_fullsized":"2024-02-10T07:50:28.445678+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3170,"log_dups_size":0,"ondisk_log_size":3170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:53.548261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207749120,"num_objects":3170,"num_object_clones":0,"num_object_copies":9510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3170,"num_write_kb":202880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":157,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.037976+0000","last_change":"2024-02-10T07:48:02.330253+0000","last_active":"2024-02-10T07:50:26.037976+0000","last_peered":"2024-02-10T07:50:26.037976+0000","last_clean":"2024-02-10T07:50:26.037976+0000","last_became_active":"2024-02-10T07:48:02.329687+0000","last_became_peered":"2024-02-10T07:48:02.329687+0000","last_unstale":"2024-02-10T07:50:26.037976+0000","last_undegraded":"2024-02-10T07:50:26.037976+0000","last_fullsized":"2024-02-10T07:50:26.037976+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:47:49.750508+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.750508+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.422647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":994,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044048+0000","last_change":"2024-02-10T07:48:15.033077+0000","last_active":"2024-02-10T07:50:26.044048+0000","last_peered":"2024-02-10T07:50:26.044048+0000","last_clean":"2024-02-10T07:50:26.044048+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:50:26.044048+0000","last_undegraded":"2024-02-10T07:50:26.044048+0000","last_fullsized":"2024-02-10T07:50:26.044048+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:48:15.032944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.032944+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":378,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044720+0000","last_change":"2024-02-10T07:49:06.257190+0000","last_active":"2024-02-10T07:50:26.044720+0000","last_peered":"2024-02-10T07:50:26.044720+0000","last_clean":"2024-02-10T07:50:26.044720+0000","last_became_active":"2024-02-10T07:49:06.256223+0000","last_became_peered":"2024-02-10T07:49:06.256223+0000","last_unstale":"2024-02-10T07:50:26.044720+0000","last_undegraded":"2024-02-10T07:50:26.044720+0000","last_fullsized":"2024-02-10T07:50:26.044720+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:17.198213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:17.198213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:30.915216+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"250'36","reported_seq":1031,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044407+0000","last_change":"2024-02-10T07:49:58.841289+0000","last_active":"2024-02-10T07:50:26.044407+0000","last_peered":"2024-02-10T07:50:26.044407+0000","last_clean":"2024-02-10T07:50:26.044407+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:50:26.044407+0000","last_undegraded":"2024-02-10T07:50:26.044407+0000","last_fullsized":"2024-02-10T07:50:26.044407+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'36","last_scrub_stamp":"2024-02-10T07:49:58.841172+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:49:58.841172+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:07.505517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":763,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044997+0000","last_change":"2024-02-10T07:49:13.369870+0000","last_active":"2024-02-10T07:50:26.044997+0000","last_peered":"2024-02-10T07:50:26.044997+0000","last_clean":"2024-02-10T07:50:26.044997+0000","last_became_active":"2024-02-10T07:46:10.714440+0000","last_became_peered":"2024-02-10T07:46:10.714440+0000","last_unstale":"2024-02-10T07:50:26.044997+0000","last_undegraded":"2024-02-10T07:50:26.044997+0000","last_fullsized":"2024-02-10T07:50:26.044997+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:13.369607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.736901+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.369607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:29.365328+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"400'3158","reported_seq":3186,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:28.437746+0000","last_change":"2024-02-10T07:49:53.283728+0000","last_active":"2024-02-10T07:50:28.437746+0000","last_peered":"2024-02-10T07:50:28.437746+0000","last_clean":"2024-02-10T07:50:28.437746+0000","last_became_active":"2024-02-10T07:49:53.283492+0000","last_became_peered":"2024-02-10T07:49:53.283492+0000","last_unstale":"2024-02-10T07:50:28.437746+0000","last_undegraded":"2024-02-10T07:50:28.437746+0000","last_fullsized":"2024-02-10T07:50:28.437746+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3158,"log_dups_size":0,"ondisk_log_size":3158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.523341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206962688,"num_objects":3158,"num_object_clones":0,"num_object_copies":9474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3158,"num_write_kb":202112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":541,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043893+0000","last_change":"2024-02-10T07:50:07.589020+0000","last_active":"2024-02-10T07:50:26.043893+0000","last_peered":"2024-02-10T07:50:26.043893+0000","last_clean":"2024-02-10T07:50:26.043893+0000","last_became_active":"2024-02-10T07:49:06.270457+0000","last_became_peered":"2024-02-10T07:49:06.270457+0000","last_unstale":"2024-02-10T07:50:26.043893+0000","last_undegraded":"2024-02-10T07:50:26.043893+0000","last_fullsized":"2024-02-10T07:50:26.043893+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.588888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.588888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:26.055454+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":755,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.046080+0000","last_change":"2024-02-10T07:49:06.267039+0000","last_active":"2024-02-10T07:50:26.046080+0000","last_peered":"2024-02-10T07:50:26.046080+0000","last_clean":"2024-02-10T07:50:26.046080+0000","last_became_active":"2024-02-10T07:49:06.266625+0000","last_became_peered":"2024-02-10T07:49:06.266625+0000","last_unstale":"2024-02-10T07:50:26.046080+0000","last_undegraded":"2024-02-10T07:50:26.046080+0000","last_fullsized":"2024-02-10T07:50:26.046080+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_clean_scrub_stamp":"2024-02-10T07:47:41.181770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1879,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:27.056251+0000","last_change":"2024-02-10T07:50:04.159154+0000","last_active":"2024-02-10T07:50:27.056251+0000","last_peered":"2024-02-10T07:50:27.056251+0000","last_clean":"2024-02-10T07:50:27.056251+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:50:27.056251+0000","last_undegraded":"2024-02-10T07:50:27.056251+0000","last_fullsized":"2024-02-10T07:50:27.056251+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:50:04.159046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.159046+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:14.041650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1110,"num_read_kb":1105,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"250'30","reported_seq":806,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.049373+0000","last_change":"2024-02-10T07:49:47.417190+0000","last_active":"2024-02-10T07:50:26.049373+0000","last_peered":"2024-02-10T07:50:26.049373+0000","last_clean":"2024-02-10T07:50:26.049373+0000","last_became_active":"2024-02-10T07:46:10.724393+0000","last_became_peered":"2024-02-10T07:46:10.724393+0000","last_unstale":"2024-02-10T07:50:26.049373+0000","last_undegraded":"2024-02-10T07:50:26.049373+0000","last_fullsized":"2024-02-10T07:50:26.049373+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:47.417096+0000","last_deep_scrub":"237'18","last_deep_scrub_stamp":"2024-02-10T07:40:31.005376+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.417096+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:05.253527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":695,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.050345+0000","last_change":"2024-02-10T07:49:52.615217+0000","last_active":"2024-02-10T07:50:26.050345+0000","last_peered":"2024-02-10T07:50:26.050345+0000","last_clean":"2024-02-10T07:50:26.050345+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:50:26.050345+0000","last_undegraded":"2024-02-10T07:50:26.050345+0000","last_fullsized":"2024-02-10T07:50:26.050345+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.615130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.615130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.239753+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"400'3228","reported_seq":3256,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:29.109635+0000","last_change":"2024-02-10T07:49:53.280610+0000","last_active":"2024-02-10T07:50:29.109635+0000","last_peered":"2024-02-10T07:50:29.109635+0000","last_clean":"2024-02-10T07:50:29.109635+0000","last_became_active":"2024-02-10T07:49:53.280272+0000","last_became_peered":"2024-02-10T07:49:53.280272+0000","last_unstale":"2024-02-10T07:50:29.109635+0000","last_undegraded":"2024-02-10T07:50:29.109635+0000","last_fullsized":"2024-02-10T07:50:29.109635+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3228,"log_dups_size":0,"ondisk_log_size":3228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:55.982947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211550208,"num_objects":3228,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3228,"num_write_kb":206592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":756,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043552+0000","last_change":"2024-02-10T07:49:59.630159+0000","last_active":"2024-02-10T07:50:26.043552+0000","last_peered":"2024-02-10T07:50:26.043552+0000","last_clean":"2024-02-10T07:50:26.043552+0000","last_became_active":"2024-02-10T07:46:10.716852+0000","last_became_peered":"2024-02-10T07:46:10.716852+0000","last_unstale":"2024-02-10T07:50:26.043552+0000","last_undegraded":"2024-02-10T07:50:26.043552+0000","last_fullsized":"2024-02-10T07:50:26.043552+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.630083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.630083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:11.435808+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":761,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044229+0000","last_change":"2024-02-10T07:49:05.193351+0000","last_active":"2024-02-10T07:50:26.044229+0000","last_peered":"2024-02-10T07:50:26.044229+0000","last_clean":"2024-02-10T07:50:26.044229+0000","last_became_active":"2024-02-10T07:46:10.718340+0000","last_became_peered":"2024-02-10T07:46:10.718340+0000","last_unstale":"2024-02-10T07:50:26.044229+0000","last_undegraded":"2024-02-10T07:50:26.044229+0000","last_fullsized":"2024-02-10T07:50:26.044229+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:05.193158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:49:05.193158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:31.087181+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"271'37","reported_seq":812,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.046722+0000","last_change":"2024-02-10T07:49:57.522503+0000","last_active":"2024-02-10T07:50:26.046722+0000","last_peered":"2024-02-10T07:50:26.046722+0000","last_clean":"2024-02-10T07:50:26.046722+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:50:26.046722+0000","last_undegraded":"2024-02-10T07:50:26.046722+0000","last_fullsized":"2024-02-10T07:50:26.046722+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'37","last_scrub_stamp":"2024-02-10T07:49:57.522391+0000","last_deep_scrub":"271'37","last_deep_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.522391+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:18.289907+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":770,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043368+0000","last_change":"2024-02-10T07:49:03.013730+0000","last_active":"2024-02-10T07:50:26.043368+0000","last_peered":"2024-02-10T07:50:26.043368+0000","last_clean":"2024-02-10T07:50:26.043368+0000","last_became_active":"2024-02-10T07:48:35.062118+0000","last_became_peered":"2024-02-10T07:48:35.062118+0000","last_unstale":"2024-02-10T07:50:26.043368+0000","last_undegraded":"2024-02-10T07:50:26.043368+0000","last_fullsized":"2024-02-10T07:50:26.043368+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:03.013625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:00.292320+0000","last_clean_scrub_stamp":"2024-02-10T07:49:03.013625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:30.915248+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"400'3199","reported_seq":3227,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:28.438902+0000","last_change":"2024-02-10T07:49:53.280094+0000","last_active":"2024-02-10T07:50:28.438902+0000","last_peered":"2024-02-10T07:50:28.438902+0000","last_clean":"2024-02-10T07:50:28.438902+0000","last_became_active":"2024-02-10T07:49:53.279765+0000","last_became_peered":"2024-02-10T07:49:53.279765+0000","last_unstale":"2024-02-10T07:50:28.438902+0000","last_undegraded":"2024-02-10T07:50:28.438902+0000","last_fullsized":"2024-02-10T07:50:28.438902+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3199,"log_dups_size":0,"ondisk_log_size":3199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:54.175208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209649664,"num_objects":3199,"num_object_clones":0,"num_object_copies":9597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3199,"num_write_kb":204736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":631,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.042856+0000","last_change":"2024-02-10T07:49:57.848098+0000","last_active":"2024-02-10T07:50:26.042856+0000","last_peered":"2024-02-10T07:50:26.042856+0000","last_clean":"2024-02-10T07:50:26.042856+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:50:26.042856+0000","last_undegraded":"2024-02-10T07:50:26.042856+0000","last_fullsized":"2024-02-10T07:50:26.042856+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:57.847989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.847989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:26.850826+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":670,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043544+0000","last_change":"2024-02-10T07:49:48.809609+0000","last_active":"2024-02-10T07:50:26.043544+0000","last_peered":"2024-02-10T07:50:26.043544+0000","last_clean":"2024-02-10T07:50:26.043544+0000","last_became_active":"2024-02-10T07:49:06.256488+0000","last_became_peered":"2024-02-10T07:49:06.256488+0000","last_unstale":"2024-02-10T07:50:26.043544+0000","last_undegraded":"2024-02-10T07:50:26.043544+0000","last_fullsized":"2024-02-10T07:50:26.043544+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:48.809513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:48.809513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:03.569894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"382'25","reported_seq":558,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044195+0000","last_change":"2024-02-10T07:49:19.084010+0000","last_active":"2024-02-10T07:50:26.044195+0000","last_peered":"2024-02-10T07:50:26.044195+0000","last_clean":"2024-02-10T07:50:26.044195+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:50:26.044195+0000","last_undegraded":"2024-02-10T07:50:26.044195+0000","last_fullsized":"2024-02-10T07:50:26.044195+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'25","last_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_deep_scrub":"382'25","last_deep_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.083856+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:34.153113+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"158'10","reported_seq":982,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.047092+0000","last_change":"2024-02-10T07:49:50.396822+0000","last_active":"2024-02-10T07:50:26.047092+0000","last_peered":"2024-02-10T07:50:26.047092+0000","last_clean":"2024-02-10T07:50:26.047092+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:50:26.047092+0000","last_undegraded":"2024-02-10T07:50:26.047092+0000","last_fullsized":"2024-02-10T07:50:26.047092+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:49:50.396730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.396730+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:52.711726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.e","version":"400'3156","reported_seq":3184,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:28.465956+0000","last_change":"2024-02-10T07:49:53.281229+0000","last_active":"2024-02-10T07:50:28.465956+0000","last_peered":"2024-02-10T07:50:28.465956+0000","last_clean":"2024-02-10T07:50:28.465956+0000","last_became_active":"2024-02-10T07:49:53.280920+0000","last_became_peered":"2024-02-10T07:49:53.280920+0000","last_unstale":"2024-02-10T07:50:28.465956+0000","last_undegraded":"2024-02-10T07:50:28.465956+0000","last_fullsized":"2024-02-10T07:50:28.465956+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3156,"log_dups_size":0,"ondisk_log_size":3156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:05.553881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206831616,"num_objects":3156,"num_object_clones":0,"num_object_copies":9468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3156,"num_write_kb":201984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":743,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045275+0000","last_change":"2024-02-10T07:48:23.271762+0000","last_active":"2024-02-10T07:50:26.045275+0000","last_peered":"2024-02-10T07:50:26.045275+0000","last_clean":"2024-02-10T07:50:26.045275+0000","last_became_active":"2024-02-10T07:46:10.719534+0000","last_became_peered":"2024-02-10T07:46:10.719534+0000","last_unstale":"2024-02-10T07:50:26.045275+0000","last_undegraded":"2024-02-10T07:50:26.045275+0000","last_fullsized":"2024-02-10T07:50:26.045275+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_clean_scrub_stamp":"2024-02-10T07:48:23.271454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:32.294804+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"296'2","reported_seq":758,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.049474+0000","last_change":"2024-02-10T07:50:03.643473+0000","last_active":"2024-02-10T07:50:26.049474+0000","last_peered":"2024-02-10T07:50:26.049474+0000","last_clean":"2024-02-10T07:50:26.049474+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:50:26.049474+0000","last_undegraded":"2024-02-10T07:50:26.049474+0000","last_fullsized":"2024-02-10T07:50:26.049474+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'2","last_scrub_stamp":"2024-02-10T07:50:03.643367+0000","last_deep_scrub":"296'2","last_deep_scrub_stamp":"2024-02-10T07:45:03.233375+0000","last_clean_scrub_stamp":"2024-02-10T07:50:03.643367+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:15.296940+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"250'28","reported_seq":787,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.038180+0000","last_change":"2024-02-10T07:49:20.216865+0000","last_active":"2024-02-10T07:50:26.038180+0000","last_peered":"2024-02-10T07:50:26.038180+0000","last_clean":"2024-02-10T07:50:26.038180+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:50:26.038180+0000","last_undegraded":"2024-02-10T07:50:26.038180+0000","last_fullsized":"2024-02-10T07:50:26.038180+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-02-10T07:49:20.216750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:20.216750+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:29.469107+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"376'12","reported_seq":769,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.041151+0000","last_change":"2024-02-10T07:49:06.282456+0000","last_active":"2024-02-10T07:50:26.041151+0000","last_peered":"2024-02-10T07:50:26.041151+0000","last_clean":"2024-02-10T07:50:26.041151+0000","last_became_active":"2024-02-10T07:49:06.255324+0000","last_became_peered":"2024-02-10T07:49:06.255324+0000","last_unstale":"2024-02-10T07:50:26.041151+0000","last_undegraded":"2024-02-10T07:50:26.041151+0000","last_fullsized":"2024-02-10T07:50:26.041151+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'11","last_scrub_stamp":"2024-02-10T07:48:58.368663+0000","last_deep_scrub":"227'9","last_deep_scrub_stamp":"2024-02-10T07:39:53.440221+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.368663+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"400'3169","reported_seq":3197,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:28.332037+0000","last_change":"2024-02-10T07:49:53.283453+0000","last_active":"2024-02-10T07:50:28.332037+0000","last_peered":"2024-02-10T07:50:28.332037+0000","last_clean":"2024-02-10T07:50:28.332037+0000","last_became_active":"2024-02-10T07:49:53.283144+0000","last_became_peered":"2024-02-10T07:49:53.283144+0000","last_unstale":"2024-02-10T07:50:28.332037+0000","last_undegraded":"2024-02-10T07:50:28.332037+0000","last_fullsized":"2024-02-10T07:50:28.332037+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3169,"log_dups_size":0,"ondisk_log_size":3169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:09.376120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207683584,"num_objects":3169,"num_object_clones":0,"num_object_copies":9507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3169,"num_write_kb":202816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":739,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.038117+0000","last_change":"2024-02-10T07:49:06.252967+0000","last_active":"2024-02-10T07:50:26.038117+0000","last_peered":"2024-02-10T07:50:26.038117+0000","last_clean":"2024-02-10T07:50:26.038117+0000","last_became_active":"2024-02-10T07:49:06.251750+0000","last_became_peered":"2024-02-10T07:49:06.251750+0000","last_unstale":"2024-02-10T07:50:26.038117+0000","last_undegraded":"2024-02-10T07:50:26.038117+0000","last_fullsized":"2024-02-10T07:50:26.038117+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:27.819612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:46:27.819612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.422550+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":230,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.045290+0000","last_change":"2024-02-10T07:47:50.948710+0000","last_active":"2024-02-10T07:50:26.045290+0000","last_peered":"2024-02-10T07:50:26.045290+0000","last_clean":"2024-02-10T07:50:26.045290+0000","last_became_active":"2024-02-10T07:47:47.966443+0000","last_became_peered":"2024-02-10T07:47:47.966443+0000","last_unstale":"2024-02-10T07:50:26.045290+0000","last_undegraded":"2024-02-10T07:50:26.045290+0000","last_fullsized":"2024-02-10T07:50:26.045290+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.948624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.948624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"250'30","reported_seq":729,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044657+0000","last_change":"2024-02-10T07:49:57.303310+0000","last_active":"2024-02-10T07:50:26.044657+0000","last_peered":"2024-02-10T07:50:26.044657+0000","last_clean":"2024-02-10T07:50:26.044657+0000","last_became_active":"2024-02-10T07:49:06.252935+0000","last_became_peered":"2024-02-10T07:49:06.252935+0000","last_unstale":"2024-02-10T07:50:26.044657+0000","last_undegraded":"2024-02-10T07:50:26.044657+0000","last_fullsized":"2024-02-10T07:50:26.044657+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:57.303200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.303200+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:15.856547+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":747,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.037960+0000","last_change":"2024-02-10T07:49:06.239655+0000","last_active":"2024-02-10T07:50:26.037960+0000","last_peered":"2024-02-10T07:50:26.037960+0000","last_clean":"2024-02-10T07:50:26.037960+0000","last_became_active":"2024-02-10T07:49:06.239190+0000","last_became_peered":"2024-02-10T07:49:06.239190+0000","last_unstale":"2024-02-10T07:50:26.037960+0000","last_undegraded":"2024-02-10T07:50:26.037960+0000","last_fullsized":"2024-02-10T07:50:26.037960+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:42.549018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.549018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.422613+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"400'3271","reported_seq":3299,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:29.141322+0000","last_change":"2024-02-10T07:49:53.282174+0000","last_active":"2024-02-10T07:50:29.141322+0000","last_peered":"2024-02-10T07:50:29.141322+0000","last_clean":"2024-02-10T07:50:29.141322+0000","last_became_active":"2024-02-10T07:49:53.281832+0000","last_became_peered":"2024-02-10T07:49:53.281832+0000","last_unstale":"2024-02-10T07:50:29.141322+0000","last_undegraded":"2024-02-10T07:50:29.141322+0000","last_fullsized":"2024-02-10T07:50:29.141322+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3271,"log_dups_size":0,"ondisk_log_size":3271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:08.492345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214368256,"num_objects":3271,"num_object_clones":0,"num_object_copies":9813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3271,"num_write_kb":209344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":255,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.040978+0000","last_change":"2024-02-10T07:49:06.274374+0000","last_active":"2024-02-10T07:50:26.040978+0000","last_peered":"2024-02-10T07:50:26.040978+0000","last_clean":"2024-02-10T07:50:26.040978+0000","last_became_active":"2024-02-10T07:48:02.371926+0000","last_became_peered":"2024-02-10T07:48:02.371926+0000","last_unstale":"2024-02-10T07:50:26.040978+0000","last_undegraded":"2024-02-10T07:50:26.040978+0000","last_fullsized":"2024-02-10T07:50:26.040978+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.274263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:49.704024+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.274263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.513614+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"395'12","reported_seq":117,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043553+0000","last_change":"2024-02-10T07:50:02.810307+0000","last_active":"2024-02-10T07:50:26.043553+0000","last_peered":"2024-02-10T07:50:26.043553+0000","last_clean":"2024-02-10T07:50:26.043553+0000","last_became_active":"2024-02-10T07:48:35.071916+0000","last_became_peered":"2024-02-10T07:48:35.071916+0000","last_unstale":"2024-02-10T07:50:26.043553+0000","last_undegraded":"2024-02-10T07:50:26.043553+0000","last_fullsized":"2024-02-10T07:50:26.043553+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'12","last_scrub_stamp":"2024-02-10T07:50:02.810229+0000","last_deep_scrub":"395'12","last_deep_scrub_stamp":"2024-02-10T07:50:02.810229+0000","last_clean_scrub_stamp":"2024-02-10T07:50:02.810229+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:28.369227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":67,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043321+0000","last_change":"2024-02-10T07:49:44.962126+0000","last_active":"2024-02-10T07:50:26.043321+0000","last_peered":"2024-02-10T07:50:26.043321+0000","last_clean":"2024-02-10T07:50:26.043321+0000","last_became_active":"2024-02-10T07:49:06.251038+0000","last_became_peered":"2024-02-10T07:49:06.251038+0000","last_unstale":"2024-02-10T07:50:26.043321+0000","last_undegraded":"2024-02-10T07:50:26.043321+0000","last_fullsized":"2024-02-10T07:50:26.043321+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:44.962041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:44.962041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:12.429908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"250'27","reported_seq":802,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.048246+0000","last_change":"2024-02-10T07:50:00.655466+0000","last_active":"2024-02-10T07:50:26.048246+0000","last_peered":"2024-02-10T07:50:26.048246+0000","last_clean":"2024-02-10T07:50:26.048246+0000","last_became_active":"2024-02-10T07:47:47.967427+0000","last_became_peered":"2024-02-10T07:47:47.967427+0000","last_unstale":"2024-02-10T07:50:26.048246+0000","last_undegraded":"2024-02-10T07:50:26.048246+0000","last_fullsized":"2024-02-10T07:50:26.048246+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:50:00.655377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.655377+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:26.934415+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"400'3302","reported_seq":3330,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:28.454624+0000","last_change":"2024-02-10T07:49:53.283708+0000","last_active":"2024-02-10T07:50:28.454624+0000","last_peered":"2024-02-10T07:50:28.454624+0000","last_clean":"2024-02-10T07:50:28.454624+0000","last_became_active":"2024-02-10T07:49:53.283415+0000","last_became_peered":"2024-02-10T07:49:53.283415+0000","last_unstale":"2024-02-10T07:50:28.454624+0000","last_undegraded":"2024-02-10T07:50:28.454624+0000","last_fullsized":"2024-02-10T07:50:28.454624+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3302,"log_dups_size":0,"ondisk_log_size":3302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.247094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216399872,"num_objects":3302,"num_object_clones":0,"num_object_copies":9906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3302,"num_write_kb":211328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":441,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.043596+0000","last_change":"2024-02-10T07:50:00.799049+0000","last_active":"2024-02-10T07:50:26.043596+0000","last_peered":"2024-02-10T07:50:26.043596+0000","last_clean":"2024-02-10T07:50:26.043596+0000","last_became_active":"2024-02-10T07:49:06.268874+0000","last_became_peered":"2024-02-10T07:49:06.268874+0000","last_unstale":"2024-02-10T07:50:26.043596+0000","last_undegraded":"2024-02-10T07:50:26.043596+0000","last_fullsized":"2024-02-10T07:50:26.043596+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.798928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.798928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:06.248341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":229,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.039993+0000","last_change":"2024-02-10T07:49:57.310447+0000","last_active":"2024-02-10T07:50:26.039993+0000","last_peered":"2024-02-10T07:50:26.039993+0000","last_clean":"2024-02-10T07:50:26.039993+0000","last_became_active":"2024-02-10T07:48:02.333689+0000","last_became_peered":"2024-02-10T07:48:02.333689+0000","last_unstale":"2024-02-10T07:50:26.039993+0000","last_undegraded":"2024-02-10T07:50:26.039993+0000","last_fullsized":"2024-02-10T07:50:26.039993+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:57.310333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.310333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:22.249646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":746,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.038043+0000","last_change":"2024-02-10T07:46:37.995480+0000","last_active":"2024-02-10T07:50:26.038043+0000","last_peered":"2024-02-10T07:50:26.038043+0000","last_clean":"2024-02-10T07:50:26.038043+0000","last_became_active":"2024-02-10T07:46:10.721886+0000","last_became_peered":"2024-02-10T07:46:10.721886+0000","last_unstale":"2024-02-10T07:50:26.038043+0000","last_undegraded":"2024-02-10T07:50:26.038043+0000","last_fullsized":"2024-02-10T07:50:26.038043+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:37.995362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.995362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.422520+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"250'24","reported_seq":826,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.040226+0000","last_change":"2024-02-10T07:49:16.249257+0000","last_active":"2024-02-10T07:50:26.040226+0000","last_peered":"2024-02-10T07:50:26.040226+0000","last_clean":"2024-02-10T07:50:26.040226+0000","last_became_active":"2024-02-10T07:48:35.062573+0000","last_became_peered":"2024-02-10T07:48:35.062573+0000","last_unstale":"2024-02-10T07:50:26.040226+0000","last_undegraded":"2024-02-10T07:50:26.040226+0000","last_fullsized":"2024-02-10T07:50:26.040226+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:49:16.248936+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.248936+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"400'3235","reported_seq":3263,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:29.178801+0000","last_change":"2024-02-10T07:49:53.285110+0000","last_active":"2024-02-10T07:50:29.178801+0000","last_peered":"2024-02-10T07:50:29.178801+0000","last_clean":"2024-02-10T07:50:29.178801+0000","last_became_active":"2024-02-10T07:49:53.284807+0000","last_became_peered":"2024-02-10T07:49:53.284807+0000","last_unstale":"2024-02-10T07:50:29.178801+0000","last_undegraded":"2024-02-10T07:50:29.178801+0000","last_fullsized":"2024-02-10T07:50:29.178801+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3235,"log_dups_size":0,"ondisk_log_size":3235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:05.874090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212008960,"num_objects":3235,"num_object_clones":0,"num_object_copies":9705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3235,"num_write_kb":207040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":67,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044526+0000","last_change":"2024-02-10T07:49:17.153443+0000","last_active":"2024-02-10T07:50:26.044526+0000","last_peered":"2024-02-10T07:50:26.044526+0000","last_clean":"2024-02-10T07:50:26.044526+0000","last_became_active":"2024-02-10T07:49:06.256104+0000","last_became_peered":"2024-02-10T07:49:06.256104+0000","last_unstale":"2024-02-10T07:50:26.044526+0000","last_undegraded":"2024-02-10T07:50:26.044526+0000","last_fullsized":"2024-02-10T07:50:26.044526+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.153368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.153368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:27.413097+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":738,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044451+0000","last_change":"2024-02-10T07:49:07.170105+0000","last_active":"2024-02-10T07:50:26.044451+0000","last_peered":"2024-02-10T07:50:26.044451+0000","last_clean":"2024-02-10T07:50:26.044451+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:50:26.044451+0000","last_undegraded":"2024-02-10T07:50:26.044451+0000","last_fullsized":"2024-02-10T07:50:26.044451+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:07.169962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:07.169962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:27.005819+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":771,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044561+0000","last_change":"2024-02-10T07:50:03.803447+0000","last_active":"2024-02-10T07:50:26.044561+0000","last_peered":"2024-02-10T07:50:26.044561+0000","last_clean":"2024-02-10T07:50:26.044561+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:50:26.044561+0000","last_undegraded":"2024-02-10T07:50:26.044561+0000","last_fullsized":"2024-02-10T07:50:26.044561+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:03.803361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:03.803361+0000","last_clean_scrub_stamp":"2024-02-10T07:50:03.803361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:28.611872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"250'39","reported_seq":841,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.039616+0000","last_change":"2024-02-10T07:48:59.066379+0000","last_active":"2024-02-10T07:50:26.039616+0000","last_peered":"2024-02-10T07:50:26.039616+0000","last_clean":"2024-02-10T07:50:26.039616+0000","last_became_active":"2024-02-10T07:48:35.062134+0000","last_became_peered":"2024-02-10T07:48:35.062134+0000","last_unstale":"2024-02-10T07:50:26.039616+0000","last_undegraded":"2024-02-10T07:50:26.039616+0000","last_fullsized":"2024-02-10T07:50:26.039616+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T07:48:59.066256+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.066256+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.422902+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"400'3241","reported_seq":3268,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:28.458047+0000","last_change":"2024-02-10T07:49:53.281316+0000","last_active":"2024-02-10T07:50:28.458047+0000","last_peered":"2024-02-10T07:50:28.458047+0000","last_clean":"2024-02-10T07:50:28.458047+0000","last_became_active":"2024-02-10T07:49:53.280975+0000","last_became_peered":"2024-02-10T07:49:53.280975+0000","last_unstale":"2024-02-10T07:50:28.458047+0000","last_undegraded":"2024-02-10T07:50:28.458047+0000","last_fullsized":"2024-02-10T07:50:28.458047+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":3241,"log_dups_size":0,"ondisk_log_size":3241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:54.358771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212402176,"num_objects":3241,"num_object_clones":0,"num_object_copies":9723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3241,"num_write_kb":207424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":734,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.040500+0000","last_change":"2024-02-10T07:49:47.275981+0000","last_active":"2024-02-10T07:50:26.040500+0000","last_peered":"2024-02-10T07:50:26.040500+0000","last_clean":"2024-02-10T07:50:26.040500+0000","last_became_active":"2024-02-10T07:46:10.717081+0000","last_became_peered":"2024-02-10T07:46:10.717081+0000","last_unstale":"2024-02-10T07:50:26.040500+0000","last_undegraded":"2024-02-10T07:50:26.040500+0000","last_fullsized":"2024-02-10T07:50:26.040500+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:47.275882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.275882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:47.745153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"313'11","reported_seq":286,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.044994+0000","last_change":"2024-02-10T07:49:14.335191+0000","last_active":"2024-02-10T07:50:26.044994+0000","last_peered":"2024-02-10T07:50:26.044994+0000","last_clean":"2024-02-10T07:50:26.044994+0000","last_became_active":"2024-02-10T07:46:10.719380+0000","last_became_peered":"2024-02-10T07:46:10.719380+0000","last_unstale":"2024-02-10T07:50:26.044994+0000","last_undegraded":"2024-02-10T07:50:26.044994+0000","last_fullsized":"2024-02-10T07:50:26.044994+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'11","last_scrub_stamp":"2024-02-10T07:49:14.335064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:14.335064+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:32.784071+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":237,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.046885+0000","last_change":"2024-02-10T07:49:06.259646+0000","last_active":"2024-02-10T07:50:26.046885+0000","last_peered":"2024-02-10T07:50:26.046885+0000","last_clean":"2024-02-10T07:50:26.046885+0000","last_became_active":"2024-02-10T07:49:06.258926+0000","last_became_peered":"2024-02-10T07:49:06.258926+0000","last_unstale":"2024-02-10T07:50:26.046885+0000","last_undegraded":"2024-02-10T07:50:26.046885+0000","last_fullsized":"2024-02-10T07:50:26.046885+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:24.220575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_clean_scrub_stamp":"2024-02-10T07:47:24.220575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:26.377013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"250'20","reported_seq":805,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-10T07:50:26.039343+0000","last_change":"2024-02-10T07:49:06.289632+0000","last_active":"2024-02-10T07:50:26.039343+0000","last_peered":"2024-02-10T07:50:26.039343+0000","last_clean":"2024-02-10T07:50:26.039343+0000","last_became_active":"2024-02-10T07:49:06.238858+0000","last_became_peered":"2024-02-10T07:49:06.238858+0000","last_unstale":"2024-02-10T07:50:26.039343+0000","last_undegraded":"2024-02-10T07:50:26.039343+0000","last_fullsized":"2024-02-10T07:50:26.039343+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:48:15.588199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.588199+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:28.422785+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3376873472,"num_objects":51527,"num_object_clones":0,"num_object_copies":154581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51527,"num_write_kb":3297728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10058072064,"data_stored":10058072064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51527,"ondisk_log_size":51527,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1110,"num_read_kb":1105,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":5804,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450560,"data_stored":11904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":773,"ondisk_log_size":773,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037988,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2297556,"kb_used_data":1638860,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":91447596,"statfs":{"total":95995035648,"available":93642338304,"internally_reserved":0,"allocated":1678192640,"data_stored":1677848435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":674494922},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:50:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.369,"15min":1.8089999999999999},"min":{"1min":0.372,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":3.3530000000000002,"5min":39.494,"15min":39.494},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":1.258,"15min":1.694},"min":{"1min":0.36599999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.2429999999999999,"5min":38.524000000000001,"15min":38.524000000000001},"last":0.67400000000000004}]},{"osd":1,"last update":"Sat Feb 10 07:50:17 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.98299999999999998,"15min":1.7150000000000001},"min":{"1min":0.39800000000000002,"5min":0.25700000000000001,"15min":0.24399999999999999},"max":{"1min":3.5089999999999999,"5min":4.2640000000000002,"15min":35.459000000000003},"last":0.443},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.95299999999999996,"15min":1.6879999999999999},"min":{"1min":0.32700000000000001,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":3.1040000000000001,"5min":4.7640000000000002,"15min":35.225000000000001},"last":0.73499999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:50:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.83099999999999996,"15min":1.2090000000000001},"min":{"1min":0.307,"5min":0.255,"15min":0.255},"max":{"1min":2.1230000000000002,"5min":5.4980000000000002,"15min":19.710000000000001},"last":2.165},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.93600000000000005,"15min":1.3799999999999999},"min":{"1min":0.45200000000000001,"5min":0.32400000000000001,"15min":0.29099999999999998},"max":{"1min":3.3130000000000002,"5min":4.2320000000000002,"15min":19.643000000000001},"last":0.59999999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:49:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.103,"15min":1.069},"min":{"1min":0.33000000000000002,"5min":0.307,"15min":0.307},"max":{"1min":4.0990000000000002,"5min":4.0990000000000002,"15min":4.0990000000000002},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.92300000000000004,"15min":0.92200000000000004},"min":{"1min":0.33400000000000002,"5min":0.313,"15min":0.313},"max":{"1min":3.9580000000000002,"5min":3.9580000000000002,"15min":3.9580000000000002},"last":2.2250000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:50:17 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.9099999999999999,"15min":1.117},"min":{"1min":0.372,"5min":0.22,"15min":0.22},"max":{"1min":3.2719999999999998,"5min":26.001000000000001,"15min":26.001000000000001},"last":3.3980000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":1.9550000000000001,"15min":1.119},"min":{"1min":0.36699999999999999,"5min":0.26800000000000002,"15min":0.24099999999999999},"max":{"1min":2.0619999999999998,"5min":27.09,"15min":27.09},"last":2.2610000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.47,"15min":1.052},"min":{"1min":0.47299999999999998,"5min":0.26600000000000001,"15min":0.245},"max":{"1min":3.5489999999999999,"5min":18.126999999999999,"15min":18.126999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.7929999999999999,"5min":1.6339999999999999,"15min":1.149},"min":{"1min":0.52900000000000003,"5min":0.318,"15min":0.29999999999999999},"max":{"1min":5.8490000000000002,"5min":19.548999999999999,"15min":19.548999999999999},"last":0.63400000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:50:17 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.6160000000000001,"15min":1.0920000000000001},"min":{"1min":0.53300000000000003,"5min":0.29399999999999998,"15min":0.28199999999999997},"max":{"1min":3.2080000000000002,"5min":17.832000000000001,"15min":17.832000000000001},"last":3.4550000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":1.792,"15min":1.151},"min":{"1min":0.42199999999999999,"5min":0.33100000000000002,"15min":0.27400000000000002},"max":{"1min":2.1509999999999998,"5min":20.449999999999999,"15min":20.449999999999999},"last":2.2010000000000001}]}]},{"osd":6,"up_from":50,"seq":214748365033,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2198420,"kb_used_data":1639688,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":91546732,"statfs":{"total":95995035648,"available":93743853568,"internally_reserved":0,"allocated":1679040512,"data_stored":1678700762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":572127686},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:49:43 2024","interfaces":[{"interface":"back","average":{"1min":2.6349999999999998,"5min":1.865,"15min":1.8069999999999999},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":19.484999999999999,"5min":24.727,"15min":63.405999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.69599999999999995,"5min":1.3240000000000001,"15min":1.77},"min":{"1min":0.25600000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.3929999999999998,"5min":29.084,"15min":67.748000000000005},"last":0.71299999999999997}]},{"osd":1,"last update":"Sat Feb 10 07:49:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":1.1719999999999999,"15min":1.631},"min":{"1min":0.307,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.4910000000000001,"5min":11.406000000000001,"15min":37.659999999999997},"last":1.216},{"interface":"front","average":{"1min":1.9470000000000001,"5min":1.6200000000000001,"15min":1.79},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.24399999999999999},"max":{"1min":19.524999999999999,"5min":19.524999999999999,"15min":37.75},"last":0.79000000000000004}]},{"osd":2,"last update":"Sat Feb 10 07:49:43 2024","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":1.4570000000000001,"15min":1.6779999999999999},"min":{"1min":0.34899999999999998,"5min":0.307,"15min":0.253},"max":{"1min":15.768000000000001,"5min":15.768000000000001,"15min":65.278999999999996},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.629,"5min":1.2190000000000001,"15min":1.645},"min":{"1min":0.36899999999999999,"5min":0.30199999999999999,"15min":0.25800000000000001},"max":{"1min":15.728999999999999,"5min":15.728999999999999,"15min":65.265000000000001},"last":0.77200000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.5600000000000001,"15min":1.6899999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.4489999999999998,"5min":13.279,"15min":13.279},"last":0.749},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.1339999999999999,"15min":1.208},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.5169999999999999,"5min":5.6470000000000002,"15min":5.6470000000000002},"last":1.1930000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:49:43 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.3660000000000001,"15min":0.877},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.23400000000000001},"max":{"1min":2.8039999999999998,"5min":7.6509999999999998,"15min":7.7839999999999998},"last":0.436},{"interface":"front","average":{"1min":0.83699999999999997,"5min":1.3340000000000001,"15min":0.84999999999999998},"min":{"1min":0.34799999999999998,"5min":0.28000000000000003,"15min":0.26000000000000001},"max":{"1min":2.1930000000000001,"5min":7.0300000000000002,"15min":7.0300000000000002},"last":1.169}]},{"osd":5,"last update":"Sat Feb 10 07:49:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.879,"15min":1.117},"min":{"1min":0.317,"5min":0.26000000000000001,"15min":0.24099999999999999},"max":{"1min":2.5680000000000001,"5min":44.689,"15min":44.689},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.88400000000000001,"5min":2.0609999999999999,"15min":1.181},"min":{"1min":0.309,"5min":0.309,"15min":0.29199999999999998},"max":{"1min":3.5699999999999998,"5min":44.465000000000003,"15min":44.465000000000003},"last":1.1279999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:50:09 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":2.2730000000000001,"15min":1.2869999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.28100000000000003},"max":{"1min":4.7750000000000004,"5min":42.631999999999998,"15min":42.631999999999998},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":1.9950000000000001,"15min":1.1859999999999999},"min":{"1min":0.43099999999999999,"5min":0.32800000000000001,"15min":0.29699999999999999},"max":{"1min":1.6819999999999999,"5min":44.792999999999999,"15min":44.792999999999999},"last":3.2010000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575107,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2794200,"kb_used_data":2059152,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":90950952,"statfs":{"total":95995035648,"available":93133774848,"internally_reserved":0,"allocated":2108571648,"data_stored":2108222547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":752679366},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,3,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:49:45 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.57599999999999996,"15min":1.7649999999999999},"min":{"1min":0.24199999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.88700000000000001,"5min":5.3799999999999999,"15min":114.27},"last":6.4379999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.67700000000000005,"15min":1.9670000000000001},"min":{"1min":0.26800000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.823,"5min":7.3470000000000004,"15min":119.499},"last":0.432}]},{"osd":2,"last update":"Sat Feb 10 07:50:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.46400000000000002,"15min":1.1639999999999999},"min":{"1min":0.253,"5min":0.219,"15min":0.19600000000000001},"max":{"1min":0.93400000000000005,"5min":5.9909999999999997,"15min":8.9209999999999994},"last":4.133},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.46300000000000002,"15min":1.2909999999999999},"min":{"1min":0.28999999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.53600000000000003,"5min":7.3879999999999999,"15min":8.8300000000000001},"last":0.46200000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:49:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":0.67400000000000004,"15min":0.55000000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.367999999999999},"last":6.2949999999999999},{"interface":"front","average":{"1min":1.4710000000000001,"5min":0.60299999999999998,"15min":0.45900000000000002},"min":{"1min":0.29399999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":22.199999999999999,"5min":22.199999999999999,"15min":22.199999999999999},"last":0.48799999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:50:03 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.4399999999999999,"15min":2.0939999999999999},"min":{"1min":0.34499999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.294,"5min":31.908999999999999,"15min":207.45699999999999},"last":6.1760000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":1.3959999999999999,"15min":1.294},"min":{"1min":0.34699999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":2.528,"5min":32.201000000000001,"15min":32.201000000000001},"last":1.0589999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:50:03 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.4530000000000001,"15min":1.9450000000000001},"min":{"1min":0.34200000000000003,"5min":0.308,"15min":0.308},"max":{"1min":2.4910000000000001,"5min":31.824999999999999,"15min":98.703000000000003},"last":6.2290000000000001},{"interface":"front","average":{"1min":0.79200000000000004,"5min":1.5089999999999999,"15min":1.923},"min":{"1min":0.39700000000000002,"5min":0.38200000000000001,"15min":0.30199999999999999},"max":{"1min":2.4740000000000002,"5min":31.841999999999999,"15min":98.664000000000001},"last":0.58699999999999997}]},{"osd":6,"last update":"Sat Feb 10 07:49:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.6419999999999999,"15min":1.4139999999999999},"min":{"1min":0.36299999999999999,"5min":0.32800000000000001,"15min":0.308},"max":{"1min":4.9669999999999996,"5min":36.808,"15min":36.808},"last":1.119},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.6459999999999999,"15min":1.581},"min":{"1min":0.40300000000000002,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":2.4929999999999999,"5min":33.774999999999999,"15min":33.774999999999999},"last":6.2629999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:50:13 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.95799999999999996,"15min":1.381},"min":{"1min":0.435,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":1.8,"5min":7.444,"15min":8.4239999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.749,"5min":1.016,"15min":1.3009999999999999},"min":{"1min":0.41899999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.3490000000000002,"5min":6.2089999999999996,"15min":11.212},"last":1.089}]}]},{"osd":0,"up_from":8,"seq":34359738632,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1969244,"kb_used_data":1230036,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":91775908,"statfs":{"total":95995035648,"available":93978529792,"internally_reserved":0,"allocated":1259556864,"data_stored":1259205084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:49:36 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":1.1850000000000001,"15min":2.3399999999999999},"min":{"1min":0.22800000000000001,"5min":0.187,"15min":0.182},"max":{"1min":2.2229999999999999,"5min":34.661999999999999,"15min":156.16200000000001},"last":2.0059999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":1.202,"15min":2.3359999999999999},"min":{"1min":0.22,"5min":0.192,"15min":0.183},"max":{"1min":1.202,"5min":34.636000000000003,"15min":156.941},"last":0.67000000000000004}]},{"osd":2,"last update":"Sat Feb 10 07:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.878,"15min":1.5760000000000001},"min":{"1min":0.25800000000000001,"5min":0.21299999999999999,"15min":0.20000000000000001},"max":{"1min":2.133,"5min":34.603000000000002,"15min":34.603000000000002},"last":0.497},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.84999999999999998,"15min":1.367},"min":{"1min":0.32500000000000001,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":1.0209999999999999,"5min":34.718000000000004,"15min":34.718000000000004},"last":0.47599999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.61899999999999999,"15min":0.628},"min":{"1min":0.29699999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.2589999999999999,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":3.125},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.60399999999999998,"15min":0.61699999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":1.3089999999999999,"5min":2.2589999999999999,"15min":2.2589999999999999},"last":1.1120000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.6619999999999999,"15min":1.3420000000000001},"min":{"1min":0.44700000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.4329999999999998,"5min":37.286000000000001,"15min":37.286000000000001},"last":2.327},{"interface":"front","average":{"1min":1.444,"5min":1.6479999999999999,"15min":1.4730000000000001},"min":{"1min":0.38100000000000001,"5min":0.315,"15min":0.26900000000000002},"max":{"1min":5.1609999999999996,"5min":34.683,"15min":34.683},"last":0.78300000000000003}]},{"osd":5,"last update":"Sat Feb 10 07:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":2.5489999999999999,"15min":2.25},"min":{"1min":0.46100000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":8.0239999999999991,"5min":41.429000000000002,"15min":41.429000000000002},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.48,"5min":2.5230000000000001,"15min":2.387},"min":{"1min":0.442,"5min":0.33500000000000002,"15min":0.308},"max":{"1min":8.0060000000000002,"5min":41.456000000000003,"15min":41.456000000000003},"last":0.746}]},{"osd":6,"last update":"Sat Feb 10 07:49:36 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":2.202,"15min":1.5660000000000001},"min":{"1min":0.40899999999999997,"5min":0.35399999999999998,"15min":0.29599999999999999},"max":{"1min":2.585,"5min":41.587000000000003,"15min":41.587000000000003},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.337,"5min":2.282,"15min":1.7150000000000001},"min":{"1min":0.48099999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":4.6509999999999998,"5min":41.256,"15min":41.256},"last":2.3650000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":1.593,"15min":1.6299999999999999},"min":{"1min":0.41999999999999998,"5min":0.34899999999999998,"15min":0.33300000000000002},"max":{"1min":2.2360000000000002,"5min":34.710999999999999,"15min":34.710999999999999},"last":1.764},{"interface":"front","average":{"1min":0.74299999999999999,"5min":1.534,"15min":1.421},"min":{"1min":0.39200000000000002,"5min":0.38500000000000001,"15min":0.379},"max":{"1min":2.0449999999999999,"5min":34.997999999999998,"15min":34.997999999999998},"last":2.1890000000000001}]}]},{"osd":2,"up_from":0,"seq":77309411581,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":347,"seq":1490353651745,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501408,"kb_used_data":1336276,"kb_used_omap":5,"kb_used_meta":165114,"kb_avail":92243744,"statfs":{"total":95995035648,"available":94457593856,"internally_reserved":0,"allocated":1368346624,"data_stored":1367993575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5625,"internal_metadata":169077255},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.58099999999999996,"15min":0.54400000000000004},"min":{"1min":0.245,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.4180000000000001,"5min":3.4180000000000001,"15min":3.4180000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.3320000000000001,"5min":0.76700000000000002,"15min":0.67300000000000004},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.939,"5min":16.939,"15min":16.939},"last":2.0600000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.443,"15min":0.41099999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.327,"5min":0.60699999999999998,"15min":0.48699999999999999},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":16.716000000000001,"5min":16.716000000000001,"15min":16.716000000000001},"last":2.0870000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.49099999999999999,"15min":0.47099999999999997},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":0.623},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.56699999999999995,"15min":0.505},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":10.122,"5min":10.122,"15min":10.122},"last":2.0270000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":2.4910000000000001,"15min":2.6680000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":10.170999999999999,"5min":28.707999999999998,"15min":28.707999999999998},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.1080000000000001,"5min":2.6800000000000002,"15min":2.9420000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":3.3860000000000001,"5min":28.760999999999999,"15min":28.760999999999999},"last":0.44400000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":0.97999999999999998,"15min":0.89100000000000001},"min":{"1min":0.34100000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":10.083,"5min":10.083,"15min":10.083},"last":2.1509999999999998},{"interface":"front","average":{"1min":1.4330000000000001,"5min":0.88500000000000001,"15min":0.79300000000000004},"min":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":10.298,"5min":10.298,"15min":10.298},"last":1.7330000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":2.363,"15min":2.5030000000000001},"min":{"1min":0.41799999999999998,"5min":0.372,"15min":0.372},"max":{"1min":10.039,"5min":28.661999999999999,"15min":28.661999999999999},"last":2.2229999999999999},{"interface":"front","average":{"1min":1.778,"5min":2.9390000000000001,"15min":3.133},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":10.217000000000001,"5min":28.785,"15min":28.785},"last":1.6299999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.1879999999999999,"15min":1.139},"min":{"1min":0.45900000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":10.26,"5min":10.26,"15min":10.26},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.3029999999999999,"15min":1.28},"min":{"1min":0.48499999999999999,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":4.2830000000000004,"5min":4.2830000000000004,"15min":4.2830000000000004},"last":0.41699999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953715,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1697876,"kb_used_data":994764,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92047276,"statfs":{"total":95995035648,"available":94256410624,"internally_reserved":0,"allocated":1018638336,"data_stored":1018293222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":1.0569999999999999,"15min":2.089},"min":{"1min":0.26400000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.504,"5min":10.859999999999999,"15min":74.533000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.2390000000000001,"15min":2.1030000000000002},"min":{"1min":0.32600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.8499999999999996,"5min":7.298,"15min":74.340999999999994},"last":0.55400000000000005}]},{"osd":1,"last update":"Sat Feb 10 07:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":1.794,"15min":2.3170000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.23200000000000001},"max":{"1min":2.3999999999999999,"5min":66.343999999999994,"15min":69.852000000000004},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.331,"15min":2.2040000000000002},"min":{"1min":0.32200000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.847,"5min":5.9909999999999997,"15min":70.692999999999998},"last":3.371}]},{"osd":2,"last update":"Sat Feb 10 07:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.3460000000000001,"15min":1.4470000000000001},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.26600000000000001},"max":{"1min":5.0700000000000003,"5min":6.0179999999999998,"15min":50.162999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.212,"15min":1.3939999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":5.9359999999999999,"5min":7.335,"15min":50.453000000000003},"last":0.41999999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":3.9710000000000001,"15min":4.4859999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.5640000000000001,"5min":66.379999999999995,"15min":66.379999999999995},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.6299999999999999,"15min":1.726},"min":{"1min":0.39600000000000002,"5min":0.309,"15min":0.309},"max":{"1min":5.0350000000000001,"5min":5.3780000000000001,"15min":5.3780000000000001},"last":2.5750000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:49:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.5609999999999999,"15min":0.95999999999999996},"min":{"1min":0.28799999999999998,"5min":0.28599999999999998,"15min":0.23100000000000001},"max":{"1min":5.9720000000000004,"5min":10.502000000000001,"15min":10.502000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":1.524,"15min":0.96299999999999997},"min":{"1min":0.33300000000000002,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":2.9820000000000002,"5min":9.2460000000000004,"15min":9.2460000000000004},"last":0.60199999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:49:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":2.306,"15min":1.1759999999999999},"min":{"1min":0.35999999999999999,"5min":0.28100000000000003,"15min":0.25900000000000001},"max":{"1min":6.0039999999999996,"5min":67.433999999999997,"15min":67.433999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.3009999999999999,"5min":2.468,"15min":1.246},"min":{"1min":0.31900000000000001,"5min":0.29099999999999998,"15min":0.24399999999999999},"max":{"1min":5.0090000000000003,"5min":66.876000000000005,"15min":66.876000000000005},"last":0.49199999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:50:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":1.514,"15min":0.94299999999999995},"min":{"1min":0.42099999999999999,"5min":0.307,"15min":0.29099999999999998},"max":{"1min":1.5269999999999999,"5min":7.4930000000000003,"15min":7.4930000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":1.474,"15min":0.996},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.29399999999999998},"max":{"1min":3.298,"5min":10.919,"15min":10.919},"last":0.81899999999999995}]}]},{"osd":5,"up_from":156,"seq":670014898344,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1448912,"kb_used_data":788420,"kb_used_omap":4,"kb_used_meta":660475,"kb_avail":92296240,"statfs":{"total":95995035648,"available":94511349760,"internally_reserved":0,"allocated":807342080,"data_stored":807006082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":676327074},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.952,"5min":1.5209999999999999,"15min":1.7370000000000001},"min":{"1min":0.22600000000000001,"5min":0.20899999999999999,"15min":0.20499999999999999},"max":{"1min":14.978,"5min":21.817,"15min":52.938000000000002},"last":0.23799999999999999},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.7330000000000001,"15min":1.974},"min":{"1min":0.23899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.643000000000001,"5min":21.347999999999999,"15min":52.805},"last":0.315}]},{"osd":1,"last update":"Sat Feb 10 07:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":1.3560000000000001,"15min":2.198},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.245},"max":{"1min":3.3130000000000002,"5min":8.0090000000000003,"15min":54.298000000000002},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.2290000000000001,"15min":2.1499999999999999},"min":{"1min":0.248,"5min":0.246,"15min":0.246},"max":{"1min":3.391,"5min":7.7649999999999997,"15min":52.969999999999999},"last":0.45700000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":1.282,"15min":1.2929999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.2879999999999998,"5min":6.3970000000000002,"15min":11.601000000000001},"last":0.379},{"interface":"front","average":{"1min":0.995,"5min":1.4630000000000001,"15min":1.3720000000000001},"min":{"1min":0.43099999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.3209999999999997,"5min":13.662000000000001,"15min":13.662000000000001},"last":0.55400000000000005}]},{"osd":3,"last update":"Sat Feb 10 07:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.5760000000000001,"15min":1.6779999999999999},"min":{"1min":0.40699999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.4340000000000002,"5min":8.1530000000000005,"15min":8.1530000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.4139999999999999,"15min":1.454},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.8470000000000004,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.60299999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.998,"15min":1.1100000000000001},"min":{"1min":0.317,"5min":0.252,"15min":0.246},"max":{"1min":6.7759999999999998,"5min":8.952,"15min":8.952},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.655,"5min":1.7170000000000001,"15min":1.0209999999999999},"min":{"1min":0.32900000000000001,"5min":0.26800000000000002,"15min":0.25900000000000001},"max":{"1min":6.202,"5min":10.180999999999999,"15min":10.180999999999999},"last":0.71599999999999997}]},{"osd":6,"last update":"Sat Feb 10 07:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.659,"15min":1.0209999999999999},"min":{"1min":0.34399999999999997,"5min":0.28699999999999998,"15min":0.23999999999999999},"max":{"1min":6.7110000000000003,"5min":9.0229999999999997,"15min":9.0229999999999997},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.7,"5min":1.5269999999999999,"15min":1.0009999999999999},"min":{"1min":0.29599999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":6.1699999999999999,"5min":6.1699999999999999,"15min":6.1699999999999999},"last":0.47399999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.7250000000000001,"15min":1.1359999999999999},"min":{"1min":0.30499999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":9.0199999999999996,"5min":9.0199999999999996,"15min":9.0199999999999996},"last":0.434},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.538,"15min":1.0229999999999999},"min":{"1min":0.35799999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":6.75,"5min":7.8109999999999999,"15min":7.8109999999999999},"last":0.48799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1284505600,"data_stored":1284505600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2146041856,"data_stored":2146041856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1376780288,"data_stored":1376780288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1045102592,"data_stored":1045102592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":826474496,"data_stored":826474496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1698234368,"data_stored":1698234368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1680932864,"data_stored":1680932864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:50:33.271 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-10T07:50:33.271 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:50:33.271 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-02-10T07:50:33.272 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd in 2 2024-02-10T07:50:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:33.273 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:33.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:33.374 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:33.421 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:33.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:33.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:33.504 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:33 smithi008 ceph-mon[40743]: pgmap v863: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 843 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-10T07:50:33.506 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:33 smithi008 ceph-mon[29816]: pgmap v863: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 843 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-10T07:50:33.523 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:33.555 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:33.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:33.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:33 smithi170 ceph-mon[33394]: pgmap v863: 148 pgs: 148 active+clean; 3.2 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 843 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-10T07:50:33.647 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 28 15 58618 58603 130.796 84.625 0.0106295 0.00764074 2024-02-10T07:50:33.657 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:33.692 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:33.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:33.794 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:33.828 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:33.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:33.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:33.891 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:33.930 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:33.968 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:33.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:33.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:34.070 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:34.110 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:34.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:34.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:34.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:50:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:50:34.213 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:34.250 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:34.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:34.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:34.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:50:34.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:50:34.352 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:34.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:34.391 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:34.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:34.472 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:50:34.493 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:34.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:34.530 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:34.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:34.631 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:34.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:50:34.647 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 29 15 60134 60119 129.553 94.75 0.00890734 0.00771409 2024-02-10T07:50:34.666 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:34.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:34.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:34.683 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:34.767 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:34.806 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:34.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:34.908 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:34.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:34.939 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:34.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:35.040 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:35.080 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:35.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:35.182 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:35.217 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:35.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:35.319 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:35.357 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:35.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:35.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:35.381 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:35 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:35.382 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:35 smithi008 ceph-mon[29816]: pgmap v864: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-10T07:50:35.382 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:35 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:35.382 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:35 smithi008 ceph-mon[40743]: pgmap v864: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-10T07:50:35.458 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:35.498 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:35.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:35.599 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:35.633 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:35.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:35 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:35.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:35 smithi170 ceph-mon[33394]: pgmap v864: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-10T07:50:35.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:35.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:35.647 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 30 16 61623 61607 128.334 93 0.00655786 0.00778737 2024-02-10T07:50:35.735 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:35.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:35.772 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:35.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:35.873 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:35.907 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:35.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:35.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:36.009 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:36.048 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:36.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:36.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:36.151 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:36.251 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:36.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:36.352 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:36.394 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:36.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:36.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:36.496 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:36.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:36.539 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:36.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:36.640 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:36.648 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 31 16 63133 63117 127.238 94.375 0.0119659 0.00785409 2024-02-10T07:50:36.679 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:36.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:36.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:36.781 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:36.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:36.825 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:36.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:36.926 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:36.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:36.967 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:36.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:37.068 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:37.103 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:37.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:37.205 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:37.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:37.249 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:37.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:37.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:37 smithi008 ceph-mon[29816]: pgmap v865: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-02-10T07:50:37.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:37 smithi008 ceph-mon[40743]: pgmap v865: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-02-10T07:50:37.351 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:37.387 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:37.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:37.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:37.489 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:37.527 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:37.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:37.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:37.630 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:37 smithi170 ceph-mon[33394]: pgmap v865: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-02-10T07:50:37.648 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 32 16 64589 64573 126.104 91 0.00926499 0.007924 2024-02-10T07:50:37.673 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:37.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:37.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:37.775 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:37.809 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:50:37.809 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:50:37.809 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-02-10T07:50:37.809 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:50:37.811 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:37.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:37.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:37.913 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:37.950 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:37.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:37.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:38.052 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:38.090 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:38.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:38.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:38.192 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:38.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:38.229 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:38.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:38.330 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:38.369 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:38.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:38.470 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:38.505 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:38.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:38.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:38.556 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:50:38.598 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:50:38.606 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:38.633 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:50:38.644 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:38.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:38.647 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 33 16 65981 65965 124.919 87 0.0076866 0.00799997 2024-02-10T07:50:38.667 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:50:38.707 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:50:38.746 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:38.753 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:50:38.792 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:38.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:38.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:38.797 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:50:38.854 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:50:38.854 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:50:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:50:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:50:38.859 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:50:38.860 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:50:38.860 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:50:38.860 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:50:38.860 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:50:38.860 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:50:38.860 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:50:38.860 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:50:38.894 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:38.935 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:38.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:39.037 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:39.074 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:39.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:39.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:39.176 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:39.225 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:39.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:39.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:39.243 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-10T07:50:39.327 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:39.363 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:39.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:39.466 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:39.512 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:39.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:39.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:39 smithi008 ceph-mon[29816]: pgmap v866: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 922 op/s 2024-02-10T07:50:39.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:39 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2112275558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:50:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:39 smithi008 ceph-mon[40743]: pgmap v866: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 922 op/s 2024-02-10T07:50:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:39 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2112275558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:50:39.613 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:39 smithi170 ceph-mon[33394]: pgmap v866: 148 pgs: 148 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 922 op/s 2024-02-10T07:50:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:39 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2112275558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:50:39.640 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:39.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:39.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:39.647 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 34 16 67338 67322 123.739 84.8125 0.00292204 0.00807295 2024-02-10T07:50:39.742 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:39.776 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:39.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:39.877 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:39.913 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:39.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:39.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:40.014 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:40.042 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:40.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:40.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:40.144 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:40.184 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:40.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:40.286 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:40.327 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:40.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:40.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:40.429 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:40.454 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:40.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:40.555 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:40.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:40 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:50:40.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:40 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:50:40.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:40 smithi008 ceph-mon[29816]: osdmap e403: 8 total, 8 up, 7 in 2024-02-10T07:50:40.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:40 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:50:40.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:40 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:50:40.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:40 smithi008 ceph-mon[40743]: osdmap e403: 8 total, 8 up, 7 in 2024-02-10T07:50:40.592 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:40.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:40 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:50:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:40 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:50:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:40 smithi170 ceph-mon[33394]: osdmap e403: 8 total, 8 up, 7 in 2024-02-10T07:50:40.648 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 35 16 68616 68600 122.485 79.875 0.00933575 0.00815773 2024-02-10T07:50:40.694 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:40.733 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:40.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:40.835 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:40.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:40.882 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:40.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:40.975 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:40.983 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:41.018 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:41.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:41.119 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:41.156 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:41.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:41.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:41.258 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:41.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:41.293 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:41.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:41.394 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:41.432 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:41.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:41.435 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:41 smithi008 ceph-mon[29816]: pgmap v868: 148 pgs: 148 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-02-10T07:50:41.435 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:41 smithi008 ceph-mon[40743]: pgmap v868: 148 pgs: 148 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-02-10T07:50:41.535 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:41.575 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:41.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:41.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:41.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:41 smithi170 ceph-mon[33394]: pgmap v868: 148 pgs: 148 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-02-10T07:50:41.650 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 36 16 69973 69957 121.439 84.8125 0.00663888 0.00822963 2024-02-10T07:50:41.677 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:41.723 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:41.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:41.750 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:41 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a[29812]: 2024-02-10T07:50:41.433+0000 7ff409c1e700 -1 mon.a@0(leader).osd e403 definitely_dead 0 2024-02-10T07:50:41.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:41.865 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:41.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:41.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:41.967 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:41.993 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:41.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:42.095 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:42.132 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:42.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:42.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:42.234 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:42.267 INFO:teuthology.orchestra.run.smithi008.stderr:marked in osd.2. 2024-02-10T07:50:42.276 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:42.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:42.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:42.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:42 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1262166569' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-10T07:50:42.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:42 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-10T07:50:42.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:42 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1262166569' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-10T07:50:42.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:42 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-10T07:50:42.377 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:42.401 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:42.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:42.503 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:42.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:42.526 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:42.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:42.627 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:42.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:42 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1262166569' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-10T07:50:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:42 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-10T07:50:42.648 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 37 16 70711 70695 119.403 46.125 0.00328327 0.0082856 2024-02-10T07:50:42.653 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:42.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:42.696 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:42.755 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:42.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:42.779 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:42.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:42.880 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:42.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:42.905 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:42.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:43.006 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:43.031 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:43.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:43.132 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:43.155 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:43.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:43.257 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:43.295 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:43.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:43 smithi170 ceph-mon[33394]: pgmap v869: 148 pgs: 148 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-02-10T07:50:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:43 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-10T07:50:43.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:43 smithi170 ceph-mon[33394]: osdmap e404: 8 total, 8 up, 8 in 2024-02-10T07:50:43.396 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:43.421 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:43.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:43.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:43.523 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:43.547 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:43.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:43.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:43 smithi008 ceph-mon[29816]: pgmap v869: 148 pgs: 148 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-02-10T07:50:43.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:43 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-10T07:50:43.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:43 smithi008 ceph-mon[29816]: osdmap e404: 8 total, 8 up, 8 in 2024-02-10T07:50:43.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:43 smithi008 ceph-mon[40743]: pgmap v869: 148 pgs: 148 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-02-10T07:50:43.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:43 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-10T07:50:43.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:43 smithi008 ceph-mon[40743]: osdmap e404: 8 total, 8 up, 8 in 2024-02-10T07:50:43.606 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:50:43.648 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 38 16 70711 70695 116.261 0 - 0.0082856 2024-02-10T07:50:43.648 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:43.673 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:43.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:43.773 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:43.799 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:43.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:43.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:43.900 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:43.927 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:43.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:43.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:44.029 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:44.054 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:44.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:44.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:44.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:50:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:50:44.156 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:44.181 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:44.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:44.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:44.186 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-02-10T07:50:44.283 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:44.310 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:44.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:44.411 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:44.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:44.436 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:44.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:44.537 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:44.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:44.576 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:44.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:44.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:44 smithi008 ceph-mon[29816]: osdmap e405: 8 total, 8 up, 8 in 2024-02-10T07:50:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:44 smithi008 ceph-mon[40743]: osdmap e405: 8 total, 8 up, 8 in 2024-02-10T07:50:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:44 smithi170 ceph-mon[33394]: osdmap e405: 8 total, 8 up, 8 in 2024-02-10T07:50:44.648 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 39 16 70711 70695 113.28 0 - 0.0082856 2024-02-10T07:50:44.678 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:44.707 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:44.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:44.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:44.809 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:44.851 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:44.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:44.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:44.953 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:45.001 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:45.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:45.103 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:45.141 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:45.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:45.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:45.244 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:45.268 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:45.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:45.370 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:45.397 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:45.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:45.435 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:45 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:45.435 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:45 smithi008 ceph-mon[29816]: pgmap v872: 148 pgs: 2 remapped+peering, 146 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 79 B/s, 1 objects/s recovering 2024-02-10T07:50:45.435 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:45 smithi008 ceph-mon[29816]: osdmap e406: 8 total, 8 up, 8 in 2024-02-10T07:50:45.436 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:45 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:45.436 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:45 smithi008 ceph-mon[40743]: pgmap v872: 148 pgs: 2 remapped+peering, 146 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 79 B/s, 1 objects/s recovering 2024-02-10T07:50:45.436 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:45 smithi008 ceph-mon[40743]: osdmap e406: 8 total, 8 up, 8 in 2024-02-10T07:50:45.498 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:45.528 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:45.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:45 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:45 smithi170 ceph-mon[33394]: pgmap v872: 148 pgs: 2 remapped+peering, 146 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.7 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 79 B/s, 1 objects/s recovering 2024-02-10T07:50:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:45 smithi170 ceph-mon[33394]: osdmap e406: 8 total, 8 up, 8 in 2024-02-10T07:50:45.633 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:45.648 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:50:45.649268+0000 min lat: 0.00135892 max lat: 2.4862 avg lat: 0.00895396 2024-02-10T07:50:45.648 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:50:45.649 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 40 16 71426 71410 111.565 14.8958 0.014701 0.00895396 2024-02-10T07:50:45.679 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:45.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:45.781 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:45.806 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:45.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:45.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:45.908 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:45.959 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:45.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:45.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:46.062 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:46.102 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:46.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:46.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:46.204 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:46.231 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:46.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:46.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:46.333 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:46.373 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:46.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:46.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:46.475 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:46.507 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:46.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:46.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:46.609 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:46.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:46.649 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:46.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:46.649 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 41 16 72119 72103 109.9 43.3125 0.026884 0.00909051 2024-02-10T07:50:46.750 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:46.801 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:46.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:46.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:46.819 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:46.862 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:50:46.862 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:50:46.903 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:46.962 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:46.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:46.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:47.019 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:50:47.064 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:47.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:47.107 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:47.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:47.207 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:47.240 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:47.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:47.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:47.342 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:47.391 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:47.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:47.440 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:47 smithi008 ceph-mon[29816]: pgmap v874: 148 pgs: 6 remapped+peering, 142 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 791 op/s; 303 B/s, 5 objects/s recovering 2024-02-10T07:50:47.440 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:47 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2107380571' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:50:47.440 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:47 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:50:47.440 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:47 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:50:47.440 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:47 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:50:47.440 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:47 smithi008 ceph-mon[29816]: osdmap e407: 8 total, 8 up, 8 in 2024-02-10T07:50:47.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:47 smithi008 ceph-mon[40743]: pgmap v874: 148 pgs: 6 remapped+peering, 142 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 791 op/s; 303 B/s, 5 objects/s recovering 2024-02-10T07:50:47.441 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:47 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2107380571' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:50:47.441 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:47 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:50:47.441 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:47 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:50:47.441 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:47 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:50:47.441 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:47 smithi008 ceph-mon[40743]: osdmap e407: 8 total, 8 up, 8 in 2024-02-10T07:50:47.493 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:47.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:47.521 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:47.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:47.622 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:47 smithi170 ceph-mon[33394]: pgmap v874: 148 pgs: 6 remapped+peering, 142 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 49 MiB/s wr, 791 op/s; 303 B/s, 5 objects/s recovering 2024-02-10T07:50:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:47 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2107380571' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:50:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:47 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:50:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:47 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:50:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:47 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:50:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:47 smithi170 ceph-mon[33394]: osdmap e407: 8 total, 8 up, 8 in 2024-02-10T07:50:47.649 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 42 16 72829 72813 108.338 44.375 0.00407661 0.00922171 2024-02-10T07:50:47.664 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:47.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:47.766 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:47.809 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:47.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:47.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:47.911 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:47.940 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:47.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:47.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:48.042 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:48.073 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:48.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:48.175 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:48.207 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:48.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:48.309 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:48.348 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:48.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:48.450 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:48.504 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:48.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:48.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:48.607 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:48.649 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 43 16 73588 73572 106.922 47.4375 0.0331635 0.00934429 2024-02-10T07:50:48.652 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:48.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:48.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:48.755 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:48.781 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:48.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:48.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:48.883 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:48.922 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:48.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:48.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:49.024 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:49.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:49.069 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:49.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:49.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:50:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:50:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:50:49.170 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:49.187 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 6, 7, 5, 3] 2024-02-10T07:50:49.187 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:50:49.187 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-02-10T07:50:49.188 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:50:49.201 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:49.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:49.303 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:49.353 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:49.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:49.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:49.455 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:49.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:49.534 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:49.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:49 smithi008 ceph-mon[29816]: pgmap v876: 148 pgs: 6 remapped+peering, 142 active+clean; 4.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 873 B/s rd, 48 MiB/s wr, 769 op/s; 310 B/s, 5 objects/s recovering 2024-02-10T07:50:49.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:50:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:49 smithi008 ceph-mon[40743]: pgmap v876: 148 pgs: 6 remapped+peering, 142 active+clean; 4.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 873 B/s rd, 48 MiB/s wr, 769 op/s; 310 B/s, 5 objects/s recovering 2024-02-10T07:50:49.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:50:49.636 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:49.637 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:49 smithi170 ceph-mon[33394]: pgmap v876: 148 pgs: 6 remapped+peering, 142 active+clean; 4.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 873 B/s rd, 48 MiB/s wr, 769 op/s; 310 B/s, 5 objects/s recovering 2024-02-10T07:50:49.637 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:50:49.649 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 44 16 74297 74281 105.499 44.3125 0.0570327 0.00947085 2024-02-10T07:50:49.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:49.681 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:49.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:49.782 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:49.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:49.810 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:49.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:49.912 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:49.940 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:49.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:49.970 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-10T07:50:50.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:50.066 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:50.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:50.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:50.168 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:50.201 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:50.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:50.256 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:50:50.303 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:50.399 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:50.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:50.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:50.501 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:50.527 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:50.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:50.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:50.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:50 smithi008 ceph-mon[29816]: Health check failed: Degraded data redundancy: 8861/221334 objects degraded (4.003%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:50:50.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:50 smithi008 ceph-mon[40743]: Health check failed: Degraded data redundancy: 8861/221334 objects degraded (4.003%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:50:50.629 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:50.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:50 smithi170 ceph-mon[33394]: Health check failed: Degraded data redundancy: 8861/221334 objects degraded (4.003%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:50:50.651 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 45 16 74942 74926 104.047 40.3125 0.012814 0.00960194 2024-02-10T07:50:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:50.657 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:50.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:50.665 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:50.746 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:50:50.758 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:50.797 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:50.798 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:50:50.847 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:50:50.859 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:50.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:50.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:50.882 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:50:50.920 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:50:50.961 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:50:50.961 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:50.990 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:50.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:51.000 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:50:51.038 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:50:51.052 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:50:51.052 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:50:51.052 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:50:51.052 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:50:51.052 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:50:51.052 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:50:51.052 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:50:51.052 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:50:51.092 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:51.125 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:51.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:51.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:51.227 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:51.272 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:51.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:51.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:51.374 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:51.402 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:51.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:51.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:51.463 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:51 smithi008 ceph-mon[29816]: pgmap v877: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 8861/221334 objects degraded (4.003%); 17224/221334 objects misplaced (7.782%); 2.5 MiB/s, 43 objects/s recovering 2024-02-10T07:50:51.463 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:51 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1809523182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:50:51.463 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:51 smithi008 ceph-mon[40743]: pgmap v877: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 8861/221334 objects degraded (4.003%); 17224/221334 objects misplaced (7.782%); 2.5 MiB/s, 43 objects/s recovering 2024-02-10T07:50:51.463 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:51 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1809523182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:50:51.504 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:51.531 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:51.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:51 smithi170 ceph-mon[33394]: pgmap v877: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 8861/221334 objects degraded (4.003%); 17224/221334 objects misplaced (7.782%); 2.5 MiB/s, 43 objects/s recovering 2024-02-10T07:50:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:51 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1809523182' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:50:51.633 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:51.651 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 46 16 75614 75598 102.697 42 0.043431 0.00972959 2024-02-10T07:50:51.680 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:51.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:51.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:51.782 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:51.821 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:51.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:51.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:51.923 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:51.968 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:51.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:52.070 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:52.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:52.122 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:52.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:52.223 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:52.253 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:52.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:52.355 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:52.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:52.380 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:52.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:52.481 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:52.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:52.530 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:52.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:52.631 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:52.653 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 47 16 76297 76281 101.418 42.6875 0.00477613 0.00985218 2024-02-10T07:50:52.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:52.674 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:52.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:52.775 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:52.814 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:52.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:52.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:52.916 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:52.948 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:52.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:52.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:53.050 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:53.079 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:53.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:53.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:53.181 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:53.224 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:53.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:53.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:53.326 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:53.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:53 smithi008 ceph-mon[29816]: osdmap e408: 8 total, 8 up, 8 in 2024-02-10T07:50:53.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:53 smithi008 ceph-mon[29816]: 5.17 scrub starts 2024-02-10T07:50:53.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:53 smithi008 ceph-mon[29816]: 5.17 scrub ok 2024-02-10T07:50:53.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:53 smithi008 ceph-mon[29816]: pgmap v879: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 651 B/s rd, 25 MiB/s wr, 403 op/s; 8861/221955 objects degraded (3.992%); 17224/221955 objects misplaced (7.760%); 2.1 MiB/s, 34 objects/s recovering 2024-02-10T07:50:53.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:53 smithi008 ceph-mon[40743]: osdmap e408: 8 total, 8 up, 8 in 2024-02-10T07:50:53.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:53 smithi008 ceph-mon[40743]: 5.17 scrub starts 2024-02-10T07:50:53.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:53 smithi008 ceph-mon[40743]: 5.17 scrub ok 2024-02-10T07:50:53.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:53 smithi008 ceph-mon[40743]: pgmap v879: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 651 B/s rd, 25 MiB/s wr, 403 op/s; 8861/221955 objects degraded (3.992%); 17224/221955 objects misplaced (7.760%); 2.1 MiB/s, 34 objects/s recovering 2024-02-10T07:50:53.355 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:53.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:53 smithi170 ceph-mon[33394]: osdmap e408: 8 total, 8 up, 8 in 2024-02-10T07:50:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:53 smithi170 ceph-mon[33394]: 5.17 scrub starts 2024-02-10T07:50:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:53 smithi170 ceph-mon[33394]: 5.17 scrub ok 2024-02-10T07:50:53.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:53 smithi170 ceph-mon[33394]: pgmap v879: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 651 B/s rd, 25 MiB/s wr, 403 op/s; 8861/221955 objects degraded (3.992%); 17224/221955 objects misplaced (7.760%); 2.1 MiB/s, 34 objects/s recovering 2024-02-10T07:50:53.457 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:53.502 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:53.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:53.604 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:53.633 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:53.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:53.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:53.653 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 48 16 77009 76993 100.232 44.5 0.026354 0.00996976 2024-02-10T07:50:53.735 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:53.775 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:53.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:53.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:53.878 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:53.906 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:53.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:53.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:54.008 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:54.036 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:54.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:54.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:50:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:50:54.138 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:54.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:54.183 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:54.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:54.284 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:50:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:54.326 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:54.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:50:54.427 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:54.455 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:54.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:54.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:54.557 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:54.603 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:54.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:54.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:54.653 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 49 16 77741 77725 99.1205 45.75 0.00559517 0.0100828 2024-02-10T07:50:54.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:54.732 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:54.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:54.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:54.834 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:54.872 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:54.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:54.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:54.974 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:55.023 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:55.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:55.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:55.125 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:55.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:55.153 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:55.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:55.254 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:55.281 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:55.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:55.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:55.383 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:55.412 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:55.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:55.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:55.514 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:55.540 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:55.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:55.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:55.575 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:55 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:55.575 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:55 smithi008 ceph-mon[29816]: pgmap v880: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 40 MiB/s wr, 635 op/s; 8861/227694 objects degraded (3.892%); 17224/227694 objects misplaced (7.565%); 2.1 MiB/s, 33 objects/s recovering 2024-02-10T07:50:55.575 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:55 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:55.575 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:55 smithi008 ceph-mon[40743]: pgmap v880: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 40 MiB/s wr, 635 op/s; 8861/227694 objects degraded (3.892%); 17224/227694 objects misplaced (7.565%); 2.1 MiB/s, 33 objects/s recovering 2024-02-10T07:50:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:55 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:50:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:55 smithi170 ceph-mon[33394]: pgmap v880: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 40 MiB/s wr, 635 op/s; 8861/227694 objects degraded (3.892%); 17224/227694 objects misplaced (7.565%); 2.1 MiB/s, 33 objects/s recovering 2024-02-10T07:50:55.643 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:55.653 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 50 16 78480 78464 98.0619 46.1875 0.0138241 0.0101913 2024-02-10T07:50:55.670 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:55.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:55.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:55.771 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:55.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:55.796 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:55.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:55.897 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:55.922 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:55.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:55.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:56.025 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:56.057 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:56.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:56.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:56.159 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:56.201 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:56.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:56.303 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:56.347 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:56.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:56.449 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:56.494 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:56.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:56.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:56.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:56 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/231537 objects degraded (3.827%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:50:56.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:56 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/231537 objects degraded (3.827%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:50:56.596 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:56.622 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:56.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:56 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/231537 objects degraded (3.827%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:50:56.653 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 51 16 79040 79024 96.8254 35 0.00805521 0.0103211 2024-02-10T07:50:56.724 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:56.753 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:56.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:56.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:56.855 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:56.885 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:56.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:56.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:56.986 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:57.031 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:57.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:57.132 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:57.164 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:57.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:57.266 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:57.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:57.324 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:57.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:57.425 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:57.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:57.452 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:57.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:57.554 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:57.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:57 smithi008 ceph-mon[40743]: pgmap v881: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 43 MiB/s wr, 683 op/s; 8861/231537 objects degraded (3.827%); 16936/231537 objects misplaced (7.315%); 3.8 MiB/s, 61 objects/s recovering 2024-02-10T07:50:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:57 smithi008 ceph-mon[40743]: osdmap e409: 8 total, 8 up, 8 in 2024-02-10T07:50:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:57 smithi008 ceph-mon[29816]: pgmap v881: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 43 MiB/s wr, 683 op/s; 8861/231537 objects degraded (3.827%); 16936/231537 objects misplaced (7.315%); 3.8 MiB/s, 61 objects/s recovering 2024-02-10T07:50:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:57 smithi008 ceph-mon[29816]: osdmap e409: 8 total, 8 up, 8 in 2024-02-10T07:50:57.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:57.603 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:57.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:57.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:57 smithi170 ceph-mon[33394]: pgmap v881: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 43 MiB/s wr, 683 op/s; 8861/231537 objects degraded (3.827%); 16936/231537 objects misplaced (7.315%); 3.8 MiB/s, 61 objects/s recovering 2024-02-10T07:50:57.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:57 smithi170 ceph-mon[33394]: osdmap e409: 8 total, 8 up, 8 in 2024-02-10T07:50:57.653 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 52 16 79789 79773 95.8636 46.8125 0.0243994 0.010423 2024-02-10T07:50:57.704 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:57.731 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:57.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:50:57.833 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:57.880 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:57.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:57.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:57.982 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:58.009 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:58.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:58.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:58.111 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:58.159 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:58.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:58.261 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:58.287 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:58.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:58.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:58.389 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:58.433 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:58.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:58.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:58.534 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:50:58.536 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:58.563 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:58.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:58.656 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 53 16 80498 80482 94.8909 44.3125 0.0192099 0.0105317 2024-02-10T07:50:58.664 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:50:58.718 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:58.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:58.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:50:58.821 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:58.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:58.870 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:58.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:58.971 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:59.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:59.015 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:59.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:59.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:50:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:50:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:50:59.116 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:59.155 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:59.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:59.256 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:50:59.287 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:59.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:59.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:50:59.390 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:50:59.419 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:59.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:59.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:50:59.521 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:50:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:59 smithi008 ceph-mon[40743]: 3.2 scrub starts 2024-02-10T07:50:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:59 smithi008 ceph-mon[40743]: 3.2 scrub ok 2024-02-10T07:50:59.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:50:59 smithi008 ceph-mon[40743]: pgmap v883: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 28 MiB/s wr, 452 op/s; 8861/232191 objects degraded (3.816%); 16936/232191 objects misplaced (7.294%); 2.2 MiB/s, 35 objects/s recovering 2024-02-10T07:50:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:59.583 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:59.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:50:59.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:59 smithi008 ceph-mon[29816]: 3.2 scrub starts 2024-02-10T07:50:59.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:59 smithi008 ceph-mon[29816]: 3.2 scrub ok 2024-02-10T07:50:59.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:50:59 smithi008 ceph-mon[29816]: pgmap v883: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 28 MiB/s wr, 452 op/s; 8861/232191 objects degraded (3.816%); 16936/232191 objects misplaced (7.294%); 2.2 MiB/s, 35 objects/s recovering 2024-02-10T07:50:59.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:59 smithi170 ceph-mon[33394]: 3.2 scrub starts 2024-02-10T07:50:59.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:59 smithi170 ceph-mon[33394]: 3.2 scrub ok 2024-02-10T07:50:59.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:50:59 smithi170 ceph-mon[33394]: pgmap v883: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 28 MiB/s wr, 452 op/s; 8861/232191 objects degraded (3.816%); 16936/232191 objects misplaced (7.294%); 2.2 MiB/s, 35 objects/s recovering 2024-02-10T07:50:59.654 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 54 16 81172 81156 93.9124 42.125 0.0249059 0.0106422 2024-02-10T07:50:59.685 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:50:59.713 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:50:59.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:59.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:50:59.815 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:50:59.869 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:50:59.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:50:59.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:50:59.973 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:50:59.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:00.000 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:00.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:00.101 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:00.130 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:00.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:00.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:00.232 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:00.261 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:00.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:00.363 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:00.390 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:00.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:00.495 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:00.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:00.537 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:00.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:00.588 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:51:00.639 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:00.657 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 55 16 81897 81881 93.0236 45.3125 0.0597275 0.0107416 2024-02-10T07:51:00.733 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:00.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:00.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:00.835 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:00.861 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:00.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:00.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:00.963 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:00.995 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:00.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:00.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:01.097 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:01.127 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:01.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:01.230 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:01.289 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:01.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:01.391 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:01.410 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:01 smithi008 ceph-mon[40743]: pgmap v884: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 826 op/s; 8861/242013 objects degraded (3.661%); 16612/242013 objects misplaced (6.864%); 4.7 MiB/s, 75 objects/s recovering 2024-02-10T07:51:01.410 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:01 smithi008 ceph-mon[29816]: pgmap v884: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 826 op/s; 8861/242013 objects degraded (3.661%); 16612/242013 objects misplaced (6.864%); 4.7 MiB/s, 75 objects/s recovering 2024-02-10T07:51:01.417 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:01.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:01.519 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:01.548 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:01.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:01.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:01 smithi170 ceph-mon[33394]: pgmap v884: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 52 MiB/s wr, 826 op/s; 8861/242013 objects degraded (3.661%); 16612/242013 objects misplaced (6.864%); 4.7 MiB/s, 75 objects/s recovering 2024-02-10T07:51:01.650 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:01.657 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 56 16 82574 82558 92.1182 42.3125 0.0123324 0.0108482 2024-02-10T07:51:01.677 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:01.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:01.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:01.779 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:01.831 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:01.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:01.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:01.933 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:01.947 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:51:01.957 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:51:01.963 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:01.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:02.065 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:02.092 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:02.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:02.194 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:02.228 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:02.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:02.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:02.330 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:02.355 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:02.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:02.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:02.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:02.483 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:02.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:02.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:02.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:02 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/242013 objects degraded (3.661%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:02 smithi008 ceph-mon[29816]: osdmap e410: 8 total, 8 up, 8 in 2024-02-10T07:51:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:02 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/242013 objects degraded (3.661%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:02 smithi008 ceph-mon[40743]: osdmap e410: 8 total, 8 up, 8 in 2024-02-10T07:51:02.584 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:02.623 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:02.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:02.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:02 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/242013 objects degraded (3.661%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:02 smithi170 ceph-mon[33394]: osdmap e410: 8 total, 8 up, 8 in 2024-02-10T07:51:02.657 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 57 16 83245 83229 91.2379 41.9375 0.0138383 0.0109517 2024-02-10T07:51:02.725 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:02.758 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:02.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:02.859 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:02.886 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:02.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:02.988 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:03.042 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:03.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:03.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:03.144 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:03.174 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:03.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:03.245 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:03 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:51:03.248 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:03 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:51:03.275 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:03.305 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:03.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:03.407 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:03.437 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:03.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:03.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:03.539 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:03.568 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:03.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:03.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:03.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:03 smithi008 ceph-mon[29816]: from='client.25886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:51:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:03 smithi008 ceph-mon[29816]: pgmap v886: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 626 op/s; 8861/242721 objects degraded (3.651%); 16612/242721 objects misplaced (6.844%); 4.8 MiB/s, 76 objects/s recovering 2024-02-10T07:51:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:03 smithi008 ceph-mon[29816]: 5.2 scrub starts 2024-02-10T07:51:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:03 smithi008 ceph-mon[29816]: 5.2 scrub ok 2024-02-10T07:51:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:03 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:51:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:03 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:51:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:03 smithi008 ceph-mon[40743]: from='client.25886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:51:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:03 smithi008 ceph-mon[40743]: pgmap v886: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 626 op/s; 8861/242721 objects degraded (3.651%); 16612/242721 objects misplaced (6.844%); 4.8 MiB/s, 76 objects/s recovering 2024-02-10T07:51:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:03 smithi008 ceph-mon[40743]: 5.2 scrub starts 2024-02-10T07:51:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:03 smithi008 ceph-mon[40743]: 5.2 scrub ok 2024-02-10T07:51:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:03 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:51:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:03 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:51:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:03 smithi170 ceph-mon[33394]: from='client.25886 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:51:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:03 smithi170 ceph-mon[33394]: pgmap v886: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 39 MiB/s wr, 626 op/s; 8861/242721 objects degraded (3.651%); 16612/242721 objects misplaced (6.844%); 4.8 MiB/s, 76 objects/s recovering 2024-02-10T07:51:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:03 smithi170 ceph-mon[33394]: 5.2 scrub starts 2024-02-10T07:51:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:03 smithi170 ceph-mon[33394]: 5.2 scrub ok 2024-02-10T07:51:03.657 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 58 16 83911 83895 90.3825 41.625 0.0223343 0.0110558 2024-02-10T07:51:03.674 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:03.701 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:03.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:03.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:03.803 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:03.829 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:03.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:03.932 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:03.984 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:03.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:03.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:04.087 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:04.113 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:04.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:04.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:51:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:51:04.191 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":884,"stamp":"2024-02-10T07:51:00.117316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5274474844,"num_objects":80671,"num_object_clones":0,"num_object_copies":242013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8861,"num_objects_misplaced":16612,"num_objects_unfound":0,"num_objects_dirty":80671,"num_whiteouts":0,"num_read":2487,"num_read_kb":2333,"num_write":81374,"num_write_kb":5152733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1172,"num_bytes_recovered":59583093,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79983,"ondisk_log_size":79983,"up":444,"acting":439,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":450,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20280480,"kb_used_data":15440204,"kb_used_omap":19,"kb_used_meta":4840108,"kb_avail":729680736,"statfs":{"total":767960285184,"available":747193073664,"internally_reserved":0,"allocated":15810768896,"data_stored":15807653535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19634,"internal_metadata":4956271438},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,9,7,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":438173696,"num_objects":6686,"num_object_clones":0,"num_object_copies":20058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-612,"num_objects_unfound":0,"num_objects_dirty":6686,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6686,"num_write_kb":427904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":612,"num_bytes_recovered":40108032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.098849"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":754,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009221+0000","last_change":"2024-02-10T07:49:56.315126+0000","last_active":"2024-02-10T07:50:57.009221+0000","last_peered":"2024-02-10T07:50:57.009221+0000","last_clean":"2024-02-10T07:50:57.009221+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:50:57.009221+0000","last_undegraded":"2024-02-10T07:50:57.009221+0000","last_fullsized":"2024-02-10T07:50:57.009221+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.315029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.315029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:19.186486+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":764,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.023806+0000","last_change":"2024-02-10T07:49:08.138553+0000","last_active":"2024-02-10T07:50:57.023806+0000","last_peered":"2024-02-10T07:50:57.023806+0000","last_clean":"2024-02-10T07:50:57.023806+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:50:57.023806+0000","last_undegraded":"2024-02-10T07:50:57.023806+0000","last_fullsized":"2024-02-10T07:50:57.023806+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:08.138442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.138442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":794,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.031047+0000","last_change":"2024-02-10T07:50:43.303349+0000","last_active":"2024-02-10T07:50:57.031047+0000","last_peered":"2024-02-10T07:50:57.031047+0000","last_clean":"2024-02-10T07:50:57.031047+0000","last_became_active":"2024-02-10T07:50:43.281905+0000","last_became_peered":"2024-02-10T07:50:43.281905+0000","last_unstale":"2024-02-10T07:50:57.031047+0000","last_undegraded":"2024-02-10T07:50:57.031047+0000","last_fullsized":"2024-02-10T07:50:57.031047+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:17.198701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:17.198701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"287'19","reported_seq":814,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009040+0000","last_change":"2024-02-10T07:46:37.008978+0000","last_active":"2024-02-10T07:50:57.009040+0000","last_peered":"2024-02-10T07:50:57.009040+0000","last_clean":"2024-02-10T07:50:57.009040+0000","last_became_active":"2024-02-10T07:46:10.717475+0000","last_became_peered":"2024-02-10T07:46:10.717475+0000","last_unstale":"2024-02-10T07:50:57.009040+0000","last_undegraded":"2024-02-10T07:50:57.009040+0000","last_fullsized":"2024-02-10T07:50:57.009040+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'19","last_scrub_stamp":"2024-02-10T07:46:37.008841+0000","last_deep_scrub":"287'19","last_deep_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.008841+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":83,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-02-10T07:50:52.017147+0000","last_change":"2024-02-10T07:49:15.079645+0000","last_active":"2024-02-10T07:50:52.017147+0000","last_peered":"2024-02-10T07:50:52.017147+0000","last_clean":"2024-02-10T07:50:52.017147+0000","last_became_active":"2024-02-10T07:49:06.236235+0000","last_became_peered":"2024-02-10T07:49:06.236235+0000","last_unstale":"2024-02-10T07:50:52.017147+0000","last_undegraded":"2024-02-10T07:50:52.017147+0000","last_fullsized":"2024-02-10T07:50:52.017147+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:15.079428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:15.079428+0000","last_clean_scrub_stamp":"2024-02-10T07:49:15.079428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":757,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009022+0000","last_change":"2024-02-10T07:50:06.335390+0000","last_active":"2024-02-10T07:50:57.009022+0000","last_peered":"2024-02-10T07:50:57.009022+0000","last_clean":"2024-02-10T07:50:57.009022+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:50:57.009022+0000","last_undegraded":"2024-02-10T07:50:57.009022+0000","last_fullsized":"2024-02-10T07:50:57.009022+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:06.335262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:50:06.335262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:25.829837+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":272,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026687+0000","last_change":"2024-02-10T07:50:43.282867+0000","last_active":"2024-02-10T07:50:57.026687+0000","last_peered":"2024-02-10T07:50:57.026687+0000","last_clean":"2024-02-10T07:50:57.026687+0000","last_became_active":"2024-02-10T07:50:43.282449+0000","last_became_peered":"2024-02-10T07:50:43.282449+0000","last_unstale":"2024-02-10T07:50:57.026687+0000","last_undegraded":"2024-02-10T07:50:57.026687+0000","last_fullsized":"2024-02-10T07:50:57.026687+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:47.812065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.812065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"250'27","reported_seq":35,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.040816+0000","last_change":"2024-02-10T07:50:43.318008+0000","last_active":"2024-02-10T07:50:57.040816+0000","last_peered":"2024-02-10T07:50:57.040816+0000","last_clean":"2024-02-10T07:50:57.040816+0000","last_became_active":"2024-02-10T07:50:43.290681+0000","last_became_peered":"2024-02-10T07:50:43.290681+0000","last_unstale":"2024-02-10T07:50:57.040816+0000","last_undegraded":"2024-02-10T07:50:57.040816+0000","last_fullsized":"2024-02-10T07:50:57.040816+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:48:13.230908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.230908+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":753,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.021229+0000","last_change":"2024-02-10T07:50:05.620136+0000","last_active":"2024-02-10T07:50:57.021229+0000","last_peered":"2024-02-10T07:50:57.021229+0000","last_clean":"2024-02-10T07:50:57.021229+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:50:57.021229+0000","last_undegraded":"2024-02-10T07:50:57.021229+0000","last_fullsized":"2024-02-10T07:50:57.021229+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:05.619958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:05.619958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:09.870665+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":726,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.028362+0000","last_change":"2024-02-10T07:49:57.669438+0000","last_active":"2024-02-10T07:50:57.028362+0000","last_peered":"2024-02-10T07:50:57.028362+0000","last_clean":"2024-02-10T07:50:57.028362+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:50:57.028362+0000","last_undegraded":"2024-02-10T07:50:57.028362+0000","last_fullsized":"2024-02-10T07:50:57.028362+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:57.669314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.669314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:17.355126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":295,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009766+0000","last_change":"2024-02-10T07:49:13.218634+0000","last_active":"2024-02-10T07:50:57.009766+0000","last_peered":"2024-02-10T07:50:57.009766+0000","last_clean":"2024-02-10T07:50:57.009766+0000","last_became_active":"2024-02-10T07:48:02.332960+0000","last_became_peered":"2024-02-10T07:48:02.332960+0000","last_unstale":"2024-02-10T07:50:57.009766+0000","last_undegraded":"2024-02-10T07:50:57.009766+0000","last_fullsized":"2024-02-10T07:50:57.009766+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.218549+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"250'15","reported_seq":779,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009328+0000","last_change":"2024-02-10T07:47:49.553645+0000","last_active":"2024-02-10T07:50:57.009328+0000","last_peered":"2024-02-10T07:50:57.009328+0000","last_clean":"2024-02-10T07:50:57.009328+0000","last_became_active":"2024-02-10T07:47:47.965930+0000","last_became_peered":"2024-02-10T07:47:47.965930+0000","last_unstale":"2024-02-10T07:50:57.009328+0000","last_undegraded":"2024-02-10T07:50:57.009328+0000","last_fullsized":"2024-02-10T07:50:57.009328+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:49.553517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.553517+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":666,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.023977+0000","last_change":"2024-02-10T07:49:56.718482+0000","last_active":"2024-02-10T07:50:57.023977+0000","last_peered":"2024-02-10T07:50:57.023977+0000","last_clean":"2024-02-10T07:50:57.023977+0000","last_became_active":"2024-02-10T07:49:06.266689+0000","last_became_peered":"2024-02-10T07:49:06.266689+0000","last_unstale":"2024-02-10T07:50:57.023977+0000","last_undegraded":"2024-02-10T07:50:57.023977+0000","last_fullsized":"2024-02-10T07:50:57.023977+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.718382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.718382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:12.254870+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":757,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030535+0000","last_change":"2024-02-10T07:48:18.069773+0000","last_active":"2024-02-10T07:50:57.030535+0000","last_peered":"2024-02-10T07:50:57.030535+0000","last_clean":"2024-02-10T07:50:57.030535+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:50:57.030535+0000","last_undegraded":"2024-02-10T07:50:57.030535+0000","last_fullsized":"2024-02-10T07:50:57.030535+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:18.069641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:18.069641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":283,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009546+0000","last_change":"2024-02-10T07:49:06.115547+0000","last_active":"2024-02-10T07:50:57.009546+0000","last_peered":"2024-02-10T07:50:57.009546+0000","last_clean":"2024-02-10T07:50:57.009546+0000","last_became_active":"2024-02-10T07:48:02.329859+0000","last_became_peered":"2024-02-10T07:48:02.329859+0000","last_unstale":"2024-02-10T07:50:57.009546+0000","last_undegraded":"2024-02-10T07:50:57.009546+0000","last_fullsized":"2024-02-10T07:50:57.009546+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.115466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.473686+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.115466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"250'7","reported_seq":767,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009715+0000","last_change":"2024-02-10T07:48:11.611397+0000","last_active":"2024-02-10T07:50:57.009715+0000","last_peered":"2024-02-10T07:50:57.009715+0000","last_clean":"2024-02-10T07:50:57.009715+0000","last_became_active":"2024-02-10T07:47:47.966093+0000","last_became_peered":"2024-02-10T07:47:47.966093+0000","last_unstale":"2024-02-10T07:50:57.009715+0000","last_undegraded":"2024-02-10T07:50:57.009715+0000","last_fullsized":"2024-02-10T07:50:57.009715+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7","last_scrub_stamp":"2024-02-10T07:48:11.608499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.608499+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":756,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.010334+0000","last_change":"2024-02-10T07:48:13.621135+0000","last_active":"2024-02-10T07:50:57.010334+0000","last_peered":"2024-02-10T07:50:57.010334+0000","last_clean":"2024-02-10T07:50:57.010334+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:50:57.010334+0000","last_undegraded":"2024-02-10T07:50:57.010334+0000","last_fullsized":"2024-02-10T07:50:57.010334+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:13.620938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.620938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":763,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.033386+0000","last_change":"2024-02-10T07:49:51.810158+0000","last_active":"2024-02-10T07:50:57.033386+0000","last_peered":"2024-02-10T07:50:57.033386+0000","last_clean":"2024-02-10T07:50:57.033386+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:50:57.033386+0000","last_undegraded":"2024-02-10T07:50:57.033386+0000","last_fullsized":"2024-02-10T07:50:57.033386+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:51.810062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:51.810062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:59.448781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":765,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030310+0000","last_change":"2024-02-10T07:49:16.313655+0000","last_active":"2024-02-10T07:50:57.030310+0000","last_peered":"2024-02-10T07:50:57.030310+0000","last_clean":"2024-02-10T07:50:57.030310+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:50:57.030310+0000","last_undegraded":"2024-02-10T07:50:57.030310+0000","last_fullsized":"2024-02-10T07:50:57.030310+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:16.313573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.313573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"250'34","reported_seq":35,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.041768+0000","last_change":"2024-02-10T07:50:43.322322+0000","last_active":"2024-02-10T07:50:57.041768+0000","last_peered":"2024-02-10T07:50:57.041768+0000","last_clean":"2024-02-10T07:50:57.041768+0000","last_became_active":"2024-02-10T07:50:43.300153+0000","last_became_peered":"2024-02-10T07:50:43.300153+0000","last_unstale":"2024-02-10T07:50:57.041768+0000","last_undegraded":"2024-02-10T07:50:57.041768+0000","last_fullsized":"2024-02-10T07:50:57.041768+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:49:00.551605+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.551605+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":451,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026798+0000","last_change":"2024-02-10T07:49:51.990345+0000","last_active":"2024-02-10T07:50:57.026798+0000","last_peered":"2024-02-10T07:50:57.026798+0000","last_clean":"2024-02-10T07:50:57.026798+0000","last_became_active":"2024-02-10T07:49:06.263790+0000","last_became_peered":"2024-02-10T07:49:06.263790+0000","last_unstale":"2024-02-10T07:50:57.026798+0000","last_undegraded":"2024-02-10T07:50:57.026798+0000","last_fullsized":"2024-02-10T07:50:57.026798+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:51.990237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:51.990237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:13.825668+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":774,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009830+0000","last_change":"2024-02-10T07:50:43.286623+0000","last_active":"2024-02-10T07:50:57.009830+0000","last_peered":"2024-02-10T07:50:57.009830+0000","last_clean":"2024-02-10T07:50:57.009830+0000","last_became_active":"2024-02-10T07:50:43.285201+0000","last_became_peered":"2024-02-10T07:50:43.285201+0000","last_unstale":"2024-02-10T07:50:57.009830+0000","last_undegraded":"2024-02-10T07:50:57.009830+0000","last_fullsized":"2024-02-10T07:50:57.009830+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.680331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.680331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":777,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009447+0000","last_change":"2024-02-10T07:50:43.282584+0000","last_active":"2024-02-10T07:50:57.009447+0000","last_peered":"2024-02-10T07:50:57.009447+0000","last_clean":"2024-02-10T07:50:57.009447+0000","last_became_active":"2024-02-10T07:50:43.281382+0000","last_became_peered":"2024-02-10T07:50:43.281382+0000","last_unstale":"2024-02-10T07:50:57.009447+0000","last_undegraded":"2024-02-10T07:50:57.009447+0000","last_fullsized":"2024-02-10T07:50:57.009447+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:04.396771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.396771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:12.835152+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"250'18","reported_seq":724,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.033171+0000","last_change":"2024-02-10T07:49:58.618564+0000","last_active":"2024-02-10T07:50:57.033171+0000","last_peered":"2024-02-10T07:50:57.033171+0000","last_clean":"2024-02-10T07:50:57.033171+0000","last_became_active":"2024-02-10T07:49:06.263261+0000","last_became_peered":"2024-02-10T07:49:06.263261+0000","last_unstale":"2024-02-10T07:50:57.033171+0000","last_undegraded":"2024-02-10T07:50:57.033171+0000","last_fullsized":"2024-02-10T07:50:57.033171+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'18","last_scrub_stamp":"2024-02-10T07:49:58.618464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:58.618464+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:59.071088+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":789,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.031571+0000","last_change":"2024-02-10T07:50:43.307775+0000","last_active":"2024-02-10T07:50:57.031571+0000","last_peered":"2024-02-10T07:50:57.031571+0000","last_clean":"2024-02-10T07:50:57.031571+0000","last_became_active":"2024-02-10T07:50:43.285254+0000","last_became_peered":"2024-02-10T07:50:43.285254+0000","last_unstale":"2024-02-10T07:50:57.031571+0000","last_undegraded":"2024-02-10T07:50:57.031571+0000","last_fullsized":"2024-02-10T07:50:57.031571+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:47:34.074746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:34.074746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":1023,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026627+0000","last_change":"2024-02-10T07:49:56.824743+0000","last_active":"2024-02-10T07:50:57.026627+0000","last_peered":"2024-02-10T07:50:57.026627+0000","last_clean":"2024-02-10T07:50:57.026627+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:50:57.026627+0000","last_undegraded":"2024-02-10T07:50:57.026627+0000","last_fullsized":"2024-02-10T07:50:57.026627+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:49:56.824601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.824601+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:04.771314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":773,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.023701+0000","last_change":"2024-02-10T07:48:30.384635+0000","last_active":"2024-02-10T07:50:57.023701+0000","last_peered":"2024-02-10T07:50:57.023701+0000","last_clean":"2024-02-10T07:50:57.023701+0000","last_became_active":"2024-02-10T07:47:47.966127+0000","last_became_peered":"2024-02-10T07:47:47.966127+0000","last_unstale":"2024-02-10T07:50:57.023701+0000","last_undegraded":"2024-02-10T07:50:57.023701+0000","last_fullsized":"2024-02-10T07:50:57.023701+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:30.384546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:30.384546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"250'25","reported_seq":837,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.048527+0000","last_change":"2024-02-10T07:49:06.308245+0000","last_active":"2024-02-10T07:50:57.048527+0000","last_peered":"2024-02-10T07:50:57.048527+0000","last_clean":"2024-02-10T07:50:57.048527+0000","last_became_active":"2024-02-10T07:49:06.261487+0000","last_became_peered":"2024-02-10T07:49:06.261487+0000","last_unstale":"2024-02-10T07:50:57.048527+0000","last_undegraded":"2024-02-10T07:50:57.048527+0000","last_fullsized":"2024-02-10T07:50:57.048527+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:48:12.278982+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-02-10T07:42:17.820959+0000","last_clean_scrub_stamp":"2024-02-10T07:48:12.278982+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":760,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.033487+0000","last_change":"2024-02-10T07:49:11.231819+0000","last_active":"2024-02-10T07:50:57.033487+0000","last_peered":"2024-02-10T07:50:57.033487+0000","last_clean":"2024-02-10T07:50:57.033487+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:50:57.033487+0000","last_undegraded":"2024-02-10T07:50:57.033487+0000","last_fullsized":"2024-02-10T07:50:57.033487+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.231704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.231704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":763,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.048743+0000","last_change":"2024-02-10T07:50:43.289955+0000","last_active":"2024-02-10T07:50:57.048743+0000","last_peered":"2024-02-10T07:50:57.048743+0000","last_clean":"2024-02-10T07:50:57.048743+0000","last_became_active":"2024-02-10T07:50:43.289011+0000","last_became_peered":"2024-02-10T07:50:43.289011+0000","last_unstale":"2024-02-10T07:50:57.048743+0000","last_undegraded":"2024-02-10T07:50:57.048743+0000","last_fullsized":"2024-02-10T07:50:57.048743+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.208451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:06.200009+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.208451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":261,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.021150+0000","last_change":"2024-02-10T07:49:06.259104+0000","last_active":"2024-02-10T07:50:57.021150+0000","last_peered":"2024-02-10T07:50:57.021150+0000","last_clean":"2024-02-10T07:50:57.021150+0000","last_became_active":"2024-02-10T07:49:06.258789+0000","last_became_peered":"2024-02-10T07:49:06.258789+0000","last_unstale":"2024-02-10T07:50:57.021150+0000","last_undegraded":"2024-02-10T07:50:57.021150+0000","last_fullsized":"2024-02-10T07:50:57.021150+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:45.224482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:47:45.224482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"250'19","reported_seq":810,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026715+0000","last_change":"2024-02-10T07:49:50.808829+0000","last_active":"2024-02-10T07:50:57.026715+0000","last_peered":"2024-02-10T07:50:57.026715+0000","last_clean":"2024-02-10T07:50:57.026715+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:50:57.026715+0000","last_undegraded":"2024-02-10T07:50:57.026715+0000","last_fullsized":"2024-02-10T07:50:57.026715+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:50.808710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.808710+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:14.055156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":776,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030755+0000","last_change":"2024-02-10T07:50:43.284049+0000","last_active":"2024-02-10T07:50:57.030755+0000","last_peered":"2024-02-10T07:50:57.030755+0000","last_clean":"2024-02-10T07:50:57.030755+0000","last_became_active":"2024-02-10T07:50:43.283688+0000","last_became_peered":"2024-02-10T07:50:43.283688+0000","last_unstale":"2024-02-10T07:50:57.030755+0000","last_undegraded":"2024-02-10T07:50:57.030755+0000","last_fullsized":"2024-02-10T07:50:57.030755+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.087614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.087614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":773,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026263+0000","last_change":"2024-02-10T07:49:54.769358+0000","last_active":"2024-02-10T07:50:57.026263+0000","last_peered":"2024-02-10T07:50:57.026263+0000","last_clean":"2024-02-10T07:50:57.026263+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:50:57.026263+0000","last_undegraded":"2024-02-10T07:50:57.026263+0000","last_fullsized":"2024-02-10T07:50:57.026263+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:54.769226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:54.769226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:24.449065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":789,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.031677+0000","last_change":"2024-02-10T07:49:06.259100+0000","last_active":"2024-02-10T07:50:57.031677+0000","last_peered":"2024-02-10T07:50:57.031677+0000","last_clean":"2024-02-10T07:50:57.031677+0000","last_became_active":"2024-02-10T07:49:06.258763+0000","last_became_peered":"2024-02-10T07:49:06.258763+0000","last_unstale":"2024-02-10T07:50:57.031677+0000","last_undegraded":"2024-02-10T07:50:57.031677+0000","last_fullsized":"2024-02-10T07:50:57.031677+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:29.188832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:29.188832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"250'10","reported_seq":89,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-02-10T07:50:52.018170+0000","last_change":"2024-02-10T07:49:16.017097+0000","last_active":"2024-02-10T07:50:52.018170+0000","last_peered":"2024-02-10T07:50:52.018170+0000","last_clean":"2024-02-10T07:50:52.018170+0000","last_became_active":"2024-02-10T07:49:06.250577+0000","last_became_peered":"2024-02-10T07:49:06.250577+0000","last_unstale":"2024-02-10T07:50:52.018170+0000","last_undegraded":"2024-02-10T07:50:52.018170+0000","last_fullsized":"2024-02-10T07:50:52.018170+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'10","last_scrub_stamp":"2024-02-10T07:49:16.017003+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.017003+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":255,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.029783+0000","last_change":"2024-02-10T07:49:06.259895+0000","last_active":"2024-02-10T07:50:57.029783+0000","last_peered":"2024-02-10T07:50:57.029783+0000","last_clean":"2024-02-10T07:50:57.029783+0000","last_became_active":"2024-02-10T07:49:06.259341+0000","last_became_peered":"2024-02-10T07:49:06.259341+0000","last_unstale":"2024-02-10T07:50:57.029783+0000","last_undegraded":"2024-02-10T07:50:57.029783+0000","last_fullsized":"2024-02-10T07:50:57.029783+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:32.258321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:32.258321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":758,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030452+0000","last_change":"2024-02-10T07:48:24.171978+0000","last_active":"2024-02-10T07:50:57.030452+0000","last_peered":"2024-02-10T07:50:57.030452+0000","last_clean":"2024-02-10T07:50:57.030452+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:50:57.030452+0000","last_undegraded":"2024-02-10T07:50:57.030452+0000","last_fullsized":"2024-02-10T07:50:57.030452+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:24.171859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:24.171859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":698,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.028874+0000","last_change":"2024-02-10T07:49:53.578357+0000","last_active":"2024-02-10T07:50:57.028874+0000","last_peered":"2024-02-10T07:50:57.028874+0000","last_clean":"2024-02-10T07:50:57.028874+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:50:57.028874+0000","last_undegraded":"2024-02-10T07:50:57.028874+0000","last_fullsized":"2024-02-10T07:50:57.028874+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:53.578274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:49:53.578274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.901374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"250'19","reported_seq":724,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030434+0000","last_change":"2024-02-10T07:49:53.245490+0000","last_active":"2024-02-10T07:50:57.030434+0000","last_peered":"2024-02-10T07:50:57.030434+0000","last_clean":"2024-02-10T07:50:57.030434+0000","last_became_active":"2024-02-10T07:49:06.239701+0000","last_became_peered":"2024-02-10T07:49:06.239701+0000","last_unstale":"2024-02-10T07:50:57.030434+0000","last_undegraded":"2024-02-10T07:50:57.030434+0000","last_fullsized":"2024-02-10T07:50:57.030434+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:53.245410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:53.245410+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:19.652831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":755,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.020930+0000","last_change":"2024-02-10T07:49:52.530010+0000","last_active":"2024-02-10T07:50:57.020930+0000","last_peered":"2024-02-10T07:50:57.020930+0000","last_clean":"2024-02-10T07:50:57.020930+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:50:57.020930+0000","last_undegraded":"2024-02-10T07:50:57.020930+0000","last_fullsized":"2024-02-10T07:50:57.020930+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.529908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:59.179342+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.529908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.765757+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":255,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.029974+0000","last_change":"2024-02-10T07:49:06.265551+0000","last_active":"2024-02-10T07:50:57.029974+0000","last_peered":"2024-02-10T07:50:57.029974+0000","last_clean":"2024-02-10T07:50:57.029974+0000","last_became_active":"2024-02-10T07:49:06.265267+0000","last_became_peered":"2024-02-10T07:49:06.265267+0000","last_unstale":"2024-02-10T07:50:57.029974+0000","last_undegraded":"2024-02-10T07:50:57.029974+0000","last_fullsized":"2024-02-10T07:50:57.029974+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:40.140557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:40.140557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":774,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030637+0000","last_change":"2024-02-10T07:50:00.265308+0000","last_active":"2024-02-10T07:50:57.030637+0000","last_peered":"2024-02-10T07:50:57.030637+0000","last_clean":"2024-02-10T07:50:57.030637+0000","last_became_active":"2024-02-10T07:47:47.966472+0000","last_became_peered":"2024-02-10T07:47:47.966472+0000","last_unstale":"2024-02-10T07:50:57.030637+0000","last_undegraded":"2024-02-10T07:50:57.030637+0000","last_fullsized":"2024-02-10T07:50:57.030637+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.265208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.265208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:23.837007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"250'19","reported_seq":598,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.024084+0000","last_change":"2024-02-10T07:50:43.310533+0000","last_active":"2024-02-10T07:50:57.024084+0000","last_peered":"2024-02-10T07:50:57.024084+0000","last_clean":"2024-02-10T07:50:57.024084+0000","last_became_active":"2024-02-10T07:50:43.296447+0000","last_became_peered":"2024-02-10T07:50:43.296447+0000","last_unstale":"2024-02-10T07:50:57.024084+0000","last_undegraded":"2024-02-10T07:50:57.024084+0000","last_fullsized":"2024-02-10T07:50:57.024084+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:09.116615+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-02-10T07:39:32.934269+0000","last_clean_scrub_stamp":"2024-02-10T07:49:09.116615+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":443,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.029284+0000","last_change":"2024-02-10T07:49:06.260883+0000","last_active":"2024-02-10T07:50:57.029284+0000","last_peered":"2024-02-10T07:50:57.029284+0000","last_clean":"2024-02-10T07:50:57.029284+0000","last_became_active":"2024-02-10T07:49:06.260569+0000","last_became_peered":"2024-02-10T07:49:06.260569+0000","last_unstale":"2024-02-10T07:50:57.029284+0000","last_undegraded":"2024-02-10T07:50:57.029284+0000","last_fullsized":"2024-02-10T07:50:57.029284+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:31.117976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:31.117976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":788,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.029318+0000","last_change":"2024-02-10T07:50:02.685941+0000","last_active":"2024-02-10T07:50:57.029318+0000","last_peered":"2024-02-10T07:50:57.029318+0000","last_clean":"2024-02-10T07:50:57.029318+0000","last_became_active":"2024-02-10T07:49:06.252380+0000","last_became_peered":"2024-02-10T07:49:06.252380+0000","last_unstale":"2024-02-10T07:50:57.029318+0000","last_undegraded":"2024-02-10T07:50:57.029318+0000","last_fullsized":"2024-02-10T07:50:57.029318+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_clean_scrub_stamp":"2024-02-10T07:50:02.685831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:09.446762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":766,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009739+0000","last_change":"2024-02-10T07:50:00.345087+0000","last_active":"2024-02-10T07:50:57.009739+0000","last_peered":"2024-02-10T07:50:57.009739+0000","last_clean":"2024-02-10T07:50:57.009739+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:50:57.009739+0000","last_undegraded":"2024-02-10T07:50:57.009739+0000","last_fullsized":"2024-02-10T07:50:57.009739+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.344993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.344993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:26.742023+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"250'24","reported_seq":826,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009614+0000","last_change":"2024-02-10T07:47:43.494877+0000","last_active":"2024-02-10T07:50:57.009614+0000","last_peered":"2024-02-10T07:50:57.009614+0000","last_clean":"2024-02-10T07:50:57.009614+0000","last_became_active":"2024-02-10T07:46:10.721336+0000","last_became_peered":"2024-02-10T07:46:10.721336+0000","last_unstale":"2024-02-10T07:50:57.009614+0000","last_undegraded":"2024-02-10T07:50:57.009614+0000","last_fullsized":"2024-02-10T07:50:57.009614+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:47:43.494350+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.494350+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":760,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.048230+0000","last_change":"2024-02-10T07:49:06.262127+0000","last_active":"2024-02-10T07:50:57.048230+0000","last_peered":"2024-02-10T07:50:57.048230+0000","last_clean":"2024-02-10T07:50:57.048230+0000","last_became_active":"2024-02-10T07:49:06.261570+0000","last_became_peered":"2024-02-10T07:49:06.261570+0000","last_unstale":"2024-02-10T07:50:57.048230+0000","last_undegraded":"2024-02-10T07:50:57.048230+0000","last_fullsized":"2024-02-10T07:50:57.048230+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:43.186809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.186809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"356'11","reported_seq":232,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.033290+0000","last_change":"2024-02-10T07:48:25.339974+0000","last_active":"2024-02-10T07:50:57.033290+0000","last_peered":"2024-02-10T07:50:57.033290+0000","last_clean":"2024-02-10T07:50:57.033290+0000","last_became_active":"2024-02-10T07:48:02.371888+0000","last_became_peered":"2024-02-10T07:48:02.371888+0000","last_unstale":"2024-02-10T07:50:57.033290+0000","last_undegraded":"2024-02-10T07:50:57.033290+0000","last_fullsized":"2024-02-10T07:50:57.033290+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'11","last_scrub_stamp":"2024-02-10T07:48:25.339855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:25.339855+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":786,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.031896+0000","last_change":"2024-02-10T07:50:43.283263+0000","last_active":"2024-02-10T07:50:57.031896+0000","last_peered":"2024-02-10T07:50:57.031896+0000","last_clean":"2024-02-10T07:50:57.031896+0000","last_became_active":"2024-02-10T07:50:43.282925+0000","last_became_peered":"2024-02-10T07:50:43.282925+0000","last_unstale":"2024-02-10T07:50:57.031896+0000","last_undegraded":"2024-02-10T07:50:57.031896+0000","last_fullsized":"2024-02-10T07:50:57.031896+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.335455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.335455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"250'25","reported_seq":387,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.023820+0000","last_change":"2024-02-10T07:49:12.150442+0000","last_active":"2024-02-10T07:50:57.023820+0000","last_peered":"2024-02-10T07:50:57.023820+0000","last_clean":"2024-02-10T07:50:57.023820+0000","last_became_active":"2024-02-10T07:48:02.329187+0000","last_became_peered":"2024-02-10T07:48:02.329187+0000","last_unstale":"2024-02-10T07:50:57.023820+0000","last_undegraded":"2024-02-10T07:50:57.023820+0000","last_fullsized":"2024-02-10T07:50:57.023820+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:49:12.150298+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:49:12.150298+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":766,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.010095+0000","last_change":"2024-02-10T07:50:43.285390+0000","last_active":"2024-02-10T07:50:57.010095+0000","last_peered":"2024-02-10T07:50:57.010095+0000","last_clean":"2024-02-10T07:50:57.010095+0000","last_became_active":"2024-02-10T07:50:43.284846+0000","last_became_peered":"2024-02-10T07:50:43.284846+0000","last_unstale":"2024-02-10T07:50:57.010095+0000","last_undegraded":"2024-02-10T07:50:57.010095+0000","last_fullsized":"2024-02-10T07:50:57.010095+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:01.513972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:01.513972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.003425+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"405'15","reported_seq":388,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.031475+0000","last_change":"2024-02-10T07:50:43.310228+0000","last_active":"2024-02-10T07:50:57.031475+0000","last_peered":"2024-02-10T07:50:57.031475+0000","last_clean":"2024-02-10T07:50:57.031475+0000","last_became_active":"2024-02-10T07:50:43.285213+0000","last_became_peered":"2024-02-10T07:50:43.285213+0000","last_unstale":"2024-02-10T07:50:57.031475+0000","last_undegraded":"2024-02-10T07:50:57.031475+0000","last_fullsized":"2024-02-10T07:50:57.031475+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'13","last_scrub_stamp":"2024-02-10T07:49:00.366610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.366610+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":781,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.047909+0000","last_change":"2024-02-10T07:50:07.501014+0000","last_active":"2024-02-10T07:50:57.047909+0000","last_peered":"2024-02-10T07:50:57.047909+0000","last_clean":"2024-02-10T07:50:57.047909+0000","last_became_active":"2024-02-10T07:49:06.257043+0000","last_became_peered":"2024-02-10T07:49:06.257043+0000","last_unstale":"2024-02-10T07:50:57.047909+0000","last_undegraded":"2024-02-10T07:50:57.047909+0000","last_fullsized":"2024-02-10T07:50:57.047909+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.500890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:33.615213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"250'15","reported_seq":519,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.031013+0000","last_change":"2024-02-10T07:49:06.277154+0000","last_active":"2024-02-10T07:50:57.031013+0000","last_peered":"2024-02-10T07:50:57.031013+0000","last_clean":"2024-02-10T07:50:57.031013+0000","last_became_active":"2024-02-10T07:49:06.255239+0000","last_became_peered":"2024-02-10T07:49:06.255239+0000","last_unstale":"2024-02-10T07:50:57.031013+0000","last_undegraded":"2024-02-10T07:50:57.031013+0000","last_fullsized":"2024-02-10T07:50:57.031013+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:39.610017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:39.610017+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":19,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.041863+0000","last_change":"2024-02-10T07:50:43.282730+0000","last_active":"2024-02-10T07:50:57.041863+0000","last_peered":"2024-02-10T07:50:57.041863+0000","last_clean":"2024-02-10T07:50:57.041863+0000","last_became_active":"2024-02-10T07:50:43.281549+0000","last_became_peered":"2024-02-10T07:50:43.281549+0000","last_unstale":"2024-02-10T07:50:57.041863+0000","last_undegraded":"2024-02-10T07:50:57.041863+0000","last_fullsized":"2024-02-10T07:50:57.041863+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:44.324046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:49:44.324046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:12.483651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":113,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026616+0000","last_change":"2024-02-10T07:50:52.097054+0000","last_active":"2024-02-10T07:50:57.026616+0000","last_peered":"2024-02-10T07:50:57.026616+0000","last_clean":"2024-02-10T07:50:57.026616+0000","last_became_active":"2024-02-10T07:49:06.262933+0000","last_became_peered":"2024-02-10T07:49:06.262933+0000","last_unstale":"2024-02-10T07:50:57.026616+0000","last_undegraded":"2024-02-10T07:50:57.026616+0000","last_fullsized":"2024-02-10T07:50:57.026616+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:52.096983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:50:52.096983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:52:09.105124+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":774,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030315+0000","last_change":"2024-02-10T07:49:59.278666+0000","last_active":"2024-02-10T07:50:57.030315+0000","last_peered":"2024-02-10T07:50:57.030315+0000","last_clean":"2024-02-10T07:50:57.030315+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:50:57.030315+0000","last_undegraded":"2024-02-10T07:50:57.030315+0000","last_fullsized":"2024-02-10T07:50:57.030315+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.278574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.278574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.042755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"250'19","reported_seq":825,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.048609+0000","last_change":"2024-02-10T07:49:06.306965+0000","last_active":"2024-02-10T07:50:57.048609+0000","last_peered":"2024-02-10T07:50:57.048609+0000","last_clean":"2024-02-10T07:50:57.048609+0000","last_became_active":"2024-02-10T07:49:06.260962+0000","last_became_peered":"2024-02-10T07:49:06.260962+0000","last_unstale":"2024-02-10T07:50:57.048609+0000","last_undegraded":"2024-02-10T07:50:57.048609+0000","last_fullsized":"2024-02-10T07:50:57.048609+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:47:42.150605+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.150605+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":753,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030554+0000","last_change":"2024-02-10T07:48:58.511630+0000","last_active":"2024-02-10T07:50:57.030554+0000","last_peered":"2024-02-10T07:50:57.030554+0000","last_clean":"2024-02-10T07:50:57.030554+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:50:57.030554+0000","last_undegraded":"2024-02-10T07:50:57.030554+0000","last_fullsized":"2024-02-10T07:50:57.030554+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:58.511251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.511251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":83,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-02-10T07:50:52.016870+0000","last_change":"2024-02-10T07:50:04.978177+0000","last_active":"2024-02-10T07:50:52.016870+0000","last_peered":"2024-02-10T07:50:52.016870+0000","last_clean":"2024-02-10T07:50:52.016870+0000","last_became_active":"2024-02-10T07:49:06.250832+0000","last_became_peered":"2024-02-10T07:49:06.250832+0000","last_unstale":"2024-02-10T07:50:52.016870+0000","last_undegraded":"2024-02-10T07:50:52.016870+0000","last_fullsized":"2024-02-10T07:50:52.016870+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:04.978097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.978097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:19.962282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":798,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026020+0000","last_change":"2024-02-10T07:49:52.832935+0000","last_active":"2024-02-10T07:50:57.026020+0000","last_peered":"2024-02-10T07:50:57.026020+0000","last_clean":"2024-02-10T07:50:57.026020+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:50:57.026020+0000","last_undegraded":"2024-02-10T07:50:57.026020+0000","last_fullsized":"2024-02-10T07:50:57.026020+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.832862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.832862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:16.134029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"250'12","reported_seq":91,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-02-10T07:50:52.017651+0000","last_change":"2024-02-10T07:49:43.919297+0000","last_active":"2024-02-10T07:50:52.017651+0000","last_peered":"2024-02-10T07:50:52.017651+0000","last_clean":"2024-02-10T07:50:52.017651+0000","last_became_active":"2024-02-10T07:49:06.258281+0000","last_became_peered":"2024-02-10T07:49:06.258281+0000","last_unstale":"2024-02-10T07:50:52.017651+0000","last_undegraded":"2024-02-10T07:50:52.017651+0000","last_fullsized":"2024-02-10T07:50:52.017651+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'12","last_scrub_stamp":"2024-02-10T07:49:43.919178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:43.919178+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:50:52.611584+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"409'5085","reported_seq":5131,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:59.129615+0000","last_change":"2024-02-10T07:49:53.374457+0000","last_active":"2024-02-10T07:50:59.129615+0000","last_peered":"2024-02-10T07:50:59.129615+0000","last_clean":"2024-02-10T07:50:59.129615+0000","last_became_active":"2024-02-10T07:49:53.374185+0000","last_became_peered":"2024-02-10T07:49:53.374185+0000","last_unstale":"2024-02-10T07:50:59.129615+0000","last_undegraded":"2024-02-10T07:50:59.129615+0000","last_fullsized":"2024-02-10T07:50:59.129615+0000","mapping_epoch":391,"log_start":"397'200","ondisk_log_start":"397'200","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4885,"log_dups_size":0,"ondisk_log_size":4885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333250560,"num_objects":5085,"num_object_clones":0,"num_object_copies":15255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":325440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":287,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026127+0000","last_change":"2024-02-10T07:50:05.847991+0000","last_active":"2024-02-10T07:50:57.026127+0000","last_peered":"2024-02-10T07:50:57.026127+0000","last_clean":"2024-02-10T07:50:57.026127+0000","last_became_active":"2024-02-10T07:48:02.366002+0000","last_became_peered":"2024-02-10T07:48:02.366002+0000","last_unstale":"2024-02-10T07:50:57.026127+0000","last_undegraded":"2024-02-10T07:50:57.026127+0000","last_fullsized":"2024-02-10T07:50:57.026127+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:05.847866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:05.847866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:23.653860+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"245'4","reported_seq":788,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.031242+0000","last_change":"2024-02-10T07:49:08.311056+0000","last_active":"2024-02-10T07:50:57.031242+0000","last_peered":"2024-02-10T07:50:57.031242+0000","last_clean":"2024-02-10T07:50:57.031242+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:50:57.031242+0000","last_undegraded":"2024-02-10T07:50:57.031242+0000","last_fullsized":"2024-02-10T07:50:57.031242+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'4","last_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_deep_scrub":"245'4","last_deep_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.310917+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"250'34","reported_seq":514,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030361+0000","last_change":"2024-02-10T07:48:11.967227+0000","last_active":"2024-02-10T07:50:57.030361+0000","last_peered":"2024-02-10T07:50:57.030361+0000","last_clean":"2024-02-10T07:50:57.030361+0000","last_became_active":"2024-02-10T07:48:02.330130+0000","last_became_peered":"2024-02-10T07:48:02.330130+0000","last_unstale":"2024-02-10T07:50:57.030361+0000","last_undegraded":"2024-02-10T07:50:57.030361+0000","last_fullsized":"2024-02-10T07:50:57.030361+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:48:11.966957+0000","last_deep_scrub":"246'22","last_deep_scrub_stamp":"2024-02-10T07:41:26.771112+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.966957+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":1004,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030550+0000","last_change":"2024-02-10T07:48:53.423840+0000","last_active":"2024-02-10T07:50:57.030550+0000","last_peered":"2024-02-10T07:50:57.030550+0000","last_clean":"2024-02-10T07:50:57.030550+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:50:57.030550+0000","last_undegraded":"2024-02-10T07:50:57.030550+0000","last_fullsized":"2024-02-10T07:50:57.030550+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:48:53.423613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:53.423613+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"409'4982","reported_seq":5657,"reported_epoch":409,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-10T07:50:59.437533+0000","last_change":"2024-02-10T07:50:44.688511+0000","last_active":"2024-02-10T07:50:59.437533+0000","last_peered":"2024-02-10T07:50:59.437533+0000","last_clean":"2024-02-10T07:50:42.243684+0000","last_became_active":"2024-02-10T07:50:44.652999+0000","last_became_peered":"2024-02-10T07:50:44.652999+0000","last_unstale":"2024-02-10T07:50:59.437533+0000","last_undegraded":"2024-02-10T07:50:59.437533+0000","last_fullsized":"2024-02-10T07:50:44.264285+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4982,"log_dups_size":0,"ondisk_log_size":4982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326500352,"num_objects":4982,"num_object_clones":0,"num_object_copies":14946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4030,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4982,"num_write_kb":318848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":20578304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,1],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1","objects":952},{"shards":"0,1,4","objects":4030}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":590,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-02-10T07:50:52.017274+0000","last_change":"2024-02-10T07:49:42.990229+0000","last_active":"2024-02-10T07:50:52.017274+0000","last_peered":"2024-02-10T07:50:52.017274+0000","last_clean":"2024-02-10T07:50:52.017274+0000","last_became_active":"2024-02-10T07:47:47.966183+0000","last_became_peered":"2024-02-10T07:47:47.966183+0000","last_unstale":"2024-02-10T07:50:52.017274+0000","last_undegraded":"2024-02-10T07:50:52.017274+0000","last_fullsized":"2024-02-10T07:50:52.017274+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:42.990098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:42.990098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.708032+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":766,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.031012+0000","last_change":"2024-02-10T07:48:56.416159+0000","last_active":"2024-02-10T07:50:57.031012+0000","last_peered":"2024-02-10T07:50:57.031012+0000","last_clean":"2024-02-10T07:50:57.031012+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:50:57.031012+0000","last_undegraded":"2024-02-10T07:50:57.031012+0000","last_fullsized":"2024-02-10T07:50:57.031012+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.416037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.416037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"250'23","reported_seq":872,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009470+0000","last_change":"2024-02-10T07:49:43.339027+0000","last_active":"2024-02-10T07:50:57.009470+0000","last_peered":"2024-02-10T07:50:57.009470+0000","last_clean":"2024-02-10T07:50:57.009470+0000","last_became_active":"2024-02-10T07:49:06.243569+0000","last_became_peered":"2024-02-10T07:49:06.243569+0000","last_unstale":"2024-02-10T07:50:57.009470+0000","last_undegraded":"2024-02-10T07:50:57.009470+0000","last_fullsized":"2024-02-10T07:50:57.009470+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-02-10T07:49:43.338928+0000","last_deep_scrub":"245'17","last_deep_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_clean_scrub_stamp":"2024-02-10T07:49:43.338928+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":19,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.040941+0000","last_change":"2024-02-10T07:50:43.285105+0000","last_active":"2024-02-10T07:50:57.040941+0000","last_peered":"2024-02-10T07:50:57.040941+0000","last_clean":"2024-02-10T07:50:57.040941+0000","last_became_active":"2024-02-10T07:50:43.284749+0000","last_became_peered":"2024-02-10T07:50:43.284749+0000","last_unstale":"2024-02-10T07:50:57.040941+0000","last_undegraded":"2024-02-10T07:50:57.040941+0000","last_fullsized":"2024-02-10T07:50:57.040941+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:04.308807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.308807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"409'5111","reported_seq":5157,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:58.344240+0000","last_change":"2024-02-10T07:49:53.374953+0000","last_active":"2024-02-10T07:50:58.344240+0000","last_peered":"2024-02-10T07:50:58.344240+0000","last_clean":"2024-02-10T07:50:58.344240+0000","last_became_active":"2024-02-10T07:49:53.374631+0000","last_became_peered":"2024-02-10T07:49:53.374631+0000","last_unstale":"2024-02-10T07:50:58.344240+0000","last_undegraded":"2024-02-10T07:50:58.344240+0000","last_fullsized":"2024-02-10T07:50:58.344240+0000","mapping_epoch":391,"log_start":"397'200","ondisk_log_start":"397'200","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4911,"log_dups_size":0,"ondisk_log_size":4911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:08.897366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334954496,"num_objects":5111,"num_object_clones":0,"num_object_copies":15333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5111,"num_write_kb":327104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":787,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030051+0000","last_change":"2024-02-10T07:50:43.281637+0000","last_active":"2024-02-10T07:50:57.030051+0000","last_peered":"2024-02-10T07:50:57.030051+0000","last_clean":"2024-02-10T07:50:57.030051+0000","last_became_active":"2024-02-10T07:50:43.281262+0000","last_became_peered":"2024-02-10T07:50:43.281262+0000","last_unstale":"2024-02-10T07:50:57.030051+0000","last_undegraded":"2024-02-10T07:50:57.030051+0000","last_fullsized":"2024-02-10T07:50:57.030051+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.361139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":19,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.041321+0000","last_change":"2024-02-10T07:50:43.281022+0000","last_active":"2024-02-10T07:50:57.041321+0000","last_peered":"2024-02-10T07:50:57.041321+0000","last_clean":"2024-02-10T07:50:57.041321+0000","last_became_active":"2024-02-10T07:50:43.280617+0000","last_became_peered":"2024-02-10T07:50:43.280617+0000","last_unstale":"2024-02-10T07:50:57.041321+0000","last_undegraded":"2024-02-10T07:50:57.041321+0000","last_fullsized":"2024-02-10T07:50:57.041321+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:21.172601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:21.172601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":783,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030845+0000","last_change":"2024-02-10T07:50:07.252577+0000","last_active":"2024-02-10T07:50:57.030845+0000","last_peered":"2024-02-10T07:50:57.030845+0000","last_clean":"2024-02-10T07:50:57.030845+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:50:57.030845+0000","last_undegraded":"2024-02-10T07:50:57.030845+0000","last_fullsized":"2024-02-10T07:50:57.030845+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.252458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.252458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:09.491987+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"250'35","reported_seq":832,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026248+0000","last_change":"2024-02-10T07:49:49.772965+0000","last_active":"2024-02-10T07:50:57.026248+0000","last_peered":"2024-02-10T07:50:57.026248+0000","last_clean":"2024-02-10T07:50:57.026248+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:50:57.026248+0000","last_undegraded":"2024-02-10T07:50:57.026248+0000","last_fullsized":"2024-02-10T07:50:57.026248+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'35","last_scrub_stamp":"2024-02-10T07:49:49.772852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:49.772852+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.199744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"409'5148","reported_seq":5193,"reported_epoch":409,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-02-10T07:50:58.340882+0000","last_change":"2024-02-10T07:50:44.679081+0000","last_active":"2024-02-10T07:50:58.340882+0000","last_peered":"2024-02-10T07:50:58.340882+0000","last_clean":"2024-02-10T07:50:42.227612+0000","last_became_active":"2024-02-10T07:50:44.671664+0000","last_became_peered":"2024-02-10T07:50:44.671664+0000","last_unstale":"2024-02-10T07:50:58.340882+0000","last_undegraded":"2024-02-10T07:50:44.289079+0000","last_fullsized":"2024-02-10T07:50:44.263721+0000","mapping_epoch":405,"log_start":"397'300","ondisk_log_start":"397'300","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4848,"log_dups_size":0,"ondisk_log_size":4848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337379328,"num_objects":5148,"num_object_clones":0,"num_object_copies":15444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5148,"num_write_kb":329472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6],"avail_no_missing":["7","1","6"],"object_location_counts":[{"shards":"1,6,7","objects":4497},{"shards":"6,7","objects":651}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":776,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.029414+0000","last_change":"2024-02-10T07:50:43.284817+0000","last_active":"2024-02-10T07:50:57.029414+0000","last_peered":"2024-02-10T07:50:57.029414+0000","last_clean":"2024-02-10T07:50:57.029414+0000","last_became_active":"2024-02-10T07:50:43.284289+0000","last_became_peered":"2024-02-10T07:50:43.284289+0000","last_unstale":"2024-02-10T07:50:57.029414+0000","last_undegraded":"2024-02-10T07:50:57.029414+0000","last_fullsized":"2024-02-10T07:50:57.029414+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:00.336175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.336175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":19,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.041320+0000","last_change":"2024-02-10T07:50:43.295279+0000","last_active":"2024-02-10T07:50:57.041320+0000","last_peered":"2024-02-10T07:50:57.041320+0000","last_clean":"2024-02-10T07:50:57.041320+0000","last_became_active":"2024-02-10T07:50:43.294972+0000","last_became_peered":"2024-02-10T07:50:43.294972+0000","last_unstale":"2024-02-10T07:50:57.041320+0000","last_undegraded":"2024-02-10T07:50:57.041320+0000","last_fullsized":"2024-02-10T07:50:57.041320+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:28.822700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:28.822700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":83,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-02-10T07:50:52.017164+0000","last_change":"2024-02-10T07:49:52.968676+0000","last_active":"2024-02-10T07:50:52.017164+0000","last_peered":"2024-02-10T07:50:52.017164+0000","last_clean":"2024-02-10T07:50:52.017164+0000","last_became_active":"2024-02-10T07:49:06.249431+0000","last_became_peered":"2024-02-10T07:49:06.249431+0000","last_unstale":"2024-02-10T07:50:52.017164+0000","last_undegraded":"2024-02-10T07:50:52.017164+0000","last_fullsized":"2024-02-10T07:50:52.017164+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.968567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.968567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:06.997404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"250'15","reported_seq":787,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030468+0000","last_change":"2024-02-10T07:48:59.304277+0000","last_active":"2024-02-10T07:50:57.030468+0000","last_peered":"2024-02-10T07:50:57.030468+0000","last_clean":"2024-02-10T07:50:57.030468+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:50:57.030468+0000","last_undegraded":"2024-02-10T07:50:57.030468+0000","last_fullsized":"2024-02-10T07:50:57.030468+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:48:59.304199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.304199+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"408'4898","reported_seq":4942,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-02-10T07:50:56.334523+0000","last_change":"2024-02-10T07:49:53.284745+0000","last_active":"2024-02-10T07:50:56.334523+0000","last_peered":"2024-02-10T07:50:56.334523+0000","last_clean":"2024-02-10T07:50:56.334523+0000","last_became_active":"2024-02-10T07:49:53.283642+0000","last_became_peered":"2024-02-10T07:49:53.283642+0000","last_unstale":"2024-02-10T07:50:56.334523+0000","last_undegraded":"2024-02-10T07:50:56.334523+0000","last_fullsized":"2024-02-10T07:50:56.334523+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4898,"log_dups_size":0,"ondisk_log_size":4898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:18.016780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320995328,"num_objects":4898,"num_object_clones":0,"num_object_copies":14694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4898,"num_write_kb":313472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":83,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-02-10T07:50:52.016956+0000","last_change":"2024-02-10T07:49:50.032577+0000","last_active":"2024-02-10T07:50:52.016956+0000","last_peered":"2024-02-10T07:50:52.016956+0000","last_clean":"2024-02-10T07:50:52.016956+0000","last_became_active":"2024-02-10T07:49:06.238367+0000","last_became_peered":"2024-02-10T07:49:06.238367+0000","last_unstale":"2024-02-10T07:50:52.016956+0000","last_undegraded":"2024-02-10T07:50:52.016956+0000","last_fullsized":"2024-02-10T07:50:52.016956+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:50.032505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.032505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:15.102164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":186,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.024229+0000","last_change":"2024-02-10T07:49:10.087939+0000","last_active":"2024-02-10T07:50:57.024229+0000","last_peered":"2024-02-10T07:50:57.024229+0000","last_clean":"2024-02-10T07:50:57.024229+0000","last_became_active":"2024-02-10T07:48:02.328316+0000","last_became_peered":"2024-02-10T07:48:02.328316+0000","last_unstale":"2024-02-10T07:50:57.024229+0000","last_undegraded":"2024-02-10T07:50:57.024229+0000","last_fullsized":"2024-02-10T07:50:57.024229+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:49:10.087795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:49:10.087795+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":759,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009327+0000","last_change":"2024-02-10T07:49:19.198419+0000","last_active":"2024-02-10T07:50:57.009327+0000","last_peered":"2024-02-10T07:50:57.009327+0000","last_clean":"2024-02-10T07:50:57.009327+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:50:57.009327+0000","last_undegraded":"2024-02-10T07:50:57.009327+0000","last_fullsized":"2024-02-10T07:50:57.009327+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:19.198312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:04.220328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.198312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"405'47","reported_seq":1175,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.029282+0000","last_change":"2024-02-10T07:50:43.327112+0000","last_active":"2024-02-10T07:50:57.029282+0000","last_peered":"2024-02-10T07:50:57.029282+0000","last_clean":"2024-02-10T07:50:57.029282+0000","last_became_active":"2024-02-10T07:50:43.297262+0000","last_became_peered":"2024-02-10T07:50:43.297262+0000","last_unstale":"2024-02-10T07:50:57.029282+0000","last_undegraded":"2024-02-10T07:50:57.029282+0000","last_fullsized":"2024-02-10T07:50:57.029282+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'46","last_scrub_stamp":"2024-02-10T07:49:55.714323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:55.714323+0000","objects_scrubbed":10,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:10.202044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":767,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030782+0000","last_change":"2024-02-10T07:49:17.357664+0000","last_active":"2024-02-10T07:50:57.030782+0000","last_peered":"2024-02-10T07:50:57.030782+0000","last_clean":"2024-02-10T07:50:57.030782+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:50:57.030782+0000","last_undegraded":"2024-02-10T07:50:57.030782+0000","last_fullsized":"2024-02-10T07:50:57.030782+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.357533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:22.786949+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.357533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"409'5100","reported_seq":5146,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:58.455957+0000","last_change":"2024-02-10T07:49:53.374436+0000","last_active":"2024-02-10T07:50:58.455957+0000","last_peered":"2024-02-10T07:50:58.455957+0000","last_clean":"2024-02-10T07:50:58.455957+0000","last_became_active":"2024-02-10T07:49:53.374133+0000","last_became_peered":"2024-02-10T07:49:53.374133+0000","last_unstale":"2024-02-10T07:50:58.455957+0000","last_undegraded":"2024-02-10T07:50:58.455957+0000","last_fullsized":"2024-02-10T07:50:58.455957+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":5100,"log_dups_size":0,"ondisk_log_size":5100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.202856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334233600,"num_objects":5100,"num_object_clones":0,"num_object_copies":15300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5100,"num_write_kb":326400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":756,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030994+0000","last_change":"2024-02-10T07:47:50.081222+0000","last_active":"2024-02-10T07:50:57.030994+0000","last_peered":"2024-02-10T07:50:57.030994+0000","last_clean":"2024-02-10T07:50:57.030994+0000","last_became_active":"2024-02-10T07:47:48.149256+0000","last_became_peered":"2024-02-10T07:47:48.149256+0000","last_unstale":"2024-02-10T07:50:57.030994+0000","last_undegraded":"2024-02-10T07:50:57.030994+0000","last_fullsized":"2024-02-10T07:50:57.030994+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.080714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:42.730875+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.080714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":785,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026093+0000","last_change":"2024-02-10T07:50:43.288780+0000","last_active":"2024-02-10T07:50:57.026093+0000","last_peered":"2024-02-10T07:50:57.026093+0000","last_clean":"2024-02-10T07:50:57.026093+0000","last_became_active":"2024-02-10T07:50:43.288388+0000","last_became_peered":"2024-02-10T07:50:43.288388+0000","last_unstale":"2024-02-10T07:50:57.026093+0000","last_undegraded":"2024-02-10T07:50:57.026093+0000","last_fullsized":"2024-02-10T07:50:57.026093+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:49.886354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:49.886354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:08.705338+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.041244+0000","last_change":"2024-02-10T07:50:43.287356+0000","last_active":"2024-02-10T07:50:57.041244+0000","last_peered":"2024-02-10T07:50:57.041244+0000","last_clean":"2024-02-10T07:50:57.041244+0000","last_became_active":"2024-02-10T07:50:43.287043+0000","last_became_peered":"2024-02-10T07:50:43.287043+0000","last_unstale":"2024-02-10T07:50:57.041244+0000","last_undegraded":"2024-02-10T07:50:57.041244+0000","last_fullsized":"2024-02-10T07:50:57.041244+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:15.234112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.234112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"250'17","reported_seq":120,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030948+0000","last_change":"2024-02-10T07:49:04.340445+0000","last_active":"2024-02-10T07:50:57.030948+0000","last_peered":"2024-02-10T07:50:57.030948+0000","last_clean":"2024-02-10T07:50:57.030948+0000","last_became_active":"2024-02-10T07:48:35.057662+0000","last_became_peered":"2024-02-10T07:48:35.057662+0000","last_unstale":"2024-02-10T07:50:57.030948+0000","last_undegraded":"2024-02-10T07:50:57.030948+0000","last_fullsized":"2024-02-10T07:50:57.030948+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'17","last_scrub_stamp":"2024-02-10T07:49:04.340366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.340366+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":23,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":19,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.041725+0000","last_change":"2024-02-10T07:50:43.280896+0000","last_active":"2024-02-10T07:50:57.041725+0000","last_peered":"2024-02-10T07:50:57.041725+0000","last_clean":"2024-02-10T07:50:57.041725+0000","last_became_active":"2024-02-10T07:50:43.279125+0000","last_became_peered":"2024-02-10T07:50:43.279125+0000","last_unstale":"2024-02-10T07:50:57.041725+0000","last_undegraded":"2024-02-10T07:50:57.041725+0000","last_fullsized":"2024-02-10T07:50:57.041725+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:50.232544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.232544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"409'4887","reported_seq":4933,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:58.425959+0000","last_change":"2024-02-10T07:49:53.374470+0000","last_active":"2024-02-10T07:50:58.425959+0000","last_peered":"2024-02-10T07:50:58.425959+0000","last_clean":"2024-02-10T07:50:58.425959+0000","last_became_active":"2024-02-10T07:49:53.374185+0000","last_became_peered":"2024-02-10T07:49:53.374185+0000","last_unstale":"2024-02-10T07:50:58.425959+0000","last_undegraded":"2024-02-10T07:50:58.425959+0000","last_fullsized":"2024-02-10T07:50:58.425959+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320274432,"num_objects":4887,"num_object_clones":0,"num_object_copies":14661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4887,"num_write_kb":312768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":175,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009840+0000","last_change":"2024-02-10T07:48:02.330253+0000","last_active":"2024-02-10T07:50:57.009840+0000","last_peered":"2024-02-10T07:50:57.009840+0000","last_clean":"2024-02-10T07:50:57.009840+0000","last_became_active":"2024-02-10T07:48:02.329687+0000","last_became_peered":"2024-02-10T07:48:02.329687+0000","last_unstale":"2024-02-10T07:50:57.009840+0000","last_undegraded":"2024-02-10T07:50:57.009840+0000","last_fullsized":"2024-02-10T07:50:57.009840+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:47:49.750508+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.750508+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1012,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030224+0000","last_change":"2024-02-10T07:48:15.033077+0000","last_active":"2024-02-10T07:50:57.030224+0000","last_peered":"2024-02-10T07:50:57.030224+0000","last_clean":"2024-02-10T07:50:57.030224+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:50:57.030224+0000","last_undegraded":"2024-02-10T07:50:57.030224+0000","last_fullsized":"2024-02-10T07:50:57.030224+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:48:15.032944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.032944+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":404,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:58.025267+0000","last_change":"2024-02-10T07:50:58.025300+0000","last_active":"2024-02-10T07:50:58.025267+0000","last_peered":"2024-02-10T07:50:58.025267+0000","last_clean":"2024-02-10T07:50:58.025267+0000","last_became_active":"2024-02-10T07:49:06.256223+0000","last_became_peered":"2024-02-10T07:49:06.256223+0000","last_unstale":"2024-02-10T07:50:58.025267+0000","last_undegraded":"2024-02-10T07:50:58.025267+0000","last_fullsized":"2024-02-10T07:50:58.025267+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:58.025193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:58.025193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:52:20.482748+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"250'36","reported_seq":1056,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026679+0000","last_change":"2024-02-10T07:49:58.841289+0000","last_active":"2024-02-10T07:50:57.026679+0000","last_peered":"2024-02-10T07:50:57.026679+0000","last_clean":"2024-02-10T07:50:57.026679+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:50:57.026679+0000","last_undegraded":"2024-02-10T07:50:57.026679+0000","last_fullsized":"2024-02-10T07:50:57.026679+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'36","last_scrub_stamp":"2024-02-10T07:49:58.841172+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:49:58.841172+0000","objects_scrubbed":8,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:07.505517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":788,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.031638+0000","last_change":"2024-02-10T07:50:43.285613+0000","last_active":"2024-02-10T07:50:57.031638+0000","last_peered":"2024-02-10T07:50:57.031638+0000","last_clean":"2024-02-10T07:50:57.031638+0000","last_became_active":"2024-02-10T07:50:43.284708+0000","last_became_peered":"2024-02-10T07:50:43.284708+0000","last_unstale":"2024-02-10T07:50:57.031638+0000","last_undegraded":"2024-02-10T07:50:57.031638+0000","last_fullsized":"2024-02-10T07:50:57.031638+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:13.369607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.736901+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.369607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"409'4941","reported_seq":4987,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:58.438221+0000","last_change":"2024-02-10T07:49:53.283728+0000","last_active":"2024-02-10T07:50:58.438221+0000","last_peered":"2024-02-10T07:50:58.438221+0000","last_clean":"2024-02-10T07:50:58.438221+0000","last_became_active":"2024-02-10T07:49:53.283492+0000","last_became_peered":"2024-02-10T07:49:53.283492+0000","last_unstale":"2024-02-10T07:50:58.438221+0000","last_undegraded":"2024-02-10T07:50:58.438221+0000","last_fullsized":"2024-02-10T07:50:58.438221+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:02.523341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323813376,"num_objects":4941,"num_object_clones":0,"num_object_copies":14823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4941,"num_write_kb":316224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":559,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.023714+0000","last_change":"2024-02-10T07:50:07.589020+0000","last_active":"2024-02-10T07:50:57.023714+0000","last_peered":"2024-02-10T07:50:57.023714+0000","last_clean":"2024-02-10T07:50:57.023714+0000","last_became_active":"2024-02-10T07:49:06.270457+0000","last_became_peered":"2024-02-10T07:49:06.270457+0000","last_unstale":"2024-02-10T07:50:57.023714+0000","last_undegraded":"2024-02-10T07:50:57.023714+0000","last_fullsized":"2024-02-10T07:50:57.023714+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.588888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.588888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:26.055454+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":774,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.029086+0000","last_change":"2024-02-10T07:49:06.267039+0000","last_active":"2024-02-10T07:50:57.029086+0000","last_peered":"2024-02-10T07:50:57.029086+0000","last_clean":"2024-02-10T07:50:57.029086+0000","last_became_active":"2024-02-10T07:49:06.266625+0000","last_became_peered":"2024-02-10T07:49:06.266625+0000","last_unstale":"2024-02-10T07:50:57.029086+0000","last_undegraded":"2024-02-10T07:50:57.029086+0000","last_fullsized":"2024-02-10T07:50:57.029086+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_clean_scrub_stamp":"2024-02-10T07:47:41.181770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1927,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.104150+0000","last_change":"2024-02-10T07:50:04.159154+0000","last_active":"2024-02-10T07:50:57.104150+0000","last_peered":"2024-02-10T07:50:57.104150+0000","last_clean":"2024-02-10T07:50:57.104150+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:50:57.104150+0000","last_undegraded":"2024-02-10T07:50:57.104150+0000","last_fullsized":"2024-02-10T07:50:57.104150+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:50:04.159046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.159046+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:14.041650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1140,"num_read_kb":1135,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"250'30","reported_seq":844,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.048345+0000","last_change":"2024-02-10T07:50:43.324808+0000","last_active":"2024-02-10T07:50:57.048345+0000","last_peered":"2024-02-10T07:50:57.048345+0000","last_clean":"2024-02-10T07:50:57.048345+0000","last_became_active":"2024-02-10T07:50:43.291771+0000","last_became_peered":"2024-02-10T07:50:43.291771+0000","last_unstale":"2024-02-10T07:50:57.048345+0000","last_undegraded":"2024-02-10T07:50:57.048345+0000","last_fullsized":"2024-02-10T07:50:57.048345+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:47.417096+0000","last_deep_scrub":"237'18","last_deep_scrub_stamp":"2024-02-10T07:40:31.005376+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.417096+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:05.958671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":713,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030136+0000","last_change":"2024-02-10T07:49:52.615217+0000","last_active":"2024-02-10T07:50:57.030136+0000","last_peered":"2024-02-10T07:50:57.030136+0000","last_clean":"2024-02-10T07:50:57.030136+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:50:57.030136+0000","last_undegraded":"2024-02-10T07:50:57.030136+0000","last_fullsized":"2024-02-10T07:50:57.030136+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.615130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.615130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.239753+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"409'4956","reported_seq":5525,"reported_epoch":409,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-10T07:50:59.210982+0000","last_change":"2024-02-10T07:50:44.712025+0000","last_active":"2024-02-10T07:50:59.210982+0000","last_peered":"2024-02-10T07:50:59.210982+0000","last_clean":"2024-02-10T07:50:42.244817+0000","last_became_active":"2024-02-10T07:50:44.652212+0000","last_became_peered":"2024-02-10T07:50:44.652212+0000","last_unstale":"2024-02-10T07:50:59.210982+0000","last_undegraded":"2024-02-10T07:50:59.210982+0000","last_fullsized":"2024-02-10T07:50:44.264876+0000","mapping_epoch":405,"log_start":"396'100","ondisk_log_start":"396'100","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4856,"log_dups_size":0,"ondisk_log_size":4856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324796416,"num_objects":4956,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4065,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4956,"num_write_kb":317184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":17039360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,4],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4","objects":891},{"shards":"1,4,7","objects":4065}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":781,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.023890+0000","last_change":"2024-02-10T07:50:43.296823+0000","last_active":"2024-02-10T07:50:57.023890+0000","last_peered":"2024-02-10T07:50:57.023890+0000","last_clean":"2024-02-10T07:50:57.023890+0000","last_became_active":"2024-02-10T07:50:43.296488+0000","last_became_peered":"2024-02-10T07:50:43.296488+0000","last_unstale":"2024-02-10T07:50:57.023890+0000","last_undegraded":"2024-02-10T07:50:57.023890+0000","last_fullsized":"2024-02-10T07:50:57.023890+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.630083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.630083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:28.581317+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":786,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.029184+0000","last_change":"2024-02-10T07:50:43.298468+0000","last_active":"2024-02-10T07:50:57.029184+0000","last_peered":"2024-02-10T07:50:57.029184+0000","last_clean":"2024-02-10T07:50:57.029184+0000","last_became_active":"2024-02-10T07:50:43.297861+0000","last_became_peered":"2024-02-10T07:50:43.297861+0000","last_unstale":"2024-02-10T07:50:57.029184+0000","last_undegraded":"2024-02-10T07:50:57.029184+0000","last_fullsized":"2024-02-10T07:50:57.029184+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:05.193158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:49:05.193158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"271'37","reported_seq":831,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030218+0000","last_change":"2024-02-10T07:49:57.522503+0000","last_active":"2024-02-10T07:50:57.030218+0000","last_peered":"2024-02-10T07:50:57.030218+0000","last_clean":"2024-02-10T07:50:57.030218+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:50:57.030218+0000","last_undegraded":"2024-02-10T07:50:57.030218+0000","last_fullsized":"2024-02-10T07:50:57.030218+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'37","last_scrub_stamp":"2024-02-10T07:49:57.522391+0000","last_deep_scrub":"271'37","last_deep_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.522391+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:18.289907+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":789,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026226+0000","last_change":"2024-02-10T07:49:03.013730+0000","last_active":"2024-02-10T07:50:57.026226+0000","last_peered":"2024-02-10T07:50:57.026226+0000","last_clean":"2024-02-10T07:50:57.026226+0000","last_became_active":"2024-02-10T07:48:35.062118+0000","last_became_peered":"2024-02-10T07:48:35.062118+0000","last_unstale":"2024-02-10T07:50:57.026226+0000","last_undegraded":"2024-02-10T07:50:57.026226+0000","last_fullsized":"2024-02-10T07:50:57.026226+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:03.013625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:00.292320+0000","last_clean_scrub_stamp":"2024-02-10T07:49:03.013625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"409'5061","reported_seq":5110,"reported_epoch":409,"state":"active+recovery_wait+remapped","last_fresh":"2024-02-10T07:50:59.244656+0000","last_change":"2024-02-10T07:50:44.689634+0000","last_active":"2024-02-10T07:50:59.244656+0000","last_peered":"2024-02-10T07:50:59.244656+0000","last_clean":"2024-02-10T07:50:42.239377+0000","last_became_active":"2024-02-10T07:50:44.668375+0000","last_became_peered":"2024-02-10T07:50:44.668375+0000","last_unstale":"2024-02-10T07:50:59.244656+0000","last_undegraded":"2024-02-10T07:50:59.244656+0000","last_fullsized":"2024-02-10T07:50:59.244656+0000","mapping_epoch":405,"log_start":"397'200","ondisk_log_start":"397'200","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4861,"log_dups_size":0,"ondisk_log_size":4861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331677696,"num_objects":5061,"num_object_clones":0,"num_object_copies":15183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4420,"num_objects_unfound":0,"num_objects_dirty":5061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5061,"num_write_kb":323904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":641}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":649,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.025936+0000","last_change":"2024-02-10T07:49:57.848098+0000","last_active":"2024-02-10T07:50:57.025936+0000","last_peered":"2024-02-10T07:50:57.025936+0000","last_clean":"2024-02-10T07:50:57.025936+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:50:57.025936+0000","last_undegraded":"2024-02-10T07:50:57.025936+0000","last_fullsized":"2024-02-10T07:50:57.025936+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:57.847989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.847989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:26.850826+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.041280+0000","last_change":"2024-02-10T07:50:43.289736+0000","last_active":"2024-02-10T07:50:57.041280+0000","last_peered":"2024-02-10T07:50:57.041280+0000","last_clean":"2024-02-10T07:50:57.041280+0000","last_became_active":"2024-02-10T07:50:43.289458+0000","last_became_peered":"2024-02-10T07:50:43.289458+0000","last_unstale":"2024-02-10T07:50:57.041280+0000","last_undegraded":"2024-02-10T07:50:57.041280+0000","last_fullsized":"2024-02-10T07:50:57.041280+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:48.809513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:48.809513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"382'25","reported_seq":576,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.024514+0000","last_change":"2024-02-10T07:49:19.084010+0000","last_active":"2024-02-10T07:50:57.024514+0000","last_peered":"2024-02-10T07:50:57.024514+0000","last_clean":"2024-02-10T07:50:57.024514+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:50:57.024514+0000","last_undegraded":"2024-02-10T07:50:57.024514+0000","last_fullsized":"2024-02-10T07:50:57.024514+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'25","last_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_deep_scrub":"382'25","last_deep_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.083856+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"158'10","reported_seq":1007,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030556+0000","last_change":"2024-02-10T07:49:50.396822+0000","last_active":"2024-02-10T07:50:57.030556+0000","last_peered":"2024-02-10T07:50:57.030556+0000","last_clean":"2024-02-10T07:50:57.030556+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:50:57.030556+0000","last_undegraded":"2024-02-10T07:50:57.030556+0000","last_fullsized":"2024-02-10T07:50:57.030556+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:49:50.396730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.396730+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.e","version":"409'5017","reported_seq":5063,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:58.465316+0000","last_change":"2024-02-10T07:49:53.281229+0000","last_active":"2024-02-10T07:50:58.465316+0000","last_peered":"2024-02-10T07:50:58.465316+0000","last_clean":"2024-02-10T07:50:58.465316+0000","last_became_active":"2024-02-10T07:49:53.280920+0000","last_became_peered":"2024-02-10T07:49:53.280920+0000","last_unstale":"2024-02-10T07:50:58.465316+0000","last_undegraded":"2024-02-10T07:50:58.465316+0000","last_fullsized":"2024-02-10T07:50:58.465316+0000","mapping_epoch":391,"log_start":"397'200","ondisk_log_start":"397'200","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4817,"log_dups_size":0,"ondisk_log_size":4817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:05.553881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328794112,"num_objects":5017,"num_object_clones":0,"num_object_copies":15051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5017,"num_write_kb":321088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":767,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.032136+0000","last_change":"2024-02-10T07:50:43.283505+0000","last_active":"2024-02-10T07:50:57.032136+0000","last_peered":"2024-02-10T07:50:57.032136+0000","last_clean":"2024-02-10T07:50:57.032136+0000","last_became_active":"2024-02-10T07:50:43.283161+0000","last_became_peered":"2024-02-10T07:50:43.283161+0000","last_unstale":"2024-02-10T07:50:57.032136+0000","last_undegraded":"2024-02-10T07:50:57.032136+0000","last_fullsized":"2024-02-10T07:50:57.032136+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_clean_scrub_stamp":"2024-02-10T07:48:23.271454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"296'2","reported_seq":776,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.048447+0000","last_change":"2024-02-10T07:50:03.643473+0000","last_active":"2024-02-10T07:50:57.048447+0000","last_peered":"2024-02-10T07:50:57.048447+0000","last_clean":"2024-02-10T07:50:57.048447+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:50:57.048447+0000","last_undegraded":"2024-02-10T07:50:57.048447+0000","last_fullsized":"2024-02-10T07:50:57.048447+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'2","last_scrub_stamp":"2024-02-10T07:50:03.643367+0000","last_deep_scrub":"296'2","last_deep_scrub_stamp":"2024-02-10T07:45:03.233375+0000","last_clean_scrub_stamp":"2024-02-10T07:50:03.643367+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:15.296940+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"250'28","reported_seq":806,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009354+0000","last_change":"2024-02-10T07:49:20.216865+0000","last_active":"2024-02-10T07:50:57.009354+0000","last_peered":"2024-02-10T07:50:57.009354+0000","last_clean":"2024-02-10T07:50:57.009354+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:50:57.009354+0000","last_undegraded":"2024-02-10T07:50:57.009354+0000","last_fullsized":"2024-02-10T07:50:57.009354+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-02-10T07:49:20.216750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:20.216750+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"376'12","reported_seq":793,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.025849+0000","last_change":"2024-02-10T07:49:06.282456+0000","last_active":"2024-02-10T07:50:57.025849+0000","last_peered":"2024-02-10T07:50:57.025849+0000","last_clean":"2024-02-10T07:50:57.025849+0000","last_became_active":"2024-02-10T07:49:06.255324+0000","last_became_peered":"2024-02-10T07:49:06.255324+0000","last_unstale":"2024-02-10T07:50:57.025849+0000","last_undegraded":"2024-02-10T07:50:57.025849+0000","last_fullsized":"2024-02-10T07:50:57.025849+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'11","last_scrub_stamp":"2024-02-10T07:48:58.368663+0000","last_deep_scrub":"227'9","last_deep_scrub_stamp":"2024-02-10T07:39:53.440221+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.368663+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"409'4982","reported_seq":5031,"reported_epoch":409,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-02-10T07:50:58.355511+0000","last_change":"2024-02-10T07:50:44.676964+0000","last_active":"2024-02-10T07:50:58.355511+0000","last_peered":"2024-02-10T07:50:58.355511+0000","last_clean":"2024-02-10T07:50:42.258059+0000","last_became_active":"2024-02-10T07:50:44.666535+0000","last_became_peered":"2024-02-10T07:50:44.666535+0000","last_unstale":"2024-02-10T07:50:58.355511+0000","last_undegraded":"2024-02-10T07:50:44.288061+0000","last_fullsized":"2024-02-10T07:50:44.263325+0000","mapping_epoch":405,"log_start":"396'100","ondisk_log_start":"396'100","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326500352,"num_objects":4982,"num_object_clones":0,"num_object_copies":14946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4364,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4982,"num_write_kb":318848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"0,1,7","objects":4364},{"shards":"1,7","objects":618}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":764,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009238+0000","last_change":"2024-02-10T07:50:43.282245+0000","last_active":"2024-02-10T07:50:57.009238+0000","last_peered":"2024-02-10T07:50:57.009238+0000","last_clean":"2024-02-10T07:50:57.009238+0000","last_became_active":"2024-02-10T07:50:43.281465+0000","last_became_peered":"2024-02-10T07:50:43.281465+0000","last_unstale":"2024-02-10T07:50:57.009238+0000","last_undegraded":"2024-02-10T07:50:57.009238+0000","last_fullsized":"2024-02-10T07:50:57.009238+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:27.819612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:46:27.819612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":248,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.031129+0000","last_change":"2024-02-10T07:47:50.948710+0000","last_active":"2024-02-10T07:50:57.031129+0000","last_peered":"2024-02-10T07:50:57.031129+0000","last_clean":"2024-02-10T07:50:57.031129+0000","last_became_active":"2024-02-10T07:47:47.966443+0000","last_became_peered":"2024-02-10T07:47:47.966443+0000","last_unstale":"2024-02-10T07:50:57.031129+0000","last_undegraded":"2024-02-10T07:50:57.031129+0000","last_fullsized":"2024-02-10T07:50:57.031129+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.948624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.948624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"250'30","reported_seq":747,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030364+0000","last_change":"2024-02-10T07:49:57.303310+0000","last_active":"2024-02-10T07:50:57.030364+0000","last_peered":"2024-02-10T07:50:57.030364+0000","last_clean":"2024-02-10T07:50:57.030364+0000","last_became_active":"2024-02-10T07:49:06.252935+0000","last_became_peered":"2024-02-10T07:49:06.252935+0000","last_unstale":"2024-02-10T07:50:57.030364+0000","last_undegraded":"2024-02-10T07:50:57.030364+0000","last_fullsized":"2024-02-10T07:50:57.030364+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:57.303200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.303200+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:15.856547+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":766,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009315+0000","last_change":"2024-02-10T07:49:06.239655+0000","last_active":"2024-02-10T07:50:57.009315+0000","last_peered":"2024-02-10T07:50:57.009315+0000","last_clean":"2024-02-10T07:50:57.009315+0000","last_became_active":"2024-02-10T07:49:06.239190+0000","last_became_peered":"2024-02-10T07:49:06.239190+0000","last_unstale":"2024-02-10T07:50:57.009315+0000","last_undegraded":"2024-02-10T07:50:57.009315+0000","last_fullsized":"2024-02-10T07:50:57.009315+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:42.549018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.549018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"409'5042","reported_seq":5088,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:59.250339+0000","last_change":"2024-02-10T07:49:53.282174+0000","last_active":"2024-02-10T07:50:59.250339+0000","last_peered":"2024-02-10T07:50:59.250339+0000","last_clean":"2024-02-10T07:50:59.250339+0000","last_became_active":"2024-02-10T07:49:53.281832+0000","last_became_peered":"2024-02-10T07:49:53.281832+0000","last_unstale":"2024-02-10T07:50:59.250339+0000","last_undegraded":"2024-02-10T07:50:59.250339+0000","last_fullsized":"2024-02-10T07:50:59.250339+0000","mapping_epoch":391,"log_start":"397'200","ondisk_log_start":"397'200","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4842,"log_dups_size":0,"ondisk_log_size":4842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:08.492345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330432512,"num_objects":5042,"num_object_clones":0,"num_object_copies":15126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5042,"num_write_kb":322688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":273,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.031028+0000","last_change":"2024-02-10T07:49:06.274374+0000","last_active":"2024-02-10T07:50:57.031028+0000","last_peered":"2024-02-10T07:50:57.031028+0000","last_clean":"2024-02-10T07:50:57.031028+0000","last_became_active":"2024-02-10T07:48:02.371926+0000","last_became_peered":"2024-02-10T07:48:02.371926+0000","last_unstale":"2024-02-10T07:50:57.031028+0000","last_undegraded":"2024-02-10T07:50:57.031028+0000","last_fullsized":"2024-02-10T07:50:57.031028+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.274263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:49.704024+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.274263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"395'12","reported_seq":142,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026522+0000","last_change":"2024-02-10T07:50:02.810307+0000","last_active":"2024-02-10T07:50:57.026522+0000","last_peered":"2024-02-10T07:50:57.026522+0000","last_clean":"2024-02-10T07:50:57.026522+0000","last_became_active":"2024-02-10T07:48:35.071916+0000","last_became_peered":"2024-02-10T07:48:35.071916+0000","last_unstale":"2024-02-10T07:50:57.026522+0000","last_undegraded":"2024-02-10T07:50:57.026522+0000","last_fullsized":"2024-02-10T07:50:57.026522+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'12","last_scrub_stamp":"2024-02-10T07:50:02.810229+0000","last_deep_scrub":"395'12","last_deep_scrub_stamp":"2024-02-10T07:50:02.810229+0000","last_clean_scrub_stamp":"2024-02-10T07:50:02.810229+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:28.369227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":83,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-02-10T07:50:52.018313+0000","last_change":"2024-02-10T07:49:44.962126+0000","last_active":"2024-02-10T07:50:52.018313+0000","last_peered":"2024-02-10T07:50:52.018313+0000","last_clean":"2024-02-10T07:50:52.018313+0000","last_became_active":"2024-02-10T07:49:06.251038+0000","last_became_peered":"2024-02-10T07:49:06.251038+0000","last_unstale":"2024-02-10T07:50:52.018313+0000","last_undegraded":"2024-02-10T07:50:52.018313+0000","last_fullsized":"2024-02-10T07:50:52.018313+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:44.962041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:44.962041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:12.429908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"250'27","reported_seq":820,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.032968+0000","last_change":"2024-02-10T07:50:00.655466+0000","last_active":"2024-02-10T07:50:57.032968+0000","last_peered":"2024-02-10T07:50:57.032968+0000","last_clean":"2024-02-10T07:50:57.032968+0000","last_became_active":"2024-02-10T07:47:47.967427+0000","last_became_peered":"2024-02-10T07:47:47.967427+0000","last_unstale":"2024-02-10T07:50:57.032968+0000","last_undegraded":"2024-02-10T07:50:57.032968+0000","last_fullsized":"2024-02-10T07:50:57.032968+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:50:00.655377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.655377+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:26.934415+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"409'5130","reported_seq":5176,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:58.457181+0000","last_change":"2024-02-10T07:49:53.283708+0000","last_active":"2024-02-10T07:50:58.457181+0000","last_peered":"2024-02-10T07:50:58.457181+0000","last_clean":"2024-02-10T07:50:58.457181+0000","last_became_active":"2024-02-10T07:49:53.283415+0000","last_became_peered":"2024-02-10T07:49:53.283415+0000","last_unstale":"2024-02-10T07:50:58.457181+0000","last_undegraded":"2024-02-10T07:50:58.457181+0000","last_fullsized":"2024-02-10T07:50:58.457181+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":5130,"log_dups_size":0,"ondisk_log_size":5130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:00.247094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336199680,"num_objects":5130,"num_object_clones":0,"num_object_copies":15390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5130,"num_write_kb":328320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":459,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026584+0000","last_change":"2024-02-10T07:50:00.799049+0000","last_active":"2024-02-10T07:50:57.026584+0000","last_peered":"2024-02-10T07:50:57.026584+0000","last_clean":"2024-02-10T07:50:57.026584+0000","last_became_active":"2024-02-10T07:49:06.268874+0000","last_became_peered":"2024-02-10T07:49:06.268874+0000","last_unstale":"2024-02-10T07:50:57.026584+0000","last_undegraded":"2024-02-10T07:50:57.026584+0000","last_fullsized":"2024-02-10T07:50:57.026584+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.798928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.798928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:06.248341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":247,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.010064+0000","last_change":"2024-02-10T07:49:57.310447+0000","last_active":"2024-02-10T07:50:57.010064+0000","last_peered":"2024-02-10T07:50:57.010064+0000","last_clean":"2024-02-10T07:50:57.010064+0000","last_became_active":"2024-02-10T07:48:02.333689+0000","last_became_peered":"2024-02-10T07:48:02.333689+0000","last_unstale":"2024-02-10T07:50:57.010064+0000","last_undegraded":"2024-02-10T07:50:57.010064+0000","last_fullsized":"2024-02-10T07:50:57.010064+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:57.310333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.310333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:22.249646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":770,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009204+0000","last_change":"2024-02-10T07:50:43.283476+0000","last_active":"2024-02-10T07:50:57.009204+0000","last_peered":"2024-02-10T07:50:57.009204+0000","last_clean":"2024-02-10T07:50:57.009204+0000","last_became_active":"2024-02-10T07:50:43.283104+0000","last_became_peered":"2024-02-10T07:50:43.283104+0000","last_unstale":"2024-02-10T07:50:57.009204+0000","last_undegraded":"2024-02-10T07:50:57.009204+0000","last_fullsized":"2024-02-10T07:50:57.009204+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:37.995362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.995362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"250'24","reported_seq":844,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.010326+0000","last_change":"2024-02-10T07:49:16.249257+0000","last_active":"2024-02-10T07:50:57.010326+0000","last_peered":"2024-02-10T07:50:57.010326+0000","last_clean":"2024-02-10T07:50:57.010326+0000","last_became_active":"2024-02-10T07:48:35.062573+0000","last_became_peered":"2024-02-10T07:48:35.062573+0000","last_unstale":"2024-02-10T07:50:57.010326+0000","last_undegraded":"2024-02-10T07:50:57.010326+0000","last_fullsized":"2024-02-10T07:50:57.010326+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:49:16.248936+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.248936+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"409'5016","reported_seq":5681,"reported_epoch":409,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-10T07:50:59.425683+0000","last_change":"2024-02-10T07:50:44.730524+0000","last_active":"2024-02-10T07:50:59.425683+0000","last_peered":"2024-02-10T07:50:59.425683+0000","last_clean":"2024-02-10T07:50:42.258341+0000","last_became_active":"2024-02-10T07:50:44.653649+0000","last_became_peered":"2024-02-10T07:50:44.653649+0000","last_unstale":"2024-02-10T07:50:59.425683+0000","last_undegraded":"2024-02-10T07:50:59.425683+0000","last_fullsized":"2024-02-10T07:50:44.263511+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328728576,"num_objects":5016,"num_object_clones":0,"num_object_copies":15048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4097,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5016,"num_write_kb":321024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":308,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[0,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3","objects":919},{"shards":"0,3,7","objects":4097}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":19,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.041789+0000","last_change":"2024-02-10T07:50:43.295417+0000","last_active":"2024-02-10T07:50:57.041789+0000","last_peered":"2024-02-10T07:50:57.041789+0000","last_clean":"2024-02-10T07:50:57.041789+0000","last_became_active":"2024-02-10T07:50:43.290295+0000","last_became_peered":"2024-02-10T07:50:43.290295+0000","last_unstale":"2024-02-10T07:50:57.041789+0000","last_undegraded":"2024-02-10T07:50:57.041789+0000","last_fullsized":"2024-02-10T07:50:57.041789+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.153368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.153368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":756,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.029376+0000","last_change":"2024-02-10T07:49:07.170105+0000","last_active":"2024-02-10T07:50:57.029376+0000","last_peered":"2024-02-10T07:50:57.029376+0000","last_clean":"2024-02-10T07:50:57.029376+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:50:57.029376+0000","last_undegraded":"2024-02-10T07:50:57.029376+0000","last_fullsized":"2024-02-10T07:50:57.029376+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:07.169962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:07.169962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":789,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.026326+0000","last_change":"2024-02-10T07:50:03.803447+0000","last_active":"2024-02-10T07:50:57.026326+0000","last_peered":"2024-02-10T07:50:57.026326+0000","last_clean":"2024-02-10T07:50:57.026326+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:50:57.026326+0000","last_undegraded":"2024-02-10T07:50:57.026326+0000","last_fullsized":"2024-02-10T07:50:57.026326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:03.803361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:03.803361+0000","last_clean_scrub_stamp":"2024-02-10T07:50:03.803361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:51:28.611872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"250'39","reported_seq":883,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009827+0000","last_change":"2024-02-10T07:50:43.316983+0000","last_active":"2024-02-10T07:50:57.009827+0000","last_peered":"2024-02-10T07:50:57.009827+0000","last_clean":"2024-02-10T07:50:57.009827+0000","last_became_active":"2024-02-10T07:50:43.281879+0000","last_became_peered":"2024-02-10T07:50:43.281879+0000","last_unstale":"2024-02-10T07:50:57.009827+0000","last_undegraded":"2024-02-10T07:50:57.009827+0000","last_fullsized":"2024-02-10T07:50:57.009827+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T07:48:59.066256+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.066256+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"409'5117","reported_seq":5163,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:58.461168+0000","last_change":"2024-02-10T07:49:53.281316+0000","last_active":"2024-02-10T07:50:58.461168+0000","last_peered":"2024-02-10T07:50:58.461168+0000","last_clean":"2024-02-10T07:50:58.461168+0000","last_became_active":"2024-02-10T07:49:53.280975+0000","last_became_peered":"2024-02-10T07:49:53.280975+0000","last_unstale":"2024-02-10T07:50:58.461168+0000","last_undegraded":"2024-02-10T07:50:58.461168+0000","last_fullsized":"2024-02-10T07:50:58.461168+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":5117,"log_dups_size":0,"ondisk_log_size":5117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335347712,"num_objects":5117,"num_object_clones":0,"num_object_copies":15351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5117,"num_write_kb":327488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":758,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.010424+0000","last_change":"2024-02-10T07:50:43.290883+0000","last_active":"2024-02-10T07:50:57.010424+0000","last_peered":"2024-02-10T07:50:57.010424+0000","last_clean":"2024-02-10T07:50:57.010424+0000","last_became_active":"2024-02-10T07:50:43.290334+0000","last_became_peered":"2024-02-10T07:50:43.290334+0000","last_unstale":"2024-02-10T07:50:57.010424+0000","last_undegraded":"2024-02-10T07:50:57.010424+0000","last_fullsized":"2024-02-10T07:50:57.010424+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:47.275882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.275882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"405'12","reported_seq":27,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.041463+0000","last_change":"2024-02-10T07:50:43.294666+0000","last_active":"2024-02-10T07:50:57.041463+0000","last_peered":"2024-02-10T07:50:57.041463+0000","last_clean":"2024-02-10T07:50:57.041463+0000","last_became_active":"2024-02-10T07:50:43.280142+0000","last_became_peered":"2024-02-10T07:50:43.280142+0000","last_unstale":"2024-02-10T07:50:57.041463+0000","last_undegraded":"2024-02-10T07:50:57.041463+0000","last_fullsized":"2024-02-10T07:50:57.041463+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'11","last_scrub_stamp":"2024-02-10T07:49:14.335064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:14.335064+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":255,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.030388+0000","last_change":"2024-02-10T07:49:06.259646+0000","last_active":"2024-02-10T07:50:57.030388+0000","last_peered":"2024-02-10T07:50:57.030388+0000","last_clean":"2024-02-10T07:50:57.030388+0000","last_became_active":"2024-02-10T07:49:06.258926+0000","last_became_peered":"2024-02-10T07:49:06.258926+0000","last_unstale":"2024-02-10T07:50:57.030388+0000","last_undegraded":"2024-02-10T07:50:57.030388+0000","last_fullsized":"2024-02-10T07:50:57.030388+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:24.220575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_clean_scrub_stamp":"2024-02-10T07:47:24.220575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"250'20","reported_seq":823,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-02-10T07:50:57.009877+0000","last_change":"2024-02-10T07:49:06.289632+0000","last_active":"2024-02-10T07:50:57.009877+0000","last_peered":"2024-02-10T07:50:57.009877+0000","last_clean":"2024-02-10T07:50:57.009877+0000","last_became_active":"2024-02-10T07:49:06.238858+0000","last_became_peered":"2024-02-10T07:49:06.238858+0000","last_unstale":"2024-02-10T07:50:57.009877+0000","last_undegraded":"2024-02-10T07:50:57.009877+0000","last_fullsized":"2024-02-10T07:50:57.009877+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:48:15.588199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.588199+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5273878528,"num_objects":80473,"num_object_clones":0,"num_object_copies":241419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8861,"num_objects_misplaced":16612,"num_objects_unfound":0,"num_objects_dirty":80473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80473,"num_write_kb":5150272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":57802752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15874785280,"data_stored":15874785280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78973,"ondisk_log_size":78973,"up":48,"acting":43,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1140,"num_read_kb":1135,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1111,"num_read_kb":868,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":6382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":774,"ondisk_log_size":774,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037994,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3179608,"kb_used_data":2502860,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":90565544,"statfs":{"total":95995035648,"available":92739117056,"internally_reserved":0,"allocated":2562928640,"data_stored":2562530069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":692976074},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:50:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.369,"15min":1.8089999999999999},"min":{"1min":0.372,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":3.3530000000000002,"5min":39.494,"15min":39.494},"last":4.5030000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":1.258,"15min":1.694},"min":{"1min":0.36599999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.2429999999999999,"5min":38.524000000000001,"15min":38.524000000000001},"last":1.7330000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:50:17 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.98299999999999998,"15min":1.7150000000000001},"min":{"1min":0.39800000000000002,"5min":0.25700000000000001,"15min":0.24399999999999999},"max":{"1min":3.5089999999999999,"5min":4.2640000000000002,"15min":35.459000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.95299999999999996,"15min":1.6879999999999999},"min":{"1min":0.32700000000000001,"5min":0.25600000000000001,"15min":0.24399999999999999},"max":{"1min":3.1040000000000001,"5min":4.7640000000000002,"15min":35.225000000000001},"last":2.6110000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:50:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.83099999999999996,"15min":1.2090000000000001},"min":{"1min":0.307,"5min":0.255,"15min":0.255},"max":{"1min":2.1230000000000002,"5min":5.4980000000000002,"15min":19.710000000000001},"last":2.1859999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.93600000000000005,"15min":1.3799999999999999},"min":{"1min":0.45200000000000001,"5min":0.32400000000000001,"15min":0.29099999999999998},"max":{"1min":3.3130000000000002,"5min":4.2320000000000002,"15min":19.643000000000001},"last":0.33500000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.1990000000000001,"15min":1.101},"min":{"1min":0.55800000000000005,"5min":0.307,"15min":0.307},"max":{"1min":4.0899999999999999,"5min":4.0990000000000002,"15min":4.0990000000000002},"last":2.6779999999999999},{"interface":"front","average":{"1min":1.298,"5min":0.998,"15min":0.94699999999999995},"min":{"1min":0.44700000000000001,"5min":0.313,"15min":0.313},"max":{"1min":4.452,"5min":4.452,"15min":4.452},"last":1.22}]},{"osd":4,"last update":"Sat Feb 10 07:50:17 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.9099999999999999,"15min":1.117},"min":{"1min":0.372,"5min":0.22,"15min":0.22},"max":{"1min":3.2719999999999998,"5min":26.001000000000001,"15min":26.001000000000001},"last":1.6479999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":1.9550000000000001,"15min":1.119},"min":{"1min":0.36699999999999999,"5min":0.26800000000000002,"15min":0.24099999999999999},"max":{"1min":2.0619999999999998,"5min":27.09,"15min":27.09},"last":2.2810000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.47,"15min":1.052},"min":{"1min":0.47299999999999998,"5min":0.26600000000000001,"15min":0.245},"max":{"1min":3.5489999999999999,"5min":18.126999999999999,"15min":18.126999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.7929999999999999,"5min":1.6339999999999999,"15min":1.149},"min":{"1min":0.52900000000000003,"5min":0.318,"15min":0.29999999999999999},"max":{"1min":5.8490000000000002,"5min":19.548999999999999,"15min":19.548999999999999},"last":1.6859999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:50:17 2024","interfaces":[{"interface":"back","average":{"1min":0.999,"5min":1.6160000000000001,"15min":1.0920000000000001},"min":{"1min":0.53300000000000003,"5min":0.29399999999999998,"15min":0.28199999999999997},"max":{"1min":3.2080000000000002,"5min":17.832000000000001,"15min":17.832000000000001},"last":2.6520000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":1.792,"15min":1.151},"min":{"1min":0.42199999999999999,"5min":0.33100000000000002,"15min":0.27400000000000002},"max":{"1min":2.1509999999999998,"5min":20.449999999999999,"15min":20.449999999999999},"last":0.93999999999999995}]}]},{"osd":6,"up_from":50,"seq":214748365039,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3161276,"kb_used_data":2580272,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":90583876,"statfs":{"total":95995035648,"available":92757889024,"internally_reserved":0,"allocated":2642198528,"data_stored":2641827646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:50:46 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":1.8620000000000001,"15min":1.7909999999999999},"min":{"1min":0.27300000000000002,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.2730000000000001,"5min":24.727,"15min":63.405999999999999},"last":3.5840000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":1.365,"15min":1.6799999999999999},"min":{"1min":0.34699999999999998,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3109999999999999,"5min":29.084,"15min":67.748000000000005},"last":3.7930000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:50:46 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.04,"15min":1.6259999999999999},"min":{"1min":0.37,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.8239999999999998,"5min":5.6180000000000003,"15min":37.659999999999997},"last":4.0199999999999996},{"interface":"front","average":{"1min":1.087,"5min":1.5449999999999999,"15min":1.7849999999999999},"min":{"1min":0.318,"5min":0.28000000000000003,"15min":0.24399999999999999},"max":{"1min":2.9940000000000002,"5min":19.524999999999999,"15min":37.75},"last":2.242}]},{"osd":2,"last update":"Sat Feb 10 07:50:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":1.4850000000000001,"15min":1.458},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.253},"max":{"1min":1.7509999999999999,"5min":15.768000000000001,"15min":62.914999999999999},"last":6.6970000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":1.2450000000000001,"15min":1.365},"min":{"1min":0.40600000000000003,"5min":0.30399999999999999,"15min":0.25800000000000001},"max":{"1min":2.0150000000000001,"5min":15.728999999999999,"15min":63.207000000000001},"last":4.8339999999999996}]},{"osd":3,"last update":"Sat Feb 10 07:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.351,"15min":1.621},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.54,"5min":13.279,"15min":13.279},"last":0.36599999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.1439999999999999,"15min":1.2110000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.7839999999999998,"5min":5.6470000000000002,"15min":5.6470000000000002},"last":0.66900000000000004}]},{"osd":4,"last update":"Sat Feb 10 07:50:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":1.3700000000000001,"15min":0.879},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.121,"5min":7.6509999999999998,"15min":7.7839999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.4179999999999999,"15min":0.88100000000000001},"min":{"1min":0.28899999999999998,"5min":0.28699999999999998,"15min":0.26000000000000001},"max":{"1min":4.8600000000000003,"5min":7.0300000000000002,"15min":7.0300000000000002},"last":0.40400000000000003}]},{"osd":5,"last update":"Sat Feb 10 07:49:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.879,"15min":1.117},"min":{"1min":0.317,"5min":0.26000000000000001,"15min":0.24099999999999999},"max":{"1min":2.5680000000000001,"5min":44.689,"15min":44.689},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":2.0609999999999999,"15min":1.181},"min":{"1min":0.309,"5min":0.309,"15min":0.29199999999999998},"max":{"1min":3.5699999999999998,"5min":44.465000000000003,"15min":44.465000000000003},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Feb 10 07:50:09 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":2.2730000000000001,"15min":1.2869999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.28100000000000003},"max":{"1min":4.7750000000000004,"5min":42.631999999999998,"15min":42.631999999999998},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":1.9950000000000001,"15min":1.1859999999999999},"min":{"1min":0.43099999999999999,"5min":0.32800000000000001,"15min":0.29699999999999999},"max":{"1min":1.6819999999999999,"5min":44.792999999999999,"15min":44.792999999999999},"last":0.56899999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575113,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3942680,"kb_used_data":3177104,"kb_used_omap":1,"kb_used_meta":765566,"kb_avail":89802472,"statfs":{"total":95995035648,"available":91957731328,"internally_reserved":0,"allocated":3253354496,"data_stored":3252951029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":783940038},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:50:47 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":0.621,"15min":1.7490000000000001},"min":{"1min":0.23300000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":6.4379999999999997,"5min":6.4379999999999997,"15min":114.27},"last":8.3100000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55100000000000005,"15min":1.849},"min":{"1min":0.25,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.7150000000000001,"5min":5.157,"15min":119.499},"last":0.72299999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:50:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.46400000000000002,"15min":1.1639999999999999},"min":{"1min":0.253,"5min":0.219,"15min":0.19600000000000001},"max":{"1min":0.93400000000000005,"5min":5.9909999999999997,"15min":8.9209999999999994},"last":2.7690000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.46300000000000002,"15min":1.2909999999999999},"min":{"1min":0.28999999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.53600000000000003,"5min":7.3879999999999999,"15min":8.8300000000000001},"last":0.746}]},{"osd":3,"last update":"Sat Feb 10 07:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":0.83099999999999996,"15min":0.60199999999999998},"min":{"1min":0.30299999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":6.2949999999999999,"5min":22.367999999999999,"15min":22.367999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.69599999999999995,"15min":0.48999999999999999},"min":{"1min":0.32000000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.298,"5min":22.199999999999999,"15min":22.199999999999999},"last":4.7839999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:50:03 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.4399999999999999,"15min":2.0939999999999999},"min":{"1min":0.34499999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.294,"5min":31.908999999999999,"15min":207.45699999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":1.3959999999999999,"15min":1.294},"min":{"1min":0.34699999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":2.528,"5min":32.201000000000001,"15min":32.201000000000001},"last":0.47399999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:50:03 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.4530000000000001,"15min":1.9450000000000001},"min":{"1min":0.34200000000000003,"5min":0.308,"15min":0.308},"max":{"1min":2.4910000000000001,"5min":31.824999999999999,"15min":98.703000000000003},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":1.5089999999999999,"15min":1.923},"min":{"1min":0.39700000000000002,"5min":0.38200000000000001,"15min":0.30199999999999999},"max":{"1min":2.4740000000000002,"5min":31.841999999999999,"15min":98.664000000000001},"last":0.69299999999999995}]},{"osd":6,"last update":"Sat Feb 10 07:50:47 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.6830000000000001,"15min":1.3899999999999999},"min":{"1min":0.34499999999999997,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":6.3719999999999999,"5min":36.808,"15min":36.808},"last":0.377},{"interface":"front","average":{"1min":1.675,"5min":1.7490000000000001,"15min":1.623},"min":{"1min":0.48399999999999999,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":6.2629999999999999,"5min":33.774999999999999,"15min":33.774999999999999},"last":2.8809999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:50:13 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.95799999999999996,"15min":1.381},"min":{"1min":0.435,"5min":0.38200000000000001,"15min":0.36099999999999999},"max":{"1min":1.8,"5min":7.444,"15min":8.4239999999999995},"last":4.8319999999999999},{"interface":"front","average":{"1min":0.749,"5min":1.016,"15min":1.3009999999999999},"min":{"1min":0.41899999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.3490000000000002,"5min":6.2089999999999996,"15min":11.212},"last":3.351}]}]},{"osd":0,"up_from":8,"seq":34359738638,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2625304,"kb_used_data":1881932,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":91119848,"statfs":{"total":95995035648,"available":93306724352,"internally_reserved":0,"allocated":1927098368,"data_stored":1926700194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.92500000000000004,"15min":2.2999999999999998},"min":{"1min":0.24199999999999999,"5min":0.187,"15min":0.182},"max":{"1min":3.0110000000000001,"5min":34.661999999999999,"15min":156.16200000000001},"last":7.2800000000000002},{"interface":"front","average":{"1min":1.0409999999999999,"5min":0.94299999999999995,"15min":2.3069999999999999},"min":{"1min":0.26000000000000001,"5min":0.192,"15min":0.183},"max":{"1min":5.7329999999999997,"5min":34.636000000000003,"15min":156.941},"last":6.9740000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.878,"15min":1.5760000000000001},"min":{"1min":0.25800000000000001,"5min":0.21299999999999999,"15min":0.20000000000000001},"max":{"1min":2.133,"5min":34.603000000000002,"15min":34.603000000000002},"last":2.7269999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.84999999999999998,"15min":1.367},"min":{"1min":0.32500000000000001,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":1.0209999999999999,"5min":34.718000000000004,"15min":34.718000000000004},"last":0.56999999999999995}]},{"osd":3,"last update":"Sat Feb 10 07:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.69099999999999995,"15min":0.65200000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.5590000000000002,"5min":3.5590000000000002,"15min":3.5590000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.0580000000000001,"5min":0.69099999999999995,"15min":0.64600000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":4.6230000000000002,"5min":4.6230000000000002,"15min":4.6230000000000002},"last":0.69299999999999995}]},{"osd":4,"last update":"Sat Feb 10 07:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.6990000000000001,"15min":1.335},"min":{"1min":0.32500000000000001,"5min":0.28299999999999997,"15min":0.27600000000000002},"max":{"1min":4.6790000000000003,"5min":37.286000000000001,"15min":37.286000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.7070000000000001,"15min":1.415},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26900000000000002},"max":{"1min":3.0670000000000002,"5min":34.683,"15min":34.683},"last":2.6200000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":2.1680000000000001,"15min":2.1139999999999999},"min":{"1min":0.42899999999999999,"5min":0.35499999999999998,"15min":0.32300000000000001},"max":{"1min":4.7149999999999999,"5min":41.429000000000002,"15min":41.429000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.1910000000000001,"5min":2.0750000000000002,"15min":2.1779999999999999},"min":{"1min":0.44600000000000001,"5min":0.34000000000000002,"15min":0.308},"max":{"1min":3.0009999999999999,"5min":41.456000000000003,"15min":41.456000000000003},"last":2.6669999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:50:37 2024","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":2.2599999999999998,"15min":1.5629999999999999},"min":{"1min":0.374,"5min":0.35399999999999998,"15min":0.29599999999999999},"max":{"1min":5.069,"5min":41.587000000000003,"15min":41.587000000000003},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.1200000000000001,"5min":2.2970000000000002,"15min":1.696},"min":{"1min":0.44600000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.9660000000000002,"5min":41.256,"15min":41.256},"last":0.746}]},{"osd":7,"last update":"Sat Feb 10 07:50:18 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":1.593,"15min":1.6299999999999999},"min":{"1min":0.41999999999999998,"5min":0.34899999999999998,"15min":0.33300000000000002},"max":{"1min":2.2360000000000002,"5min":34.710999999999999,"15min":34.710999999999999},"last":2.6989999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":1.534,"15min":1.421},"min":{"1min":0.39200000000000002,"5min":0.38500000000000001,"15min":0.379},"max":{"1min":2.0449999999999999,"5min":34.997999999999998,"15min":34.997999999999998},"last":0.63100000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411587,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799860,"kb_used_data":273064,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":92945292,"statfs":{"total":95995035648,"available":95175979008,"internally_reserved":0,"allocated":279617536,"data_stored":279254892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:50:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.92900000000000005,"15min":2.1520000000000001},"min":{"1min":0.223,"5min":0.191,"15min":0.191},"max":{"1min":7.0460000000000003,"5min":20.335999999999999,"15min":90.046999999999997},"last":7.3550000000000004},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.93500000000000005,"15min":1.8879999999999999},"min":{"1min":0.26600000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":6.694,"5min":20.370000000000001,"15min":90.016000000000005},"last":5.8440000000000003}]},{"osd":1,"last update":"Sat Feb 10 07:50:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.52500000000000002,"15min":1.631},"min":{"1min":0.25,"5min":0.20899999999999999,"15min":0.19500000000000001},"max":{"1min":0.90900000000000003,"5min":3.6699999999999999,"15min":65.795000000000002},"last":5.157},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.47399999999999998,"15min":1.415},"min":{"1min":0.248,"5min":0.219,"15min":0.19500000000000001},"max":{"1min":1.0009999999999999,"5min":2.6030000000000002,"15min":9.5630000000000006},"last":1.8009999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.64500000000000002,"15min":0.59499999999999997},"min":{"1min":0.30399999999999999,"5min":0.217,"15min":0.217},"max":{"1min":5.125,"5min":5.125,"15min":5.125},"last":5.125},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.55400000000000005,"15min":0.51000000000000001},"min":{"1min":0.29099999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":4.4729999999999999,"5min":4.4729999999999999,"15min":4.4729999999999999},"last":2.7040000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:49:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.1599999999999999,"15min":1.6080000000000001},"min":{"1min":0.44,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":11.395,"5min":11.395,"15min":65.494},"last":5.0250000000000004},{"interface":"front","average":{"1min":1.028,"5min":0.998,"15min":1.3839999999999999},"min":{"1min":0.32900000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.476,"5min":4.4340000000000002,"15min":33.554000000000002},"last":1.7589999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.262,"15min":1.3680000000000001},"min":{"1min":0.40799999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":2.0339999999999998,"5min":25.533000000000001,"15min":25.533000000000001},"last":1.2350000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.272,"15min":1.615},"min":{"1min":0.35599999999999998,"5min":0.34899999999999998,"15min":0.34499999999999997},"max":{"1min":1.5,"5min":25.503,"15min":65.834000000000003},"last":4.9749999999999996}]},{"osd":6,"last update":"Sat Feb 10 07:50:37 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.8069999999999999,"15min":1.7490000000000001},"min":{"1min":0.41399999999999998,"5min":0.35199999999999998,"15min":0.33800000000000002},"max":{"1min":1.9390000000000001,"5min":35.603999999999999,"15min":65.748000000000005},"last":5.069},{"interface":"front","average":{"1min":0.89800000000000002,"5min":1.2809999999999999,"15min":1.343},"min":{"1min":0.371,"5min":0.34599999999999997,"15min":0.315},"max":{"1min":4.2240000000000002,"5min":25.901,"15min":25.901},"last":1.325}]},{"osd":7,"last update":"Sat Feb 10 07:50:15 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":1.0860000000000001,"15min":1.3899999999999999},"min":{"1min":0.44400000000000001,"5min":0.36699999999999999,"15min":0.33000000000000002},"max":{"1min":2.7829999999999999,"5min":6.5700000000000003,"15min":8.2189999999999994},"last":1.29},{"interface":"front","average":{"1min":0.76600000000000001,"5min":1.0049999999999999,"15min":1.5649999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":2.556,"5min":5.8940000000000001,"15min":65.539000000000001},"last":5.0979999999999999}]}]},{"osd":3,"up_from":347,"seq":1490353651751,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2381512,"kb_used_data":2198336,"kb_used_omap":5,"kb_used_meta":183162,"kb_avail":91363640,"statfs":{"total":95995035648,"available":93556367360,"internally_reserved":0,"allocated":2251096064,"data_stored":2250700299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5625,"internal_metadata":187558407},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.63600000000000001,"15min":0.56200000000000006},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.4740000000000002,"5min":6.4740000000000002,"15min":6.4740000000000002},"last":6.4740000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80300000000000005,"15min":0.68500000000000005},"min":{"1min":0.27200000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.5649999999999999,"5min":16.939,"15min":16.939},"last":0.56399999999999995}]},{"osd":1,"last update":"Sat Feb 10 07:50:50 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":0.59899999999999998,"15min":0.46300000000000002},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":6.4429999999999996,"5min":6.4429999999999996,"15min":6.4429999999999996},"last":6.4429999999999996},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.68999999999999995,"15min":0.51500000000000001},"min":{"1min":0.311,"5min":0.22,"15min":0.22},"max":{"1min":2.5960000000000001,"5min":16.716000000000001,"15min":16.716000000000001},"last":0.59699999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.55700000000000005,"15min":0.49299999999999999},"min":{"1min":0.28399999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":6.3410000000000002,"5min":6.3410000000000002,"15min":6.3410000000000002},"last":6.3410000000000002},{"interface":"front","average":{"1min":0.999,"5min":0.67100000000000004,"15min":0.54000000000000004},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.1680000000000001,"5min":10.122,"15min":10.122},"last":3.9950000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:50:50 2024","interfaces":[{"interface":"back","average":{"1min":1.956,"5min":2.331,"15min":2.6150000000000002},"min":{"1min":0.40500000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":15.505000000000001,"5min":28.707999999999998,"15min":28.707999999999998},"last":6.149},{"interface":"front","average":{"1min":1.8540000000000001,"5min":2.4359999999999999,"15min":2.8599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":15.551,"5min":28.760999999999999,"15min":28.760999999999999},"last":6.4119999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:50:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7729999999999999,"5min":1.165,"15min":0.95299999999999996},"min":{"1min":0.40899999999999997,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":15.525,"5min":15.525,"15min":15.525},"last":0.65100000000000002},{"interface":"front","average":{"1min":1.145,"5min":0.96399999999999997,"15min":0.81999999999999995},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":6.2290000000000001,"5min":10.298,"15min":10.298},"last":6.2290000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:50:50 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":2.0960000000000001,"15min":2.4140000000000001},"min":{"1min":0.443,"5min":0.372,"15min":0.372},"max":{"1min":3.629,"5min":28.661999999999999,"15min":28.661999999999999},"last":0.624},{"interface":"front","average":{"1min":1.236,"5min":2.54,"15min":3},"min":{"1min":0.45300000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":6.1870000000000003,"5min":28.785,"15min":28.785},"last":6.1870000000000003}]},{"osd":7,"last update":"Sat Feb 10 07:50:50 2024","interfaces":[{"interface":"back","average":{"1min":2.1699999999999999,"5min":1.399,"15min":1.2090000000000001},"min":{"1min":0.40999999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":15.577999999999999,"5min":15.577999999999999,"15min":15.577999999999999},"last":6.2619999999999996},{"interface":"front","average":{"1min":1.327,"5min":1.3149999999999999,"15min":1.284},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":6.3780000000000001,"5min":6.3780000000000001,"15min":6.3780000000000001},"last":6.3780000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953721,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2254036,"kb_used_data":1550924,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":91491116,"statfs":{"total":95995035648,"available":93686902784,"internally_reserved":0,"allocated":1588146176,"data_stored":1587746696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":1.0569999999999999,"15min":2.089},"min":{"1min":0.26400000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.504,"5min":10.859999999999999,"15min":74.533000000000001},"last":5.0819999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.2390000000000001,"15min":2.1030000000000002},"min":{"1min":0.32600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.8499999999999996,"5min":7.298,"15min":74.340999999999994},"last":1.8720000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":1.794,"15min":2.3170000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.23200000000000001},"max":{"1min":2.3999999999999999,"5min":66.343999999999994,"15min":69.852000000000004},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.331,"15min":2.2040000000000002},"min":{"1min":0.32200000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.847,"5min":5.9909999999999997,"15min":70.692999999999998},"last":0.45800000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:49:59 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.3460000000000001,"15min":1.4470000000000001},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.26600000000000001},"max":{"1min":5.0700000000000003,"5min":6.0179999999999998,"15min":50.162999999999997},"last":2.681},{"interface":"front","average":{"1min":0.96799999999999997,"5min":1.212,"15min":1.3939999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":5.9359999999999999,"5min":7.335,"15min":50.453000000000003},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Feb 10 07:50:50 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":3.3580000000000001,"15min":4.282},"min":{"1min":0.33500000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":6.1520000000000001,"5min":66.379999999999995,"15min":66.379999999999995},"last":3.4489999999999998},{"interface":"front","average":{"1min":1.206,"5min":1.516,"15min":1.6879999999999999},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.309},"max":{"1min":4.8940000000000001,"5min":5.3780000000000001,"15min":5.3780000000000001},"last":0.36499999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:49:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.5609999999999999,"15min":0.95999999999999996},"min":{"1min":0.28799999999999998,"5min":0.28599999999999998,"15min":0.23100000000000001},"max":{"1min":5.9720000000000004,"5min":10.502000000000001,"15min":10.502000000000001},"last":0.499},{"interface":"front","average":{"1min":0.79300000000000004,"5min":1.524,"15min":0.96299999999999997},"min":{"1min":0.33300000000000002,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":2.9820000000000002,"5min":9.2460000000000004,"15min":9.2460000000000004},"last":0.313}]},{"osd":6,"last update":"Sat Feb 10 07:50:41 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":2.3370000000000002,"15min":1.2},"min":{"1min":0.30499999999999999,"5min":0.28100000000000003,"15min":0.25900000000000001},"max":{"1min":2.9049999999999998,"5min":67.433999999999997,"15min":67.433999999999997},"last":0.44},{"interface":"front","average":{"1min":0.95599999999999996,"5min":2.4289999999999998,"15min":1.274},"min":{"1min":0.35099999999999998,"5min":0.29499999999999998,"15min":0.24399999999999999},"max":{"1min":2.4350000000000001,"5min":66.876000000000005,"15min":66.876000000000005},"last":0.41899999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:50:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":1.514,"15min":0.94299999999999995},"min":{"1min":0.42099999999999999,"5min":0.307,"15min":0.29099999999999998},"max":{"1min":1.5269999999999999,"5min":7.4930000000000003,"15min":7.4930000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":1.474,"15min":0.996},"min":{"1min":0.30299999999999999,"5min":0.30199999999999999,"15min":0.29399999999999998},"max":{"1min":3.298,"5min":10.919,"15min":10.919},"last":0.48199999999999998}]}]},{"osd":5,"up_from":156,"seq":670014898350,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1936204,"kb_used_data":1275712,"kb_used_omap":4,"kb_used_meta":660475,"kb_avail":91808948,"statfs":{"total":95995035648,"available":94012362752,"internally_reserved":0,"allocated":1306329088,"data_stored":1305942710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":676327074},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.514,"15min":1.681},"min":{"1min":0.23799999999999999,"5min":0.20899999999999999,"15min":0.20499999999999999},"max":{"1min":3.3370000000000002,"5min":21.817,"15min":52.938000000000002},"last":24.292999999999999},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.446,"15min":1.8979999999999999},"min":{"1min":0.25800000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.8290000000000002,"5min":21.347999999999999,"15min":52.805},"last":23.908000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.1399999999999999,"15min":2.0169999999999999},"min":{"1min":0.27700000000000002,"5min":0.26900000000000002,"15min":0.245},"max":{"1min":2.5840000000000001,"5min":6.423,"15min":54.298000000000002},"last":28.780999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.2270000000000001,"15min":1.968},"min":{"1min":0.26400000000000001,"5min":0.246,"15min":0.246},"max":{"1min":3.032,"5min":7.7649999999999997,"15min":52.969999999999999},"last":27.600000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.1200000000000001,"15min":1.214},"min":{"1min":0.379,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.6309999999999998,"5min":6.3970000000000002,"15min":11.601000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.423,"15min":1.296},"min":{"1min":0.376,"5min":0.28199999999999997,"15min":0.28100000000000003},"max":{"1min":3.2080000000000002,"5min":13.662000000000001,"15min":13.662000000000001},"last":0.45200000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.409,"15min":1.6220000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":3.0899999999999999,"5min":8.1530000000000005,"15min":8.1530000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.278,"15min":1.409},"min":{"1min":0.32600000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.3639999999999999,"5min":5.8470000000000004,"15min":5.8470000000000004},"last":0.434}]},{"osd":4,"last update":"Sat Feb 10 07:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":1.97,"15min":1.1299999999999999},"min":{"1min":0.28599999999999998,"5min":0.252,"15min":0.246},"max":{"1min":3.222,"5min":8.952,"15min":8.952},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.755,"5min":1.702,"15min":1.0409999999999999},"min":{"1min":0.315,"5min":0.26800000000000002,"15min":0.25900000000000001},"max":{"1min":3.2970000000000002,"5min":10.180999999999999,"15min":10.180999999999999},"last":0.61499999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":1.6299999999999999,"15min":1.0409999999999999},"min":{"1min":0.32400000000000001,"5min":0.28699999999999998,"15min":0.23999999999999999},"max":{"1min":3.2589999999999999,"5min":9.0229999999999997,"15min":9.0229999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.80400000000000005,"5min":1.528,"15min":1.008},"min":{"1min":0.308,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":3.3879999999999999,"5min":6.1699999999999999,"15min":6.1699999999999999},"last":0.66000000000000003}]},{"osd":7,"last update":"Sat Feb 10 07:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.4690000000000001,"15min":1.155},"min":{"1min":0.40899999999999997,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.657,"5min":9.0199999999999996,"15min":9.0199999999999996},"last":0.379},{"interface":"front","average":{"1min":0.96299999999999997,"5min":1.4119999999999999,"15min":1.0469999999999999},"min":{"1min":0.371,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":3.234,"5min":7.8109999999999999,"15min":7.8109999999999999},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1938161664,"data_stored":1938161664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3268018176,"data_stored":3268018176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295763968,"data_stored":295763968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2251227136,"data_stored":2251227136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1597374464,"data_stored":1597374464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1310261248,"data_stored":1310261248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2652504064,"data_stored":2652504064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2561474560,"data_stored":2561474560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:51:04.199 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-02-10T07:51:04.214 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:04.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:04.329 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:04.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:04.431 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:04.458 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:04.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:04.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:04.499 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:04 smithi008 ceph-mon[40743]: 5.7 scrub starts 2024-02-10T07:51:04.500 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:04 smithi008 ceph-mon[40743]: 11.a scrub starts 2024-02-10T07:51:04.500 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:04 smithi008 ceph-mon[40743]: 5.7 scrub ok 2024-02-10T07:51:04.500 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:51:04.500 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:51:04.500 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:04 smithi008 ceph-mon[29816]: 5.7 scrub starts 2024-02-10T07:51:04.500 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:04 smithi008 ceph-mon[29816]: 11.a scrub starts 2024-02-10T07:51:04.500 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:04 smithi008 ceph-mon[29816]: 5.7 scrub ok 2024-02-10T07:51:04.500 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:51:04.500 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:51:04.560 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:04.608 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:04.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:04.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:04 smithi170 ceph-mon[33394]: 5.7 scrub starts 2024-02-10T07:51:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:04 smithi170 ceph-mon[33394]: 11.a scrub starts 2024-02-10T07:51:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:04 smithi170 ceph-mon[33394]: 5.7 scrub ok 2024-02-10T07:51:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:51:04.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:51:04.658 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 59 16 84463 84447 89.4347 34.5 0.0315753 0.0111735 2024-02-10T07:51:04.711 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:04.741 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:04.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:04.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:04.843 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:04.893 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:04.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:04.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:04.995 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:05.008 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:51:05.024 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:05.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:05.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:05.125 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:05.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:05.187 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:05.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:05.288 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:05.318 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:05.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:05.408 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:51:05.408 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:75223 11.4 e407/405 11:2172ace1:::benchmark_data_smithi008_186804_object75222:head v 407'4806, mlcod=0'0)", 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:51.119326+0000", 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 14.283838491999999, 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.064326306999999999, 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.119326+0000", 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.409 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.119326+0000", 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.119327+0000", 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9999999999999995e-07 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.119380+0000", 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2732999999999998e-05 2024-02-10T07:51:05.410 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.411 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.411 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.411 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.119396+0000", 2024-02-10T07:51:05.411 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6433e-05 2024-02-10T07:51:05.411 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.411 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.411 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.411 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.119404+0000", 2024-02-10T07:51:05.411 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.977e-06 2024-02-10T07:51:05.411 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.411 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.411 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.411 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.122971+0000", 2024-02-10T07:51:05.411 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003566895 2024-02-10T07:51:05.412 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.412 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.412 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.412 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.123028+0000", 2024-02-10T07:51:05.412 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7074000000000003e-05 2024-02-10T07:51:05.412 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.412 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.412 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.412 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.183559+0000", 2024-02-10T07:51:05.412 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.060530860999999998 2024-02-10T07:51:05.412 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.412 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.412 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.413 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.183653+0000", 2024-02-10T07:51:05.413 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3333999999999998e-05 2024-02-10T07:51:05.413 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.413 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.413 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.413 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.413 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.413 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDPGPush(11.2 407/405 [PushOp(11:44cfccad:::benchmark_data_smithi008_186804_object2154:head, version: 396'119, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:44cfccad:::benchmark_data_smithi008_186804_object2154:head@396'119, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-10T07:51:05.413 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:51.307558+0000", 2024-02-10T07:51:05.413 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 14.095606348, 2024-02-10T07:51:05.413 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.088330948000000006, 2024-02-10T07:51:05.413 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.413 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.307558+0000", 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.307558+0000", 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.307559+0000", 2024-02-10T07:51:05.414 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8999999999999996e-07 2024-02-10T07:51:05.415 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.415 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.415 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.415 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.307624+0000", 2024-02-10T07:51:05.415 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4581999999999999e-05 2024-02-10T07:51:05.415 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.415 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.415 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.415 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.307644+0000", 2024-02-10T07:51:05.415 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0608e-05 2024-02-10T07:51:05.415 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.415 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.415 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.415 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.307660+0000", 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5577999999999999e-05 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.393274+0000", 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.085614661999999994 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.393317+0000", 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2157999999999999e-05 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.416 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:51.395889+0000", 2024-02-10T07:51:05.417 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0025727699999999998 2024-02-10T07:51:05.417 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.417 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.417 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.417 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.417 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.417 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "MOSDPGPush(11.c 408/405 [PushOp(11:35259e66:::benchmark_data_smithi008_186804_object2020:head, version: 396'142, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:35259e66:::benchmark_data_smithi008_186804_object2020:head@396'142, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-10T07:51:05.417 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:52.438445+0000", 2024-02-10T07:51:05.417 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.964720155, 2024-02-10T07:51:05.417 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.064038859000000004, 2024-02-10T07:51:05.417 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.417 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:51:05.417 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.417 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.418 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.418 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.438445+0000", 2024-02-10T07:51:05.418 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.418 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.418 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.418 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.418 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.438445+0000", 2024-02-10T07:51:05.418 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.418 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.418 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.418 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.418 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.438446+0000", 2024-02-10T07:51:05.418 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1209999999999999e-06 2024-02-10T07:51:05.418 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.419 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.419 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.419 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.438636+0000", 2024-02-10T07:51:05.419 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019023 2024-02-10T07:51:05.419 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.419 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.419 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.419 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.438658+0000", 2024-02-10T07:51:05.419 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1926999999999999e-05 2024-02-10T07:51:05.419 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.419 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.419 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.419 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.438674+0000", 2024-02-10T07:51:05.419 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6035000000000001e-05 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.496749+0000", 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.058075387999999999 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.496763+0000", 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4226000000000001e-05 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.502483+0000", 2024-02-10T07:51:05.420 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0057199319999999996 2024-02-10T07:51:05.421 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.421 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.421 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.421 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.421 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.421 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:76237 11.2 e408/405 11:495acb2f:::benchmark_data_smithi008_186804_object76236:head v 408'4755, mlcod=397'147)", 2024-02-10T07:51:05.421 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:52.579126+0000", 2024-02-10T07:51:05.421 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.824038722999999, 2024-02-10T07:51:05.421 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.063539292999999997, 2024-02-10T07:51:05.421 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.421 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.421 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.421 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.579126+0000", 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.579126+0000", 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.579127+0000", 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0100000000000005e-07 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.422 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.579148+0000", 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1729000000000002e-05 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.579161+0000", 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2325e-05 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.579167+0000", 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8240000000000002e-06 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.423 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.587854+0000", 2024-02-10T07:51:05.424 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0086867349999999992 2024-02-10T07:51:05.424 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.424 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.424 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.424 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.587884+0000", 2024-02-10T07:51:05.424 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9382000000000001e-05 2024-02-10T07:51:05.424 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.424 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.424 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.424 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.642600+0000", 2024-02-10T07:51:05.425 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.054716088000000003 2024-02-10T07:51:05.425 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.425 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.425 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.425 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.642665+0000", 2024-02-10T07:51:05.425 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5609000000000002e-05 2024-02-10T07:51:05.425 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.425 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.425 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.425 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.425 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.425 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:76452 11.c e408/405 11:3b0ba73f:::benchmark_data_smithi008_186804_object76451:head v 408'4691, mlcod=396'147)", 2024-02-10T07:51:05.425 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:52.884323+0000", 2024-02-10T07:51:05.425 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.518841277, 2024-02-10T07:51:05.426 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.070407839999999999, 2024-02-10T07:51:05.426 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.426 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.426 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.426 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.426 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.426 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.884323+0000", 2024-02-10T07:51:05.426 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.426 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.426 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.426 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.427 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.884323+0000", 2024-02-10T07:51:05.427 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.427 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.427 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.427 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.427 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.884325+0000", 2024-02-10T07:51:05.427 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.186e-06 2024-02-10T07:51:05.427 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.427 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.427 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.427 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.884356+0000", 2024-02-10T07:51:05.427 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1143000000000003e-05 2024-02-10T07:51:05.427 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.428 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.428 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.428 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.884367+0000", 2024-02-10T07:51:05.428 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1684e-05 2024-02-10T07:51:05.428 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.428 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.428 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.428 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.884376+0000", 2024-02-10T07:51:05.428 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8119999999999997e-06 2024-02-10T07:51:05.428 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.428 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.428 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.428 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.909737+0000", 2024-02-10T07:51:05.428 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.025361174 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.911449+0000", 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001712079 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.954525+0000", 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.043075892999999997 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.954731+0000", 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00020586899999999999 2024-02-10T07:51:05.429 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:76590 11.4 e408/405 11:20e4a5c3:::benchmark_data_smithi008_186804_object76589:head v 408'4896, mlcod=0'0)", 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:53.062370+0000", 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.340795109, 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.073321035000000007, 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.062370+0000", 2024-02-10T07:51:05.430 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.062370+0000", 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.062371+0000", 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2389999999999999e-06 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.062405+0000", 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4208999999999997e-05 2024-02-10T07:51:05.431 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.432 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.432 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.432 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.062422+0000", 2024-02-10T07:51:05.432 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6674e-05 2024-02-10T07:51:05.432 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.432 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.432 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.432 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.062431+0000", 2024-02-10T07:51:05.432 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.5109999999999992e-06 2024-02-10T07:51:05.432 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.432 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.432 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.433 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.086613+0000", 2024-02-10T07:51:05.433 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.024181735999999999 2024-02-10T07:51:05.433 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.433 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.433 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.433 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.086641+0000", 2024-02-10T07:51:05.433 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8194000000000001e-05 2024-02-10T07:51:05.433 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.433 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.433 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.433 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.135620+0000", 2024-02-10T07:51:05.433 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.048978499000000002 2024-02-10T07:51:05.434 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.434 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.434 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.434 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.135691+0000", 2024-02-10T07:51:05.434 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0973e-05 2024-02-10T07:51:05.434 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.434 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.434 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.434 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.434 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.434 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:05.436 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:76595 11.c e408/405 11:3238ca04:::benchmark_data_smithi008_186804_object76594:head v 408'4701, mlcod=396'149)", 2024-02-10T07:51:05.436 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:53.069883+0000", 2024-02-10T07:51:05.436 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.333282055, 2024-02-10T07:51:05.436 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.065733142999999994, 2024-02-10T07:51:05.436 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.436 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.436 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.437 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.437 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.437 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.069883+0000", 2024-02-10T07:51:05.437 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.437 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.437 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.437 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.437 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.069883+0000", 2024-02-10T07:51:05.437 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.437 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.437 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.437 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.437 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.069884+0000", 2024-02-10T07:51:05.438 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.063e-06 2024-02-10T07:51:05.438 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.438 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.438 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.438 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.069917+0000", 2024-02-10T07:51:05.438 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.2926000000000003e-05 2024-02-10T07:51:05.438 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.438 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.438 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.438 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.069930+0000", 2024-02-10T07:51:05.438 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3088000000000001e-05 2024-02-10T07:51:05.439 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.439 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.439 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.439 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.069938+0000", 2024-02-10T07:51:05.439 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6149999999999997e-06 2024-02-10T07:51:05.439 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.439 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.439 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.439 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.085282+0000", 2024-02-10T07:51:05.439 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.015344059 2024-02-10T07:51:05.440 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.440 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.440 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.440 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.085311+0000", 2024-02-10T07:51:05.440 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9053000000000001e-05 2024-02-10T07:51:05.440 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.440 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.440 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.440 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.135444+0000", 2024-02-10T07:51:05.441 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.050132471999999997 2024-02-10T07:51:05.441 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.441 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.441 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.441 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.135616+0000", 2024-02-10T07:51:05.441 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000171867 2024-02-10T07:51:05.441 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.441 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.441 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.441 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.441 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.441 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:76816 11.d e408/405 11:b97b99db:::benchmark_data_smithi008_186804_object76815:head v 408'4822, mlcod=0'0)", 2024-02-10T07:51:05.441 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:53.366174+0000", 2024-02-10T07:51:05.441 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 12.036990825, 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.064987967999999993, 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.366174+0000", 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.366174+0000", 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.442 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.366175+0000", 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1699999999999997e-07 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.366203+0000", 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7963999999999999e-05 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.366217+0000", 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4399000000000001e-05 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.443 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.444 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.444 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.366225+0000", 2024-02-10T07:51:05.444 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0989999999999999e-06 2024-02-10T07:51:05.444 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.444 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.444 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.444 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.373873+0000", 2024-02-10T07:51:05.444 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0076475129999999999 2024-02-10T07:51:05.444 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.444 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.444 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.444 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.373900+0000", 2024-02-10T07:51:05.444 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7013e-05 2024-02-10T07:51:05.444 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.431114+0000", 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.057214005999999998 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:53.431162+0000", 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8056999999999997e-05 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:78727 11.4 e408/405 11:2c5805ea:::benchmark_data_smithi008_186804_object78726:head v 408'5038, mlcod=0'0)", 2024-02-10T07:51:05.445 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:56.182118+0000", 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 9.2210469939999999, 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.072483605000000007, 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.182118+0000", 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.182118+0000", 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.446 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.182118+0000", 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6100000000000002e-07 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.182154+0000", 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.5481000000000001e-05 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.182161+0000", 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5109999999999997e-06 2024-02-10T07:51:05.447 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.182166+0000", 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9309999999999999e-06 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.193702+0000", 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.011535672 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.193734+0000", 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.252e-05 2024-02-10T07:51:05.448 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.254516+0000", 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.060781992 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.254601+0000", 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5136999999999995e-05 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:78737 11.4 e408/405 11:24f7012d:::benchmark_data_smithi008_186804_object78736:head v 408'5039, mlcod=0'0)", 2024-02-10T07:51:05.449 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:56.191228+0000", 2024-02-10T07:51:05.450 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 9.2119362079999991, 2024-02-10T07:51:05.450 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.069413140999999998, 2024-02-10T07:51:05.450 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.450 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.450 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.450 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.450 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.450 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.191228+0000", 2024-02-10T07:51:05.450 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.450 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.450 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.450 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.450 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.191228+0000", 2024-02-10T07:51:05.450 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.191229+0000", 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4399999999999998e-07 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.191273+0000", 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3628000000000003e-05 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.191295+0000", 2024-02-10T07:51:05.451 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1628e-05 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.191304+0000", 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.9050000000000003e-06 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.205237+0000", 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013933041 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.205307+0000", 2024-02-10T07:51:05.452 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.9999999999999994e-05 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.260606+0000", 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.055298893000000002 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.260642+0000", 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6102000000000001e-05 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.453 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:78729 11.c e408/405 11:3f7ac733:::benchmark_data_smithi008_186804_object78728:head v 408'4850, mlcod=397'200)", 2024-02-10T07:51:05.454 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:56.194257+0000", 2024-02-10T07:51:05.454 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 9.2089074820000008, 2024-02-10T07:51:05.454 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.066340700000000002, 2024-02-10T07:51:05.454 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.454 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.454 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.454 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.454 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.454 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.194257+0000", 2024-02-10T07:51:05.454 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.454 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.454 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.454 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.454 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.194257+0000", 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.194258+0000", 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0999999999999998e-07 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.194283+0000", 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4808999999999999e-05 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.455 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.194292+0000", 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.611e-06 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.194299+0000", 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4489999999999996e-06 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.203914+0000", 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0096148259999999999 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.456 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.203944+0000", 2024-02-10T07:51:05.457 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0627999999999999e-05 2024-02-10T07:51:05.457 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.457 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.457 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.457 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.260546+0000", 2024-02-10T07:51:05.457 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.056601823000000002 2024-02-10T07:51:05.457 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.457 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.457 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.457 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:56.260598+0000", 2024-02-10T07:51:05.457 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.1944000000000003e-05 2024-02-10T07:51:05.457 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.457 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.458 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.458 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.458 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.458 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:82498 11.4 e409/405 11:27dc9e34:::benchmark_data_smithi008_186804_object82497:head v 409'5288, mlcod=0'0)", 2024-02-10T07:51:05.458 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:01.540202+0000", 2024-02-10T07:51:05.458 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.8629625980000002, 2024-02-10T07:51:05.458 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.061515558999999997, 2024-02-10T07:51:05.458 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.458 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.458 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.458 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.458 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.458 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.540202+0000", 2024-02-10T07:51:05.458 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.540202+0000", 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.540202+0000", 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.2800000000000003e-07 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.540236+0000", 2024-02-10T07:51:05.459 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.3118999999999999e-05 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.540248+0000", 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2058999999999999e-05 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.540253+0000", 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4149999999999998e-06 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.549152+0000", 2024-02-10T07:51:05.460 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0088985959999999999 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.549179+0000", 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6951e-05 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.601668+0000", 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.052489131000000001 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.601718+0000", 2024-02-10T07:51:05.461 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9960000000000003e-05 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:82616 11.4 e409/405 11:2d17b065:::benchmark_data_smithi008_186804_object82615:head v 409'5299, mlcod=0'0)", 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:01.717787+0000", 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.6853781350000001, 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.062406957999999998, 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.462 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.717787+0000", 2024-02-10T07:51:05.463 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.463 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.463 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.463 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.463 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.717787+0000", 2024-02-10T07:51:05.463 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.463 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.463 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.463 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.463 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.717788+0000", 2024-02-10T07:51:05.463 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.06e-06 2024-02-10T07:51:05.464 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.464 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.464 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.464 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.717851+0000", 2024-02-10T07:51:05.464 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3235e-05 2024-02-10T07:51:05.465 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.465 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.465 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.465 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.717866+0000", 2024-02-10T07:51:05.465 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4657999999999999e-05 2024-02-10T07:51:05.465 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.465 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.465 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.465 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.717874+0000", 2024-02-10T07:51:05.465 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7760000000000003e-06 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.738983+0000", 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.021108608000000001 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.739014+0000", 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0979000000000003e-05 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.780143+0000", 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.041128888000000002 2024-02-10T07:51:05.466 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.467 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.467 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.467 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.780194+0000", 2024-02-10T07:51:05.467 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0754000000000002e-05 2024-02-10T07:51:05.467 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.467 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.467 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.467 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.467 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.467 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:82759 11.2 e409/405 11:4b75bab4:::benchmark_data_smithi008_186804_object82758:head v 409'5120, mlcod=397'355)", 2024-02-10T07:51:05.468 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:01.941479+0000", 2024-02-10T07:51:05.468 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 3.461685524, 2024-02-10T07:51:05.468 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.069110106000000004, 2024-02-10T07:51:05.468 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.468 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.468 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.468 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.468 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.468 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.941479+0000", 2024-02-10T07:51:05.468 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.468 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.468 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.468 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.469 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.941479+0000", 2024-02-10T07:51:05.469 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.469 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.469 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.469 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.469 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.941480+0000", 2024-02-10T07:51:05.469 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.187e-06 2024-02-10T07:51:05.469 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.469 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.469 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.469 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.941508+0000", 2024-02-10T07:51:05.470 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8050000000000001e-05 2024-02-10T07:51:05.470 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.470 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.470 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.470 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.941521+0000", 2024-02-10T07:51:05.470 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3010000000000001e-05 2024-02-10T07:51:05.470 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.470 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.470 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.470 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.941531+0000", 2024-02-10T07:51:05.470 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8330000000000006e-06 2024-02-10T07:51:05.470 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.470 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.470 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.471 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.960947+0000", 2024-02-10T07:51:05.471 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.019415822999999999 2024-02-10T07:51:05.471 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.471 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.471 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.471 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:01.960977+0000", 2024-02-10T07:51:05.471 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9930999999999999e-05 2024-02-10T07:51:05.471 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.471 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.471 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.471 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.010497+0000", 2024-02-10T07:51:05.472 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.049519658000000001 2024-02-10T07:51:05.472 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.472 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.472 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.472 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.010589+0000", 2024-02-10T07:51:05.472 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2614000000000005e-05 2024-02-10T07:51:05.472 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.472 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.472 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.472 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.472 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.472 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:83342 11.d e410/405 11:b817498d:::benchmark_data_smithi008_186804_object83341:head v 410'5211, mlcod=0'0)", 2024-02-10T07:51:05.473 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:02.802145+0000", 2024-02-10T07:51:05.473 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.6010200449999998, 2024-02-10T07:51:05.473 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.069608619999999996, 2024-02-10T07:51:05.473 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.473 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.473 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.473 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.473 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.473 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.802145+0000", 2024-02-10T07:51:05.473 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.473 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.473 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.474 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.474 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.802145+0000", 2024-02-10T07:51:05.474 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.474 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.474 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.474 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.474 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.802145+0000", 2024-02-10T07:51:05.474 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9500000000000002e-07 2024-02-10T07:51:05.474 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.474 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.474 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.475 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.802186+0000", 2024-02-10T07:51:05.475 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0399999999999999e-05 2024-02-10T07:51:05.475 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.475 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.475 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.475 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.802199+0000", 2024-02-10T07:51:05.475 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2880999999999999e-05 2024-02-10T07:51:05.475 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.475 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.475 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.475 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.802204+0000", 2024-02-10T07:51:05.476 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.925e-06 2024-02-10T07:51:05.476 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.476 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.476 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.476 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.802285+0000", 2024-02-10T07:51:05.476 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0622000000000002e-05 2024-02-10T07:51:05.476 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.476 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.476 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.476 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.802314+0000", 2024-02-10T07:51:05.477 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8801000000000001e-05 2024-02-10T07:51:05.477 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.477 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.477 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.477 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.871560+0000", 2024-02-10T07:51:05.477 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.069246153000000005 2024-02-10T07:51:05.477 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.477 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.477 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.477 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.871753+0000", 2024-02-10T07:51:05.477 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019324300000000001 2024-02-10T07:51:05.478 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.478 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.478 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.478 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.478 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.478 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:83346 11.d e410/405 11:b71dbd24:::benchmark_data_smithi008_186804_object83345:head v 410'5212, mlcod=0'0)", 2024-02-10T07:51:05.478 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:02.809722+0000", 2024-02-10T07:51:05.478 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.5934428949999999, 2024-02-10T07:51:05.478 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.070656900999999994, 2024-02-10T07:51:05.478 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.478 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.478 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.478 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.478 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.809722+0000", 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.809722+0000", 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.809723+0000", 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.091e-06 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.479 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.480 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.809747+0000", 2024-02-10T07:51:05.480 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4170000000000001e-05 2024-02-10T07:51:05.480 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.480 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.480 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.480 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.809760+0000", 2024-02-10T07:51:05.480 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2488e-05 2024-02-10T07:51:05.480 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.480 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.480 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.480 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.809768+0000", 2024-02-10T07:51:05.480 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5529999999999993e-06 2024-02-10T07:51:05.480 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.480 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.817313+0000", 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0075448310000000001 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.817335+0000", 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1721000000000001e-05 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.880326+0000", 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.062991426000000003 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.481 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.880379+0000", 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2621000000000003e-05 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:83349 11.d e410/405 11:b586b063:::benchmark_data_smithi008_186804_object83348:head v 410'5213, mlcod=0'0)", 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:02.817023+0000", 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 2.5861420979999998, 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.063388804000000007, 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.482 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.817023+0000", 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.817023+0000", 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.817023+0000", 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8799999999999998e-07 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.483 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.817050+0000", 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7396999999999999e-05 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.817059+0000", 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1499999999999999e-06 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.817065+0000", 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0009999999999998e-06 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.484 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.837483+0000", 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.020418380999999999 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.837515+0000", 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1785000000000001e-05 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.880382+0000", 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.042867290000000002 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.485 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:02.880411+0000", 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.9411999999999999e-05 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:84173 11.2 e410/405 11:4a6b0531:::benchmark_data_smithi008_186804_object84172:head v 410'5227, mlcod=397'387)", 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:04.109384+0000", 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 1.2937802299999999, 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.063931618999999995, 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.486 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.109384+0000", 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.109384+0000", 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.109385+0000", 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7000000000000004e-07 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.487 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.109411+0000", 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6188999999999999e-05 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.109426+0000", 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4472e-05 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.109433+0000", 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5290000000000003e-06 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.488 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.119280+0000", 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0098461729999999997 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.119303+0000", 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3572000000000001e-05 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.173089+0000", 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.053785359999999997 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.489 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.173316+0000", 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00022755400000000001 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:84371 11.4 e410/405 11:22ca1a73:::benchmark_data_smithi008_186804_object84370:head v 410'5403, mlcod=0'0)", 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:04.476948+0000", 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.926217136, 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.072828198999999996, 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.490 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.476948+0000", 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.476948+0000", 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.476949+0000", 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0100000000000001e-06 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.491 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.476983+0000", 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.4368e-05 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.477003+0000", 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.965e-05 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.477013+0000", 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0193e-05 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.492 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.477211+0000", 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019859700000000001 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.477278+0000", 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6684000000000001e-05 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.549727+0000", 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.072448811000000002 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.493 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.549776+0000", 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8885999999999999e-05 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:84373 11.4 e410/405 11:2397e640:::benchmark_data_smithi008_186804_object84372:head v 410'5404, mlcod=0'0)", 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:04.479361+0000", 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.923803296, 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.070475022999999998, 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.494 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.479361+0000", 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.479361+0000", 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.479362+0000", 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9199999999999999e-07 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:51:05.495 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.479407+0000", 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4574999999999999e-05 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.479423+0000", 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6089000000000001e-05 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.479432+0000", 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5839999999999995e-06 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:51:05.496 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.492292+0000", 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012860376999999999 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.492320+0000", 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.7945000000000001e-05 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.549812+0000", 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.057491958000000003 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:04.549836+0000", 2024-02-10T07:51:05.497 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4502999999999999e-05 2024-02-10T07:51:05.498 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.498 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.498 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.498 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:51:05.498 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:51:05.498 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:51:05.513 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:05.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:05.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:05.564 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:05 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:05.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:05 smithi008 ceph-mon[40743]: pgmap v887: 148 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 708 op/s; 8861/248535 objects degraded (3.565%); 16612/248535 objects misplaced (6.684%); 2.5 MiB/s, 40 objects/s recovering 2024-02-10T07:51:05.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:05 smithi008 ceph-mon[40743]: 3.16 scrub starts 2024-02-10T07:51:05.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:05 smithi008 ceph-mon[40743]: 3.16 scrub ok 2024-02-10T07:51:05.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:05 smithi008 ceph-mon[40743]: 11.3 scrub starts 2024-02-10T07:51:05.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:05 smithi008 ceph-mon[40743]: 11.a scrub ok 2024-02-10T07:51:05.565 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:05 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:05.565 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:05 smithi008 ceph-mon[29816]: pgmap v887: 148 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 708 op/s; 8861/248535 objects degraded (3.565%); 16612/248535 objects misplaced (6.684%); 2.5 MiB/s, 40 objects/s recovering 2024-02-10T07:51:05.565 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:05 smithi008 ceph-mon[29816]: 3.16 scrub starts 2024-02-10T07:51:05.565 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:05 smithi008 ceph-mon[29816]: 3.16 scrub ok 2024-02-10T07:51:05.565 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:05 smithi008 ceph-mon[29816]: 11.3 scrub starts 2024-02-10T07:51:05.565 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:05 smithi008 ceph-mon[29816]: 11.a scrub ok 2024-02-10T07:51:05.615 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:05 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:05 smithi170 ceph-mon[33394]: pgmap v887: 148 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 708 op/s; 8861/248535 objects degraded (3.565%); 16612/248535 objects misplaced (6.684%); 2.5 MiB/s, 40 objects/s recovering 2024-02-10T07:51:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:05 smithi170 ceph-mon[33394]: 3.16 scrub starts 2024-02-10T07:51:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:05 smithi170 ceph-mon[33394]: 3.16 scrub ok 2024-02-10T07:51:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:05 smithi170 ceph-mon[33394]: 11.3 scrub starts 2024-02-10T07:51:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:05 smithi170 ceph-mon[33394]: 11.a scrub ok 2024-02-10T07:51:05.658 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:51:05.659563+0000 min lat: 0.00135892 max lat: 2.4862 avg lat: 0.0112861 2024-02-10T07:51:05.659 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:51:05.659 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 60 16 85044 85028 88.5488 36.3125 0.0184314 0.0112861 2024-02-10T07:51:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:05.662 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:05.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:05.763 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:05.799 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:05.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:05.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:05.901 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:05.929 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:05.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:05.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:06.030 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:06.060 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:06.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:06.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:06.162 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:06.188 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:06.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:06.290 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:06.321 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:06.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:06.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:06.375 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:51:06.423 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:06.452 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:06.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:06.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:06.554 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:06.595 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:06.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:06.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:06 smithi170 ceph-mon[33394]: 3.1e scrub starts 2024-02-10T07:51:06.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:06 smithi170 ceph-mon[33394]: 3.1e scrub ok 2024-02-10T07:51:06.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:06 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2324750811' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:51:06.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:06 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:51:06.659 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 61 16 85715 85699 87.7847 41.9375 0.0331708 0.0113856 2024-02-10T07:51:06.697 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:06.732 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:06.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:06.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:06 smithi008 ceph-mon[29816]: 3.1e scrub starts 2024-02-10T07:51:06.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:06 smithi008 ceph-mon[29816]: 3.1e scrub ok 2024-02-10T07:51:06.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:06 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2324750811' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:51:06.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:06 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:51:06.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:06 smithi008 ceph-mon[40743]: 3.1e scrub starts 2024-02-10T07:51:06.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:06 smithi008 ceph-mon[40743]: 3.1e scrub ok 2024-02-10T07:51:06.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:06 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2324750811' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:51:06.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:06 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:51:06.834 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:06.864 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:06.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:06.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:06.966 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:07.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:07.005 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:07.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:07.106 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:07.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:07.150 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:07.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:07.251 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:07.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:07.292 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:07.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:07.393 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:07.401 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-10T07:51:07.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:07.430 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:07.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:07.532 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:07.574 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:07.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:07 smithi170 ceph-mon[33394]: 11.3 scrub ok 2024-02-10T07:51:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:07 smithi170 ceph-mon[33394]: pgmap v888: 148 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 736 op/s; 8861/252276 objects degraded (3.512%); 16322/252276 objects misplaced (6.470%); 4.2 MiB/s, 67 objects/s recovering 2024-02-10T07:51:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:07 smithi170 ceph-mon[33394]: 11.8 scrub starts 2024-02-10T07:51:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:07 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:51:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:07 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:51:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:07 smithi170 ceph-mon[33394]: osdmap e411: 8 total, 8 up, 8 in 2024-02-10T07:51:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:07 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/252276 objects degraded (3.512%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:07 smithi170 ceph-mon[33394]: osdmap e412: 8 total, 8 up, 8 in 2024-02-10T07:51:07.659 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 62 16 86408 86392 87.0675 43.3125 0.00669607 0.0114792 2024-02-10T07:51:07.676 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:07.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:07.710 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:07.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:07.811 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:07.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[29816]: 11.3 scrub ok 2024-02-10T07:51:07.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[29816]: pgmap v888: 148 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 736 op/s; 8861/252276 objects degraded (3.512%); 16322/252276 objects misplaced (6.470%); 4.2 MiB/s, 67 objects/s recovering 2024-02-10T07:51:07.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[29816]: 11.8 scrub starts 2024-02-10T07:51:07.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:51:07.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:51:07.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[29816]: osdmap e411: 8 total, 8 up, 8 in 2024-02-10T07:51:07.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/252276 objects degraded (3.512%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:07.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[29816]: osdmap e412: 8 total, 8 up, 8 in 2024-02-10T07:51:07.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[40743]: 11.3 scrub ok 2024-02-10T07:51:07.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[40743]: pgmap v888: 148 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 736 op/s; 8861/252276 objects degraded (3.512%); 16322/252276 objects misplaced (6.470%); 4.2 MiB/s, 67 objects/s recovering 2024-02-10T07:51:07.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[40743]: 11.8 scrub starts 2024-02-10T07:51:07.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:51:07.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:51:07.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[40743]: osdmap e411: 8 total, 8 up, 8 in 2024-02-10T07:51:07.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/252276 objects degraded (3.512%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:07.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:07 smithi008 ceph-mon[40743]: osdmap e412: 8 total, 8 up, 8 in 2024-02-10T07:51:07.888 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:07.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:07.991 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:08.036 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:08.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:08.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:08.056 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:51:08.138 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:08.175 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:08.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:08.276 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:08.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:08.322 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:08.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:08.350 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:51:08.423 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:08.454 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:08.454 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:08 smithi170 ceph-mon[33394]: 11.1 scrub starts 2024-02-10T07:51:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:08.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:08.556 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:08.612 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:08.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:08.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:08.659 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 63 16 87048 87032 86.3198 40 0.0506586 0.011579 2024-02-10T07:51:08.683 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:08 smithi008 ceph-mon[40743]: 11.1 scrub starts 2024-02-10T07:51:08.684 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:08 smithi008 ceph-mon[29816]: 11.1 scrub starts 2024-02-10T07:51:08.714 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:08.765 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:08.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:08.866 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:08.914 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:08.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:08.978 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:51:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:51:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:51:09.016 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:09.053 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:09.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:09.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:09.155 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:09.186 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:09.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:09.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:09.287 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:09.317 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:09.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:09.419 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:09.462 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:09.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:09.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:09.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:09 smithi170 ceph-mon[33394]: 11.8 scrub ok 2024-02-10T07:51:09.542 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:09 smithi170 ceph-mon[33394]: pgmap v891: 148 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 838 B/s rd, 35 MiB/s wr, 554 op/s; 8861/252870 objects degraded (3.504%); 16322/252870 objects misplaced (6.455%); 3.0 MiB/s, 47 objects/s recovering 2024-02-10T07:51:09.563 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:09.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:09.609 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:09.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:09.659 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 64 16 87760 87744 85.6664 44.5 0.0169192 0.011667 2024-02-10T07:51:09.710 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:09.749 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:09.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:09.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:09.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:09 smithi008 ceph-mon[29816]: 11.8 scrub ok 2024-02-10T07:51:09.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:09 smithi008 ceph-mon[29816]: pgmap v891: 148 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 838 B/s rd, 35 MiB/s wr, 554 op/s; 8861/252870 objects degraded (3.504%); 16322/252870 objects misplaced (6.455%); 3.0 MiB/s, 47 objects/s recovering 2024-02-10T07:51:09.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:09 smithi008 ceph-mon[40743]: 11.8 scrub ok 2024-02-10T07:51:09.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:09 smithi008 ceph-mon[40743]: pgmap v891: 148 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 838 B/s rd, 35 MiB/s wr, 554 op/s; 8861/252870 objects degraded (3.504%); 16322/252870 objects misplaced (6.455%); 3.0 MiB/s, 47 objects/s recovering 2024-02-10T07:51:09.851 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:09.882 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:09.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:09.890 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:51:09.983 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:10.047 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:10.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:10.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:10.149 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:10.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:10.241 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:10.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:10.342 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:10.373 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:10.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:10.475 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:10.503 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:10.504 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:10 smithi008 ceph-mon[29816]: 11.1 scrub ok 2024-02-10T07:51:10.505 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:10 smithi008 ceph-mon[40743]: 11.1 scrub ok 2024-02-10T07:51:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:10.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:10.606 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:10 smithi170 ceph-mon[33394]: 11.1 scrub ok 2024-02-10T07:51:10.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:10.634 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:10.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:10.659 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 65 16 88399 88383 84.9629 39.9375 0.00983573 0.0117639 2024-02-10T07:51:10.736 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:10.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:10.766 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:10.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:10.867 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:10.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:10.893 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:10.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:10.995 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:11.023 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:11.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:11.126 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:11.136 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:51:11.150 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:11.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:11.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:11.181 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:51:11.252 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:51:11.252 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:11.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:11.278 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:11.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:11.300 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:51:11.355 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:51:11.378 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:11.399 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:51:11.422 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:11.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:11.428 INFO:teuthology.orchestra.run.smithi008.stderr:set pool 11 pgp_num to 16 2024-02-10T07:51:11.439 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:51:11.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[40743]: pgmap v892: 148 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 793 op/s; 8861/261747 objects degraded (3.385%); 16020/261747 objects misplaced (6.120%); 4.6 MiB/s, 73 objects/s recovering 2024-02-10T07:51:11.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[40743]: 4.a scrub starts 2024-02-10T07:51:11.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[40743]: 4.a scrub ok 2024-02-10T07:51:11.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:51:11.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:51:11.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1593096894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:51:11.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/455303439' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-10T07:51:11.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-10T07:51:11.441 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[29816]: pgmap v892: 148 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 793 op/s; 8861/261747 objects degraded (3.385%); 16020/261747 objects misplaced (6.120%); 4.6 MiB/s, 73 objects/s recovering 2024-02-10T07:51:11.441 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[29816]: 4.a scrub starts 2024-02-10T07:51:11.441 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[29816]: 4.a scrub ok 2024-02-10T07:51:11.441 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:51:11.441 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:51:11.442 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1593096894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:51:11.442 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/455303439' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-10T07:51:11.442 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:11 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-10T07:51:11.479 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:51:11.524 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:11.531 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:51:11.548 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:51:11.548 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:51:11.549 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:51:11.549 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:51:11.549 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:51:11.549 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:51:11.549 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:51:11.549 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:51:11.551 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:11.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:11.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:11.657 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:11.660 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 66 16 88952 88936 84.1993 34.5625 0.029382 0.0118678 2024-02-10T07:51:11.702 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:11.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:11.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:11.804 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:11.831 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:11.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:11.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:11.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:11 smithi170 ceph-mon[33394]: pgmap v892: 148 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 793 op/s; 8861/261747 objects degraded (3.385%); 16020/261747 objects misplaced (6.120%); 4.6 MiB/s, 73 objects/s recovering 2024-02-10T07:51:11.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:11 smithi170 ceph-mon[33394]: 4.a scrub starts 2024-02-10T07:51:11.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:11 smithi170 ceph-mon[33394]: 4.a scrub ok 2024-02-10T07:51:11.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:11 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:51:11.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:11 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:51:11.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:11 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1593096894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:51:11.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:11 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/455303439' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-10T07:51:11.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:11 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-10T07:51:11.933 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:11.960 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:11.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:11.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:12.062 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:12.105 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:12.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:12.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:12.207 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:12.233 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:51:12.233 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:51:12.239 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:12.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:12.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:12.341 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:12.373 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:12.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:12.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:12.474 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:12.504 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:12.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:12.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:12.606 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:12 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-02-10T07:51:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:12 smithi170 ceph-mon[33394]: osdmap e413: 8 total, 8 up, 8 in 2024-02-10T07:51:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:12 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/261747 objects degraded (3.385%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:12 smithi170 ceph-mon[33394]: osdmap e414: 8 total, 8 up, 8 in 2024-02-10T07:51:12.648 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:12.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:12.660 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 67 16 89671 89655 83.6134 44.9375 0.00335561 0.0119533 2024-02-10T07:51:12.742 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:12 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-02-10T07:51:12.742 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:12 smithi008 ceph-mon[29816]: osdmap e413: 8 total, 8 up, 8 in 2024-02-10T07:51:12.742 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:12 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/261747 objects degraded (3.385%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:12.742 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:12 smithi008 ceph-mon[29816]: osdmap e414: 8 total, 8 up, 8 in 2024-02-10T07:51:12.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:12 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-02-10T07:51:12.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:12 smithi008 ceph-mon[40743]: osdmap e413: 8 total, 8 up, 8 in 2024-02-10T07:51:12.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:12 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/261747 objects degraded (3.385%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:12.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:12 smithi008 ceph-mon[40743]: osdmap e414: 8 total, 8 up, 8 in 2024-02-10T07:51:12.750 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:12.811 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:12.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:12.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:12.913 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:12.942 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:12.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:13.044 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:13.084 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:13.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:13.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:13.186 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:13.230 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:13.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:13.332 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:13.359 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:13.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:13.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:13.461 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:13.508 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:13.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:13.610 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:13 smithi170 ceph-mon[33394]: pgmap v895: 148 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 887 B/s rd, 36 MiB/s wr, 583 op/s; 8861/262356 objects degraded (3.377%); 16020/262356 objects misplaced (6.106%); 3.3 MiB/s, 52 objects/s recovering 2024-02-10T07:51:13.659 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:13.660 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 68 16 90379 90363 83.0345 44.25 0.030224 0.0120365 2024-02-10T07:51:13.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:13.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:13.762 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:13.790 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:13.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:13.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:13 smithi008 ceph-mon[29816]: pgmap v895: 148 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 887 B/s rd, 36 MiB/s wr, 583 op/s; 8861/262356 objects degraded (3.377%); 16020/262356 objects misplaced (6.106%); 3.3 MiB/s, 52 objects/s recovering 2024-02-10T07:51:13.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:13 smithi008 ceph-mon[40743]: pgmap v895: 148 pgs: 1 active+recovery_wait+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 5.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 887 B/s rd, 36 MiB/s wr, 583 op/s; 8861/262356 objects degraded (3.377%); 16020/262356 objects misplaced (6.106%); 3.3 MiB/s, 52 objects/s recovering 2024-02-10T07:51:13.892 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:13.926 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-10T07:51:13.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:13.929 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:13.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:14.030 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:14.062 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:14.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:14.112 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:51:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:51:14.164 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:14.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:14.195 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:14.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:14.263 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:51:14.296 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:14.338 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:14.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:14.439 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:14.481 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:14.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:14.582 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:14.613 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:14.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:14.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:14.660 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 69 16 91073 91057 82.4598 43.375 0.051081 0.0121211 2024-02-10T07:51:14.715 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:14.736 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:14 smithi170 ceph-mon[33394]: 5.1f deep-scrub starts 2024-02-10T07:51:14.737 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:14 smithi170 ceph-mon[33394]: 5.1f deep-scrub ok 2024-02-10T07:51:14.743 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:14.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:14.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:14 smithi008 ceph-mon[29816]: 5.1f deep-scrub starts 2024-02-10T07:51:14.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:14 smithi008 ceph-mon[29816]: 5.1f deep-scrub ok 2024-02-10T07:51:14.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:14 smithi008 ceph-mon[40743]: 5.1f deep-scrub starts 2024-02-10T07:51:14.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:14 smithi008 ceph-mon[40743]: 5.1f deep-scrub ok 2024-02-10T07:51:14.847 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:14.877 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:14.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:14.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:14.979 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:15.011 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:15.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:15.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:15.114 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:15.142 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:15.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:15.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:15.244 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:15.269 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:15.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:15.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:15.273 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:51:15.372 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:15.398 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:15.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:15.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:15.501 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:15.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:15 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:15.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:15 smithi008 ceph-mon[29816]: pgmap v896: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 44 MiB/s wr, 708 op/s; 8861/267948 objects degraded (3.307%); 16020/267948 objects misplaced (5.979%); 2.7 MiB/s, 42 objects/s recovering 2024-02-10T07:51:15.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:15 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:15.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:15 smithi008 ceph-mon[40743]: pgmap v896: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 44 MiB/s wr, 708 op/s; 8861/267948 objects degraded (3.307%); 16020/267948 objects misplaced (5.979%); 2.7 MiB/s, 42 objects/s recovering 2024-02-10T07:51:15.609 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:15.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:15.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:15.660 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 70 16 91718 91702 81.8575 40.3125 0.00962974 0.0122107 2024-02-10T07:51:15.711 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:15.737 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:15 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:15.737 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:15 smithi170 ceph-mon[33394]: pgmap v896: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 44 MiB/s wr, 708 op/s; 8861/267948 objects degraded (3.307%); 16020/267948 objects misplaced (5.979%); 2.7 MiB/s, 42 objects/s recovering 2024-02-10T07:51:15.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:15.775 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:15.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:15.778 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:51:15.876 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:15.911 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:15.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:16.013 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:16.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:16.047 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:16.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:16.148 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:16.183 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:16.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:16.284 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:16.328 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:16.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:16.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:16.429 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:16.452 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:51:16.462 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:16.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:16.469 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:16 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3226641772' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:51:16.469 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:16 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3226641772' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:51:16.564 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:16.592 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:16.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:16.660 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 71 16 92370 92354 81.2786 40.75 0.00679472 0.0122974 2024-02-10T07:51:16.694 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:16.724 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:16.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:16.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:16.826 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:16.857 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:16.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:16.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:16 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3226641772' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:51:16.959 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:16.998 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:16.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:16.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:17.100 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:17.143 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:17.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:17.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:17.245 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:17.286 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:17.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:17.387 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:17.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:17.414 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:17.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:17.515 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:17.558 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:17.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:17.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:17.661 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 72 16 93061 93045 80.7496 43.1875 0.0342003 0.0123765 2024-02-10T07:51:17.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:17.695 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:17.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:17.795 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 6, 7, 5, 3] 2024-02-10T07:51:17.795 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 0 1 2024-02-10T07:51:17.795 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:17.825 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:17.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:17.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:17 smithi008 ceph-mon[29816]: pgmap v897: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 788 op/s; 8861/271770 objects degraded (3.260%); 15714/271770 objects misplaced (5.782%); 4.7 MiB/s, 75 objects/s recovering 2024-02-10T07:51:17.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:17 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:51:17.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:17 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3226641772' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:51:17.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:17 smithi008 ceph-mon[29816]: osdmap e415: 8 total, 8 up, 8 in 2024-02-10T07:51:17.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:17 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/271770 objects degraded (3.260%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:17.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:17 smithi008 ceph-mon[29816]: osdmap e416: 8 total, 8 up, 8 in 2024-02-10T07:51:17.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:17 smithi008 ceph-mon[40743]: pgmap v897: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 788 op/s; 8861/271770 objects degraded (3.260%); 15714/271770 objects misplaced (5.782%); 4.7 MiB/s, 75 objects/s recovering 2024-02-10T07:51:17.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:17 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:51:17.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:17 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3226641772' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:51:17.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:17 smithi008 ceph-mon[40743]: osdmap e415: 8 total, 8 up, 8 in 2024-02-10T07:51:17.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:17 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/271770 objects degraded (3.260%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:17.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:17 smithi008 ceph-mon[40743]: osdmap e416: 8 total, 8 up, 8 in 2024-02-10T07:51:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:17 smithi170 ceph-mon[33394]: pgmap v897: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 788 op/s; 8861/271770 objects degraded (3.260%); 15714/271770 objects misplaced (5.782%); 4.7 MiB/s, 75 objects/s recovering 2024-02-10T07:51:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:17 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:51:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:17 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3226641772' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:51:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:17 smithi170 ceph-mon[33394]: osdmap e415: 8 total, 8 up, 8 in 2024-02-10T07:51:17.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:17 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/271770 objects degraded (3.260%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:17.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:17 smithi170 ceph-mon[33394]: osdmap e416: 8 total, 8 up, 8 in 2024-02-10T07:51:17.927 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:17.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:17.977 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:17.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:18.078 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:18.106 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:18.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:18.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:18.207 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:18.235 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:18.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:18.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:18.337 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:18.370 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:18.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:18.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:18.472 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:18.502 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:18.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:18.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:18 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:51:18.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:18 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:51:18.603 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:18.631 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:18.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:18.660 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 73 16 93714 93698 80.2025 40.8125 0.00851867 0.0124623 2024-02-10T07:51:18.733 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:18.762 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:18.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:18.864 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:18.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:18 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:51:18.915 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:18.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:18.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:19.017 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:19.053 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:19.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:19.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:51:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:51:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:51:19.155 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:19.185 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:19.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:19.286 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:19.333 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:19.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:19.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:19.435 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:19.472 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:19.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:19.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:19.574 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:19.602 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:19.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:19.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:19.661 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 74 16 94434 94418 79.7268 45 0.0124554 0.0125367 2024-02-10T07:51:19.704 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:19.739 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:19.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:19.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:19 smithi008 ceph-mon[29816]: pgmap v900: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 838 B/s rd, 34 MiB/s wr, 551 op/s; 8861/272439 objects degraded (3.252%); 15714/272439 objects misplaced (5.768%); 3.1 MiB/s, 50 objects/s recovering 2024-02-10T07:51:19.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:19 smithi008 ceph-mon[40743]: pgmap v900: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 838 B/s rd, 34 MiB/s wr, 551 op/s; 8861/272439 objects degraded (3.252%); 15714/272439 objects misplaced (5.768%); 3.1 MiB/s, 50 objects/s recovering 2024-02-10T07:51:19.840 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:19.854 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:51:19.854 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:51:19.855 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:19 smithi170 ceph-mon[33394]: pgmap v900: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 838 B/s rd, 34 MiB/s wr, 551 op/s; 8861/272439 objects degraded (3.252%); 15714/272439 objects misplaced (5.768%); 3.1 MiB/s, 50 objects/s recovering 2024-02-10T07:51:19.870 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:19.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:19.972 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:20.003 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:20.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:20.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:20.104 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:20.130 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:20.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:20.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:20.231 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:20.257 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:20.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:20.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:20.359 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:20.400 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:20.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:20.502 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:20.551 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:20.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:20.652 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:20.661 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 75 16 95088 95072 79.2088 40.875 0.00872128 0.0126166 2024-02-10T07:51:20.681 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:20.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:20.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:20.783 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:20.808 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:20.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:20.910 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:20.953 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:20.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:20.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:21.055 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:21.098 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:21.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:21.200 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:21.229 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:21.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:21.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:21.330 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:21.374 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:21.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:21.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:21.476 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:21.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:21.518 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:21.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:21.619 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:21.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:21.657 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:21.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:21.661 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 76 16 95735 95719 78.6986 40.4375 0.0254743 0.0126994 2024-02-10T07:51:21.758 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:21.784 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:21.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:21.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:21.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:21 smithi008 ceph-mon[29816]: pgmap v901: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 813 op/s; 8861/281850 objects degraded (3.144%); 15406/281850 objects misplaced (5.466%); 4.8 MiB/s, 76 objects/s recovering 2024-02-10T07:51:21.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:21 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:51:21.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:21 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:51:21.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:21 smithi008 ceph-mon[40743]: pgmap v901: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 813 op/s; 8861/281850 objects degraded (3.144%); 15406/281850 objects misplaced (5.466%); 4.8 MiB/s, 76 objects/s recovering 2024-02-10T07:51:21.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:21 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:51:21.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:21 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:51:21.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:21 smithi170 ceph-mon[33394]: pgmap v901: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 813 op/s; 8861/281850 objects degraded (3.144%); 15406/281850 objects misplaced (5.466%); 4.8 MiB/s, 76 objects/s recovering 2024-02-10T07:51:21.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:21 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:51:21.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:21 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:51:21.886 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:21.910 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:21.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:21.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:21.937 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-10T07:51:22.012 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:22.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:22.058 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:22.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:22.159 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:22.195 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:22.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:22.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:22.297 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:22.304 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:51:22.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:22.338 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:22.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:22.439 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:22.478 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:22.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:22.579 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:22.617 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:22.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:22.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:22.661 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 77 16 96338 96322 78.166 37.6875 0.00767148 0.0127883 2024-02-10T07:51:22.719 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:22.720 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:22 smithi170 ceph-mon[33394]: osdmap e417: 8 total, 8 up, 8 in 2024-02-10T07:51:22.720 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:22 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/281850 objects degraded (3.144%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:22.720 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:22 smithi170 ceph-mon[33394]: osdmap e418: 8 total, 8 up, 8 in 2024-02-10T07:51:22.766 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:22.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:22.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:22 smithi008 ceph-mon[29816]: osdmap e417: 8 total, 8 up, 8 in 2024-02-10T07:51:22.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:22 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/281850 objects degraded (3.144%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:22.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:22 smithi008 ceph-mon[29816]: osdmap e418: 8 total, 8 up, 8 in 2024-02-10T07:51:22.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:22 smithi008 ceph-mon[40743]: osdmap e417: 8 total, 8 up, 8 in 2024-02-10T07:51:22.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:22 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/281850 objects degraded (3.144%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:22.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:22 smithi008 ceph-mon[40743]: osdmap e418: 8 total, 8 up, 8 in 2024-02-10T07:51:22.868 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:22.892 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:22.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:22.994 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:23.023 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:23.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:23.125 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:23.152 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:23.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:23.254 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:23.279 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:23.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:23.380 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:23.421 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:23.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:23.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:23.523 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:23.550 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:23.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:23.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:23.653 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:23.661 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 78 16 96851 96835 77.5746 32.0625 0.0353038 0.0128827 2024-02-10T07:51:23.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:23.702 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:23.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:23.743 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:23 smithi170 ceph-mon[33394]: pgmap v904: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 899 B/s rd, 37 MiB/s wr, 587 op/s; 8861/282456 objects degraded (3.137%); 15406/282456 objects misplaced (5.454%); 3.4 MiB/s, 54 objects/s recovering 2024-02-10T07:51:23.803 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:23 smithi008 ceph-mon[29816]: pgmap v904: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 899 B/s rd, 37 MiB/s wr, 587 op/s; 8861/282456 objects degraded (3.137%); 15406/282456 objects misplaced (5.454%); 3.4 MiB/s, 54 objects/s recovering 2024-02-10T07:51:23.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:23 smithi008 ceph-mon[40743]: pgmap v904: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 899 B/s rd, 37 MiB/s wr, 587 op/s; 8861/282456 objects degraded (3.137%); 15406/282456 objects misplaced (5.454%); 3.4 MiB/s, 54 objects/s recovering 2024-02-10T07:51:23.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:23.834 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:23.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:23.936 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:23.976 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:23.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:23.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:24.077 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:24.080 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:51:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:51:24.101 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:24.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:24.203 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:24.249 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:24.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:24.255 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:51:24.255 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:51:24.351 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:24.379 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:24.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:24.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:24.482 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:24.512 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:24.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:24.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:24.614 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:24.662 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 79 16 97513 97497 77.1163 41.375 0.00429226 0.0129592 2024-02-10T07:51:24.668 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:24.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:24.770 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:24.792 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:51:24.839 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:24.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:24.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:24.941 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:24.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:24.979 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:24.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:25.080 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:25.104 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:25.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:25.162 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:51:25.206 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:25.233 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:25.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:25.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:25.335 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:25.357 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-10T07:51:25.392 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:25.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:25.494 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:25.543 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:25.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:25.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:25.645 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:25.662 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:2024-02-10T07:51:25.663256+0000 min lat: 0.00135892 max lat: 2.4862 avg lat: 0.0130548 2024-02-10T07:51:25.662 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-10T07:51:25.663 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 80 16 98037 98021 76.5611 32.75 0.0525265 0.0130548 2024-02-10T07:51:25.671 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:51:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:25.688 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:25.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:25.790 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:25.803 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:25 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:25.803 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:25 smithi170 ceph-mon[33394]: pgmap v905: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 45 MiB/s wr, 718 op/s; 8861/287736 objects degraded (3.080%); 15406/287736 objects misplaced (5.354%); 2.7 MiB/s, 43 objects/s recovering 2024-02-10T07:51:25.831 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:25.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:25.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:25 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:25.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:25 smithi008 ceph-mon[29816]: pgmap v905: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 45 MiB/s wr, 718 op/s; 8861/287736 objects degraded (3.080%); 15406/287736 objects misplaced (5.354%); 2.7 MiB/s, 43 objects/s recovering 2024-02-10T07:51:25.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:25 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:25.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:25 smithi008 ceph-mon[40743]: pgmap v905: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.4 KiB/s rd, 45 MiB/s wr, 718 op/s; 8861/287736 objects degraded (3.080%); 15406/287736 objects misplaced (5.354%); 2.7 MiB/s, 43 objects/s recovering 2024-02-10T07:51:25.933 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:25.959 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:25.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:25.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:26.061 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:26.093 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:26.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:26.194 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:26.228 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:26.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:26.328 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:26.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:26.381 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:26.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:26.482 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:26.530 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:26.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:26.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:26.632 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:26.660 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:26.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:26.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:26.662 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 81 16 98768 98752 76.1799 45.6875 0.00385669 0.0131195 2024-02-10T07:51:26.762 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:26.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:26.828 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:26.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:26.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:26.973 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:26.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:26.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:27.075 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:27.100 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:27.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:27.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:27.202 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:27.231 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:27.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:27.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:27.333 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:27.375 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:27.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:27.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:27.477 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:27.508 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:27.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:27.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:27.610 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:27 smithi170 ceph-mon[33394]: pgmap v906: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 788 op/s; 8861/291333 objects degraded (3.042%); 15099/291333 objects misplaced (5.183%); 4.8 MiB/s, 76 objects/s recovering 2024-02-10T07:51:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:27 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/291333 objects degraded (3.042%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:27.638 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:27.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:27.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:27.662 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 82 15 99468 99453 75.7852 43.8125 0.0129011 0.0131887 2024-02-10T07:51:27.740 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:27.774 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:27.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:27.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:27.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:27 smithi008 ceph-mon[29816]: pgmap v906: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 788 op/s; 8861/291333 objects degraded (3.042%); 15099/291333 objects misplaced (5.183%); 4.8 MiB/s, 76 objects/s recovering 2024-02-10T07:51:27.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:27 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/291333 objects degraded (3.042%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:27.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:27 smithi008 ceph-mon[40743]: pgmap v906: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 788 op/s; 8861/291333 objects degraded (3.042%); 15099/291333 objects misplaced (5.183%); 4.8 MiB/s, 76 objects/s recovering 2024-02-10T07:51:27.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:27 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/291333 objects degraded (3.042%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:27.876 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:27.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:27.915 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:27.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:28.011 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:51:28.011 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:51:28.016 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:28.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:28.055 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:28.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:28.156 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:28.183 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:28.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:28.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:28.285 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:28.333 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:28.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:28.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:28.434 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:28.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:28.476 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:28.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:28.577 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:28.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:28.614 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:28.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:28.663 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 83 16 100180 100164 75.4075 44.4375 0.0205102 0.0132551 2024-02-10T07:51:28.677 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-10T07:51:28.714 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:28.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:28.759 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:28.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:28.861 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:28.892 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:28.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:28.993 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:29.021 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:51:29.046 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:29.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:29.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:29.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:51:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:51:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:51:29.148 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:29.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:29.177 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:29.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:29.278 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:29.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:29.307 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:29.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:29.409 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:29.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:29.449 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:29.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:29.550 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:29.596 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:29.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:29.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:29.677 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 84 16 100886 100870 75.0351 44.125 0.0157502 0.0133206 2024-02-10T07:51:29.699 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:29.729 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:29.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:29.829 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:29.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:29 smithi008 ceph-mon[29816]: pgmap v907: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 26 MiB/s wr, 417 op/s; 8861/291870 objects degraded (3.036%); 15099/291870 objects misplaced (5.173%); 2.4 MiB/s, 38 objects/s recovering 2024-02-10T07:51:29.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:29 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:51:29.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:29 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:51:29.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:29 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:51:29.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:29 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:51:29.839 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:29 smithi008 ceph-mon[40743]: pgmap v907: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 26 MiB/s wr, 417 op/s; 8861/291870 objects degraded (3.036%); 15099/291870 objects misplaced (5.173%); 2.4 MiB/s, 38 objects/s recovering 2024-02-10T07:51:29.839 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:29 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:51:29.839 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:29 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:51:29.839 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:29 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:51:29.839 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:29 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:51:29.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:29.862 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:29.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:29.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:29 smithi170 ceph-mon[33394]: pgmap v907: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 26 MiB/s wr, 417 op/s; 8861/291870 objects degraded (3.036%); 15099/291870 objects misplaced (5.173%); 2.4 MiB/s, 38 objects/s recovering 2024-02-10T07:51:29.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:29 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:51:29.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:29 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:51:29.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:29 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:51:29.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:29 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:51:29.963 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:29.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:29.995 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:29.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:30.096 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:30.130 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:30.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:30.232 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:30.272 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:30.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:30.374 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:30.412 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:30.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:30.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:30.514 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:30.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:30.553 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:30.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:30.654 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:30.663 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 85 16 101590 101574 74.6699 44 0.0159217 0.0133855 2024-02-10T07:51:30.690 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:30.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:30.740 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:30 smithi170 ceph-mon[33394]: osdmap e419: 8 total, 8 up, 8 in 2024-02-10T07:51:30.792 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:30.828 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:30.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:30.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:30.831 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:30 smithi008 ceph-mon[40743]: osdmap e419: 8 total, 8 up, 8 in 2024-02-10T07:51:30.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:30 smithi008 ceph-mon[29816]: osdmap e419: 8 total, 8 up, 8 in 2024-02-10T07:51:30.930 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:30.987 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:30.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:30.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:31.089 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:31.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:31.121 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:31.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:31.223 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:31.234 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:51:31.234 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:51:31.272 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:31.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:31.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:31.375 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:31.404 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:31.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:31.506 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:31.566 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:31.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:31.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:31.666 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 86 16 102250 102234 74.2786 41.25 0.0369881 0.0134547 2024-02-10T07:51:31.669 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:31.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:31.698 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:31.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:31.799 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:31.820 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:31 smithi008 ceph-mon[29816]: pgmap v909: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 48 MiB/s wr, 769 op/s; 8861/301149 objects degraded (2.942%); 14731/301149 objects misplaced (4.892%); 5.2 MiB/s, 83 objects/s recovering 2024-02-10T07:51:31.821 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:31 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:51:31.824 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:31 smithi008 ceph-mon[40743]: pgmap v909: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 48 MiB/s wr, 769 op/s; 8861/301149 objects degraded (2.942%); 14731/301149 objects misplaced (4.892%); 5.2 MiB/s, 83 objects/s recovering 2024-02-10T07:51:31.824 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:31 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:51:31.865 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:51:31.883 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:31.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:31 smithi170 ceph-mon[33394]: pgmap v909: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 48 MiB/s wr, 769 op/s; 8861/301149 objects degraded (2.942%); 14731/301149 objects misplaced (4.892%); 5.2 MiB/s, 83 objects/s recovering 2024-02-10T07:51:31.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:31 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:51:31.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:31.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:31.936 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:51:31.950 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-10T07:51:31.985 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:31.988 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:51:32.033 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:32.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:32.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:32.045 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:51:32.052 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:51:32.100 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:51:32.135 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:32.147 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:51:32.162 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:32.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:32.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:32.196 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:51:32.255 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:51:32.264 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:32.269 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:51:32.276 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:51:32.276 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:51:32.276 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:51:32.276 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:51:32.276 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:51:32.276 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:51:32.276 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:51:32.277 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:51:32.292 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:32.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:32.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:32.394 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:32.465 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:32.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:32.567 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:32.599 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:32.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:32.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:32.650 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:32 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4118352555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:51:32.650 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:32 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/301149 objects degraded (2.942%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:32.650 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:32 smithi170 ceph-mon[33394]: osdmap e420: 8 total, 8 up, 8 in 2024-02-10T07:51:32.650 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:32 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/493728839' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:51:32.651 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:32 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:51:32.666 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 87 16 102743 102727 73.779 30.8125 0.0567878 0.0135474 2024-02-10T07:51:32.702 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:32.753 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:32.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:32.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:32.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:32 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4118352555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:51:32.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:32 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/301149 objects degraded (2.942%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:32.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:32 smithi008 ceph-mon[29816]: osdmap e420: 8 total, 8 up, 8 in 2024-02-10T07:51:32.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:32 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/493728839' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:51:32.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:32 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:51:32.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:32 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4118352555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:51:32.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:32 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/301149 objects degraded (2.942%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:32.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:32 smithi008 ceph-mon[40743]: osdmap e420: 8 total, 8 up, 8 in 2024-02-10T07:51:32.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:32 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/493728839' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:51:32.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:32 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:51:32.854 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:32.902 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:32.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:33.002 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:33.044 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:33.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:33.068 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.0 to 1 (10000) 2024-02-10T07:51:33.147 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:33.173 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:33.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:33.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:33.275 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:33.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:33.324 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:33.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:33.425 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:33.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:33.465 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:33.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:33.566 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:33.610 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:33.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:33.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:33.668 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 88 15 103432 103417 73.4307 43.125 0.0147245 0.0136119 2024-02-10T07:51:33.711 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:33.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:33.743 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:33.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:33.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:33 smithi008 ceph-mon[29816]: pgmap v911: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 37 MiB/s wr, 586 op/s; 8861/301815 objects degraded (2.936%); 14731/301815 objects misplaced (4.881%); 5.3 MiB/s, 84 objects/s recovering 2024-02-10T07:51:33.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:51:33.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:51:33.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:51:33.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:33 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:51:33.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:33 smithi008 ceph-mon[29816]: osdmap e421: 8 total, 8 up, 8 in 2024-02-10T07:51:33.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:51:33.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:33 smithi008 ceph-mon[40743]: pgmap v911: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 37 MiB/s wr, 586 op/s; 8861/301815 objects degraded (2.936%); 14731/301815 objects misplaced (4.881%); 5.3 MiB/s, 84 objects/s recovering 2024-02-10T07:51:33.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:51:33.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:51:33.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:51:33.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:33 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:51:33.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:33 smithi008 ceph-mon[40743]: osdmap e421: 8 total, 8 up, 8 in 2024-02-10T07:51:33.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:51:33.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:33 smithi170 ceph-mon[33394]: pgmap v911: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 37 MiB/s wr, 586 op/s; 8861/301815 objects degraded (2.936%); 14731/301815 objects misplaced (4.881%); 5.3 MiB/s, 84 objects/s recovering 2024-02-10T07:51:33.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:51:33.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:51:33.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:51:33.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:33 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:51:33.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:33 smithi170 ceph-mon[33394]: osdmap e421: 8 total, 8 up, 8 in 2024-02-10T07:51:33.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:51:33.844 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:33.883 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:33.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:33.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:33.985 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:34.013 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:34.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:34.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:34.115 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:34.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:51:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:51:34.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:34.163 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:34.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:34.264 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:34.305 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:34.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:34.407 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:34.437 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:34.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:34.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:34.539 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:34.576 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:51:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2024-02-10T07:51:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2024-02-10T07:51:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:51:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:41919 10.f 10:fe80f644:::benchmark_data_smithi008_161452_object41918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-02-10T07:51:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:39.102785+0000", 2024-02-10T07:51:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 295.46735022399997, 2024-02-10T07:51:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1426675949999998, 2024-02-10T07:51:34.576 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 41919 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.102785+0000", 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.577 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.102785+0000", 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.102786+0000", 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.499e-06 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.102819+0000", 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.3141e-05 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.102828+0000", 2024-02-10T07:51:34.578 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1330000000000005e-06 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.102835+0000", 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3740000000000003e-06 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.104418+0000", 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0015829399999999999 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.104593+0000", 2024-02-10T07:51:34.579 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000175594 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.104807+0000", 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000213527 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.123742+0000", 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.018934790999999999 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.123754+0000", 2024-02-10T07:51:34.580 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2807e-05 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116058+0000", 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.992304042 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116069+0000", 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0338e-05 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.232651+0000", 2024-02-10T07:51:34.581 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.116582145 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for degraded object", 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.233055+0000", 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00040451799999999998 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.245377+0000", 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.012321937 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.245452+0000", 2024-02-10T07:51:34.582 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4808999999999995e-05 2024-02-10T07:51:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:41925 10.f 10:fd1202af:::benchmark_data_smithi008_161452_object41924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-02-10T07:51:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:39.116289+0000", 2024-02-10T07:51:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 295.45384606900001, 2024-02-10T07:51:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1290358039999999, 2024-02-10T07:51:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.583 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 41925 2024-02-10T07:51:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.116289+0000", 2024-02-10T07:51:34.584 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.116289+0000", 2024-02-10T07:51:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.116290+0000", 2024-02-10T07:51:34.585 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.471e-06 2024-02-10T07:51:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.116328+0000", 2024-02-10T07:51:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.79e-05 2024-02-10T07:51:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.116341+0000", 2024-02-10T07:51:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3389e-05 2024-02-10T07:51:34.586 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.116350+0000", 2024-02-10T07:51:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.568e-06 2024-02-10T07:51:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.128171+0000", 2024-02-10T07:51:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011821038000000001 2024-02-10T07:51:34.587 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.128184+0000", 2024-02-10T07:51:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3083e-05 2024-02-10T07:51:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116095+0000", 2024-02-10T07:51:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.987910909 2024-02-10T07:51:34.588 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116104+0000", 2024-02-10T07:51:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6640000000000004e-06 2024-02-10T07:51:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.235143+0000", 2024-02-10T07:51:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1190396680000001 2024-02-10T07:51:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.589 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.235295+0000", 2024-02-10T07:51:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015196799999999999 2024-02-10T07:51:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:51:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.235471+0000", 2024-02-10T07:51:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017589999999999999 2024-02-10T07:51:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.243262+0000", 2024-02-10T07:51:34.590 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0077909909999999997 2024-02-10T07:51:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.591 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.243372+0000", 2024-02-10T07:51:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000109372 2024-02-10T07:51:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.592 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.243426+0000", 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3993999999999998e-05 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.243479+0000", 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3488e-05 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.245325+0000", 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001845401 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.593 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:41928 10.f 10:f9cfc315:::benchmark_data_smithi008_161452_object41927:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-02-10T07:51:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:39.121388+0000", 2024-02-10T07:51:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 295.44874713399997, 2024-02-10T07:51:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1443620619999999, 2024-02-10T07:51:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 41928 2024-02-10T07:51:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.594 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.121388+0000", 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.121388+0000", 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.121389+0000", 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.144e-06 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.595 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.122070+0000", 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00068071199999999998 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.122082+0000", 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2508e-05 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.122090+0000", 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2660000000000001e-06 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.596 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.128211+0000", 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0061210140000000001 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.128219+0000", 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0069999999999997e-06 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116129+0000", 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.98790985499999995 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116138+0000", 2024-02-10T07:51:34.597 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.799e-06 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.247168+0000", 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.131029662 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.247309+0000", 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000141614 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.247466+0000", 2024-02-10T07:51:34.598 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015640599999999999 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.254260+0000", 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0067942829999999999 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.254472+0000", 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00021164799999999999 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.263451+0000", 2024-02-10T07:51:34.599 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0089796189999999994 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.263510+0000", 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8499999999999999e-05 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.265750+0000", 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0022400250000000001 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.600 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:41947 10.f 10:f09ca940:::benchmark_data_smithi008_161452_object41946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-02-10T07:51:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:39.137085+0000", 2024-02-10T07:51:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 295.43304936499999, 2024-02-10T07:51:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1309200609999999, 2024-02-10T07:51:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 41947 2024-02-10T07:51:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.601 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137085+0000", 2024-02-10T07:51:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137085+0000", 2024-02-10T07:51:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.602 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137086+0000", 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.023e-06 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137116+0000", 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.9118000000000001e-05 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137126+0000", 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0001e-05 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.603 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137133+0000", 2024-02-10T07:51:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6389999999999992e-06 2024-02-10T07:51:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137245+0000", 2024-02-10T07:51:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011220799999999999 2024-02-10T07:51:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137256+0000", 2024-02-10T07:51:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0628000000000001e-05 2024-02-10T07:51:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.604 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116163+0000", 2024-02-10T07:51:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.97890684100000003 2024-02-10T07:51:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116171+0000", 2024-02-10T07:51:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5669999999999995e-06 2024-02-10T07:51:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.605 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.249091+0000", 2024-02-10T07:51:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1329191320000001 2024-02-10T07:51:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.249256+0000", 2024-02-10T07:51:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016493899999999999 2024-02-10T07:51:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:51:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.249424+0000", 2024-02-10T07:51:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016830400000000001 2024-02-10T07:51:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.254454+0000", 2024-02-10T07:51:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0050305569999999997 2024-02-10T07:51:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.263399+0000", 2024-02-10T07:51:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0089446109999999999 2024-02-10T07:51:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.607 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.265887+0000", 2024-02-10T07:51:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0024884640000000001 2024-02-10T07:51:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.266002+0000", 2024-02-10T07:51:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000114545 2024-02-10T07:51:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.268005+0000", 2024-02-10T07:51:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020034839999999998 2024-02-10T07:51:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.608 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:41948 10.7 10:ee9e0489:::benchmark_data_smithi008_161452_object41947:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-02-10T07:51:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:39.137156+0000", 2024-02-10T07:51:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 295.43297918500002, 2024-02-10T07:51:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1499865140000001, 2024-02-10T07:51:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 41948 2024-02-10T07:51:34.609 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137156+0000", 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137156+0000", 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137156+0000", 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3e-07 2024-02-10T07:51:34.610 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137282+0000", 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012545699999999999 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137290+0000", 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0479999999999999e-06 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137297+0000", 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8079999999999999e-06 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.611 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137357+0000", 2024-02-10T07:51:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0056999999999998e-05 2024-02-10T07:51:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137366+0000", 2024-02-10T07:51:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.7610000000000002e-06 2024-02-10T07:51:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.115543+0000", 2024-02-10T07:51:34.612 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.97817686599999998 2024-02-10T07:51:34.613 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.613 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.613 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.613 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.115556+0000", 2024-02-10T07:51:34.613 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2368000000000001e-05 2024-02-10T07:51:34.613 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.613 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.613 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.613 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.254298+0000", 2024-02-10T07:51:34.613 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1387420749999999 2024-02-10T07:51:34.613 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.614 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.614 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.614 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.254448+0000", 2024-02-10T07:51:34.614 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000150653 2024-02-10T07:51:34.614 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.614 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.614 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:34.614 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.254641+0000", 2024-02-10T07:51:34.614 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000192473 2024-02-10T07:51:34.614 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.614 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.614 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.614 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.265768+0000", 2024-02-10T07:51:34.615 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011127497 2024-02-10T07:51:34.615 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.615 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.615 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.615 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.265828+0000", 2024-02-10T07:51:34.615 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9722999999999997e-05 2024-02-10T07:51:34.615 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.615 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.615 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.615 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.285720+0000", 2024-02-10T07:51:34.616 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019892152 2024-02-10T07:51:34.616 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.616 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.616 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:34.616 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.285790+0000", 2024-02-10T07:51:34.616 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9791999999999999e-05 2024-02-10T07:51:34.616 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.616 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.616 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.616 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.287142+0000", 2024-02-10T07:51:34.616 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0013521539999999999 2024-02-10T07:51:34.616 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.617 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.617 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.617 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.617 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.617 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:41949 10.f 10:f2c31d47:::benchmark_data_smithi008_161452_object41948:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-02-10T07:51:34.617 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:39.137836+0000", 2024-02-10T07:51:34.617 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 295.432298635, 2024-02-10T07:51:34.617 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1321038059999999, 2024-02-10T07:51:34.617 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.617 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:34.617 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.618 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.618 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.618 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 41949 2024-02-10T07:51:34.618 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.618 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.618 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.618 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.618 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137836+0000", 2024-02-10T07:51:34.618 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.618 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.618 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.619 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.619 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137836+0000", 2024-02-10T07:51:34.619 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.619 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.619 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.619 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.619 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137837+0000", 2024-02-10T07:51:34.619 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.16e-07 2024-02-10T07:51:34.619 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.619 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.619 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.619 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137863+0000", 2024-02-10T07:51:34.619 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.5877e-05 2024-02-10T07:51:34.620 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.620 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.620 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.620 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137877+0000", 2024-02-10T07:51:34.620 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3637e-05 2024-02-10T07:51:34.620 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.620 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.620 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.620 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137884+0000", 2024-02-10T07:51:34.620 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4250000000000001e-06 2024-02-10T07:51:34.620 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.620 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.620 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.620 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.137995+0000", 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000111335 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.138005+0000", 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.2920000000000001e-06 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116196+0000", 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.97819139600000005 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.621 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116204+0000", 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.2050000000000002e-06 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.250843+0000", 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1346386079999999 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.250949+0000", 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010659500000000001 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.251059+0000", 2024-02-10T07:51:34.622 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000109969 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.256311+0000", 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0052517750000000002 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.263407+0000", 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0070953379999999996 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.268077+0000", 2024-02-10T07:51:34.623 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0046699949999999997 2024-02-10T07:51:34.624 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.624 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.624 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:34.624 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.268140+0000", 2024-02-10T07:51:34.624 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3110999999999999e-05 2024-02-10T07:51:34.624 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.624 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.624 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.624 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.269940+0000", 2024-02-10T07:51:34.624 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001800332 2024-02-10T07:51:34.624 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.624 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.624 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:41959 10.7 10:e9ded313:::benchmark_data_smithi008_161452_object41958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:39.144450+0000", 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 295.42568438199999, 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.143750426, 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 41959 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.625 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.626 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.144450+0000", 2024-02-10T07:51:34.626 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.626 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.626 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.626 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.626 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.144450+0000", 2024-02-10T07:51:34.626 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.626 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.626 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.626 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.626 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.144452+0000", 2024-02-10T07:51:34.626 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.404e-06 2024-02-10T07:51:34.626 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.626 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.144484+0000", 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.1680000000000002e-05 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.144495+0000", 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.169e-05 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.144503+0000", 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.6249999999999998e-06 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.627 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.144565+0000", 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2197000000000001e-05 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.144576+0000", 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0881e-05 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.115912+0000", 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.97133569799999997 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.628 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.115923+0000", 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1371e-05 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.256383+0000", 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.140459876 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.256512+0000", 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012919900000000001 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.629 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.256672+0000", 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015958500000000001 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.265775+0000", 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0091033799999999995 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.268062+0000", 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0022874380000000001 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.630 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.287292+0000", 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019229593 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.287353+0000", 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1285999999999996e-05 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.288201+0000", 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00084752299999999998 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.631 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.632 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.632 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.632 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:41964 10.7 10:ed5f8ca6:::benchmark_data_smithi008_161452_object41963:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-02-10T07:51:34.632 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:39.145792+0000", 2024-02-10T07:51:34.632 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 295.424343126, 2024-02-10T07:51:34.632 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1433444709999998, 2024-02-10T07:51:34.632 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.632 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:34.632 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.632 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.632 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.632 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 41964 2024-02-10T07:51:34.632 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.633 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:34.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:34.634 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.634 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.635 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.635 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.145792+0000", 2024-02-10T07:51:34.635 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.635 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.635 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.635 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.635 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.145792+0000", 2024-02-10T07:51:34.635 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.635 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.635 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.635 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.635 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.145793+0000", 2024-02-10T07:51:34.635 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.161e-06 2024-02-10T07:51:34.635 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.636 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.636 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.636 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.145829+0000", 2024-02-10T07:51:34.636 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5738999999999998e-05 2024-02-10T07:51:34.636 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.636 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.636 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.636 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.145841+0000", 2024-02-10T07:51:34.636 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1929999999999999e-05 2024-02-10T07:51:34.636 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.636 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.636 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.636 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.145849+0000", 2024-02-10T07:51:34.636 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.1210000000000007e-06 2024-02-10T07:51:34.637 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.637 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.637 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.637 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.145905+0000", 2024-02-10T07:51:34.637 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6740000000000002e-05 2024-02-10T07:51:34.637 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.637 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.637 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.637 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.145915+0000", 2024-02-10T07:51:34.637 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.8779999999999993e-06 2024-02-10T07:51:34.637 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.637 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.637 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.637 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.115964+0000", 2024-02-10T07:51:34.638 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.97004878900000002 2024-02-10T07:51:34.638 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.638 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.638 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.638 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.115973+0000", 2024-02-10T07:51:34.638 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1249999999999999e-06 2024-02-10T07:51:34.638 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.638 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.638 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.638 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.258132+0000", 2024-02-10T07:51:34.638 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1421590909999999 2024-02-10T07:51:34.638 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.638 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.638 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.258274+0000", 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014182100000000001 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.258472+0000", 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019808599999999999 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.265781+0000", 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0073091320000000003 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.639 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.269995+0000", 2024-02-10T07:51:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0042135430000000001 2024-02-10T07:51:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.288236+0000", 2024-02-10T07:51:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.018240722000000001 2024-02-10T07:51:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.288271+0000", 2024-02-10T07:51:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5454000000000003e-05 2024-02-10T07:51:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.641 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.289136+0000", 2024-02-10T07:51:34.641 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00086513900000000001 2024-02-10T07:51:34.641 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.641 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.641 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.641 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.641 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.641 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:41966 10.7 10:eacdf0da:::benchmark_data_smithi008_161452_object41965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-02-10T07:51:34.641 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:39.149048+0000", 2024-02-10T07:51:34.641 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 295.42108708299997, 2024-02-10T07:51:34.641 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1410599669999999, 2024-02-10T07:51:34.641 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.641 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:34.641 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 41966 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.149048+0000", 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.149048+0000", 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.642 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.149049+0000", 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5489999999999999e-06 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.149079+0000", 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.993e-05 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.149098+0000", 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8457e-05 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.643 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.149106+0000", 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6920000000000008e-06 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.149171+0000", 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4546000000000002e-05 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.149181+0000", 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.7010000000000008e-06 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.644 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116010+0000", 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.96682916100000005 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116019+0000", 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0699999999999996e-06 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.260044+0000", 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1440251930000001 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.645 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.260169+0000", 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000124685 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.260325+0000", 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015620099999999999 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.265788+0000", 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.005462929 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.270036+0000", 2024-02-10T07:51:34.646 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.004248618 2024-02-10T07:51:34.647 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.647 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.647 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.647 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.289193+0000", 2024-02-10T07:51:34.647 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.019156436999999998 2024-02-10T07:51:34.647 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.647 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.647 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:34.647 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.289298+0000", 2024-02-10T07:51:34.647 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000104769 2024-02-10T07:51:34.647 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.647 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.647 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.648 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.290108+0000", 2024-02-10T07:51:34.648 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00081002899999999998 2024-02-10T07:51:34.648 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.648 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.648 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.648 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.648 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.648 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:41987 10.f 10:fa1b6008:::benchmark_data_smithi008_161452_object41986:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-02-10T07:51:34.648 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:39.167355+0000", 2024-02-10T07:51:34.648 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 295.40277938200001, 2024-02-10T07:51:34.648 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1058154450000002, 2024-02-10T07:51:34.648 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.649 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:34.649 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.649 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.649 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.649 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 41987 2024-02-10T07:51:34.649 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.649 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.649 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.649 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.649 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.167355+0000", 2024-02-10T07:51:34.649 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.649 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.650 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.650 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.650 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.167355+0000", 2024-02-10T07:51:34.650 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.650 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.650 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.650 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.650 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.167357+0000", 2024-02-10T07:51:34.650 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5009999999999999e-06 2024-02-10T07:51:34.650 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.650 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.650 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.650 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.167421+0000", 2024-02-10T07:51:34.650 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4108000000000004e-05 2024-02-10T07:51:34.651 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.651 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.651 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.651 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.167433+0000", 2024-02-10T07:51:34.651 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1681e-05 2024-02-10T07:51:34.651 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.651 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.651 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.651 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.167441+0000", 2024-02-10T07:51:34.651 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6920000000000008e-06 2024-02-10T07:51:34.651 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.652 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.652 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.652 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.167505+0000", 2024-02-10T07:51:34.652 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3419999999999999e-05 2024-02-10T07:51:34.652 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.652 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.652 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.652 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.167513+0000", 2024-02-10T07:51:34.652 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3459999999999994e-06 2024-02-10T07:51:34.652 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.652 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.652 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.652 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116230+0000", 2024-02-10T07:51:34.652 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.94871699099999995 2024-02-10T07:51:34.653 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.653 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.653 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.653 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116239+0000", 2024-02-10T07:51:34.653 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3680000000000002e-06 2024-02-10T07:51:34.653 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.653 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.653 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.653 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.252531+0000", 2024-02-10T07:51:34.653 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1362928130000001 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.252640+0000", 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000108968 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.252765+0000", 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000124275 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.256350+0000", 2024-02-10T07:51:34.654 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0035854020000000001 2024-02-10T07:51:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.263413+0000", 2024-02-10T07:51:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.007062848 2024-02-10T07:51:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.270089+0000", 2024-02-10T07:51:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0066766539999999997 2024-02-10T07:51:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.655 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.270152+0000", 2024-02-10T07:51:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2481e-05 2024-02-10T07:51:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.273171+0000", 2024-02-10T07:51:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.003018897 2024-02-10T07:51:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.656 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:41991 10.7 10:eec2ffec:::benchmark_data_smithi008_161452_object41990:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-02-10T07:51:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:39.172032+0000", 2024-02-10T07:51:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 295.39810315, 2024-02-10T07:51:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.119106859, 2024-02-10T07:51:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 41991 2024-02-10T07:51:34.657 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.172032+0000", 2024-02-10T07:51:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.172032+0000", 2024-02-10T07:51:34.658 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.172033+0000", 2024-02-10T07:51:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6700000000000002e-07 2024-02-10T07:51:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.172088+0000", 2024-02-10T07:51:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5569999999999998e-05 2024-02-10T07:51:34.659 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.172107+0000", 2024-02-10T07:51:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9258000000000001e-05 2024-02-10T07:51:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.172115+0000", 2024-02-10T07:51:34.660 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.909e-06 2024-02-10T07:51:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.172167+0000", 2024-02-10T07:51:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2154999999999998e-05 2024-02-10T07:51:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.172175+0000", 2024-02-10T07:51:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9529999999999999e-06 2024-02-10T07:51:34.661 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116266+0000", 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.94409096000000003 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.116275+0000", 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.4279999999999996e-06 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.261677+0000", 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1454021480000001 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.662 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.663 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.663 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.261782+0000", 2024-02-10T07:51:34.663 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010550199999999999 2024-02-10T07:51:34.663 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.663 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.663 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:34.663 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.261928+0000", 2024-02-10T07:51:34.663 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014512499999999999 2024-02-10T07:51:34.663 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.663 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.663 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.663 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.265843+0000", 2024-02-10T07:51:34.663 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0039151409999999996 2024-02-10T07:51:34.663 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.270147+0000", 2024-02-10T07:51:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0043042840000000002 2024-02-10T07:51:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.290168+0000", 2024-02-10T07:51:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.020021466000000002 2024-02-10T07:51:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:34.664 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.290234+0000", 2024-02-10T07:51:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6033999999999998e-05 2024-02-10T07:51:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.291139+0000", 2024-02-10T07:51:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00090405900000000003 2024-02-10T07:51:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:92273 10.f 10:f6c5c12c:::benchmark_data_smithi008_161452_object92272:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-02-10T07:51:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:33.171159+0000", 2024-02-10T07:51:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 241.39897542200001, 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0152455140000001, 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 92273 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.171159+0000", 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.666 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.171159+0000", 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.171161+0000", 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.389e-06 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.171192+0000", 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.1260999999999998e-05 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.667 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.171203+0000", 2024-02-10T07:51:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1007e-05 2024-02-10T07:51:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.171211+0000", 2024-02-10T07:51:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7209999999999994e-06 2024-02-10T07:51:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.172330+0000", 2024-02-10T07:51:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001119173 2024-02-10T07:51:34.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.172478+0000", 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014768299999999999 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.172639+0000", 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00016093499999999999 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.177498+0000", 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.004859866 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.195498+0000", 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.017999953999999999 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.195514+0000", 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5823000000000001e-05 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:34.186321+0000", 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.99080663599999996 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.670 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:34.186405+0000", 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.4066000000000005e-05 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:92276 10.f 10:f17fd61c:::benchmark_data_smithi008_161452_object92275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e339)", 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:33.176155+0000", 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 241.39398026999999, 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0106151640000001, 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.671 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 92276 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.176155+0000", 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.176155+0000", 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.672 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.176156+0000", 2024-02-10T07:51:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.578e-06 2024-02-10T07:51:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.176195+0000", 2024-02-10T07:51:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8872999999999999e-05 2024-02-10T07:51:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.176210+0000", 2024-02-10T07:51:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4808e-05 2024-02-10T07:51:34.673 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.176218+0000", 2024-02-10T07:51:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3999999999999992e-06 2024-02-10T07:51:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.179138+0000", 2024-02-10T07:51:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0029201539999999999 2024-02-10T07:51:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.179287+0000", 2024-02-10T07:51:34.674 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014892200000000001 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.179458+0000", 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017122200000000001 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.195550+0000", 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016091594000000001 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:33.195560+0000", 2024-02-10T07:51:34.675 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0244999999999999e-05 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:34.186698+0000", 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.99113770899999998 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:34.186770+0000", 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.1658999999999999e-05 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.676 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:99598 10.7 10:eb55143c:::benchmark_data_smithi008_161452_object99597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-02-10T07:51:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:40.221282+0000", 2024-02-10T07:51:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 234.348852626, 2024-02-10T07:51:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0285609120000001, 2024-02-10T07:51:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:51:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 99598 2024-02-10T07:51:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.677 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.221282+0000", 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.221282+0000", 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.221283+0000", 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.288e-06 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.678 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.221345+0000", 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1111999999999994e-05 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.221354+0000", 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.7110000000000007e-06 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.221361+0000", 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.7680000000000003e-06 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.223838+0000", 2024-02-10T07:51:34.679 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0024771070000000001 2024-02-10T07:51:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.223972+0000", 2024-02-10T07:51:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013365600000000001 2024-02-10T07:51:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.224119+0000", 2024-02-10T07:51:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014738699999999999 2024-02-10T07:51:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.680 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.232364+0000", 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0082445120000000007 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.235180+0000", 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0028166459999999999 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.250983+0000", 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.015802687999999999 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.250995+0000", 2024-02-10T07:51:34.681 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2213e-05 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:41.249741+0000", 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.99874615200000005 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:41.249843+0000", 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000101672 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.682 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:99600 10.7 10:e393e642:::benchmark_data_smithi008_161452_object99599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:47:40.223591+0000", 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 234.34654371400001, 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.02678045, 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 99600 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.223591+0000", 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.223591+0000", 2024-02-10T07:51:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.223593+0000", 2024-02-10T07:51:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5069999999999999e-06 2024-02-10T07:51:34.684 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.223649+0000", 2024-02-10T07:51:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6747999999999999e-05 2024-02-10T07:51:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.223663+0000", 2024-02-10T07:51:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3662999999999999e-05 2024-02-10T07:51:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.685 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.223670+0000", 2024-02-10T07:51:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3050000000000004e-06 2024-02-10T07:51:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.226729+0000", 2024-02-10T07:51:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0030591239999999999 2024-02-10T07:51:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.227171+0000", 2024-02-10T07:51:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00044157899999999999 2024-02-10T07:51:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.227853+0000", 2024-02-10T07:51:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00068167100000000001 2024-02-10T07:51:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.235272+0000", 2024-02-10T07:51:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0074195650000000004 2024-02-10T07:51:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.687 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.239361+0000", 2024-02-10T07:51:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0040889339999999998 2024-02-10T07:51:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.251031+0000", 2024-02-10T07:51:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011669766999999999 2024-02-10T07:51:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:40.251041+0000", 2024-02-10T07:51:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0258000000000001e-05 2024-02-10T07:51:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:41.250288+0000", 2024-02-10T07:51:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.99924676999999995 2024-02-10T07:51:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:47:41.250372+0000", 2024-02-10T07:51:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3559000000000004e-05 2024-02-10T07:51:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.689 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:196249 10.f 10:f39b67d7:::benchmark_data_smithi008_161452_object87788:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-02-10T07:51:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:49:05.224978+0000", 2024-02-10T07:51:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 149.34515680199999, 2024-02-10T07:51:34.691 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9145773749999999, 2024-02-10T07:51:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:51:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 196249 2024-02-10T07:51:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.224978+0000", 2024-02-10T07:51:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.224978+0000", 2024-02-10T07:51:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.224979+0000", 2024-02-10T07:51:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9000000000000005e-07 2024-02-10T07:51:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.693 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.224984+0000", 2024-02-10T07:51:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5299999999999998e-06 2024-02-10T07:51:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.224994+0000", 2024-02-10T07:51:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0424999999999999e-05 2024-02-10T07:51:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.225002+0000", 2024-02-10T07:51:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.4989999999999994e-06 2024-02-10T07:51:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.225058+0000", 2024-02-10T07:51:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5609000000000002e-05 2024-02-10T07:51:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.225204+0000", 2024-02-10T07:51:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000145458 2024-02-10T07:51:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.695 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,4", 2024-02-10T07:51:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.225321+0000", 2024-02-10T07:51:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011777699999999999 2024-02-10T07:51:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.230185+0000", 2024-02-10T07:51:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0048640760000000002 2024-02-10T07:51:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.246823+0000", 2024-02-10T07:51:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.016637748000000001 2024-02-10T07:51:34.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.264140+0000", 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.017316489000000001 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.264154+0000", 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3923e-05 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:06.233503+0000", 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.96934903100000003 2024-02-10T07:51:34.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:06.233517+0000", 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.431e-05 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.139488+0000", 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.90597083 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.139555+0000", 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.7680000000000003e-05 2024-02-10T07:51:34.698 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:196272 10.f 10:fad55a16:::benchmark_data_smithi008_161452_object87811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:49:05.279853+0000", 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 149.290282249, 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8740174349999998, 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 196272 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.699 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.279853+0000", 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.279853+0000", 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.279854+0000", 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.4750000000000001e-06 2024-02-10T07:51:34.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.279859+0000", 2024-02-10T07:51:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.164e-06 2024-02-10T07:51:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.279876+0000", 2024-02-10T07:51:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6452000000000001e-05 2024-02-10T07:51:34.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.279884+0000", 2024-02-10T07:51:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9740000000000002e-06 2024-02-10T07:51:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.280336+0000", 2024-02-10T07:51:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00045254399999999998 2024-02-10T07:51:34.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.280347+0000", 2024-02-10T07:51:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0501e-05 2024-02-10T07:51:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:06.233546+0000", 2024-02-10T07:51:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.953199303 2024-02-10T07:51:34.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:06.233555+0000", 2024-02-10T07:51:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.9199999999999993e-06 2024-02-10T07:51:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.140169+0000", 2024-02-10T07:51:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.90661400599999997 2024-02-10T07:51:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.704 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.140271+0000", 2024-02-10T07:51:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000102082 2024-02-10T07:51:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3,4", 2024-02-10T07:51:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.140376+0000", 2024-02-10T07:51:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000104857 2024-02-10T07:51:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.148294+0000", 2024-02-10T07:51:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0079177540000000008 2024-02-10T07:51:34.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.148337+0000", 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.316e-05 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.153523+0000", 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0051864199999999997 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.153693+0000", 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000170185 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.153760+0000", 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6647000000000001e-05 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.153870+0000", 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000109991 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:196275 10.f 10:ff58d29d:::benchmark_data_smithi008_161452_object87814:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-02-10T07:51:34.707 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:49:05.279909+0000", 2024-02-10T07:51:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 149.29022531300001, 2024-02-10T07:51:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8746776440000001, 2024-02-10T07:51:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 196275 2024-02-10T07:51:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.279909+0000", 2024-02-10T07:51:34.708 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.279909+0000", 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.279910+0000", 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3300000000000002e-07 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.279913+0000", 2024-02-10T07:51:34.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.841e-06 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.279920+0000", 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9870000000000002e-06 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.279925+0000", 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3879999999999999e-06 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.280390+0000", 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00046439400000000003 2024-02-10T07:51:34.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.280398+0000", 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7460000000000006e-06 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:06.233580+0000", 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.95318180500000005 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:06.233589+0000", 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.1730000000000001e-06 2024-02-10T07:51:34.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.142779+0000", 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.90919013299999996 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.142891+0000", 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000111827 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3,4", 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.142997+0000", 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010563700000000001 2024-02-10T07:51:34.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.148350+0000", 2024-02-10T07:51:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0053525949999999999 2024-02-10T07:51:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.153579+0000", 2024-02-10T07:51:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0052296210000000003 2024-02-10T07:51:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.153957+0000", 2024-02-10T07:51:34.713 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000377337 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.153995+0000", 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8095e-05 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.154466+0000", 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000471822 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.154587+0000", 2024-02-10T07:51:34.714 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012071 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:196277 10.f 10:fb8f3b14:::benchmark_data_smithi008_161452_object87816:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:49:05.284080+0000", 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 149.286054635, 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8708861940000001, 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.715 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 196277 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.284080+0000", 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.284080+0000", 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.716 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.284082+0000", 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.719e-06 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.284087+0000", 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0919999999999997e-06 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.284097+0000", 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0441e-05 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.717 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.284105+0000", 2024-02-10T07:51:34.718 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.5669999999999997e-06 2024-02-10T07:51:34.718 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.718 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.718 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.718 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.284177+0000", 2024-02-10T07:51:34.718 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2282999999999999e-05 2024-02-10T07:51:34.718 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.718 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.718 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.719 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 89 16 104139 104123 73.1014 44.125 0.0156634 0.0136736 2024-02-10T07:51:34.719 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.284190+0000", 2024-02-10T07:51:34.719 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2981e-05 2024-02-10T07:51:34.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.719 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.719 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.719 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:06.233614+0000", 2024-02-10T07:51:34.719 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.94942417800000001 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:06.233623+0000", 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0760000000000003e-06 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.145164+0000", 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.91154134399999998 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.145278+0000", 2024-02-10T07:51:34.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011383399999999999 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3,4", 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.145419+0000", 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014105899999999999 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.153537+0000", 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0081183260000000004 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.154641+0000", 2024-02-10T07:51:34.721 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0011036900000000001 2024-02-10T07:51:34.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.154688+0000", 2024-02-10T07:51:34.722 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7049000000000003e-05 2024-02-10T07:51:34.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.154745+0000", 2024-02-10T07:51:34.722 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6694000000000003e-05 2024-02-10T07:51:34.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:34.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.154846+0000", 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000101406 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.154966+0000", 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000120455 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:196286 10.f 10:ff72c6b4:::benchmark_data_smithi008_161452_object87825:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:49:05.295029+0000", 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 149.27510551, 2024-02-10T07:51:34.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.862918165, 2024-02-10T07:51:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 196286 2024-02-10T07:51:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.295029+0000", 2024-02-10T07:51:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.295029+0000", 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.295031+0000", 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9240000000000001e-06 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.295037+0000", 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6169999999999998e-06 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.295051+0000", 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3845999999999999e-05 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.295060+0000", 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4120000000000007e-06 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.295129+0000", 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8518999999999999e-05 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:05.295137+0000", 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.8230000000000001e-06 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:06.233647+0000", 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.93850986999999997 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:06.233656+0000", 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3159999999999997e-06 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.728 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:34.728 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.150857+0000", 2024-02-10T07:51:34.728 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.91720149100000004 2024-02-10T07:51:34.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.728 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:34.728 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.150966+0000", 2024-02-10T07:51:34.728 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000109268 2024-02-10T07:51:34.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.729 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,3,4", 2024-02-10T07:51:34.729 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.151083+0000", 2024-02-10T07:51:34.729 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000116406 2024-02-10T07:51:34.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.729 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:34.729 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.153882+0000", 2024-02-10T07:51:34.729 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0027989550000000001 2024-02-10T07:51:34.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.729 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.729 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.155023+0000", 2024-02-10T07:51:34.729 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0011408410000000001 2024-02-10T07:51:34.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.730 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.730 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.155439+0000", 2024-02-10T07:51:34.730 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000416432 2024-02-10T07:51:34.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.730 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:34.730 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.157763+0000", 2024-02-10T07:51:34.730 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0023240629999999999 2024-02-10T07:51:34.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.731 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:34.731 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.157825+0000", 2024-02-10T07:51:34.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1660000000000003e-05 2024-02-10T07:51:34.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:34.731 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:34.731 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:34.731 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:49:07.157947+0000", 2024-02-10T07:51:34.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012272199999999999 2024-02-10T07:51:34.731 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.731 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.731 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.731 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:34.731 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:34.731 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:51:34.734 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:34.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:34.779 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:34.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:34.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:34 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:34.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:34 smithi008 ceph-mon[29816]: pgmap v913: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 54 MiB/s wr, 865 op/s; 8861/307434 objects degraded (2.882%); 14731/307434 objects misplaced (4.792%); 3.8 MiB/s, 61 objects/s recovering 2024-02-10T07:51:34.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:34 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:34.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:34 smithi008 ceph-mon[40743]: pgmap v913: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 54 MiB/s wr, 865 op/s; 8861/307434 objects degraded (2.882%); 14731/307434 objects misplaced (4.792%); 3.8 MiB/s, 61 objects/s recovering 2024-02-10T07:51:34.880 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:34.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:34 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:34.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:34 smithi170 ceph-mon[33394]: pgmap v913: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.2 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 54 MiB/s wr, 865 op/s; 8861/307434 objects degraded (2.882%); 14731/307434 objects misplaced (4.792%); 3.8 MiB/s, 61 objects/s recovering 2024-02-10T07:51:34.930 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:34.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:35.033 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:35.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:35.071 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:35.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:35.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:35.204 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:35.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:35.305 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:35.349 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:35.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:35.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:35.451 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:35.483 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:35.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:35.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:35.585 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:35.596 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-10T07:51:35.615 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:35.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:35.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:35.666 INFO:tasks.radosbench.radosbench.0.smithi008.stdout: 90 1 104759 104758 72.7302 39.6875 0.0459505 0.0137437 2024-02-10T07:51:35.717 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:35.741 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:35.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:35.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:35.775 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:51:35.843 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:35.862 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:51:35.867 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:35.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:35.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:35.970 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:35.998 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:35.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:35.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:36.100 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:36.127 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:36.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:36.229 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:36.256 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:36.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:36.358 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:36.386 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:36.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:36.487 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:36.514 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:36.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:36.615 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:36.640 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:36.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:36.667 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Total time run: 90.0235 2024-02-10T07:51:36.667 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Total writes made: 104759 2024-02-10T07:51:36.667 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Write size: 65536 2024-02-10T07:51:36.667 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Object size: 65536 2024-02-10T07:51:36.667 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Bandwidth (MB/sec): 72.7303 2024-02-10T07:51:36.667 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev Bandwidth: 47.5695 2024-02-10T07:51:36.667 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max bandwidth (MB/sec): 226.875 2024-02-10T07:51:36.667 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min bandwidth (MB/sec): 0 2024-02-10T07:51:36.667 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Average IOPS: 1163 2024-02-10T07:51:36.667 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev IOPS: 761.117 2024-02-10T07:51:36.668 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max IOPS: 3630 2024-02-10T07:51:36.668 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min IOPS: 0 2024-02-10T07:51:36.668 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Average Latency(s): 0.0137438 2024-02-10T07:51:36.668 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Stddev Latency(s): 0.032978 2024-02-10T07:51:36.668 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Max latency(s): 2.4862 2024-02-10T07:51:36.668 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Min latency(s): 0.00135892 2024-02-10T07:51:36.678 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Cleaning up (deleting benchmark objects) 2024-02-10T07:51:36.742 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:36.784 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:36.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:36.886 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:36.915 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:36.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:36.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:37.017 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:37.063 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:37.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:37.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:37.165 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:37.205 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:37.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:37.307 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:37.340 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:37.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:37.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:37.446 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:37.475 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:37.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:37.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:37.523 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:37 smithi170 ceph-mon[33394]: pgmap v914: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 774 B/s rd, 35 MiB/s wr, 560 op/s; 8861/311034 objects degraded (2.849%); 14414/311034 objects misplaced (4.634%); 5.4 MiB/s, 85 objects/s recovering 2024-02-10T07:51:37.523 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:37 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/311034 objects degraded (2.849%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:37.523 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:37 smithi170 ceph-mon[33394]: osdmap e422: 8 total, 8 up, 8 in 2024-02-10T07:51:37.577 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:37.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:37 smithi008 ceph-mon[29816]: pgmap v914: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 774 B/s rd, 35 MiB/s wr, 560 op/s; 8861/311034 objects degraded (2.849%); 14414/311034 objects misplaced (4.634%); 5.4 MiB/s, 85 objects/s recovering 2024-02-10T07:51:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:37 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/311034 objects degraded (2.849%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:37 smithi008 ceph-mon[29816]: osdmap e422: 8 total, 8 up, 8 in 2024-02-10T07:51:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:37 smithi008 ceph-mon[40743]: pgmap v914: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 774 B/s rd, 35 MiB/s wr, 560 op/s; 8861/311034 objects degraded (2.849%); 14414/311034 objects misplaced (4.634%); 5.4 MiB/s, 85 objects/s recovering 2024-02-10T07:51:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:37 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/311034 objects degraded (2.849%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:37 smithi008 ceph-mon[40743]: osdmap e422: 8 total, 8 up, 8 in 2024-02-10T07:51:37.622 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:37.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:37.724 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:37.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:37.752 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:37.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:37.853 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:37.898 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:37.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:38.000 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:38.034 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:51:38.034 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:51:38.034 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.034 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25838.0:105719 11.4 11:23acead3:::benchmark_data_smithi008_186804_object957:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:51:38.027209+0000", 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.0059949770000000003, 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0060189340000000001, 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25838", 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 105719 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.027209+0000", 2024-02-10T07:51:38.035 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:38.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.036 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:38.036 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.027209+0000", 2024-02-10T07:51:38.036 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:38.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.036 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:38.036 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.027210+0000", 2024-02-10T07:51:38.036 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3090000000000001e-06 2024-02-10T07:51:38.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.036 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.036 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.027214+0000", 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.1930000000000003e-06 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.027226+0000", 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1891e-05 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.027232+0000", 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1870000000000002e-06 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.037 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:38.038 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.027289+0000", 2024-02-10T07:51:38.038 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6399e-05 2024-02-10T07:51:38.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.038 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.038 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:38.038 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.027465+0000", 2024-02-10T07:51:38.038 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000176737 2024-02-10T07:51:38.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.038 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.038 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 2,6", 2024-02-10T07:51:38.038 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.027647+0000", 2024-02-10T07:51:38.038 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00018167899999999999 2024-02-10T07:51:38.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.038 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.029785+0000", 2024-02-10T07:51:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0021379630000000001 2024-02-10T07:51:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.030623+0000", 2024-02-10T07:51:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00083801299999999997 2024-02-10T07:51:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.039 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop_reply(client.25838.0:105698 11.f e422/405 ondisk, result = 0)", 2024-02-10T07:51:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:51:38.031768+0000", 2024-02-10T07:51:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.001435239, 2024-02-10T07:51:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0015128100000000001, 2024-02-10T07:51:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:51:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031768+0000", 2024-02-10T07:51:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:38.040 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031768+0000", 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031769+0000", 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2900000000000003e-07 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031773+0000", 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.9870000000000001e-06 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:38.041 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031785+0000", 2024-02-10T07:51:38.042 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1623e-05 2024-02-10T07:51:38.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.042 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.042 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:38.042 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031790+0000", 2024-02-10T07:51:38.042 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3340000000000001e-06 2024-02-10T07:51:38.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.042 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.042 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:38.042 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031833+0000", 2024-02-10T07:51:38.042 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.3090999999999998e-05 2024-02-10T07:51:38.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.042 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.042 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031840+0000", 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.2370000000000003e-06 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_repop_reply(client.25838.0:105700 11.f e422/405)", 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:51:38.031915+0000", 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.0012887739999999999, 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001398437, 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "queued for pg", 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:38.043 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031915+0000", 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031915+0000", 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031915+0000", 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.2599999999999998e-07 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.044 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.045 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:38.045 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031918+0000", 2024-02-10T07:51:38.045 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7819999999999998e-06 2024-02-10T07:51:38.045 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.045 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.045 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:38.045 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031924+0000", 2024-02-10T07:51:38.045 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4570000000000002e-06 2024-02-10T07:51:38.045 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.045 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.045 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:38.045 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031928+0000", 2024-02-10T07:51:38.045 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8850000000000001e-06 2024-02-10T07:51:38.045 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25838.0:105698 11.f 11:f19fd652:::benchmark_data_smithi008_186804_object936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:51:38.002314+0000", 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.030889244, 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.031028038000000001, 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25838", 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 105698 2024-02-10T07:51:38.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.002314+0000", 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.002314+0000", 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.002315+0000", 2024-02-10T07:51:38.047 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2599999999999998e-07 2024-02-10T07:51:38.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.048 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.048 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:38.048 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.002317+0000", 2024-02-10T07:51:38.048 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.351e-06 2024-02-10T07:51:38.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.048 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.048 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:38.048 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.002323+0000", 2024-02-10T07:51:38.048 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9710000000000001e-06 2024-02-10T07:51:38.048 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.048 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.048 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:38.048 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.002328+0000", 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8579999999999999e-06 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.004191+0000", 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0018625200000000001 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.004326+0000", 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00013590799999999999 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:51:38.049 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.004483+0000", 2024-02-10T07:51:38.050 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015694 2024-02-10T07:51:38.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.050 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.050 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:38.050 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.010409+0000", 2024-02-10T07:51:38.050 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0059253379999999996 2024-02-10T07:51:38.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.050 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.050 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:38.050 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.011029+0000", 2024-02-10T07:51:38.050 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00062068400000000004 2024-02-10T07:51:38.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.050 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.050 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:38.051 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031846+0000", 2024-02-10T07:51:38.051 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.020816715 2024-02-10T07:51:38.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.051 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.051 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:38.051 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.031884+0000", 2024-02-10T07:51:38.051 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7944999999999997e-05 2024-02-10T07:51:38.051 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:38.051 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:38.051 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:38.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.051 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.051 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25838.0:105700 11.f 11:fcacc2ce:::benchmark_data_smithi008_186804_object938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-02-10T07:51:38.051 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:51:38.002359+0000", 2024-02-10T07:51:38.052 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 0.030844466000000001, 2024-02-10T07:51:38.052 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.031041975999999999, 2024-02-10T07:51:38.052 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:38.052 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:51:38.052 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:38.052 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25838", 2024-02-10T07:51:38.052 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:51:38.052 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 105700 2024-02-10T07:51:38.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.052 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:38.052 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.052 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:38.052 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.002359+0000", 2024-02-10T07:51:38.052 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:38.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.053 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.053 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:38.053 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.002359+0000", 2024-02-10T07:51:38.053 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:38.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.053 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.053 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:38.053 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.002360+0000", 2024-02-10T07:51:38.053 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.27e-07 2024-02-10T07:51:38.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.053 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.053 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:38.053 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.002362+0000", 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.3460000000000001e-06 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.002368+0000", 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2659999999999998e-06 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.002373+0000", 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9119999999999997e-06 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:38.054 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.007223+0000", 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.004849936 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.007370+0000", 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014703599999999999 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.007524+0000", 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015366499999999999 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.010440+0000", 2024-02-10T07:51:38.055 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0029158640000000002 2024-02-10T07:51:38.056 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:38.056 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:38.056 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:38.056 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.013994+0000", 2024-02-10T07:51:38.056 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.003554733 2024-02-10T07:51:38.056 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:38.056 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:38.056 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:38.056 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:38.056 INFO:teuthology.orchestra.run.smithi170.stdout: ], 2024-02-10T07:51:38.056 INFO:teuthology.orchestra.run.smithi170.stdout: "num_ops": 5 2024-02-10T07:51:38.056 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:51:38.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:38.058 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:38.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:38.159 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:38.190 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:38.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:38.292 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:38.328 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:38.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:38.430 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:38.454 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:38.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:38.556 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:38.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:38.596 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:38.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:38.697 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:38.721 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:38.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:38.726 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-10T07:51:38.823 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:38.871 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:38.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:38.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:38.973 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:39.016 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:39.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:39.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:39.067 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:51:39.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:51:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:51:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:51:39.118 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:39.147 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:39.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:39.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:39.249 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:39.276 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:39.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:39.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:39.378 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:39.401 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:39.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:39.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:39.503 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:39.510 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:39 smithi170 ceph-mon[33394]: pgmap v916: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 839 B/s rd, 33 MiB/s wr, 534 op/s; 8861/311586 objects degraded (2.844%); 14414/311586 objects misplaced (4.626%); 3.2 MiB/s, 51 objects/s recovering 2024-02-10T07:51:39.545 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:39.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:39.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:39 smithi008 ceph-mon[29816]: pgmap v916: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 839 B/s rd, 33 MiB/s wr, 534 op/s; 8861/311586 objects degraded (2.844%); 14414/311586 objects misplaced (4.626%); 3.2 MiB/s, 51 objects/s recovering 2024-02-10T07:51:39.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:39 smithi008 ceph-mon[40743]: pgmap v916: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 839 B/s rd, 33 MiB/s wr, 534 op/s; 8861/311586 objects degraded (2.844%); 14414/311586 objects misplaced (4.626%); 3.2 MiB/s, 51 objects/s recovering 2024-02-10T07:51:39.646 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:39.674 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:39.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:39.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:39.776 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:39.823 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:39.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:39.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:39.925 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:39.955 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:39.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:39.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:40.057 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:40.099 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:40.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:40.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:40.201 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:40.239 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:40.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:40.341 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:40.376 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:40.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:40.478 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:40.507 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:40.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:40.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:40.609 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:40.644 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:40.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:40.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:40.747 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:40.774 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:40.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:40.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:40.876 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:40.922 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:40.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:41.024 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:41.065 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:41.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:41.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:41.193 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:41.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:41.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:41.289 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:41 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:51:41.289 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:41 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:51:41.289 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:41 smithi170 ceph-mon[33394]: pgmap v917: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 34 MiB/s wr, 721 op/s; 8861/310647 objects degraded (2.852%); 13905/310647 objects misplaced (4.476%); 6.5 MiB/s, 103 objects/s recovering 2024-02-10T07:51:41.289 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:41 smithi170 ceph-mon[33394]: osdmap e423: 8 total, 8 up, 8 in 2024-02-10T07:51:41.295 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:41.341 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:41.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:41.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:41.361 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:51:41.362 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [], 2024-02-10T07:51:41.362 INFO:teuthology.orchestra.run.smithi170.stdout: "complaint_time": 30, 2024-02-10T07:51:41.362 INFO:teuthology.orchestra.run.smithi170.stdout: "num_blocked_ops": 0 2024-02-10T07:51:41.362 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:51:41.442 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:41.481 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:41.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:41.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:41 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:51:41.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:41 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:51:41.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:41 smithi008 ceph-mon[40743]: pgmap v917: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 34 MiB/s wr, 721 op/s; 8861/310647 objects degraded (2.852%); 13905/310647 objects misplaced (4.476%); 6.5 MiB/s, 103 objects/s recovering 2024-02-10T07:51:41.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:41 smithi008 ceph-mon[40743]: osdmap e423: 8 total, 8 up, 8 in 2024-02-10T07:51:41.583 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:41.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:41 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:51:41.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:41 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:51:41.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:41 smithi008 ceph-mon[29816]: pgmap v917: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 34 MiB/s wr, 721 op/s; 8861/310647 objects degraded (2.852%); 13905/310647 objects misplaced (4.476%); 6.5 MiB/s, 103 objects/s recovering 2024-02-10T07:51:41.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:41 smithi008 ceph-mon[29816]: osdmap e423: 8 total, 8 up, 8 in 2024-02-10T07:51:41.613 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:41.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:41.715 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:41.750 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 1 1 2024-02-10T07:51:41.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:41.768 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:41.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:41.869 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:41.944 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:41.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:42.046 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:42.076 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:42.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:42.143 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-10T07:51:42.178 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:42.251 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:42.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:42.353 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:42 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/310647 objects degraded (2.852%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:42.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:42 smithi170 ceph-mon[33394]: osdmap e424: 8 total, 8 up, 8 in 2024-02-10T07:51:42.395 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:42.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:42.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:42.463 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:51:42.466 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:51:42.497 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:42.524 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:42.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:42.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:42.526 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:42 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/310647 objects degraded (2.852%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:42.526 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:42 smithi008 ceph-mon[29816]: osdmap e424: 8 total, 8 up, 8 in 2024-02-10T07:51:42.526 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:42 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/310647 objects degraded (2.852%), 2 pgs degraded (PG_DEGRADED) 2024-02-10T07:51:42.526 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:42 smithi008 ceph-mon[40743]: osdmap e424: 8 total, 8 up, 8 in 2024-02-10T07:51:42.562 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:51:42.626 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:42.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:42.684 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:42.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:42.785 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:42.835 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:42.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:42.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:42.937 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:42.977 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:42.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:43.079 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:43.119 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:43.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:43.220 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:43.258 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:51:43.278 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:43.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:43.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:43.380 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:43.417 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:43.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:43.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:43.519 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:43.549 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:43.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:43.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:43.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:43 smithi008 ceph-mon[29816]: pgmap v920: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 13 MiB/s wr, 481 op/s; 8861/310068 objects degraded (2.858%); 13905/310068 objects misplaced (4.485%); 5.3 MiB/s, 84 objects/s recovering 2024-02-10T07:51:43.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:43 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3999095825' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:51:43.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:43 smithi008 ceph-mon[40743]: pgmap v920: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 13 MiB/s wr, 481 op/s; 8861/310068 objects degraded (2.858%); 13905/310068 objects misplaced (4.485%); 5.3 MiB/s, 84 objects/s recovering 2024-02-10T07:51:43.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:43 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3999095825' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:51:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:43 smithi170 ceph-mon[33394]: pgmap v920: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 13 MiB/s wr, 481 op/s; 8861/310068 objects degraded (2.858%); 13905/310068 objects misplaced (4.485%); 5.3 MiB/s, 84 objects/s recovering 2024-02-10T07:51:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:43 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3999095825' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:51:43.651 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:43.702 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:43.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:43.803 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:43.828 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:43.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:43.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:43.929 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:43.957 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:43.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:44.059 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:44.107 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:44.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:44.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:51:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:51:44.208 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:44.258 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:44.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:44.359 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:44.394 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:44 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:51:44.394 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:44 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3999095825' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:51:44.394 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:44 smithi008 ceph-mon[29816]: osdmap e425: 8 total, 8 up, 8 in 2024-02-10T07:51:44.395 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:44 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:51:44.395 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:44 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3999095825' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:51:44.395 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:44 smithi008 ceph-mon[40743]: osdmap e425: 8 total, 8 up, 8 in 2024-02-10T07:51:44.398 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:44.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:44.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:44.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:44 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:51:44.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:44 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3999095825' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:51:44.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:44 smithi170 ceph-mon[33394]: osdmap e425: 8 total, 8 up, 8 in 2024-02-10T07:51:44.501 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:44.530 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:44.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:44.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:44.631 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:44.661 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-02-10T07:51:44.661 INFO:teuthology.orchestra.run.smithi170.stdout: "size": 20, 2024-02-10T07:51:44.661 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 600, 2024-02-10T07:51:44.662 INFO:teuthology.orchestra.run.smithi170.stdout: "ops": [ 2024-02-10T07:51:44.662 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.662 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25283.0:216555 9.d 9:b6f64057:::benchmark_data_smithi008_134920_object88378:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e300)", 2024-02-10T07:51:44.662 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:45:14.186897+0000", 2024-02-10T07:51:44.662 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 390.46938738599999, 2024-02-10T07:51:44.662 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.136967228, 2024-02-10T07:51:44.662 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.662 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:44.662 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.662 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25283", 2024-02-10T07:51:44.662 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/3568554563", 2024-02-10T07:51:44.662 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 216555 2024-02-10T07:51:44.662 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.662 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:14.186897+0000", 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:14.186897+0000", 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:14.186898+0000", 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.3500000000000005e-07 2024-02-10T07:51:44.663 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.664 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.664 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.664 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:14.186902+0000", 2024-02-10T07:51:44.664 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.391e-06 2024-02-10T07:51:44.664 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.664 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.664 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.664 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:14.186916+0000", 2024-02-10T07:51:44.664 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3468e-05 2024-02-10T07:51:44.664 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.664 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.664 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.664 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:14.186923+0000", 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3629999999999996e-06 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:14.291954+0000", 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.10503127 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:14.292065+0000", 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000110931 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,5", 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:14.292176+0000", 2024-02-10T07:51:44.665 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011089799999999999 2024-02-10T07:51:44.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.666 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.666 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.666 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:14.301152+0000", 2024-02-10T07:51:44.666 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0089758119999999997 2024-02-10T07:51:44.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.666 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.666 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:44.666 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:14.301262+0000", 2024-02-10T07:51:44.666 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000110238 2024-02-10T07:51:44.666 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.666 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.666 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.666 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:14.323774+0000", 2024-02-10T07:51:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.022511384999999998 2024-02-10T07:51:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:14.323826+0000", 2024-02-10T07:51:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2139999999999999e-05 2024-02-10T07:51:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:45:14.323864+0000", 2024-02-10T07:51:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8396999999999998e-05 2024-02-10T07:51:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.667 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24356.0:3132 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 93827140437120 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:39.099189+0000", 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 305.55709523799999, 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.027226051, 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24356", 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1933562162", 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 3132 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.668 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.099189+0000", 2024-02-10T07:51:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.099189+0000", 2024-02-10T07:51:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.099191+0000", 2024-02-10T07:51:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5859999999999999e-06 2024-02-10T07:51:44.669 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.099196+0000", 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.5999999999999997e-06 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.099214+0000", 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.7642000000000002e-05 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.099222+0000", 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3820000000000004e-06 2024-02-10T07:51:44.670 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.099728+0000", 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00050553499999999999 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op must wait for map", 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.099744+0000", 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6280999999999999e-05 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.126258+0000", 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.026513756999999999 2024-02-10T07:51:44.671 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:39.126269+0000", 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1011e-05 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.121163+0000", 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.99489466599999998 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.121176+0000", 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2937e-05 2024-02-10T07:51:44.672 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.118784+0000", 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.99760719799999997 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for degraded object", 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.119390+0000", 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00060612399999999999 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.123542+0000", 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0041522989999999999 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.673 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.123693+0000", 2024-02-10T07:51:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00015089899999999999 2024-02-10T07:51:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.126415+0000", 2024-02-10T07:51:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0027221340000000002 2024-02-10T07:51:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.674 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.24356.0:3134 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 93827140437120 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-02-10T07:51:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:46:40.115889+0000", 2024-02-10T07:51:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 304.54039564599998, 2024-02-10T07:51:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.011044855, 2024-02-10T07:51:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "started", 2024-02-10T07:51:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.24356", 2024-02-10T07:51:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/1933562162", 2024-02-10T07:51:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 3134 2024-02-10T07:51:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.676 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.115889+0000", 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.115889+0000", 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.115890+0000", 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8500000000000001e-06 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.677 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.115895+0000", 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.8999999999999997e-06 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.115918+0000", 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2416e-05 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.115929+0000", 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1541e-05 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.678 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.122100+0000", 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0061709169999999997 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:40.122112+0000", 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1318000000000001e-05 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.119565+0000", 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.99745368400000001 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.679 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.680 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for degraded object", 2024-02-10T07:51:44.680 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.119825+0000", 2024-02-10T07:51:44.680 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00026007599999999998 2024-02-10T07:51:44.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.680 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.680 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.124265+0000", 2024-02-10T07:51:44.680 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0044393929999999998 2024-02-10T07:51:44.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.680 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.680 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.680 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.124345+0000", 2024-02-10T07:51:44.680 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0044999999999997e-05 2024-02-10T07:51:44.680 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:46:41.126933+0000", 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0025887150000000001 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:120693 10.9 10:97fea47f:::benchmark_data_smithi008_161452_object12232:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e355)", 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:01.312077+0000", 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 223.34420768499999, 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0263828639999999, 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.681 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 120693 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312077+0000", 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312077+0000", 2024-02-10T07:51:44.682 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.683 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.683 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.683 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312078+0000", 2024-02-10T07:51:44.683 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5060000000000001e-06 2024-02-10T07:51:44.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.683 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.683 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.683 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312082+0000", 2024-02-10T07:51:44.683 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7589999999999999e-06 2024-02-10T07:51:44.683 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.683 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.683 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.684 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312092+0000", 2024-02-10T07:51:44.684 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.6949999999999995e-06 2024-02-10T07:51:44.684 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.684 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.684 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312099+0000", 2024-02-10T07:51:44.684 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9920000000000001e-06 2024-02-10T07:51:44.684 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.684 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.684 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.684 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.320886+0000", 2024-02-10T07:51:44.684 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0087871200000000007 2024-02-10T07:51:44.684 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.685 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:44.685 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.320897+0000", 2024-02-10T07:51:44.685 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1077000000000001e-05 2024-02-10T07:51:44.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.685 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.685 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.338396+0000", 2024-02-10T07:51:44.685 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.017499551 2024-02-10T07:51:44.685 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.685 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.685 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.685 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.338459+0000", 2024-02-10T07:51:44.685 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3163999999999994e-05 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:120694 10.9 10:92494e13:::benchmark_data_smithi008_161452_object12233:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e355)", 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:01.312109+0000", 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 223.34417540300001, 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.026594655, 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:44.686 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 120694 2024-02-10T07:51:44.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.687 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.687 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.687 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312109+0000", 2024-02-10T07:51:44.687 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.687 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.687 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312109+0000", 2024-02-10T07:51:44.687 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.687 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.687 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.687 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312109+0000", 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5299999999999999e-07 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312112+0000", 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.7250000000000002e-06 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312118+0000", 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2899999999999999e-06 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.688 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312124+0000", 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.694e-06 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.320924+0000", 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0088004669999999993 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.320933+0000", 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.4290000000000001e-06 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.689 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.338658+0000", 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0177253740000001 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.338703+0000", 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5222999999999997e-05 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:120696 10.9 10:9a6e9288:::benchmark_data_smithi008_161452_object12235:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e355)", 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:01.312130+0000", 2024-02-10T07:51:44.690 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 223.34415399299999, 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.02666865, 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 120696 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312130+0000", 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.691 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312130+0000", 2024-02-10T07:51:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312131+0000", 2024-02-10T07:51:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.4200000000000001e-07 2024-02-10T07:51:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312133+0000", 2024-02-10T07:51:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.0990000000000001e-06 2024-02-10T07:51:44.692 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312139+0000", 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.19e-06 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312144+0000", 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9289999999999997e-06 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.320959+0000", 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0088154630000000008 2024-02-10T07:51:44.693 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.320968+0000", 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3250000000000008e-06 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.338747+0000", 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.017779628 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.338799+0000", 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.1573999999999998e-05 2024-02-10T07:51:44.694 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:120708 10.9 10:91b593c6:::benchmark_data_smithi008_161452_object12247:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e355)", 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:01.312152+0000", 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 223.344132124, 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0381779710000001, 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "reached pg", 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 120708 2024-02-10T07:51:44.695 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.696 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.696 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.696 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312152+0000", 2024-02-10T07:51:44.696 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.696 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.696 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312152+0000", 2024-02-10T07:51:44.696 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.696 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.696 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.696 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.696 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312153+0000", 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8500000000000002e-07 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312154+0000", 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9470000000000002e-06 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312160+0000", 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9290000000000003e-06 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.697 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312166+0000", 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.2129999999999999e-06 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.320993+0000", 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0088278560000000002 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.321002+0000", 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.0490000000000004e-06 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.338828+0000", 2024-02-10T07:51:44.698 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0178264669999999 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for missing object", 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.338921+0000", 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.2927e-05 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.350241+0000", 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.011320514 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.350330+0000", 2024-02-10T07:51:44.699 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.8684000000000007e-05 2024-02-10T07:51:44.700 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.700 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.700 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.700 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.700 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.700 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:120710 10.9 10:9ba4cca3:::benchmark_data_smithi008_161452_object12249:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-02-10T07:51:44.700 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:01.312379+0000", 2024-02-10T07:51:44.700 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 223.34390564500001, 2024-02-10T07:51:44.700 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.036246204, 2024-02-10T07:51:44.700 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.700 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:44.700 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.700 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:44.700 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 120710 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312379+0000", 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312379+0000", 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.701 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312379+0000", 2024-02-10T07:51:44.702 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.4700000000000001e-07 2024-02-10T07:51:44.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.702 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.702 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312383+0000", 2024-02-10T07:51:44.702 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7340000000000002e-06 2024-02-10T07:51:44.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.702 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.702 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312392+0000", 2024-02-10T07:51:44.702 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.9390000000000003e-06 2024-02-10T07:51:44.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.702 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.702 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.312398+0000", 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3430000000000001e-06 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.321028+0000", 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0086296259999999996 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.321036+0000", 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.3180000000000007e-06 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.703 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.704 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.339063+0000", 2024-02-10T07:51:44.704 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0180269639999999 2024-02-10T07:51:44.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.704 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.704 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.339180+0000", 2024-02-10T07:51:44.704 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011684900000000001 2024-02-10T07:51:44.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.704 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:44.704 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.339596+0000", 2024-02-10T07:51:44.704 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000416322 2024-02-10T07:51:44.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.704 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.348418+0000", 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.008821888 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.348488+0000", 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.9467000000000004e-05 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.348503+0000", 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5051e-05 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.705 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.348566+0000", 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3622999999999997e-05 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.348625+0000", 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.8332999999999999e-05 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:120712 10.9 10:987befbd:::benchmark_data_smithi008_161452_object12251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-02-10T07:51:44.706 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:01.327063+0000", 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 223.329221566, 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0235674509999999, 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 120712 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.327063+0000", 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.327063+0000", 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.327064+0000", 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.598e-06 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.327068+0000", 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.726e-06 2024-02-10T07:51:44.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.327078+0000", 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0448000000000001e-05 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.327088+0000", 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.251e-06 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.327917+0000", 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00082891200000000001 2024-02-10T07:51:44.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.327927+0000", 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0657e-05 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.346006+0000", 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0180788039999999 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.346104+0000", 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.7552000000000004e-05 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.710 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.346190+0000", 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6093000000000004e-05 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.348631+0000", 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0024415589999999998 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.350071+0000", 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001440006 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.711 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.350485+0000", 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000413268 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.350535+0000", 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0599000000000001e-05 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.350630+0000", 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.4977999999999998e-05 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.712 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:120717 10.9 10:9a81e745:::benchmark_data_smithi008_161452_object12256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:01.335747+0000", 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 223.320537694, 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.016859535, 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 120717 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.713 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.335747+0000", 2024-02-10T07:51:44.714 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.714 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.335747+0000", 2024-02-10T07:51:44.714 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.714 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.714 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.714 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.714 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.335749+0000", 2024-02-10T07:51:44.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:44.715 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:44.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:44.716 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9599999999999999e-06 2024-02-10T07:51:44.716 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.716 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.716 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.335755+0000", 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.703e-06 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.335776+0000", 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.1229999999999998e-05 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.335786+0000", 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.5259999999999999e-06 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.717 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.335857+0000", 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.0821000000000005e-05 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for peered", 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:01.335868+0000", 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0868e-05 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.348664+0000", 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.012796279 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.718 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.348804+0000", 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014020300000000001 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.348939+0000", 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000134542 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.350434+0000", 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0014957740000000001 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.719 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.352409+0000", 2024-02-10T07:51:44.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001974689 2024-02-10T07:51:44.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.720 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.352455+0000", 2024-02-10T07:51:44.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5636000000000001e-05 2024-02-10T07:51:44.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.720 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:44.720 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.352519+0000", 2024-02-10T07:51:44.720 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4380000000000004e-05 2024-02-10T07:51:44.720 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.720 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:02.352606+0000", 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6923999999999996e-05 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:125415 10.9 10:904b9b77:::benchmark_data_smithi008_161452_object16954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:07.369724+0000", 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 217.28656016599999, 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.122842057, 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:44.721 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 125415 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.369724+0000", 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.369724+0000", 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.722 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.369726+0000", 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9199999999999998e-06 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.369736+0000", 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0139999999999999e-05 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.369749+0000", 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.2771e-05 2024-02-10T07:51:44.723 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.369758+0000", 2024-02-10T07:51:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.6689999999999995e-06 2024-02-10T07:51:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.370892+0000", 2024-02-10T07:51:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.001134536 2024-02-10T07:51:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.370998+0000", 2024-02-10T07:51:44.724 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000105559 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.371102+0000", 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000103902 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.375939+0000", 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0048375620000000001 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.378119+0000", 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0021803450000000002 2024-02-10T07:51:44.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.492473+0000", 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11435376 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.492530+0000", 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6932000000000003e-05 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.492566+0000", 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.5960999999999997e-05 2024-02-10T07:51:44.726 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:125421 10.9 10:9dcb00cb:::benchmark_data_smithi008_161452_object16960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:07.370764+0000", 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 217.28552034800001, 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12199818, 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 125421 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.727 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.370764+0000", 2024-02-10T07:51:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.370764+0000", 2024-02-10T07:51:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.370765+0000", 2024-02-10T07:51:44.728 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.0100000000000003e-07 2024-02-10T07:51:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.370769+0000", 2024-02-10T07:51:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2100000000000003e-06 2024-02-10T07:51:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.370781+0000", 2024-02-10T07:51:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1593000000000001e-05 2024-02-10T07:51:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.729 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.370789+0000", 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.977e-06 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.373528+0000", 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0027393439999999999 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.373631+0000", 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010298699999999999 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:44.730 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.373736+0000", 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010516799999999999 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.375946+0000", 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0022097110000000001 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.378148+0000", 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0022021250000000001 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.492664+0000", 2024-02-10T07:51:44.731 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.114516494 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.492715+0000", 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0658999999999997e-05 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.492762+0000", 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.7011000000000001e-05 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.732 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:125425 10.9 10:9e890f47:::benchmark_data_smithi008_161452_object16964:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:07.371714+0000", 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 217.28456977799999, 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.125963453, 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 125425 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.733 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.371714+0000", 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.371714+0000", 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.371715+0000", 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.0189999999999999e-06 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.734 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.371720+0000", 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.2799999999999997e-06 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.371736+0000", 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6223999999999999e-05 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.371744+0000", 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.7889999999999995e-06 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.375974+0000", 2024-02-10T07:51:44.735 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0042301259999999999 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.376076+0000", 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010203 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.376176+0000", 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00010025499999999999 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.378182+0000", 2024-02-10T07:51:44.736 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0020055619999999998 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.378419+0000", 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00023685399999999999 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.497584+0000", 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.11916498 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.497640+0000", 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.6217000000000001e-05 2024-02-10T07:51:44.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:07.497678+0000", 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8117000000000001e-05 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:138945 10.c 10:360caef6:::benchmark_data_smithi008_161452_object30484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:18.568261+0000", 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 206.08802369200001, 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.17422686000000001, 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.738 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 138945 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.568261+0000", 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.568261+0000", 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.739 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.568261+0000", 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.37e-07 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.568264+0000", 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.706e-06 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.568271+0000", 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8549999999999996e-06 2024-02-10T07:51:44.740 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.568277+0000", 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.9440000000000002e-06 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.574241+0000", 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0059648649999999998 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.574341+0000", 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.9190000000000004e-05 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.741 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,4", 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.574408+0000", 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.7638000000000005e-05 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.577312+0000", 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0029036660000000001 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.578873+0000", 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0015608460000000001 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.742 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.742414+0000", 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.16354117300000001 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.742459+0000", 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.5422e-05 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.742487+0000", 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.8017999999999998e-05 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.743 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:139213 10.c 10:3ec12cd2:::benchmark_data_smithi008_161452_object30752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:18.955865+0000", 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 205.700418896, 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.16217379100000001, 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 139213 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.744 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.955865+0000", 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.955865+0000", 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.955867+0000", 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.24e-06 2024-02-10T07:51:44.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.955872+0000", 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9509999999999997e-06 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.955885+0000", 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.3402e-05 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.955892+0000", 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.3799999999999996e-06 2024-02-10T07:51:44.746 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.955961+0000", 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.8923000000000004e-05 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.956091+0000", 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000129648 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 0,4", 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.956209+0000", 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000118168 2024-02-10T07:51:44.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.748 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.748 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.748 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.960686+0000", 2024-02-10T07:51:44.748 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0044768350000000002 2024-02-10T07:51:44.748 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.748 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.748 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:44.748 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.963215+0000", 2024-02-10T07:51:44.748 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.002529108 2024-02-10T07:51:44.748 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.748 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.748 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.748 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:19.117742+0000", 2024-02-10T07:51:44.748 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.15452663899999999 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:19.117927+0000", 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00018536900000000001 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:19.118039+0000", 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00011212799999999999 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.749 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:139219 10.9 10:90641e63:::benchmark_data_smithi008_161452_object30758:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:18.959929+0000", 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 205.69635557199999, 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.157732123, 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 139219 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.959929+0000", 2024-02-10T07:51:44.750 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.959929+0000", 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.959931+0000", 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.305e-06 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.959937+0000", 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.0569999999999998e-06 2024-02-10T07:51:44.751 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.959959+0000", 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2084999999999999e-05 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.959969+0000", 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.8339999999999994e-06 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.960031+0000", 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.2327000000000001e-05 2024-02-10T07:51:44.752 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.960152+0000", 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000120426 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.960263+0000", 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000110884 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.963203+0000", 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0029399579999999999 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.753 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.754 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.754 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.963628+0000", 2024-02-10T07:51:44.754 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00042569100000000001 2024-02-10T07:51:44.754 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.754 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.754 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.754 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:19.117494+0000", 2024-02-10T07:51:44.754 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.153865798 2024-02-10T07:51:44.754 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.754 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.754 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:44.754 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:19.117581+0000", 2024-02-10T07:51:44.754 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.7377e-05 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:19.117661+0000", 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 7.9381000000000005e-05 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.15630.0:139224 10.9 10:98ae6f45:::benchmark_data_smithi008_161452_object30763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:48:18.967395+0000", 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 205.688889612, 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.15972101599999999, 2024-02-10T07:51:44.755 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.756 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:44.756 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.756 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.15630", 2024-02-10T07:51:44.756 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/414024437", 2024-02-10T07:51:44.756 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 139224 2024-02-10T07:51:44.756 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.756 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.756 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.756 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.756 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.967395+0000", 2024-02-10T07:51:44.756 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.756 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.756 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.756 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.967395+0000", 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.967396+0000", 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.5179999999999999e-06 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.967401+0000", 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.6959999999999996e-06 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.757 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.967420+0000", 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.8763999999999999e-05 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.967429+0000", 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 9.7310000000000005e-06 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.967493+0000", 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.4028000000000004e-05 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.758 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.759 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.967619+0000", 2024-02-10T07:51:44.759 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000125566 2024-02-10T07:51:44.759 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.759 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.759 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 4,5", 2024-02-10T07:51:44.759 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.967731+0000", 2024-02-10T07:51:44.759 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000111815 2024-02-10T07:51:44.759 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.759 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.759 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:44.759 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.969714+0000", 2024-02-10T07:51:44.759 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0019832809999999999 2024-02-10T07:51:44.759 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.759 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:18.969910+0000", 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019555500000000001 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:19.126806+0000", 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.156896339 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:19.126993+0000", 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00018735900000000001 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.760 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:48:19.127116+0000", 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000122364 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25838.0:104213 11.f 11:fdf17b83:::benchmark_data_smithi008_186804_object104212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e421)", 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:51:34.778421+0000", 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 9.8778628790000003, 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.115207252, 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:44.761 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25838", 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 104213 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:34.778421+0000", 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:34.778421+0000", 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.762 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:34.778423+0000", 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.714e-06 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:34.778461+0000", 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.7968e-05 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:34.778478+0000", 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.6484e-05 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.763 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:34.778486+0000", 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 8.5269999999999998e-06 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:34.778549+0000", 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.3391000000000003e-05 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:34.778744+0000", 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00019475499999999999 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.764 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:34.778892+0000", 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00014775300000000001 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:34.781143+0000", 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0022508350000000001 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:34.786831+0000", 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0056877580000000002 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.765 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:34.890097+0000", 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.10326654 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:34.890163+0000", 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.6174999999999996e-05 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:34.893629+0000", 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0034653520000000001 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.766 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.767 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.767 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25838.0:105945 11.f 11:f159ef99:::benchmark_data_smithi008_186804_object1183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-02-10T07:51:44.767 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:51:38.371719+0000", 2024-02-10T07:51:44.767 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 6.2845649809999999, 2024-02-10T07:51:44.767 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.13587703100000001, 2024-02-10T07:51:44.767 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.767 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:44.767 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.767 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25838", 2024-02-10T07:51:44.767 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:51:44.767 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 105945 2024-02-10T07:51:44.767 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.767 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.767 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.769 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.769 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.371719+0000", 2024-02-10T07:51:44.769 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.769 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.769 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.769 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.769 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.371719+0000", 2024-02-10T07:51:44.769 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.769 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.769 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.769 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.769 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.371721+0000", 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.9999999999999999e-06 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.371726+0000", 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.9339999999999997e-06 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.371748+0000", 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 2.2064000000000001e-05 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.770 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.371785+0000", 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.6480000000000003e-05 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.372621+0000", 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00083576199999999998 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.372800+0000", 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00017965099999999999 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:51:44.771 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.372930+0000", 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000129591 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.375199+0000", 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0022694400000000002 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.380228+0000", 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.005029286 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.772 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.504423+0000", 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.124194263 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.504477+0000", 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.3814999999999999e-05 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.507596+0000", 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0031197450000000002 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: "description": "osd_op(client.25838.0:105955 11.f 11:fccd0153:::benchmark_data_smithi008_186804_object1193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: "initiated_at": "2024-02-10T07:51:38.387251+0000", 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: "age": 6.2690328089999996, 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.12611899800000001, 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: "type_data": { 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: "client_info": { 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: "client": "client.25838", 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: "tid": 105955 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: "events": [ 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "initiated", 2024-02-10T07:51:44.774 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.387251+0000", 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "header_read", 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.387251+0000", 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "throttled", 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.387253+0000", 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1009999999999999e-06 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "all_read", 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.387258+0000", 2024-02-10T07:51:44.775 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 4.972e-06 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "dispatched", 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.387269+0000", 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 1.1439e-05 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "queued_for_pg", 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.387275+0000", 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 6.1820000000000003e-06 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "reached_pg", 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.387326+0000", 2024-02-10T07:51:44.776 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 5.0557999999999999e-05 2024-02-10T07:51:44.777 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.777 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.777 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "started", 2024-02-10T07:51:44.777 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.387441+0000", 2024-02-10T07:51:44.777 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.000114819 2024-02-10T07:51:44.777 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.777 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.777 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "waiting for subops from 1,2", 2024-02-10T07:51:44.777 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.387561+0000", 2024-02-10T07:51:44.777 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.00012084200000000001 2024-02-10T07:51:44.777 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.777 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.777 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "op_commit", 2024-02-10T07:51:44.777 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.389730+0000", 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0021685860000000001 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.393540+0000", 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.0038100370000000001 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.507641+0000", 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.114100585 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "commit_sent", 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.507680+0000", 2024-02-10T07:51:44.778 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 3.8948e-05 2024-02-10T07:51:44.779 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-02-10T07:51:44.779 INFO:teuthology.orchestra.run.smithi170.stdout: { 2024-02-10T07:51:44.779 INFO:teuthology.orchestra.run.smithi170.stdout: "event": "done", 2024-02-10T07:51:44.779 INFO:teuthology.orchestra.run.smithi170.stdout: "time": "2024-02-10T07:51:38.513370+0000", 2024-02-10T07:51:44.779 INFO:teuthology.orchestra.run.smithi170.stdout: "duration": 0.005690929 2024-02-10T07:51:44.779 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.779 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.779 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.779 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-02-10T07:51:44.779 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-02-10T07:51:44.779 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-02-10T07:51:44.816 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:44.844 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:44.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:44.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:44.946 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:44.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:44.998 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:44.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:45.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:45.127 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:45.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:45.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:45.229 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:45.258 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:45.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:45.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:45 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:45 smithi008 ceph-mon[29816]: pgmap v922: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.8 KiB/s rd, 11 MiB/s wr, 763 op/s; 8861/304455 objects degraded (2.910%); 13905/304455 objects misplaced (4.567%); 5.3 MiB/s, 84 objects/s recovering 2024-02-10T07:51:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:45 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:51:45.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:45 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:51:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:45 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:45 smithi008 ceph-mon[40743]: pgmap v922: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.8 KiB/s rd, 11 MiB/s wr, 763 op/s; 8861/304455 objects degraded (2.910%); 13905/304455 objects misplaced (4.567%); 5.3 MiB/s, 84 objects/s recovering 2024-02-10T07:51:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:45 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:51:45.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:45 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:51:45.360 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:45.389 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:45.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:45.491 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:45.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:45.540 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:45.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:45 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:45 smithi170 ceph-mon[33394]: pgmap v922: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.8 KiB/s rd, 11 MiB/s wr, 763 op/s; 8861/304455 objects degraded (2.910%); 13905/304455 objects misplaced (4.567%); 5.3 MiB/s, 84 objects/s recovering 2024-02-10T07:51:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:45 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:51:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:45 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:51:45.642 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:45.667 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:45.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:45.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:45.707 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T07:51:45.770 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:45.820 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:45.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:45.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:45.923 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:45.950 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:45.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:45.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:46.052 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:46.053 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:51:46.077 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:46.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:46.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:46.179 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:46.209 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:46.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:46.311 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:46.350 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:46.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:46.452 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:46.481 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:46 smithi170 ceph-mon[33394]: osdmap e426: 8 total, 8 up, 8 in 2024-02-10T07:51:46.502 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:46.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:46.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:46.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:46 smithi008 ceph-mon[29816]: osdmap e426: 8 total, 8 up, 8 in 2024-02-10T07:51:46.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:46 smithi008 ceph-mon[40743]: osdmap e426: 8 total, 8 up, 8 in 2024-02-10T07:51:46.604 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:46.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:46.667 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:46.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:46.768 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:46.800 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:46.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:46.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:46.902 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:46.941 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:46.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:46.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:47.043 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:47.071 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:47.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:47.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:47.173 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:47.220 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:47.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:47.322 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:47.367 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:47.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:47.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:47.468 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:47.522 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:47.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:47.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:47 smithi170 ceph-mon[33394]: pgmap v924: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 865 B/s rd, 0 B/s wr, 567 op/s; 8861/300597 objects degraded (2.948%); 13572/300597 objects misplaced (4.515%); 3.5 MiB/s, 56 objects/s recovering 2024-02-10T07:51:47.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:47 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/300597 objects degraded (2.948%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:51:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:47 smithi008 ceph-mon[29816]: pgmap v924: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 865 B/s rd, 0 B/s wr, 567 op/s; 8861/300597 objects degraded (2.948%); 13572/300597 objects misplaced (4.515%); 3.5 MiB/s, 56 objects/s recovering 2024-02-10T07:51:47.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:47 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/300597 objects degraded (2.948%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:51:47.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:47 smithi008 ceph-mon[40743]: pgmap v924: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 865 B/s rd, 0 B/s wr, 567 op/s; 8861/300597 objects degraded (2.948%); 13572/300597 objects misplaced (4.515%); 3.5 MiB/s, 56 objects/s recovering 2024-02-10T07:51:47.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:47 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/300597 objects degraded (2.948%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:51:47.624 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:47.652 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:47.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:47.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:47.754 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:47.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:47.813 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:47.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:47.901 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:51:47.914 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:47.942 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:47.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:47.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:47.949 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:51:47.991 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:51:48.041 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:51:48.044 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:48.079 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:48.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:48.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:48.083 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:51:48.128 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:51:48.178 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:51:48.181 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:48.207 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:48.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:48.221 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:51:48.232 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:51:48.232 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:51:48.232 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:51:48.232 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:51:48.232 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:51:48.232 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:51:48.232 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:51:48.232 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:51:48.309 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:48.327 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.1 to 1 (10000) 2024-02-10T07:51:48.355 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:48.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:48.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:48.457 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:48.495 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:48.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:48.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:48.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:48 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2187540459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:51:48.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:48 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/394020052' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:51:48.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:48 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:51:48.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:48 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2187540459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:51:48.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:48 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/394020052' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:51:48.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:48 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:51:48.598 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:48 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2187540459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:51:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:48 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/394020052' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:51:48.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:48 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:51:48.637 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:48.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:48.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:48.738 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:48.780 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:48.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:48.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:48.882 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:48.883 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:51:48.913 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:48.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:48.924 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:51:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:51:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:51:49.015 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:49.040 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:49.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:49.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:49.142 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:49.190 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:49.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:49.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:49.226 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:51:49.226 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:51:49.292 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:49.334 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:49.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:49.440 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:49.481 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:49.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:49.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:49.499 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:49 smithi008 ceph-mon[29816]: pgmap v925: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 841 B/s rd, 0 B/s wr, 558 op/s; 8861/299886 objects degraded (2.955%); 13572/299886 objects misplaced (4.526%); 3.4 MiB/s, 54 objects/s recovering 2024-02-10T07:51:49.500 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:49 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:51:49.500 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:49 smithi008 ceph-mon[29816]: osdmap e427: 8 total, 8 up, 8 in 2024-02-10T07:51:49.500 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:51:49.500 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:49 smithi008 ceph-mon[40743]: pgmap v925: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 841 B/s rd, 0 B/s wr, 558 op/s; 8861/299886 objects degraded (2.955%); 13572/299886 objects misplaced (4.526%); 3.4 MiB/s, 54 objects/s recovering 2024-02-10T07:51:49.500 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:49 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:51:49.500 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:49 smithi008 ceph-mon[40743]: osdmap e427: 8 total, 8 up, 8 in 2024-02-10T07:51:49.500 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:51:49.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:49 smithi170 ceph-mon[33394]: pgmap v925: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 841 B/s rd, 0 B/s wr, 558 op/s; 8861/299886 objects degraded (2.955%); 13572/299886 objects misplaced (4.526%); 3.4 MiB/s, 54 objects/s recovering 2024-02-10T07:51:49.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:49 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:51:49.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:49 smithi170 ceph-mon[33394]: osdmap e427: 8 total, 8 up, 8 in 2024-02-10T07:51:49.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:51:49.584 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:49.630 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:49.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:49.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:49.732 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:49.761 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:49.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:49.863 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:49.889 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 3 1 2024-02-10T07:51:49.890 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:49.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:49.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:49.958 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-10T07:51:49.992 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:50.010 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:51:50.029 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:50.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:50.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:50.131 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:50.158 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:50.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:50.260 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:50.268 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:51:50.314 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:50.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:50.415 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:50.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:50.468 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:50.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:50.569 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:50.602 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:50.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:50.704 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:50.742 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:50.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:50.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:50.844 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:50.872 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:50.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:50.974 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:51.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:51.025 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:51.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:51.126 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:51.161 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:51:51.177 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:51.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:51.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:51.279 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:51.307 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:51.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:51.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:51.409 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:51.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:51.436 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:51.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:51.537 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:51.574 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:51.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:51.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:51 smithi008 ceph-mon[29816]: pgmap v927: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 968 op/s; 8861/290046 objects degraded (3.055%); 13226/290046 objects misplaced (4.560%); 6.1 MiB/s, 98 objects/s recovering 2024-02-10T07:51:51.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:51 smithi008 ceph-mon[40743]: pgmap v927: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 968 op/s; 8861/290046 objects degraded (3.055%); 13226/290046 objects misplaced (4.560%); 6.1 MiB/s, 98 objects/s recovering 2024-02-10T07:51:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:51 smithi170 ceph-mon[33394]: pgmap v927: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 968 op/s; 8861/290046 objects degraded (3.055%); 13226/290046 objects misplaced (4.560%); 6.1 MiB/s, 98 objects/s recovering 2024-02-10T07:51:51.676 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:51.707 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:51.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:51.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:51.809 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:51.856 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:51.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:51.958 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:51.997 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:51.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:52.099 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:52.146 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:52.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:52.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:52.247 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:52.276 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:52.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:52.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:52.378 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:52.382 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:51:52.434 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:52.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:52.472 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:52 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/290046 objects degraded (3.055%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:51:52.472 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:52 smithi170 ceph-mon[33394]: osdmap e428: 8 total, 8 up, 8 in 2024-02-10T07:51:52.523 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:51:52.523 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:51:52.537 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:52.570 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:52.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:52.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:52.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:52 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/290046 objects degraded (3.055%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:51:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:52 smithi008 ceph-mon[29816]: osdmap e428: 8 total, 8 up, 8 in 2024-02-10T07:51:52.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:52 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/290046 objects degraded (3.055%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:51:52.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:52 smithi008 ceph-mon[40743]: osdmap e428: 8 total, 8 up, 8 in 2024-02-10T07:51:52.672 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:52.700 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:52.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:52.802 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:52.835 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:52.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:52.936 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:52.962 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:52.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:53.063 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:53.104 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:53.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:53.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:53.205 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:53.251 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:53.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:53.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:53.352 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:53.380 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:53.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:53.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:53.453 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:53 smithi170 ceph-mon[33394]: pgmap v929: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 744 B/s rd, 0 B/s wr, 541 op/s; 8861/289431 objects degraded (3.062%); 13226/289431 objects misplaced (4.570%); 3.1 MiB/s, 50 objects/s recovering 2024-02-10T07:51:53.483 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:53.492 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-10T07:51:53.524 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:53.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:53.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:53.611 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:53 smithi008 ceph-mon[29816]: pgmap v929: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 744 B/s rd, 0 B/s wr, 541 op/s; 8861/289431 objects degraded (3.062%); 13226/289431 objects misplaced (4.570%); 3.1 MiB/s, 50 objects/s recovering 2024-02-10T07:51:53.611 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:53 smithi008 ceph-mon[40743]: pgmap v929: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 744 B/s rd, 0 B/s wr, 541 op/s; 8861/289431 objects degraded (3.062%); 13226/289431 objects misplaced (4.570%); 3.1 MiB/s, 50 objects/s recovering 2024-02-10T07:51:53.626 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:53.654 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:53.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:53.756 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:53.786 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:53.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:53.815 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:51:53.879 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:51:53.887 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:53.954 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:53.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:54.056 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:54.083 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:54.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:54.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:54.185 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:54.222 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:51:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:51:54.231 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:54.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:54.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:54.333 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:54.376 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:54.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:54.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:54.478 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:54.528 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:54.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:54.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:54.630 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:54.658 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:54.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:54.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:54.760 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:54.827 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:54.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:54.929 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:54.956 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:54.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:55.058 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:55.088 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:55.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:55.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:55.190 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:55.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:55.222 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:55.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:55.323 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:55.380 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:55.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:55.390 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:51:55.482 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:55.513 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:55.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:55.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:55.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:55 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:55.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:55 smithi170 ceph-mon[33394]: pgmap v930: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 722 op/s; 8861/283272 objects degraded (3.128%); 13226/283272 objects misplaced (4.669%); 2.7 MiB/s, 43 objects/s recovering 2024-02-10T07:51:55.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:55 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/368275341' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:51:55.544 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:55 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:51:55.576 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:55 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:55.576 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:55 smithi008 ceph-mon[29816]: pgmap v930: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 722 op/s; 8861/283272 objects degraded (3.128%); 13226/283272 objects misplaced (4.669%); 2.7 MiB/s, 43 objects/s recovering 2024-02-10T07:51:55.576 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:55 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/368275341' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:51:55.576 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:55 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:51:55.576 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:55 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:51:55.576 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:55 smithi008 ceph-mon[40743]: pgmap v930: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 722 op/s; 8861/283272 objects degraded (3.128%); 13226/283272 objects misplaced (4.669%); 2.7 MiB/s, 43 objects/s recovering 2024-02-10T07:51:55.576 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:55 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/368275341' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:51:55.577 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:55 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:51:55.615 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:55.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:55.647 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:55.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:55.703 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:51:55.704 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:51:55.748 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:55.794 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:55.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:55.895 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:55.944 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:55.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:55.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:56.046 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:56.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:56.086 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:56.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:56.187 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:56.231 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:56.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:56.332 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:56.367 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-10T07:51:56.371 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:56.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:56.387 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.3 to 1 (10000) 2024-02-10T07:51:56.473 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:56.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:56.503 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:56.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:56.605 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:56 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:51:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:56 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:51:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:56 smithi170 ceph-mon[33394]: osdmap e429: 8 total, 8 up, 8 in 2024-02-10T07:51:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:56 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3358596134' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:51:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:56 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:51:56.634 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:56.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:56.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:56.674 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:56 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:51:56.674 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:56 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:51:56.674 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:56 smithi008 ceph-mon[29816]: osdmap e429: 8 total, 8 up, 8 in 2024-02-10T07:51:56.674 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:56 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3358596134' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:51:56.674 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:56 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:51:56.674 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:56 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:51:56.674 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:56 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:51:56.674 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:56 smithi008 ceph-mon[40743]: osdmap e429: 8 total, 8 up, 8 in 2024-02-10T07:51:56.674 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:56 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3358596134' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:51:56.674 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:56 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:51:56.736 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:56.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:56.794 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:56.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:56.895 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:56.940 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:56.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:57.041 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:57.097 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:57.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:57.199 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:57.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:57.227 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:57.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:57.328 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:57.354 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:57.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:57.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:57.456 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:57.497 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:57.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:57.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:57 smithi008 ceph-mon[29816]: pgmap v932: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 881 op/s; 8861/279213 objects degraded (3.174%); 12857/279213 objects misplaced (4.605%); 5.7 MiB/s, 91 objects/s recovering 2024-02-10T07:51:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:57 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:51:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:57 smithi008 ceph-mon[29816]: osdmap e430: 8 total, 8 up, 8 in 2024-02-10T07:51:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:57 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/279213 objects degraded (3.174%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:51:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:57 smithi008 ceph-mon[40743]: pgmap v932: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 881 op/s; 8861/279213 objects degraded (3.174%); 12857/279213 objects misplaced (4.605%); 5.7 MiB/s, 91 objects/s recovering 2024-02-10T07:51:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:57 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:51:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:57 smithi008 ceph-mon[40743]: osdmap e430: 8 total, 8 up, 8 in 2024-02-10T07:51:57.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:57 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/279213 objects degraded (3.174%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:51:57.599 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:57 smithi170 ceph-mon[33394]: pgmap v932: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 881 op/s; 8861/279213 objects degraded (3.174%); 12857/279213 objects misplaced (4.605%); 5.7 MiB/s, 91 objects/s recovering 2024-02-10T07:51:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:57 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:51:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:57 smithi170 ceph-mon[33394]: osdmap e430: 8 total, 8 up, 8 in 2024-02-10T07:51:57.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:57 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/279213 objects degraded (3.174%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:51:57.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:57.645 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:57.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:57.713 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 4 1 2024-02-10T07:51:57.746 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:57.774 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:57.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:57.875 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:57.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:57.905 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:57.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:58.006 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:58.032 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:58.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:58.133 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:58.170 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:58.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:58.272 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:51:58.301 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:58.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:51:58.403 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:51:58.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:58.467 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:58.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:51:58.568 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:58.597 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:58.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:58.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:51:58.699 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:58.728 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:58.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:58.830 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:58.860 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:58.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:58.961 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:51:59.005 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:59.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:59.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:51:59.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:51:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:51:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:51:59.107 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:59.137 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:59.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:59.239 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:51:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:59.279 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:59.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:51:59.380 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:59.407 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:59.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:59.475 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:51:59.508 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:51:59.537 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:59.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:59.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:51:59.629 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:51:59 smithi008 ceph-mon[40743]: pgmap v934: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 841 B/s rd, 0 B/s wr, 596 op/s; 8861/278565 objects degraded (3.181%); 12857/278565 objects misplaced (4.615%); 3.8 MiB/s, 60 objects/s recovering 2024-02-10T07:51:59.637 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:51:59 smithi008 ceph-mon[29816]: pgmap v934: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 841 B/s rd, 0 B/s wr, 596 op/s; 8861/278565 objects degraded (3.181%); 12857/278565 objects misplaced (4.615%); 3.8 MiB/s, 60 objects/s recovering 2024-02-10T07:51:59.638 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:51:59 smithi170 ceph-mon[33394]: pgmap v934: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 841 B/s rd, 0 B/s wr, 596 op/s; 8861/278565 objects degraded (3.181%); 12857/278565 objects misplaced (4.615%); 3.8 MiB/s, 60 objects/s recovering 2024-02-10T07:51:59.639 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:51:59.687 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:51:59.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:59.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:51:59.789 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:51:59.819 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:59.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:59.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:51:59.921 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:51:59.948 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:51:59.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:51:59.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:00.050 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:00.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:00.091 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:00.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:00.192 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:00.232 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:00.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:00.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:00.334 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:00.367 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:00.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:00.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:00.469 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:00.509 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:00.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:00.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:00.581 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:00 smithi008 ceph-mon[40743]: 5.1b scrub starts 2024-02-10T07:52:00.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:00 smithi008 ceph-mon[40743]: 5.1b scrub ok 2024-02-10T07:52:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:00 smithi008 ceph-mon[29816]: 5.1b scrub starts 2024-02-10T07:52:00.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:00 smithi008 ceph-mon[29816]: 5.1b scrub ok 2024-02-10T07:52:00.611 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:00.643 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:00.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:00.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:00.745 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:00.776 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:00.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:00.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:00.878 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:00.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:00 smithi170 ceph-mon[33394]: 5.1b scrub starts 2024-02-10T07:52:00.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:00 smithi170 ceph-mon[33394]: 5.1b scrub ok 2024-02-10T07:52:00.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:00.943 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:00.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:00.963 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:52:01.012 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:52:01.044 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:01.050 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:52:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:01.096 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:01.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:01.103 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:52:01.142 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:52:01.191 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:52:01.197 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:01.231 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:01.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:01.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:01.234 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:52:01.272 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:52:01.280 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:52:01.280 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:52:01.280 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:52:01.280 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:52:01.280 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:52:01.280 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:52:01.281 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:52:01.281 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:52:01.333 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:01.381 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:01.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:01.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:01.472 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:52:01.483 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:01.509 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:01 smithi170 ceph-mon[33394]: 6.c scrub starts 2024-02-10T07:52:01.509 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:01 smithi170 ceph-mon[33394]: 6.c scrub ok 2024-02-10T07:52:01.509 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:01 smithi170 ceph-mon[33394]: pgmap v935: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 8.7 KiB/s rd, 0 B/s wr, 873 op/s; 8861/268785 objects degraded (3.297%); 12496/268785 objects misplaced (4.649%); 5.7 MiB/s, 91 objects/s recovering 2024-02-10T07:52:01.509 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:01 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2001041768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:52:01.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:01.513 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:01.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:01.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:01 smithi008 ceph-mon[29816]: 6.c scrub starts 2024-02-10T07:52:01.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:01 smithi008 ceph-mon[29816]: 6.c scrub ok 2024-02-10T07:52:01.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:01 smithi008 ceph-mon[29816]: pgmap v935: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 8.7 KiB/s rd, 0 B/s wr, 873 op/s; 8861/268785 objects degraded (3.297%); 12496/268785 objects misplaced (4.649%); 5.7 MiB/s, 91 objects/s recovering 2024-02-10T07:52:01.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:01 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2001041768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:52:01.526 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:01 smithi008 ceph-mon[40743]: 6.c scrub starts 2024-02-10T07:52:01.526 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:01 smithi008 ceph-mon[40743]: 6.c scrub ok 2024-02-10T07:52:01.526 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:01 smithi008 ceph-mon[40743]: pgmap v935: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 8.7 KiB/s rd, 0 B/s wr, 873 op/s; 8861/268785 objects degraded (3.297%); 12496/268785 objects misplaced (4.649%); 5.7 MiB/s, 91 objects/s recovering 2024-02-10T07:52:01.526 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:01 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2001041768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:52:01.614 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:01.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:01.689 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:01.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:01.790 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:01.855 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:01.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:01.958 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:01.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:01.994 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:01.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:02.095 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:02.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:02.125 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:02.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:02.140 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:52:02.226 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:02.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:02.254 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:02.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:02.355 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:02.385 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:52:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:02.425 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:02.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:02.527 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:02.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:02.577 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:02.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:02.678 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:02.732 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:02.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:02.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:02 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/268785 objects degraded (3.297%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:02.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:02 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/268785 objects degraded (3.297%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:02.835 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:02.877 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:02.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:02.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:02.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:02 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/268785 objects degraded (3.297%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:02.978 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:03.008 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:03.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:03.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:03.110 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:03.156 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:03.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:03.257 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:03.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:03.289 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:03.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:03.390 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:03.440 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:03.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:03.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:03.542 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:03.570 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:03.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:03.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:03.672 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:03.697 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:03.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:03.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:03.799 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:03.828 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:03.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:03.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:03.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:03 smithi008 ceph-mon[29816]: pgmap v936: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 649 op/s; 8861/268059 objects degraded (3.306%); 12496/268059 objects misplaced (4.662%); 5.7 MiB/s, 91 objects/s recovering 2024-02-10T07:52:03.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:03 smithi008 ceph-mon[40743]: pgmap v936: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 649 op/s; 8861/268059 objects degraded (3.306%); 12496/268059 objects misplaced (4.662%); 5.7 MiB/s, 91 objects/s recovering 2024-02-10T07:52:03.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:03 smithi170 ceph-mon[33394]: pgmap v936: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 649 op/s; 8861/268059 objects degraded (3.306%); 12496/268059 objects misplaced (4.662%); 5.7 MiB/s, 91 objects/s recovering 2024-02-10T07:52:03.930 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:03.959 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:03.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:03.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:04.064 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:04.092 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:04.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:04.183 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:52:04.193 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:04.223 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:04.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:04.325 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:04.354 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:04.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:04.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:04.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:52:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:52:04.456 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:04.522 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:04.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:04.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:04.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:52:04.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:52:04.624 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:04.664 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:04.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:04.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:04.766 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:04.818 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:04.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:04.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:52:04.920 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:04.951 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:04.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:05.053 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:05.091 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:05.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:05.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:05.193 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:05.235 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:05.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:05.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:05.337 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:05.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:05.365 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:05.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:05.466 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:05.499 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:05.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:05.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:05.601 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:05.630 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:05.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:05.732 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:05.775 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:05.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:05.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:05 smithi008 ceph-mon[29816]: 4.5 scrub starts 2024-02-10T07:52:05.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:05 smithi008 ceph-mon[29816]: 4.5 scrub ok 2024-02-10T07:52:05.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:05 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:05.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:05 smithi008 ceph-mon[29816]: pgmap v937: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 723 op/s; 8861/261891 objects degraded (3.383%); 12496/261891 objects misplaced (4.771%); 2.6 MiB/s, 41 objects/s recovering 2024-02-10T07:52:05.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:05 smithi008 ceph-mon[40743]: 4.5 scrub starts 2024-02-10T07:52:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:05 smithi008 ceph-mon[40743]: 4.5 scrub ok 2024-02-10T07:52:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:05 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:05.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:05 smithi008 ceph-mon[40743]: pgmap v937: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 723 op/s; 8861/261891 objects degraded (3.383%); 12496/261891 objects misplaced (4.771%); 2.6 MiB/s, 41 objects/s recovering 2024-02-10T07:52:05.876 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:05 smithi170 ceph-mon[33394]: 4.5 scrub starts 2024-02-10T07:52:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:05 smithi170 ceph-mon[33394]: 4.5 scrub ok 2024-02-10T07:52:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:05 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:05 smithi170 ceph-mon[33394]: pgmap v937: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 723 op/s; 8861/261891 objects degraded (3.383%); 12496/261891 objects misplaced (4.771%); 2.6 MiB/s, 41 objects/s recovering 2024-02-10T07:52:05.902 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:05.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:06.004 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:06.031 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:06.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:06.132 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:06.176 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:06.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:06.277 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:06.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:06.330 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:06.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:06.431 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:06.459 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:06.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:06.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:06.561 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:06.605 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:06.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:06.706 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:06.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:06.747 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:06.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:06.848 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:06.885 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:06.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:06.987 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:07.014 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:07.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:07.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:07.116 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:07.145 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:07.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:07.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:07.247 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:07.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:07.290 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:07.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:07.391 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:07.418 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:07.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:07.520 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:07.566 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:07.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:07.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:07.668 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:07.711 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:07.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:07.813 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:07.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:07 smithi008 ceph-mon[29816]: pgmap v938: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 784 op/s; 8861/257841 objects degraded (3.437%); 12140/257841 objects misplaced (4.708%); 4.5 MiB/s, 71 objects/s recovering 2024-02-10T07:52:07.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:07 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/257841 objects degraded (3.437%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:07.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:07 smithi008 ceph-mon[40743]: pgmap v938: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 784 op/s; 8861/257841 objects degraded (3.437%); 12140/257841 objects misplaced (4.708%); 4.5 MiB/s, 71 objects/s recovering 2024-02-10T07:52:07.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:07 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/257841 objects degraded (3.437%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:07.844 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:07.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:07.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:07.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:07 smithi170 ceph-mon[33394]: pgmap v938: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 784 op/s; 8861/257841 objects degraded (3.437%); 12140/257841 objects misplaced (4.708%); 4.5 MiB/s, 71 objects/s recovering 2024-02-10T07:52:07.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:07 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/257841 objects degraded (3.437%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:07.946 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:07.974 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:07.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:07.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:08.076 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:08.119 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:08.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:08.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:08.221 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:08.253 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:08.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:08.354 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:08.402 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:08.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:08.504 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:08.575 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:08.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:08.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:08.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:08 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:52:08.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:08 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:52:08.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:08 smithi008 ceph-mon[29816]: osdmap e431: 8 total, 8 up, 8 in 2024-02-10T07:52:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:08 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:52:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:08 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:52:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:08 smithi008 ceph-mon[40743]: osdmap e431: 8 total, 8 up, 8 in 2024-02-10T07:52:08.610 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:52:08.610 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:52:08.610 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.610 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:127291 11.b 11:dc456f20:::benchmark_data_smithi008_186804_object22529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-02-10T07:52:08.610 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:08.577892+0000", 2024-02-10T07:52:08.610 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.030761150000000001, 2024-02-10T07:52:08.610 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.030800398999999999, 2024-02-10T07:52:08.610 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:08.610 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 127291 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.577892+0000", 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.577892+0000", 2024-02-10T07:52:08.611 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:08.612 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.612 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.612 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:08.612 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.577893+0000", 2024-02-10T07:52:08.612 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4600000000000003e-07 2024-02-10T07:52:08.612 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.612 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.612 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:08.612 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.577897+0000", 2024-02-10T07:52:08.612 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.5000000000000001e-06 2024-02-10T07:52:08.612 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.612 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.612 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:08.612 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.577908+0000", 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0659000000000001e-05 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.577916+0000", 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1070000000000005e-06 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.584938+0000", 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0070220689999999997 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:08.613 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.585088+0000", 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014952499999999999 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.585220+0000", 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000132461 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.588448+0000", 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032275580000000002 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:08.614 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.589777+0000", 2024-02-10T07:52:08.615 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0013289510000000001 2024-02-10T07:52:08.615 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:08.615 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:08.615 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:08.615 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.615 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.615 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:127297 11.0 e431/391 11:0c982e84:::benchmark_data_smithi008_186804_object22535:head v 431'7972, mlcod=431'7970)", 2024-02-10T07:52:08.615 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:08.594315+0000", 2024-02-10T07:52:08.615 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.014338392, 2024-02-10T07:52:08.615 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014480407000000001, 2024-02-10T07:52:08.615 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:08.615 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:52:08.615 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:08.615 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594315+0000", 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594315+0000", 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594316+0000", 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.108e-06 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.616 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594323+0000", 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1910000000000003e-06 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594342+0000", 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8678999999999999e-05 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594354+0000", 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1800999999999999e-05 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.617 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.618 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:08.618 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594833+0000", 2024-02-10T07:52:08.618 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00047981800000000002 2024-02-10T07:52:08.618 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.618 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.618 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:08.618 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594905+0000", 2024-02-10T07:52:08.618 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1919e-05 2024-02-10T07:52:08.618 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:08.618 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:08.618 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:08.618 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.618 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.618 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:127300 11.1 e431/391 11:8cfff4da:::benchmark_data_smithi008_186804_object22538:head v 431'8159, mlcod=431'8156)", 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:08.594396+0000", 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.014257061999999999, 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014465171000000001, 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594396+0000", 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594396+0000", 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:08.619 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.620 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.620 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:08.620 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594397+0000", 2024-02-10T07:52:08.620 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9900000000000001e-07 2024-02-10T07:52:08.620 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.620 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.620 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:08.620 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594407+0000", 2024-02-10T07:52:08.620 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8849999999999994e-06 2024-02-10T07:52:08.620 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.620 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.620 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:08.620 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594416+0000", 2024-02-10T07:52:08.620 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1400000000000006e-06 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594421+0000", 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.694e-06 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.599779+0000", 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0053575999999999997 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.599860+0000", 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.1181999999999996e-05 2024-02-10T07:52:08.621 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:127301 11.8 e431/391 11:1743082e:::benchmark_data_smithi008_186804_object22539:head v 431'8005, mlcod=431'8003)", 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:08.594436+0000", 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.014217584, 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.014488551000000001, 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594436+0000", 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:08.622 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594436+0000", 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594436+0000", 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.91e-07 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594440+0000", 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.805e-06 2024-02-10T07:52:08.623 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594445+0000", 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4759999999999997e-06 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594450+0000", 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2729999999999996e-06 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594797+0000", 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000347542 2024-02-10T07:52:08.624 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.594948+0000", 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015093699999999999 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:127305 11.9 e431/391 11:9783d40c:::benchmark_data_smithi008_186804_object22543:head v 431'7850, mlcod=431'7848)", 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:08.598675+0000", 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0099787030000000002, 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.010313601, 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:08.625 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.598675+0000", 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.598675+0000", 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.598675+0000", 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.4e-07 2024-02-10T07:52:08.626 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.627 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.627 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:08.627 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.598688+0000", 2024-02-10T07:52:08.627 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2167000000000001e-05 2024-02-10T07:52:08.627 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.627 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.627 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:08.627 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.598703+0000", 2024-02-10T07:52:08.627 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5812e-05 2024-02-10T07:52:08.627 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.627 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.627 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:08.627 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.598712+0000", 2024-02-10T07:52:08.627 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.721e-06 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.602009+0000", 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0032972930000000002 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.602062+0000", 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2913e-05 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.628 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:127307 11.0 e431/391 11:06ddaa40:::benchmark_data_smithi008_186804_object22545:head v 431'7973, mlcod=431'7970)", 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:08.600113+0000", 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.0085398260000000004, 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.008936711, 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.600113+0000", 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.600113+0000", 2024-02-10T07:52:08.629 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.600114+0000", 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4099999999999999e-07 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.600120+0000", 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9089999999999997e-06 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.600132+0000", 2024-02-10T07:52:08.630 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1632e-05 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.600139+0000", 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0840000000000003e-06 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.603659+0000", 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.003520423 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:08.603690+0000", 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.0713000000000002e-05 2024-02-10T07:52:08.631 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:08.632 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:08.632 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:08.632 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:08.632 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:52:08.632 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 6 2024-02-10T07:52:08.632 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:52:08.678 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:08.709 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:08.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:08.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:08.811 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:08.855 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:08.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:08.857 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:52:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:52:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:52:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:08 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:52:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:08 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:52:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:08 smithi170 ceph-mon[33394]: osdmap e431: 8 total, 8 up, 8 in 2024-02-10T07:52:08.957 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:08.996 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:08.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:08.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:09.098 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:09.126 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:09.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:09.228 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:09.261 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:09.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:09.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:09.363 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:09.394 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:09.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:09.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:09.496 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:09.529 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:09.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:09.631 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:09.661 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:09.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:09.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:09.763 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:09.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:09.794 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:09.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:09.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:09 smithi008 ceph-mon[29816]: pgmap v940: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 783 op/s; 8861/257223 objects degraded (3.445%); 12140/257223 objects misplaced (4.720%); 4.5 MiB/s, 71 objects/s recovering 2024-02-10T07:52:09.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:09 smithi008 ceph-mon[29816]: 11.9 scrub starts 2024-02-10T07:52:09.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:09 smithi008 ceph-mon[40743]: pgmap v940: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 783 op/s; 8861/257223 objects degraded (3.445%); 12140/257223 objects misplaced (4.720%); 4.5 MiB/s, 71 objects/s recovering 2024-02-10T07:52:09.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:09 smithi008 ceph-mon[40743]: 11.9 scrub starts 2024-02-10T07:52:09.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:09 smithi170 ceph-mon[33394]: pgmap v940: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 783 op/s; 8861/257223 objects degraded (3.445%); 12140/257223 objects misplaced (4.720%); 4.5 MiB/s, 71 objects/s recovering 2024-02-10T07:52:09.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:09 smithi170 ceph-mon[33394]: 11.9 scrub starts 2024-02-10T07:52:09.895 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:09.928 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:09.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:10.030 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:10.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:10.078 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:10.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:10.179 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:10.209 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:10.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:10.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:10.311 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:10.337 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:10.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:10.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:10.439 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:10.474 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:10.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:10.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:10.575 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:10.603 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:10.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:10.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:10.672 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:10 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:52:10.672 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:10 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:52:10.673 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:10 smithi008 ceph-mon[29816]: 11.9 scrub ok 2024-02-10T07:52:10.673 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:10 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:52:10.673 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:10 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:52:10.673 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:10 smithi008 ceph-mon[40743]: 11.9 scrub ok 2024-02-10T07:52:10.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:10.733 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:10.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:10.834 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:10.878 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:10.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:10.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:10 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:52:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:10 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:52:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:10 smithi170 ceph-mon[33394]: 11.9 scrub ok 2024-02-10T07:52:10.980 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:11.016 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:11.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:11.117 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:11.162 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:11.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:11.264 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:11.307 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:11.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:11.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:11.409 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:11.453 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:11.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:11.554 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:11.581 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:11.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:11.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:11.682 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:11.712 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:11.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:11.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:11.814 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:11.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:11 smithi008 ceph-mon[40743]: 5.7 scrub starts 2024-02-10T07:52:11.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:11 smithi008 ceph-mon[40743]: 5.7 scrub ok 2024-02-10T07:52:11.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:11 smithi008 ceph-mon[40743]: pgmap v941: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 751 op/s; 8861/248115 objects degraded (3.571%); 12021/248115 objects misplaced (4.845%); 3.0 MiB/s, 47 objects/s recovering 2024-02-10T07:52:11.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:11 smithi008 ceph-mon[40743]: osdmap e432: 8 total, 8 up, 8 in 2024-02-10T07:52:11.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:11 smithi008 ceph-mon[40743]: 6.1f scrub starts 2024-02-10T07:52:11.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:11 smithi008 ceph-mon[40743]: 6.1f scrub ok 2024-02-10T07:52:11.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:11 smithi008 ceph-mon[29816]: 5.7 scrub starts 2024-02-10T07:52:11.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:11 smithi008 ceph-mon[29816]: 5.7 scrub ok 2024-02-10T07:52:11.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:11 smithi008 ceph-mon[29816]: pgmap v941: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 751 op/s; 8861/248115 objects degraded (3.571%); 12021/248115 objects misplaced (4.845%); 3.0 MiB/s, 47 objects/s recovering 2024-02-10T07:52:11.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:11 smithi008 ceph-mon[29816]: osdmap e432: 8 total, 8 up, 8 in 2024-02-10T07:52:11.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:11 smithi008 ceph-mon[29816]: 6.1f scrub starts 2024-02-10T07:52:11.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:11 smithi008 ceph-mon[29816]: 6.1f scrub ok 2024-02-10T07:52:11.857 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:11 smithi170 ceph-mon[33394]: 5.7 scrub starts 2024-02-10T07:52:11.857 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:11 smithi170 ceph-mon[33394]: 5.7 scrub ok 2024-02-10T07:52:11.857 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:11 smithi170 ceph-mon[33394]: pgmap v941: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 751 op/s; 8861/248115 objects degraded (3.571%); 12021/248115 objects misplaced (4.845%); 3.0 MiB/s, 47 objects/s recovering 2024-02-10T07:52:11.857 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:11 smithi170 ceph-mon[33394]: osdmap e432: 8 total, 8 up, 8 in 2024-02-10T07:52:11.857 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:11 smithi170 ceph-mon[33394]: 6.1f scrub starts 2024-02-10T07:52:11.857 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:11 smithi170 ceph-mon[33394]: 6.1f scrub ok 2024-02-10T07:52:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:11.859 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:11.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:11.960 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:11.992 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:11.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:11.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:12.094 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:12.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:12.125 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:12.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:12.130 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:52:12.162 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-10T07:52:12.226 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:12.285 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:12.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:12.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:12.387 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:12.412 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:12.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:12.514 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:12.564 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:12.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:12.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:12.667 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:12.715 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:12.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:12.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:12.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:12 smithi008 ceph-mon[29816]: 4.1f scrub starts 2024-02-10T07:52:12.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:12 smithi008 ceph-mon[29816]: 4.1f scrub ok 2024-02-10T07:52:12.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:12 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/248115 objects degraded (3.571%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:12.767 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:12 smithi008 ceph-mon[40743]: 4.1f scrub starts 2024-02-10T07:52:12.767 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:12 smithi008 ceph-mon[40743]: 4.1f scrub ok 2024-02-10T07:52:12.767 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:12 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/248115 objects degraded (3.571%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:12.818 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:12.859 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:12.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:12.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:12 smithi170 ceph-mon[33394]: 4.1f scrub starts 2024-02-10T07:52:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:12 smithi170 ceph-mon[33394]: 4.1f scrub ok 2024-02-10T07:52:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:12 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/248115 objects degraded (3.571%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:12.961 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:12.990 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:12.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:13.092 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:13.119 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:13.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:13.221 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:13.247 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:13.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:13.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:13.349 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:13.378 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:13.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:13.480 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:13.523 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:13.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:13.624 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:13.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:13.668 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:13.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:13.711 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:13 smithi008 ceph-mon[29816]: pgmap v943: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 677 op/s; 8861/246099 objects degraded (3.601%); 11752/246099 objects misplaced (4.775%); 5.8 MiB/s, 92 objects/s recovering 2024-02-10T07:52:13.712 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:13 smithi008 ceph-mon[40743]: pgmap v943: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 677 op/s; 8861/246099 objects degraded (3.601%); 11752/246099 objects misplaced (4.775%); 5.8 MiB/s, 92 objects/s recovering 2024-02-10T07:52:13.769 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:13.814 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:52:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:13.821 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:13.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:13 smithi170 ceph-mon[33394]: pgmap v943: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 677 op/s; 8861/246099 objects degraded (3.601%); 11752/246099 objects misplaced (4.775%); 5.8 MiB/s, 92 objects/s recovering 2024-02-10T07:52:13.922 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:13.949 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:13.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:14.051 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:14.099 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:14.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:14.200 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:14.266 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:14.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:14.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:14.367 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:14.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:52:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:52:14.411 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:14.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:14.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:14.482 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.4 to 1 (10000) 2024-02-10T07:52:14.482 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:52:14.514 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:14.538 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:14.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:14.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:14 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3218511087' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:52:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:14 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:52:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:14 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1948373917' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:52:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:14 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:52:14.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:14 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3218511087' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:52:14.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:14 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:52:14.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:14 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1948373917' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:52:14.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:14 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:52:14.640 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:14.679 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:14.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:14.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:14.781 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:14.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:14.837 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:14.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:14 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3218511087' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:52:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:14 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:52:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:14 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1948373917' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:52:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:14 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:52:14.939 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:14.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:14.975 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:14.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:15.076 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:15.111 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:15.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:15.212 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:15.256 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:15.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:15.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:15.358 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:15.403 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:15.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:15.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:15.506 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:15.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:15.548 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:15.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:15.649 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:15.694 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:15.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:15.796 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:15.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:15 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:15.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:15 smithi008 ceph-mon[29816]: pgmap v944: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 746 op/s; 8861/239940 objects degraded (3.693%); 11752/239940 objects misplaced (4.898%); 3.0 MiB/s, 48 objects/s recovering 2024-02-10T07:52:15.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:15 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:52:15.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:15 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:52:15.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:15 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:52:15.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:15 smithi008 ceph-mon[29816]: osdmap e433: 8 total, 8 up, 8 in 2024-02-10T07:52:15.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:15 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:15.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:15 smithi008 ceph-mon[40743]: pgmap v944: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 746 op/s; 8861/239940 objects degraded (3.693%); 11752/239940 objects misplaced (4.898%); 3.0 MiB/s, 48 objects/s recovering 2024-02-10T07:52:15.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:15 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:52:15.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:15 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:52:15.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:15 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:52:15.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:15 smithi008 ceph-mon[40743]: osdmap e433: 8 total, 8 up, 8 in 2024-02-10T07:52:15.845 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:15.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:15.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:15 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:15 smithi170 ceph-mon[33394]: pgmap v944: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 746 op/s; 8861/239940 objects degraded (3.693%); 11752/239940 objects misplaced (4.898%); 3.0 MiB/s, 48 objects/s recovering 2024-02-10T07:52:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:15 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:52:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:15 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:52:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:15 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:52:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:15 smithi170 ceph-mon[33394]: osdmap e433: 8 total, 8 up, 8 in 2024-02-10T07:52:15.947 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:15.974 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:15.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:15.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:16.076 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:16.124 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:16.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:16.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:16.226 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:16.251 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:16.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:16.352 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:16.392 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:16.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:16.494 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:16.525 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:16.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:16.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:16.626 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:16.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:16.676 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:16.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:16.777 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:16.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:16.824 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:16.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:16.925 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:16.970 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:16.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:16.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:17.072 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:17.110 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:17.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:17.212 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:17.242 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:17.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:17.344 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:17.386 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:17.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:17.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:17.489 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:17.527 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:17.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:17.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:17.629 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:17.664 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:17.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:17.765 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:17.793 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:17.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:17.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:17.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:17 smithi008 ceph-mon[29816]: pgmap v946: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 885 op/s; 8861/235992 objects degraded (3.755%); 11377/235992 objects misplaced (4.821%); 6.0 MiB/s, 95 objects/s recovering 2024-02-10T07:52:17.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:17 smithi008 ceph-mon[29816]: 3.10 scrub starts 2024-02-10T07:52:17.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:17 smithi008 ceph-mon[29816]: 3.10 scrub ok 2024-02-10T07:52:17.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:17 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/235992 objects degraded (3.755%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:17.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:17 smithi008 ceph-mon[40743]: pgmap v946: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 885 op/s; 8861/235992 objects degraded (3.755%); 11377/235992 objects misplaced (4.821%); 6.0 MiB/s, 95 objects/s recovering 2024-02-10T07:52:17.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:17 smithi008 ceph-mon[40743]: 3.10 scrub starts 2024-02-10T07:52:17.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:17 smithi008 ceph-mon[40743]: 3.10 scrub ok 2024-02-10T07:52:17.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:17 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/235992 objects degraded (3.755%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:17 smithi170 ceph-mon[33394]: pgmap v946: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 885 op/s; 8861/235992 objects degraded (3.755%); 11377/235992 objects misplaced (4.821%); 6.0 MiB/s, 95 objects/s recovering 2024-02-10T07:52:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:17 smithi170 ceph-mon[33394]: 3.10 scrub starts 2024-02-10T07:52:17.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:17 smithi170 ceph-mon[33394]: 3.10 scrub ok 2024-02-10T07:52:17.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:17 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/235992 objects degraded (3.755%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:17.895 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:17.937 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:17.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:18.038 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:18.067 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:18.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:18.169 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:18.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:18.217 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:18.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:18.318 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:18.351 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:18.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:18.452 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:18.480 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:18.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:18.583 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:18.620 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:18.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:18.724 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:18.757 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:18.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:18.858 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:18.886 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:18.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:18.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:18.988 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:19.018 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:19.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:19.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:19.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:52:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:52:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:52:19.120 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:19.159 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:19.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:19.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:19.261 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:19.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:19.293 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:19.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:19.394 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:19.436 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:19.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:19.537 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:19.579 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:19.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:19.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:19 smithi008 ceph-mon[29816]: pgmap v947: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 533 op/s; 8861/235311 objects degraded (3.766%); 11377/235311 objects misplaced (4.835%); 5.0 MiB/s, 80 objects/s recovering 2024-02-10T07:52:19.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:19 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:52:19.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:19 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:52:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:19 smithi008 ceph-mon[40743]: pgmap v947: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 533 op/s; 8861/235311 objects degraded (3.766%); 11377/235311 objects misplaced (4.835%); 5.0 MiB/s, 80 objects/s recovering 2024-02-10T07:52:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:19 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:52:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:19 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:52:19.682 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:19.724 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:19.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:19.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:19.785 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 5 1 2024-02-10T07:52:19.827 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:19.856 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:19.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:19 smithi170 ceph-mon[33394]: pgmap v947: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 533 op/s; 8861/235311 objects degraded (3.766%); 11377/235311 objects misplaced (4.835%); 5.0 MiB/s, 80 objects/s recovering 2024-02-10T07:52:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:19 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:52:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:19 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:52:19.960 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:19.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:19.987 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:19.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:20.087 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:20.147 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:20.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:20.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:20.249 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:20.299 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:20.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:20.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:20.400 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:20.430 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:20.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:20.532 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:20.581 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:20.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:20.682 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:20.737 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:20.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:20.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:20.839 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:20.866 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:20.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:20.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:20.968 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:20.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:20.996 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:20.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:21.097 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:21.143 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:21.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:21.244 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:21.290 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:21.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:21.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:21.391 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:21.457 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:21.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:21.559 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:21.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:21 smithi008 ceph-mon[29816]: pgmap v948: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 697 op/s; 8861/226500 objects degraded (3.912%); 11259/226500 objects misplaced (4.971%); 3.2 MiB/s, 50 objects/s recovering 2024-02-10T07:52:21.561 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:21 smithi008 ceph-mon[40743]: pgmap v948: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 697 op/s; 8861/226500 objects degraded (3.912%); 11259/226500 objects misplaced (4.971%); 3.2 MiB/s, 50 objects/s recovering 2024-02-10T07:52:21.623 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:21.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:21.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:21.725 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:21.741 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:52:21.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:21.776 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:21.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:21.784 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:52:21.799 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:52:21.844 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:52:21.878 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:21.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:21 smithi170 ceph-mon[33394]: pgmap v948: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 697 op/s; 8861/226500 objects degraded (3.912%); 11259/226500 objects misplaced (4.971%); 3.2 MiB/s, 50 objects/s recovering 2024-02-10T07:52:21.905 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:52:21.954 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:21.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:21.956 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:52:21.995 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:52:22.038 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:52:22.057 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:22.078 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:52:22.081 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:22.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:22.082 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:52:22.082 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:52:22.083 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:52:22.083 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:52:22.083 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:52:22.083 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:52:22.083 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:52:22.083 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:52:22.182 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:22.226 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:22.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:22.327 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:22.379 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:22.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:22.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:22.481 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:22.503 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:52:22.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:22.513 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:22.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:22.614 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:22.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:22.707 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:22.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:22.808 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:22.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:22 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3060081562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:52:22.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:22 smithi008 ceph-mon[29816]: 6.a scrub starts 2024-02-10T07:52:22.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:22 smithi008 ceph-mon[29816]: 6.a scrub ok 2024-02-10T07:52:22.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:22 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/226500 objects degraded (3.912%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:22.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:22 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3060081562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:52:22.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:22 smithi008 ceph-mon[40743]: 6.a scrub starts 2024-02-10T07:52:22.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:22 smithi008 ceph-mon[40743]: 6.a scrub ok 2024-02-10T07:52:22.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:22 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/226500 objects degraded (3.912%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:22.836 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:22.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:22.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:22 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3060081562' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:52:22.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:22 smithi170 ceph-mon[33394]: 6.a scrub starts 2024-02-10T07:52:22.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:22 smithi170 ceph-mon[33394]: 6.a scrub ok 2024-02-10T07:52:22.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:22 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/226500 objects degraded (3.912%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:22.937 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:22.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:22.963 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:22.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:23.063 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:23.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:23.125 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:23.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:23.226 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:23.256 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:23.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:23.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:23.357 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:23.383 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:23.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:23.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:23.485 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:23.490 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:52:23.490 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:52:23.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:23.511 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:23.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:23.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:23 smithi008 ceph-mon[29816]: pgmap v949: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 725 op/s; 8861/224343 objects degraded (3.950%); 10976/224343 objects misplaced (4.893%); 4.8 MiB/s, 77 objects/s recovering 2024-02-10T07:52:23.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:23 smithi008 ceph-mon[29816]: 5.11 scrub starts 2024-02-10T07:52:23.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:23 smithi008 ceph-mon[29816]: 5.11 scrub ok 2024-02-10T07:52:23.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:23 smithi008 ceph-mon[40743]: pgmap v949: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 725 op/s; 8861/224343 objects degraded (3.950%); 10976/224343 objects misplaced (4.893%); 4.8 MiB/s, 77 objects/s recovering 2024-02-10T07:52:23.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:23 smithi008 ceph-mon[40743]: 5.11 scrub starts 2024-02-10T07:52:23.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:23 smithi008 ceph-mon[40743]: 5.11 scrub ok 2024-02-10T07:52:23.613 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:23.641 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:23.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:23.742 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:23.785 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:23.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:23.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:23 smithi170 ceph-mon[33394]: pgmap v949: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 725 op/s; 8861/224343 objects degraded (3.950%); 10976/224343 objects misplaced (4.893%); 4.8 MiB/s, 77 objects/s recovering 2024-02-10T07:52:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:23 smithi170 ceph-mon[33394]: 5.11 scrub starts 2024-02-10T07:52:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:23 smithi170 ceph-mon[33394]: 5.11 scrub ok 2024-02-10T07:52:23.887 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:23.928 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:23.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:24.030 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:24.058 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:24.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:24.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:24.160 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:24.203 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:24.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:24.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:24.305 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:24.340 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:24.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:24.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:52:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:52:24.443 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:24.470 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:24.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:24.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:24.572 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:24.600 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:24.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:24.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:24.702 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:24.729 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:24.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:24.831 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:24.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:24 smithi008 ceph-mon[29816]: 6.0 scrub starts 2024-02-10T07:52:24.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:24 smithi008 ceph-mon[29816]: 6.0 scrub ok 2024-02-10T07:52:24.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:24 smithi008 ceph-mon[40743]: 6.0 scrub starts 2024-02-10T07:52:24.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:24 smithi008 ceph-mon[40743]: 6.0 scrub ok 2024-02-10T07:52:24.876 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:24.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:24.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:24 smithi170 ceph-mon[33394]: 6.0 scrub starts 2024-02-10T07:52:24.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:24 smithi170 ceph-mon[33394]: 6.0 scrub ok 2024-02-10T07:52:24.978 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:52:24.978 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:25.004 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:25.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:25.105 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:25.131 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:25.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:25.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:25.233 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:25.286 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:25.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:25.388 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:25.420 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:25.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:25.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:25.522 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:25.548 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:25.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:25.652 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:25.682 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:25.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:25.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:25.784 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:25.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:25.812 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:25.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:25.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:25 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:25.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:25 smithi008 ceph-mon[29816]: pgmap v950: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 726 op/s; 8861/218178 objects degraded (4.061%); 10976/218178 objects misplaced (5.031%); 4.8 MiB/s, 77 objects/s recovering 2024-02-10T07:52:25.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:25 smithi008 ceph-mon[29816]: 5.e deep-scrub starts 2024-02-10T07:52:25.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:25 smithi008 ceph-mon[29816]: 5.e deep-scrub ok 2024-02-10T07:52:25.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:25 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:25.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:25 smithi008 ceph-mon[40743]: pgmap v950: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 726 op/s; 8861/218178 objects degraded (4.061%); 10976/218178 objects misplaced (5.031%); 4.8 MiB/s, 77 objects/s recovering 2024-02-10T07:52:25.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:25 smithi008 ceph-mon[40743]: 5.e deep-scrub starts 2024-02-10T07:52:25.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:25 smithi008 ceph-mon[40743]: 5.e deep-scrub ok 2024-02-10T07:52:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:25 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:25 smithi170 ceph-mon[33394]: pgmap v950: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 726 op/s; 8861/218178 objects degraded (4.061%); 10976/218178 objects misplaced (5.031%); 4.8 MiB/s, 77 objects/s recovering 2024-02-10T07:52:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:25 smithi170 ceph-mon[33394]: 5.e deep-scrub starts 2024-02-10T07:52:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:25 smithi170 ceph-mon[33394]: 5.e deep-scrub ok 2024-02-10T07:52:25.914 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:25.956 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:25.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:26.057 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:26.099 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:26.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:26.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:26.201 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:26.224 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:26.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:26.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:26.326 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:26.355 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:26.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:26.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:26.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:26.482 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:26.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:26.584 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:26.610 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:26.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:26.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:26.712 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:26.740 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:26.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:26.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:26.842 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:26.891 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:26.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:26.993 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:27.022 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:27.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:27.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:27.124 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:27.170 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:27.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:27.272 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:27.313 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:27.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:27.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:27.415 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:27.452 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:27.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:27.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:27.554 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:27.595 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:27.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:27.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:27.662 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-10T07:52:27.697 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:27.720 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:27 smithi008 ceph-mon[29816]: pgmap v951: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 875 B/s rd, 0 B/s wr, 738 op/s; 8861/214071 objects degraded (4.139%); 10561/214071 objects misplaced (4.933%); 6.4 MiB/s, 101 objects/s recovering 2024-02-10T07:52:27.720 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:27 smithi008 ceph-mon[29816]: 3.a scrub starts 2024-02-10T07:52:27.720 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:27 smithi008 ceph-mon[29816]: 3.a scrub ok 2024-02-10T07:52:27.720 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:27 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8861/214071 objects degraded (4.139%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:27.720 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:27 smithi008 ceph-mon[40743]: pgmap v951: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 875 B/s rd, 0 B/s wr, 738 op/s; 8861/214071 objects degraded (4.139%); 10561/214071 objects misplaced (4.933%); 6.4 MiB/s, 101 objects/s recovering 2024-02-10T07:52:27.720 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:27 smithi008 ceph-mon[40743]: 3.a scrub starts 2024-02-10T07:52:27.720 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:27 smithi008 ceph-mon[40743]: 3.a scrub ok 2024-02-10T07:52:27.721 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:27 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8861/214071 objects degraded (4.139%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:27.757 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:27.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:27.860 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:27.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:27 smithi170 ceph-mon[33394]: pgmap v951: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 875 B/s rd, 0 B/s wr, 738 op/s; 8861/214071 objects degraded (4.139%); 10561/214071 objects misplaced (4.933%); 6.4 MiB/s, 101 objects/s recovering 2024-02-10T07:52:27.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:27 smithi170 ceph-mon[33394]: 3.a scrub starts 2024-02-10T07:52:27.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:27 smithi170 ceph-mon[33394]: 3.a scrub ok 2024-02-10T07:52:27.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:27 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8861/214071 objects degraded (4.139%), 2 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:27.887 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:27.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:27.989 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:28.044 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:28.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:28.146 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:28.177 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:28.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:28.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:28.277 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:52:28.279 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:28.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:28.308 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:28.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:28.409 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:28.427 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:52:28.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:28.437 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:28.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:28.485 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:52:28.538 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:28.582 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:28.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:28.683 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:28.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:28.714 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:28.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:28.789 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:28 smithi008 ceph-mon[29816]: 3.16 scrub starts 2024-02-10T07:52:28.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:28 smithi008 ceph-mon[29816]: 3.16 scrub ok 2024-02-10T07:52:28.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:28 smithi008 ceph-mon[40743]: 3.16 scrub starts 2024-02-10T07:52:28.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:28 smithi008 ceph-mon[40743]: 3.16 scrub ok 2024-02-10T07:52:28.814 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:28.857 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:28.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:28.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:28 smithi170 ceph-mon[33394]: 3.16 scrub starts 2024-02-10T07:52:28.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:28 smithi170 ceph-mon[33394]: 3.16 scrub ok 2024-02-10T07:52:28.958 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:28.987 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:28.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:29.079 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:52:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:52:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:52:29.088 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:29.136 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:29.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:29.237 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:29.289 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:29.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:29.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:29.390 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:29.420 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:29.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:29.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:29.522 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:29.550 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:29.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:29.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:29.701 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:29.727 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:29.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:29.829 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:29.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:29 smithi008 ceph-mon[29816]: pgmap v952: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 628 op/s; 8861/213375 objects degraded (4.153%); 10561/213375 objects misplaced (4.950%); 4.2 MiB/s, 67 objects/s recovering 2024-02-10T07:52:29.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:29 smithi008 ceph-mon[40743]: pgmap v952: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 628 op/s; 8861/213375 objects degraded (4.153%); 10561/213375 objects misplaced (4.950%); 4.2 MiB/s, 67 objects/s recovering 2024-02-10T07:52:29.874 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:29.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:29.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:29.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:29 smithi170 ceph-mon[33394]: pgmap v952: 148 pgs: 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 142 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 628 op/s; 8861/213375 objects degraded (4.153%); 10561/213375 objects misplaced (4.950%); 4.2 MiB/s, 67 objects/s recovering 2024-02-10T07:52:29.976 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:30.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:30.004 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:30.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:30.105 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:30.132 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:30.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:30.234 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:30.262 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:30.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:30.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:30.363 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:30.413 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:30.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:30.514 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:30.540 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:30.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:30.642 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:30.670 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:30.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:30.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:30.772 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:30.798 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:30.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:30.900 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:30.928 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:30.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:31.030 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:31.075 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:31.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:31.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:31.177 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:31.223 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:31.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:31.325 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:31.363 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:31.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:31.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:31.465 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:31.507 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:31.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:31 smithi008 ceph-mon[29816]: pgmap v953: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 142 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 863 op/s; 8919/204267 objects degraded (4.366%); 10383/204267 objects misplaced (5.083%); 4.9 MiB/s, 77 objects/s recovering 2024-02-10T07:52:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:31 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1142311149' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:52:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:31 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:52:31.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:31 smithi008 ceph-mon[40743]: pgmap v953: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 142 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 863 op/s; 8919/204267 objects degraded (4.366%); 10383/204267 objects misplaced (5.083%); 4.9 MiB/s, 77 objects/s recovering 2024-02-10T07:52:31.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:31 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1142311149' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:52:31.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:31 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:52:31.609 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:31.660 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:31.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:31.762 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:31.788 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:31.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:31.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:31 smithi170 ceph-mon[33394]: pgmap v953: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 142 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 863 op/s; 8919/204267 objects degraded (4.366%); 10383/204267 objects misplaced (5.083%); 4.9 MiB/s, 77 objects/s recovering 2024-02-10T07:52:31.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:31 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1142311149' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:52:31.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:31 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:52:31.889 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:31.917 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:31.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:31.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:32.019 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:32.048 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:32.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:32.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:32.154 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:32.192 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:32.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:32.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:32.293 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:32.323 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:32.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:32.424 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:32.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:32.455 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:32.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:32.556 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:32.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:32.615 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:32.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:32.673 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:32 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:52:32.674 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:32 smithi008 ceph-mon[29816]: osdmap e434: 8 total, 8 up, 8 in 2024-02-10T07:52:32.674 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:32 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8919/204267 objects degraded (4.366%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:32.674 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:32 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:52:32.674 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:32 smithi008 ceph-mon[40743]: osdmap e434: 8 total, 8 up, 8 in 2024-02-10T07:52:32.675 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:32 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8919/204267 objects degraded (4.366%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:32.717 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:32.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:32.754 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:32.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:32.855 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:32.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:32 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:52:32.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:32 smithi170 ceph-mon[33394]: osdmap e434: 8 total, 8 up, 8 in 2024-02-10T07:52:32.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:32 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8919/204267 objects degraded (4.366%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:32.905 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:32.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:32.972 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.5 to 1 (10000) 2024-02-10T07:52:33.007 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:33.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:33.052 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:33.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:33.153 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:33.184 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:33.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:33.285 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:33.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:33.322 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:33.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:33.423 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:33.493 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:33.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:33.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:33 smithi008 ceph-mon[29816]: pgmap v955: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 142 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 740 op/s; 8919/202161 objects degraded (4.412%); 10053/202161 objects misplaced (4.973%); 5.4 MiB/s, 86 objects/s recovering 2024-02-10T07:52:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:52:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:33 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:52:33.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:33 smithi008 ceph-mon[40743]: pgmap v955: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 142 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 740 op/s; 8919/202161 objects degraded (4.412%); 10053/202161 objects misplaced (4.973%); 5.4 MiB/s, 86 objects/s recovering 2024-02-10T07:52:33.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:52:33.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:33 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:52:33.596 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:33.633 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:33.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:33.734 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:33.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:33.781 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:33.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:33.882 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:33.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:33 smithi170 ceph-mon[33394]: pgmap v955: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 142 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 740 op/s; 8919/202161 objects degraded (4.412%); 10053/202161 objects misplaced (4.973%); 5.4 MiB/s, 86 objects/s recovering 2024-02-10T07:52:33.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:52:33.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:33 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:52:33.912 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:33.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:33.954 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 6 1 2024-02-10T07:52:34.015 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:34.041 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:34.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:34.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:34.143 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:34.214 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:34.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:34.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:34.315 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:34.365 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:34.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:34.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:34.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:52:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:52:34.467 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:34.514 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:34.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:34.615 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:34.640 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:34.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:34.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:34.742 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:34.770 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:34.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:34.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:34.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:52:34.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:52:34.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:52:34.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:52:34.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:52:34.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:52:34.872 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:34.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:52:34.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:52:34.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:52:34.921 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:34.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:35.022 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:35.061 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:35.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:35.162 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:35.202 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:35.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:35.304 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:35.331 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:35.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:35.433 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:35.464 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:35.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:35.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:35.566 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:35.602 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:35.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:35.703 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:35.735 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:35.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:35 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:35 smithi008 ceph-mon[29816]: pgmap v956: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 142 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 744 op/s; 8919/195855 objects degraded (4.554%); 10053/195855 objects misplaced (5.133%); 5.4 MiB/s, 86 objects/s recovering 2024-02-10T07:52:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:35 smithi008 ceph-mon[29816]: 5.17 scrub starts 2024-02-10T07:52:35.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:35 smithi008 ceph-mon[29816]: 5.17 scrub ok 2024-02-10T07:52:35.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:35 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:35.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:35 smithi008 ceph-mon[40743]: pgmap v956: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 142 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 744 op/s; 8919/195855 objects degraded (4.554%); 10053/195855 objects misplaced (5.133%); 5.4 MiB/s, 86 objects/s recovering 2024-02-10T07:52:35.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:35 smithi008 ceph-mon[40743]: 5.17 scrub starts 2024-02-10T07:52:35.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:35 smithi008 ceph-mon[40743]: 5.17 scrub ok 2024-02-10T07:52:35.837 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:35.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:35 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:35.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:35 smithi170 ceph-mon[33394]: pgmap v956: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 142 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 744 op/s; 8919/195855 objects degraded (4.554%); 10053/195855 objects misplaced (5.133%); 5.4 MiB/s, 86 objects/s recovering 2024-02-10T07:52:35.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:35 smithi170 ceph-mon[33394]: 5.17 scrub starts 2024-02-10T07:52:35.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:35 smithi170 ceph-mon[33394]: 5.17 scrub ok 2024-02-10T07:52:35.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:35.885 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:35.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:35.986 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:36.017 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:36.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:36.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:36.118 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:36.163 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:36.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:36.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:36.265 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:36.291 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:36.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:36.393 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:36.423 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:36.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:36.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:36.525 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:36.536 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:52:36.555 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:36.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:36.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:36.657 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:36.681 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:36.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:36.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:36.783 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:36.814 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:36 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3892164089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:52:36.814 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:36 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:52:36.814 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:36 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3892164089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:52:36.814 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:36 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:52:36.822 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:36.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:36.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:36 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3892164089' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:52:36.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:36 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:52:36.923 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:36.955 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:36.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:37.057 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:37.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:37.113 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:37.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:37.214 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:37.256 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:37.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:37.357 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:37.396 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:37.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:37.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:37.407 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:52:37.466 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:52:37.497 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:37.511 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:52:37.522 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:37.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:37.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:37.554 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:52:37.602 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:52:37.624 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:37.647 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:52:37.650 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:37.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:37.679 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:52:37.679 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:37 smithi008 ceph-mon[29816]: pgmap v957: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 746 op/s; 9056/191709 objects degraded (4.724%); 9455/191709 objects misplaced (4.932%); 5.7 MiB/s, 91 objects/s recovering 2024-02-10T07:52:37.679 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:37 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:52:37.679 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:37 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:52:37.680 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:37 smithi008 ceph-mon[29816]: osdmap e435: 8 total, 8 up, 8 in 2024-02-10T07:52:37.680 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:37 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 9056/191709 objects degraded (4.724%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:37.680 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:37 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/849885709' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:52:37.680 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:37 smithi008 ceph-mon[40743]: pgmap v957: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 746 op/s; 9056/191709 objects degraded (4.724%); 9455/191709 objects misplaced (4.932%); 5.7 MiB/s, 91 objects/s recovering 2024-02-10T07:52:37.680 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:37 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:52:37.680 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:37 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:52:37.680 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:37 smithi008 ceph-mon[40743]: osdmap e435: 8 total, 8 up, 8 in 2024-02-10T07:52:37.680 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:37 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 9056/191709 objects degraded (4.724%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:37.680 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:37 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/849885709' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:52:37.699 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:52:37.700 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:52:37.700 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:52:37.700 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:52:37.701 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:52:37.701 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:52:37.701 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:52:37.701 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:52:37.701 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:52:37.752 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:37.780 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:37.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:37.882 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:37.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:37 smithi170 ceph-mon[33394]: pgmap v957: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 746 op/s; 9056/191709 objects degraded (4.724%); 9455/191709 objects misplaced (4.932%); 5.7 MiB/s, 91 objects/s recovering 2024-02-10T07:52:37.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:37 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:52:37.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:37 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:52:37.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:37 smithi170 ceph-mon[33394]: osdmap e435: 8 total, 8 up, 8 in 2024-02-10T07:52:37.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:37 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 9056/191709 objects degraded (4.724%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:37.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:37 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/849885709' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:52:37.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:37.923 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:37.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:38.024 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:38.050 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:38.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:38.152 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:38.178 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:38.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:38.185 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:52:38.185 INFO:teuthology.orchestra.run.smithi008.stdout: "size": 20, 2024-02-10T07:52:38.186 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 600, 2024-02-10T07:52:38.186 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:52:38.186 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.186 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:55752 11.b 11:db921083:::benchmark_data_smithi008_186804_object55751:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-02-10T07:52:38.186 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:31.512411+0000", 2024-02-10T07:52:38.186 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 126.66983760399999, 2024-02-10T07:52:38.186 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.081278510999999998, 2024-02-10T07:52:38.186 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.186 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.186 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.186 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.186 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.186 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 55752 2024-02-10T07:52:38.186 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.187 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.187 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.187 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.187 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:31.512411+0000", 2024-02-10T07:52:38.187 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.187 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.187 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.187 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.187 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:31.512411+0000", 2024-02-10T07:52:38.187 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.187 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.187 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.187 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:31.512412+0000", 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.499e-06 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:31.512434+0000", 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1203999999999999e-05 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:31.512455+0000", 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1083000000000001e-05 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.188 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:31.512464+0000", 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.0899999999999994e-06 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:31.512510+0000", 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.668e-05 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:31.512597+0000", 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6185000000000002e-05 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.189 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:31.512684+0000", 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.7610999999999999e-05 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:31.514386+0000", 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001701436 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:31.516588+0000", 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022023519999999999 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.190 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:31.593577+0000", 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.076988741999999999 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:31.593648+0000", 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.1464000000000006e-05 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:31.593689+0000", 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1165000000000003e-05 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.191 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.192 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.192 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.192 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:57543 11.b 11:dddbdd9c:::benchmark_data_smithi008_186804_object57542:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-02-10T07:52:38.192 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:32.830535+0000", 2024-02-10T07:52:38.192 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 125.351713389, 2024-02-10T07:52:38.192 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.076175202999999997, 2024-02-10T07:52:38.192 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.192 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.192 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.192 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.192 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.192 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 57543 2024-02-10T07:52:38.192 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.192 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:32.830535+0000", 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:32.830535+0000", 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:32.830536+0000", 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2619999999999999e-06 2024-02-10T07:52:38.193 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:32.830559+0000", 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2283999999999999e-05 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:32.830569+0000", 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0542e-05 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:32.830575+0000", 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1399999999999997e-06 2024-02-10T07:52:38.194 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.195 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.195 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.195 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:32.830636+0000", 2024-02-10T07:52:38.195 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0982e-05 2024-02-10T07:52:38.195 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.195 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.195 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.195 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:32.830759+0000", 2024-02-10T07:52:38.195 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012247500000000001 2024-02-10T07:52:38.195 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.195 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.195 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.195 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:32.830902+0000", 2024-02-10T07:52:38.195 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000143008 2024-02-10T07:52:38.196 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.196 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.196 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.196 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:32.832812+0000", 2024-02-10T07:52:38.196 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001910206 2024-02-10T07:52:38.196 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.196 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.196 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.196 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:32.905467+0000", 2024-02-10T07:52:38.196 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.072655308000000002 2024-02-10T07:52:38.196 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.199 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.199 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.200 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:32.906354+0000", 2024-02-10T07:52:38.200 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00088642099999999998 2024-02-10T07:52:38.200 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.200 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.200 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.201 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:32.906517+0000", 2024-02-10T07:52:38.201 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016355800000000001 2024-02-10T07:52:38.201 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.201 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.201 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.201 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:32.906710+0000", 2024-02-10T07:52:38.201 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000193017 2024-02-10T07:52:38.202 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.202 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.202 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.202 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.202 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.202 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:76085 11.b 11:d6206f51:::benchmark_data_smithi008_186804_object76084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-02-10T07:52:38.203 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:52.327476+0000", 2024-02-10T07:52:38.203 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 105.854772748, 2024-02-10T07:52:38.203 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.080681596999999994, 2024-02-10T07:52:38.203 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.203 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.203 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.203 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.205 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.205 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 76085 2024-02-10T07:52:38.205 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.206 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.206 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.206 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.206 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.327476+0000", 2024-02-10T07:52:38.206 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.206 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.207 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.207 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.207 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.327476+0000", 2024-02-10T07:52:38.207 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.207 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.207 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.208 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.208 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.327477+0000", 2024-02-10T07:52:38.208 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.55e-06 2024-02-10T07:52:38.208 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.208 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.208 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.209 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.327502+0000", 2024-02-10T07:52:38.209 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4539e-05 2024-02-10T07:52:38.209 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.209 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.209 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.209 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.327614+0000", 2024-02-10T07:52:38.210 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000112429 2024-02-10T07:52:38.210 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.210 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.210 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.210 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.327624+0000", 2024-02-10T07:52:38.210 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0186999999999999e-05 2024-02-10T07:52:38.211 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.211 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.211 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.211 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.328023+0000", 2024-02-10T07:52:38.211 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00039819699999999998 2024-02-10T07:52:38.211 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.212 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.212 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.212 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.328462+0000", 2024-02-10T07:52:38.212 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00043975500000000001 2024-02-10T07:52:38.212 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.212 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.212 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.213 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.328985+0000", 2024-02-10T07:52:38.213 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000522757 2024-02-10T07:52:38.213 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.213 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.213 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.213 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.336585+0000", 2024-02-10T07:52:38.213 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0075996429999999997 2024-02-10T07:52:38.213 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.214 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.214 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.214 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.342656+0000", 2024-02-10T07:52:38.214 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0060712259999999999 2024-02-10T07:52:38.214 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.214 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.214 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.214 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.407923+0000", 2024-02-10T07:52:38.215 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.065267059000000002 2024-02-10T07:52:38.215 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.215 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.215 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.215 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.408115+0000", 2024-02-10T07:52:38.215 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019204700000000001 2024-02-10T07:52:38.215 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.215 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.215 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.216 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.408157+0000", 2024-02-10T07:52:38.216 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2208e-05 2024-02-10T07:52:38.216 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.216 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.216 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.216 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.216 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.216 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:76089 11.b 11:da16ee53:::benchmark_data_smithi008_186804_object76088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-02-10T07:52:38.217 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:52.328492+0000", 2024-02-10T07:52:38.217 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 105.853756352, 2024-02-10T07:52:38.217 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.079945489999999994, 2024-02-10T07:52:38.217 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.217 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.217 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.217 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.217 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.218 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 76089 2024-02-10T07:52:38.218 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.218 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.218 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.218 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.218 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.328492+0000", 2024-02-10T07:52:38.218 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.218 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.218 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.218 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.218 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.328492+0000", 2024-02-10T07:52:38.219 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.219 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.219 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.219 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.219 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.328493+0000", 2024-02-10T07:52:38.219 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2020000000000001e-06 2024-02-10T07:52:38.219 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.219 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.219 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.220 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.328517+0000", 2024-02-10T07:52:38.220 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3404000000000002e-05 2024-02-10T07:52:38.220 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.220 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.220 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.220 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.328535+0000", 2024-02-10T07:52:38.220 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8014999999999999e-05 2024-02-10T07:52:38.220 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.220 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.220 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.220 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.328544+0000", 2024-02-10T07:52:38.220 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8950000000000004e-06 2024-02-10T07:52:38.220 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.332083+0000", 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035396949999999998 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.332227+0000", 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014405 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.332377+0000", 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014958500000000001 2024-02-10T07:52:38.221 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.337187+0000", 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0048099609999999998 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.342695+0000", 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0055083279999999998 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.408194+0000", 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.065499207000000004 2024-02-10T07:52:38.222 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.408347+0000", 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015206599999999999 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.408438+0000", 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1082000000000006e-05 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.223 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:76093 11.b 11:d6ffac9d:::benchmark_data_smithi008_186804_object76092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-02-10T07:52:38.224 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:52.333589+0000", 2024-02-10T07:52:38.224 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 105.848659783, 2024-02-10T07:52:38.224 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.075474468000000003, 2024-02-10T07:52:38.224 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.224 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.224 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.224 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.224 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.224 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 76093 2024-02-10T07:52:38.224 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.224 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.224 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.224 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.225 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.333589+0000", 2024-02-10T07:52:38.225 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.225 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.225 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.225 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.225 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.333589+0000", 2024-02-10T07:52:38.225 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.225 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.225 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.225 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.225 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.333591+0000", 2024-02-10T07:52:38.225 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0619999999999999e-06 2024-02-10T07:52:38.225 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.225 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.333614+0000", 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.3569000000000001e-05 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.333631+0000", 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6704000000000001e-05 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.333640+0000", 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2909999999999996e-06 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.226 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.337226+0000", 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0035853819999999998 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.337356+0000", 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013059000000000001 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.337493+0000", 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013683799999999999 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.227 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.228 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.340844+0000", 2024-02-10T07:52:38.228 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033513089999999998 2024-02-10T07:52:38.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.228 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.348675+0000", 2024-02-10T07:52:38.228 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0078307589999999996 2024-02-10T07:52:38.228 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.228 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.228 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.228 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.408857+0000", 2024-02-10T07:52:38.228 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.060182016999999997 2024-02-10T07:52:38.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.229 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.229 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.229 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.409027+0000", 2024-02-10T07:52:38.229 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000169292 2024-02-10T07:52:38.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.229 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.229 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.229 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.409063+0000", 2024-02-10T07:52:38.229 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6655000000000001e-05 2024-02-10T07:52:38.229 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.229 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.229 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.229 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:76098 11.b 11:df5d63f2:::benchmark_data_smithi008_186804_object76097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:52.341944+0000", 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 105.840304203, 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.076800048999999995, 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 76098 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.230 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.231 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.341944+0000", 2024-02-10T07:52:38.231 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.231 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.231 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.231 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.341944+0000", 2024-02-10T07:52:38.231 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.231 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.231 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.231 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.341946+0000", 2024-02-10T07:52:38.231 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5820000000000001e-06 2024-02-10T07:52:38.231 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.231 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.232 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.341970+0000", 2024-02-10T07:52:38.232 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4372e-05 2024-02-10T07:52:38.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.232 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.341986+0000", 2024-02-10T07:52:38.232 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5945999999999999e-05 2024-02-10T07:52:38.232 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.232 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.232 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.233 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.341994+0000", 2024-02-10T07:52:38.233 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7829999999999999e-06 2024-02-10T07:52:38.233 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.233 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.233 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.233 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.342729+0000", 2024-02-10T07:52:38.233 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00073487400000000005 2024-02-10T07:52:38.233 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.233 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.234 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.234 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.342881+0000", 2024-02-10T07:52:38.234 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015218 2024-02-10T07:52:38.234 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.234 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.234 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.234 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.343041+0000", 2024-02-10T07:52:38.234 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015998400000000001 2024-02-10T07:52:38.234 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.235 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.235 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.235 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.346029+0000", 2024-02-10T07:52:38.235 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0029877440000000001 2024-02-10T07:52:38.235 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.235 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.235 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.235 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.348819+0000", 2024-02-10T07:52:38.235 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002790004 2024-02-10T07:52:38.235 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.236 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.236 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.418337+0000", 2024-02-10T07:52:38.236 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.069517920999999996 2024-02-10T07:52:38.236 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.236 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.236 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.237 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.418590+0000", 2024-02-10T07:52:38.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00025289300000000002 2024-02-10T07:52:38.237 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.237 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.237 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.237 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:52.418744+0000", 2024-02-10T07:52:38.237 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000154766 2024-02-10T07:52:38.237 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.238 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.238 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.238 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.238 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.238 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:79789 11.b 11:dda345bb:::benchmark_data_smithi008_186804_object79788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-02-10T07:52:38.239 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:50:57.653518+0000", 2024-02-10T07:52:38.239 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 100.528730552, 2024-02-10T07:52:38.239 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.073390545000000001, 2024-02-10T07:52:38.239 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.239 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.239 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.239 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.240 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.240 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 79789 2024-02-10T07:52:38.240 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.240 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.240 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.240 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.240 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:57.653518+0000", 2024-02-10T07:52:38.241 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.241 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.241 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.241 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.241 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:57.653518+0000", 2024-02-10T07:52:38.241 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.241 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.241 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.242 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:57.653520+0000", 2024-02-10T07:52:38.242 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.66e-06 2024-02-10T07:52:38.242 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.242 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.242 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.242 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:57.653543+0000", 2024-02-10T07:52:38.242 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.2940999999999999e-05 2024-02-10T07:52:38.243 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.243 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.243 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:57.653559+0000", 2024-02-10T07:52:38.243 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6243e-05 2024-02-10T07:52:38.243 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.243 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.244 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.244 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:57.653568+0000", 2024-02-10T07:52:38.244 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3200000000000006e-06 2024-02-10T07:52:38.244 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.244 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.245 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.245 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:57.653627+0000", 2024-02-10T07:52:38.245 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9234000000000002e-05 2024-02-10T07:52:38.245 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.245 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.245 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.246 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:57.653734+0000", 2024-02-10T07:52:38.246 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000106953 2024-02-10T07:52:38.246 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.246 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.246 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.246 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:57.653836+0000", 2024-02-10T07:52:38.246 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000101908 2024-02-10T07:52:38.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.247 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.247 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:57.655826+0000", 2024-02-10T07:52:38.247 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0019902320000000002 2024-02-10T07:52:38.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.247 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.247 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:57.721625+0000", 2024-02-10T07:52:38.247 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.065798454000000006 2024-02-10T07:52:38.247 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.247 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.247 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:57.726554+0000", 2024-02-10T07:52:38.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0049292190000000003 2024-02-10T07:52:38.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.248 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:57.726763+0000", 2024-02-10T07:52:38.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00020878199999999999 2024-02-10T07:52:38.248 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.248 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.248 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.248 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:50:57.726908+0000", 2024-02-10T07:52:38.248 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000145599 2024-02-10T07:52:38.248 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.249 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.249 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.249 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.249 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.249 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:83909 11.b 11:dbf3bd99:::benchmark_data_smithi008_186804_object83908:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-02-10T07:52:38.249 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:03.649479+0000", 2024-02-10T07:52:38.249 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 94.532769713999997, 2024-02-10T07:52:38.249 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.077612348999999997, 2024-02-10T07:52:38.249 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.249 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.250 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.250 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.250 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.250 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 83909 2024-02-10T07:52:38.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.250 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.250 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:03.649479+0000", 2024-02-10T07:52:38.250 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.250 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.250 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.250 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.250 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:03.649479+0000", 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:03.649481+0000", 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7600000000000001e-06 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:03.649507+0000", 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.6361000000000001e-05 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.251 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:03.649532+0000", 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.4909000000000002e-05 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:03.649541+0000", 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4190000000000008e-06 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:03.651539+0000", 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0019976740000000001 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.252 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:03.651705+0000", 2024-02-10T07:52:38.253 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00016619599999999999 2024-02-10T07:52:38.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.253 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.253 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:03.651898+0000", 2024-02-10T07:52:38.253 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00019318499999999999 2024-02-10T07:52:38.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.253 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.253 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:03.664560+0000", 2024-02-10T07:52:38.253 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012661732 2024-02-10T07:52:38.253 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.253 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.253 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:03.685169+0000", 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.020609025 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:03.726986+0000", 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.041816751999999999 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:03.727045+0000", 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9045000000000001e-05 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.254 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.255 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:03.727091+0000", 2024-02-10T07:52:38.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6291000000000001e-05 2024-02-10T07:52:38.255 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.255 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.255 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.255 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.255 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.255 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:84783 11.b 11:d8b24208:::benchmark_data_smithi008_186804_object84782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-02-10T07:52:38.255 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:05.200540+0000", 2024-02-10T07:52:38.255 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 92.981708783000002, 2024-02-10T07:52:38.255 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.071473250000000002, 2024-02-10T07:52:38.256 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.256 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.256 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.256 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.256 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.256 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 84783 2024-02-10T07:52:38.256 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.256 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.256 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.256 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.256 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:05.200540+0000", 2024-02-10T07:52:38.256 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.256 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:05.200540+0000", 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:05.200542+0000", 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0080000000000001e-06 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:05.200603+0000", 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0798000000000003e-05 2024-02-10T07:52:38.257 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:05.200631+0000", 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.8175999999999999e-05 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:05.200646+0000", 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5012e-05 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:05.200757+0000", 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011082300000000001 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.258 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.259 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.259 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:05.200928+0000", 2024-02-10T07:52:38.259 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017103800000000001 2024-02-10T07:52:38.259 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.259 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.259 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.259 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:05.201102+0000", 2024-02-10T07:52:38.259 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00017461500000000001 2024-02-10T07:52:38.259 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.260 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.260 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.260 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:05.208244+0000", 2024-02-10T07:52:38.261 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0071416409999999998 2024-02-10T07:52:38.261 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.261 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.261 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:05.230735+0000", 2024-02-10T07:52:38.261 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.022490739999999999 2024-02-10T07:52:38.261 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.261 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.261 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:05.258215+0000", 2024-02-10T07:52:38.261 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.027480919999999999 2024-02-10T07:52:38.261 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.261 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.262 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.262 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:05.271954+0000", 2024-02-10T07:52:38.262 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.013738057 2024-02-10T07:52:38.262 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.262 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.262 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.262 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:05.272013+0000", 2024-02-10T07:52:38.262 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.9422000000000001e-05 2024-02-10T07:52:38.262 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.262 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.262 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.262 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.262 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:105344 11.b 11:de7fb0b1:::benchmark_data_smithi008_186804_object582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:37.433290+0000", 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 60.748958268999999, 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.099615756, 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 105344 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.263 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.433290+0000", 2024-02-10T07:52:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.433290+0000", 2024-02-10T07:52:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.433292+0000", 2024-02-10T07:52:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4440000000000001e-06 2024-02-10T07:52:38.264 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.433297+0000", 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.8969999999999999e-06 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.433309+0000", 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1932e-05 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.433317+0000", 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.6009999999999995e-06 2024-02-10T07:52:38.265 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.266 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.266 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.433406+0000", 2024-02-10T07:52:38.266 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.8430999999999994e-05 2024-02-10T07:52:38.266 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.266 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.266 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.266 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.433522+0000", 2024-02-10T07:52:38.266 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000116718 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.433626+0000", 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000104146 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.437757+0000", 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.004130178 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.443563+0000", 2024-02-10T07:52:38.267 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0058064930000000002 2024-02-10T07:52:38.268 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.268 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.268 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.268 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.532597+0000", 2024-02-10T07:52:38.268 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.089034297999999998 2024-02-10T07:52:38.268 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.268 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.268 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.268 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.532778+0000", 2024-02-10T07:52:38.268 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00018085 2024-02-10T07:52:38.268 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.268 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.269 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.269 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.532906+0000", 2024-02-10T07:52:38.269 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000127768 2024-02-10T07:52:38.269 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.269 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.269 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.269 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.269 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.269 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:105353 11.b 11:d1020612:::benchmark_data_smithi008_186804_object591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-02-10T07:52:38.269 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:37.444270+0000", 2024-02-10T07:52:38.269 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 60.737978032000001, 2024-02-10T07:52:38.269 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.10100611700000001, 2024-02-10T07:52:38.269 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.269 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 105353 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.444270+0000", 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.444270+0000", 2024-02-10T07:52:38.270 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.444272+0000", 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6279999999999999e-06 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.444278+0000", 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0730000000000001e-06 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.444291+0000", 2024-02-10T07:52:38.271 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.331e-05 2024-02-10T07:52:38.272 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.272 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.272 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.272 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.444301+0000", 2024-02-10T07:52:38.272 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3049999999999998e-06 2024-02-10T07:52:38.272 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.272 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.272 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.272 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.444361+0000", 2024-02-10T07:52:38.272 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0359000000000003e-05 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.444473+0000", 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000112005 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.444584+0000", 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000110375 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.447318+0000", 2024-02-10T07:52:38.273 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002734046 2024-02-10T07:52:38.274 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.274 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.274 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.447976+0000", 2024-02-10T07:52:38.274 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00065796400000000003 2024-02-10T07:52:38.274 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.274 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.274 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.545176+0000", 2024-02-10T07:52:38.274 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.097200644000000003 2024-02-10T07:52:38.274 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.274 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.274 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.274 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.545233+0000", 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.6781999999999999e-05 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.545277+0000", 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.3625999999999999e-05 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:105356 11.b 11:d114b0ed:::benchmark_data_smithi008_186804_object594:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:37.464349+0000", 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 60.717899158999998, 2024-02-10T07:52:38.275 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.084411862000000004, 2024-02-10T07:52:38.276 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.276 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.276 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.276 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.276 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.276 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 105356 2024-02-10T07:52:38.276 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.276 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.276 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.277 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.277 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.464349+0000", 2024-02-10T07:52:38.277 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.277 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.277 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.277 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.277 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.464349+0000", 2024-02-10T07:52:38.277 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.277 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.277 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.278 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.278 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.464352+0000", 2024-02-10T07:52:38.278 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1149999999999998e-06 2024-02-10T07:52:38.278 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.278 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.278 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.278 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.464364+0000", 2024-02-10T07:52:38.278 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1042e-05 2024-02-10T07:52:38.278 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.278 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.278 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.279 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.464385+0000", 2024-02-10T07:52:38.279 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.1248999999999999e-05 2024-02-10T07:52:38.279 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.279 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.279 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.279 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.464399+0000", 2024-02-10T07:52:38.279 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4204e-05 2024-02-10T07:52:38.279 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.279 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.279 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.280 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.464480+0000", 2024-02-10T07:52:38.280 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0747000000000005e-05 2024-02-10T07:52:38.280 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.280 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.280 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.280 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.464625+0000", 2024-02-10T07:52:38.280 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000145764 2024-02-10T07:52:38.281 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.281 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.281 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.281 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.464709+0000", 2024-02-10T07:52:38.281 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.3373000000000003e-05 2024-02-10T07:52:38.281 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.281 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.281 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.281 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.466942+0000", 2024-02-10T07:52:38.281 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0022327810000000001 2024-02-10T07:52:38.281 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.281 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.281 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.281 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.468350+0000", 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0014080589999999999 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.548438+0000", 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.080088582000000005 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.548622+0000", 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00018353600000000001 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.282 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:37.548761+0000", 2024-02-10T07:52:38.283 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013941 2024-02-10T07:52:38.283 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.283 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.283 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.283 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.283 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.283 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:105748 11.b 11:d007a4d1:::benchmark_data_smithi008_186804_object986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-02-10T07:52:38.283 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:38.062992+0000", 2024-02-10T07:52:38.283 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 60.119256694999997, 2024-02-10T07:52:38.283 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.099537871999999999, 2024-02-10T07:52:38.283 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.283 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.283 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.283 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.284 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.284 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 105748 2024-02-10T07:52:38.284 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.284 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.284 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.284 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.284 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:38.062992+0000", 2024-02-10T07:52:38.284 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.284 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.284 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.284 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.284 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:38.062992+0000", 2024-02-10T07:52:38.284 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.284 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.285 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.285 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.285 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:38.062993+0000", 2024-02-10T07:52:38.285 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.635e-06 2024-02-10T07:52:38.285 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.285 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.285 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.285 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:38.063000+0000", 2024-02-10T07:52:38.285 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2110000000000003e-06 2024-02-10T07:52:38.285 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.285 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.285 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.285 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:38.063012+0000", 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.2262999999999999e-05 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:38.063022+0000", 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6979999999999993e-06 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:38.063956+0000", 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00093485900000000002 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:38.064105+0000", 2024-02-10T07:52:38.286 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014892400000000001 2024-02-10T07:52:38.287 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.287 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.287 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.287 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:38.064228+0000", 2024-02-10T07:52:38.287 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000122626 2024-02-10T07:52:38.287 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.287 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.287 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.287 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:38.068570+0000", 2024-02-10T07:52:38.287 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.004342154 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:38.071875+0000", 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0033044099999999998 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:38.162296+0000", 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.090420985999999995 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:38.162483+0000", 2024-02-10T07:52:38.288 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000187796 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:38.162530+0000", 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6310000000000002e-05 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:108549 11.b 11:dcd6b86d:::benchmark_data_smithi008_186804_object3787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:42.289213+0000", 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 55.893035613999999, 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.078329299000000005, 2024-02-10T07:52:38.289 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 108549 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:42.289213+0000", 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.290 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:42.289213+0000", 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:42.289215+0000", 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6130000000000001e-06 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:42.289221+0000", 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3759999999999996e-06 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.291 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:42.289236+0000", 2024-02-10T07:52:38.292 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4629e-05 2024-02-10T07:52:38.292 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.292 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.292 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.292 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:42.289245+0000", 2024-02-10T07:52:38.292 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3389999999999999e-06 2024-02-10T07:52:38.292 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.292 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.292 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.292 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:42.289295+0000", 2024-02-10T07:52:38.292 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0642000000000001e-05 2024-02-10T07:52:38.292 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.292 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.292 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:42.289409+0000", 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000113899 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:42.289519+0000", 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000110097 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:42.296298+0000", 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.006778905 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.293 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:42.298691+0000", 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0023927800000000002 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:42.367073+0000", 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.068382009999999993 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:42.367300+0000", 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000226775 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.294 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.295 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:42.367542+0000", 2024-02-10T07:52:38.295 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000242234 2024-02-10T07:52:38.295 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.295 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.295 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.295 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.295 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.295 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:111089 11.b 11:da4990de:::benchmark_data_smithi008_186804_object6327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-02-10T07:52:38.295 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:51:45.969765+0000", 2024-02-10T07:52:38.295 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 52.212483718999998, 2024-02-10T07:52:38.295 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.13371216, 2024-02-10T07:52:38.295 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.295 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.295 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 111089 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:45.969765+0000", 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:45.969765+0000", 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.296 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.297 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.297 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.297 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:45.969766+0000", 2024-02-10T07:52:38.297 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4750000000000001e-06 2024-02-10T07:52:38.297 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.297 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.297 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.297 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:45.969773+0000", 2024-02-10T07:52:38.297 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0849999999999999e-06 2024-02-10T07:52:38.297 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.297 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.298 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.298 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:45.969786+0000", 2024-02-10T07:52:38.298 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3113e-05 2024-02-10T07:52:38.298 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.298 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.298 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.298 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:45.969802+0000", 2024-02-10T07:52:38.298 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.5048999999999999e-05 2024-02-10T07:52:38.298 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.298 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.298 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.298 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:45.969865+0000", 2024-02-10T07:52:38.298 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.3378000000000002e-05 2024-02-10T07:52:38.298 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:45.970014+0000", 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014935999999999999 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:45.970160+0000", 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014602199999999999 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:45.972903+0000", 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.002743124 2024-02-10T07:52:38.299 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:45.976771+0000", 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0038673420000000002 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:46.103372+0000", 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.126601563 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:46.103434+0000", 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2038999999999993e-05 2024-02-10T07:52:38.300 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.301 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.301 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.301 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:51:46.103477+0000", 2024-02-10T07:52:38.301 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.261e-05 2024-02-10T07:52:38.301 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.301 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.301 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.301 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.301 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.301 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:124367 11.b 11:d2083bc1:::benchmark_data_smithi008_186804_object19605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-02-10T07:52:38.301 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:04.635147+0000", 2024-02-10T07:52:38.301 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 33.547101398000002, 2024-02-10T07:52:38.301 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.082090814999999998, 2024-02-10T07:52:38.301 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.302 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.302 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.302 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.302 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.302 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 124367 2024-02-10T07:52:38.302 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.302 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.302 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.302 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.302 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.635147+0000", 2024-02-10T07:52:38.302 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.302 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.302 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.302 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.635147+0000", 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.635149+0000", 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.7290000000000001e-06 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.635155+0000", 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.0719999999999996e-06 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.303 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.635174+0000", 2024-02-10T07:52:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9477e-05 2024-02-10T07:52:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.635185+0000", 2024-02-10T07:52:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0241000000000001e-05 2024-02-10T07:52:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.635286+0000", 2024-02-10T07:52:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000100906 2024-02-10T07:52:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.304 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.635412+0000", 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012603399999999999 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.635535+0000", 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012339499999999999 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.640873+0000", 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0053384540000000003 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.305 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.306 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.306 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.648239+0000", 2024-02-10T07:52:38.306 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0073653909999999998 2024-02-10T07:52:38.306 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.306 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.306 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.306 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.717125+0000", 2024-02-10T07:52:38.306 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.068885818000000001 2024-02-10T07:52:38.306 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.306 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.306 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.306 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.717191+0000", 2024-02-10T07:52:38.306 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5982999999999995e-05 2024-02-10T07:52:38.306 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.717238+0000", 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7314999999999997e-05 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:124375 11.b 11:d59bcf9f:::benchmark_data_smithi008_186804_object19613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:04.650311+0000", 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 33.531937165000002, 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.075741262000000004, 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.307 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.308 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.308 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.308 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.308 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 124375 2024-02-10T07:52:38.308 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.308 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.308 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.308 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.309 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.650311+0000", 2024-02-10T07:52:38.309 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.309 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.309 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.309 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.309 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.650311+0000", 2024-02-10T07:52:38.309 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.309 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.309 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.309 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.309 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.650313+0000", 2024-02-10T07:52:38.309 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.79e-06 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.650319+0000", 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2419999999999997e-06 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.650334+0000", 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4966999999999999e-05 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.650344+0000", 2024-02-10T07:52:38.310 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.3749999999999992e-06 2024-02-10T07:52:38.311 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.311 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.311 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.311 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.650419+0000", 2024-02-10T07:52:38.311 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5366999999999999e-05 2024-02-10T07:52:38.311 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.311 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.311 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.311 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.650566+0000", 2024-02-10T07:52:38.311 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014655899999999999 2024-02-10T07:52:38.311 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.311 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.311 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.311 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.650708+0000", 2024-02-10T07:52:38.312 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000142779 2024-02-10T07:52:38.312 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.312 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.312 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.312 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.656281+0000", 2024-02-10T07:52:38.312 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.005572558 2024-02-10T07:52:38.312 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.312 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.312 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.312 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.656643+0000", 2024-02-10T07:52:38.312 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00036223599999999999 2024-02-10T07:52:38.312 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.313 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.313 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.313 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.725958+0000", 2024-02-10T07:52:38.313 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.069315225999999994 2024-02-10T07:52:38.313 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.313 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.313 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.313 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.726011+0000", 2024-02-10T07:52:38.313 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2434e-05 2024-02-10T07:52:38.313 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.313 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.313 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.314 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.726053+0000", 2024-02-10T07:52:38.314 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.1729e-05 2024-02-10T07:52:38.314 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.314 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.314 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.314 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.314 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.314 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:124376 11.b 11:d110f499:::benchmark_data_smithi008_186804_object19614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-02-10T07:52:38.314 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:04.652823+0000", 2024-02-10T07:52:38.314 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 33.529425586999999, 2024-02-10T07:52:38.314 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.073620605000000006, 2024-02-10T07:52:38.314 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.314 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.314 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.315 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.315 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.315 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 124376 2024-02-10T07:52:38.315 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.315 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.315 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.315 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.315 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.652823+0000", 2024-02-10T07:52:38.315 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.315 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.315 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.315 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.316 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.652823+0000", 2024-02-10T07:52:38.316 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.316 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.316 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.316 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.316 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.652825+0000", 2024-02-10T07:52:38.316 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.9230000000000001e-06 2024-02-10T07:52:38.316 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.316 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.316 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.316 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.652832+0000", 2024-02-10T07:52:38.316 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.7880000000000001e-06 2024-02-10T07:52:38.316 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.317 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.317 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.317 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.652851+0000", 2024-02-10T07:52:38.317 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.965e-05 2024-02-10T07:52:38.317 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.317 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.317 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.317 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.652861+0000", 2024-02-10T07:52:38.317 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.6849999999999996e-06 2024-02-10T07:52:38.317 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.317 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.317 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.318 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.653290+0000", 2024-02-10T07:52:38.318 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:38.319 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00042891700000000001 2024-02-10T07:52:38.319 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.320 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.320 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.320 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.653444+0000", 2024-02-10T07:52:38.320 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015384900000000001 2024-02-10T07:52:38.320 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.320 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.320 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.320 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.653594+0000", 2024-02-10T07:52:38.320 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00015055099999999999 2024-02-10T07:52:38.320 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.320 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.320 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.320 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.656384+0000", 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0027897759999999999 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.656692+0000", 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00030768700000000001 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.726377+0000", 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.069685121000000003 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.321 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.726408+0000", 2024-02-10T07:52:38.322 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.1473000000000002e-05 2024-02-10T07:52:38.322 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.322 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.322 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.322 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:04.726444+0000", 2024-02-10T07:52:38.322 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.5185000000000003e-05 2024-02-10T07:52:38.322 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.322 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.322 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.322 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.322 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.322 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:128019 11.b 11:d15c933d:::benchmark_data_smithi008_186804_object23257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-02-10T07:52:38.322 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:09.551206+0000", 2024-02-10T07:52:38.322 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 28.631042101999999, 2024-02-10T07:52:38.323 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.076658105000000004, 2024-02-10T07:52:38.323 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.323 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.323 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.323 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.323 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.323 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 128019 2024-02-10T07:52:38.323 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.323 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.323 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.323 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.324 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:09.551206+0000", 2024-02-10T07:52:38.324 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.324 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.324 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.324 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.324 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:09.551206+0000", 2024-02-10T07:52:38.324 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.324 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.324 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.324 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.324 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:09.551208+0000", 2024-02-10T07:52:38.325 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.308e-06 2024-02-10T07:52:38.325 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.325 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.325 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.325 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:09.551214+0000", 2024-02-10T07:52:38.325 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.4999999999999996e-06 2024-02-10T07:52:38.325 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.325 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.325 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.325 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:09.551227+0000", 2024-02-10T07:52:38.325 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.3098000000000001e-05 2024-02-10T07:52:38.325 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.325 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:09.551237+0000", 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2410000000000001e-06 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:09.551299+0000", 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.2378999999999993e-05 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:09.551417+0000", 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00011763 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.326 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:09.551494+0000", 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.7690999999999999e-05 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:09.605253+0000", 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.053758918000000003 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:09.610475+0000", 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.005221954 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.327 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.328 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.328 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:09.627004+0000", 2024-02-10T07:52:38.328 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.016528516 2024-02-10T07:52:38.328 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.328 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.328 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.328 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:09.627817+0000", 2024-02-10T07:52:38.328 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00081333399999999998 2024-02-10T07:52:38.328 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.328 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.328 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.328 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:09.627865+0000", 2024-02-10T07:52:38.328 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7536e-05 2024-02-10T07:52:38.328 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.329 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.329 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.329 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.329 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.329 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:132533 11.b 11:dcf17f07:::benchmark_data_smithi008_186804_object27771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-02-10T07:52:38.329 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:15.973224+0000", 2024-02-10T07:52:38.329 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 22.209024195000001, 2024-02-10T07:52:38.329 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.073148296000000002, 2024-02-10T07:52:38.329 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:38.329 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 132533 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:15.973224+0000", 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:15.973224+0000", 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:38.330 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.331 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.331 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:38.331 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:15.973225+0000", 2024-02-10T07:52:38.331 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.2699999999999999e-07 2024-02-10T07:52:38.331 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.331 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.331 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:38.331 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:15.973229+0000", 2024-02-10T07:52:38.331 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.7170000000000002e-06 2024-02-10T07:52:38.331 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.331 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.331 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:38.331 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:15.973238+0000", 2024-02-10T07:52:38.331 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.1929999999999999e-06 2024-02-10T07:52:38.332 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.332 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.332 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:38.332 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:15.973244+0000", 2024-02-10T07:52:38.332 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8459999999999996e-06 2024-02-10T07:52:38.332 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.332 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.332 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:38.332 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:15.974665+0000", 2024-02-10T07:52:38.332 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.001421243 2024-02-10T07:52:38.332 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.332 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.332 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:38.332 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:15.974734+0000", 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8541999999999995e-05 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 0,1", 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:15.974804+0000", 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.0815000000000006e-05 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:15.975775+0000", 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00097083499999999997 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.333 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:16.041866+0000", 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.066091021 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:16.046266+0000", 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0043995689999999999 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:16.046332+0000", 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.6076999999999998e-05 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "done", 2024-02-10T07:52:38.334 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:16.046373+0000", 2024-02-10T07:52:38.335 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.0710999999999997e-05 2024-02-10T07:52:38.335 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.335 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.335 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.335 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:38.335 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:38.335 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:52:38.345 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:38.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:38.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:38.447 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:38.482 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:38.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:38.583 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:38.631 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:38.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:38.732 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:38.778 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:38.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:38.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:38.880 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:38.909 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:38.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:38.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:39.011 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:39.039 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:39.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:39.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:39.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:52:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:52:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:52:39.141 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:39.179 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:39.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:39.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:39.281 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:39.319 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:39.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:39.421 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:39.462 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:39.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:39.564 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:39.594 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:39.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:39.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:39.696 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:39.735 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:39.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:39.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:39.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:39 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:52:39.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:39 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:52:39.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:39 smithi008 ceph-mon[29816]: pgmap v959: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 552 op/s; 9056/191019 objects degraded (4.741%); 9455/191019 objects misplaced (4.950%); 6.2 MiB/s, 98 objects/s recovering 2024-02-10T07:52:39.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:39 smithi008 ceph-mon[29816]: osdmap e436: 8 total, 8 up, 8 in 2024-02-10T07:52:39.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:39 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:52:39.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:39 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:52:39.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:39 smithi008 ceph-mon[40743]: pgmap v959: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 552 op/s; 9056/191019 objects degraded (4.741%); 9455/191019 objects misplaced (4.950%); 6.2 MiB/s, 98 objects/s recovering 2024-02-10T07:52:39.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:39 smithi008 ceph-mon[40743]: osdmap e436: 8 total, 8 up, 8 in 2024-02-10T07:52:39.839 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:39.869 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:39.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:39.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:39 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:52:39.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:39 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:52:39.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:39 smithi170 ceph-mon[33394]: pgmap v959: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 552 op/s; 9056/191019 objects degraded (4.741%); 9455/191019 objects misplaced (4.950%); 6.2 MiB/s, 98 objects/s recovering 2024-02-10T07:52:39.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:39 smithi170 ceph-mon[33394]: osdmap e436: 8 total, 8 up, 8 in 2024-02-10T07:52:39.971 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:40.001 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:40.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:40.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:40.103 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:40.153 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:40.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:40.255 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:40.287 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:40.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:40.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:40.389 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:40.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:40.434 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:40.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:40.536 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:40.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:40.580 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:40.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:40.653 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-10T07:52:40.681 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:40.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:40.763 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:40.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:40.864 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:40.892 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:40.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:40.994 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:41.055 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:52:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:41.065 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:41.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:41.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:41.194 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:41.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:41.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:41.296 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:41.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:41.329 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:41.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:41.430 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:41.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:41.477 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:41.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:41.537 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:52:41.578 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:41.603 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:41.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:41.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:41.736 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:41.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:41.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:41.835 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:41 smithi008 ceph-mon[29816]: pgmap v961: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 832 op/s; 9158/182196 objects degraded (5.026%); 9230/182196 objects misplaced (5.066%); 4.6 MiB/s, 72 objects/s recovering 2024-02-10T07:52:41.835 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:41 smithi008 ceph-mon[40743]: pgmap v961: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 832 op/s; 9158/182196 objects degraded (5.026%); 9230/182196 objects misplaced (5.066%); 4.6 MiB/s, 72 objects/s recovering 2024-02-10T07:52:41.837 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:41.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:41 smithi170 ceph-mon[33394]: pgmap v961: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 832 op/s; 9158/182196 objects degraded (5.026%); 9230/182196 objects misplaced (5.066%); 4.6 MiB/s, 72 objects/s recovering 2024-02-10T07:52:41.897 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:41.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:41.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:42.002 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:42.031 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:42.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:42.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:42.037 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:52:42.133 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:42.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:42.180 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:42.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:42.281 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:42.310 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:42.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:42.412 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:42.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:42.441 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:42.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:42.542 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:42.585 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:42.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:42.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:42.687 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:42.713 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:42.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:42.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:42.801 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:42 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 9158/182196 objects degraded (5.026%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:42.801 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:42 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:52:42.801 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:42 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:52:42.801 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:42 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 9158/182196 objects degraded (5.026%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:42.801 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:42 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:52:42.801 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:42 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:52:42.815 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:42.843 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:42.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:42.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:42 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 9158/182196 objects degraded (5.026%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:42.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:42 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:52:42.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:42 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:52:42.946 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:42.973 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:42.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:42.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:43.049 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:52:43.075 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:43.105 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:43.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:43.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:43.207 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:43.240 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:43.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:43.344 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:43.392 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:43.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:43.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:43.494 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:43.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:43.522 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:43.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:43.623 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:43.654 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:43.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:43.755 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:43.783 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:43.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:43.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:43 smithi008 ceph-mon[29816]: pgmap v962: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 655 op/s; 9260/180132 objects degraded (5.141%); 8790/180132 objects misplaced (4.880%); 7.2 MiB/s, 115 objects/s recovering 2024-02-10T07:52:43.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:43 smithi008 ceph-mon[29816]: osdmap e437: 8 total, 8 up, 8 in 2024-02-10T07:52:43.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:43 smithi008 ceph-mon[40743]: pgmap v962: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 655 op/s; 9260/180132 objects degraded (5.141%); 8790/180132 objects misplaced (4.880%); 7.2 MiB/s, 115 objects/s recovering 2024-02-10T07:52:43.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:43 smithi008 ceph-mon[40743]: osdmap e437: 8 total, 8 up, 8 in 2024-02-10T07:52:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:43 smithi170 ceph-mon[33394]: pgmap v962: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 655 op/s; 9260/180132 objects degraded (5.141%); 8790/180132 objects misplaced (4.880%); 7.2 MiB/s, 115 objects/s recovering 2024-02-10T07:52:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:43 smithi170 ceph-mon[33394]: osdmap e437: 8 total, 8 up, 8 in 2024-02-10T07:52:43.885 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:43.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:43.915 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:43.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:44.016 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:44.043 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:44.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:44.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:44.145 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:44.176 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:44.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:44.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:44.278 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:44.308 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:44.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:44.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:52:43 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:52:44.410 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:44.464 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:44.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:44.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:44.567 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:44.596 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:44.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:44.697 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:44.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:44.729 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:44.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:44.830 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:44.856 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:44.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:44.958 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:45.000 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:45.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:45.102 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:45.144 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:45.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:45.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:45.246 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:45.271 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:45.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:45.373 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:45.402 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:45.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:45.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:45.504 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:45.533 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:45.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:45.635 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:45.662 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:45.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:45.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:45.764 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:45.803 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:45 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:45.803 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:45 smithi008 ceph-mon[29816]: pgmap v964: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 764 op/s; 9260/173508 objects degraded (5.337%); 8790/173508 objects misplaced (5.066%); 3.8 MiB/s, 60 objects/s recovering 2024-02-10T07:52:45.803 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:45 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:45.803 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:45 smithi008 ceph-mon[40743]: pgmap v964: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 764 op/s; 9260/173508 objects degraded (5.337%); 8790/173508 objects misplaced (5.066%); 3.8 MiB/s, 60 objects/s recovering 2024-02-10T07:52:45.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:45.816 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:45.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:45.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:45 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:45.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:45 smithi170 ceph-mon[33394]: pgmap v964: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 764 op/s; 9260/173508 objects degraded (5.337%); 8790/173508 objects misplaced (5.066%); 3.8 MiB/s, 60 objects/s recovering 2024-02-10T07:52:45.916 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:45.955 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:45.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:45.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:46.057 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:46.086 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:46.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:46.188 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:46.228 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:46.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:46.330 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:46.357 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:46.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:46.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:46.458 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:46.486 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:46.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:46.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:46.587 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:46.613 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:46.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:46.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:46.715 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:46.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:46.758 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:46.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:46.859 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:46.896 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:46.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:46.997 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:47.035 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:47.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:47.137 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:47.185 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:47.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:47.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:47.287 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:47.317 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:47.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:47.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:47.419 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:47.456 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:47.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:47.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:47.558 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:47.602 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:47.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:47.704 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:47.735 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:47.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:47.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:47.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:47 smithi008 ceph-mon[29816]: pgmap v965: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 902 op/s; 9433/169371 objects degraded (5.569%); 8126/169371 objects misplaced (4.798%); 7.4 MiB/s, 118 objects/s recovering 2024-02-10T07:52:47.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:47 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 9433/169371 objects degraded (5.569%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:47.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:47 smithi008 ceph-mon[40743]: pgmap v965: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 902 op/s; 9433/169371 objects degraded (5.569%); 8126/169371 objects misplaced (4.798%); 7.4 MiB/s, 118 objects/s recovering 2024-02-10T07:52:47.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:47 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 9433/169371 objects degraded (5.569%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:47.836 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:47.866 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:47.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:47.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:47 smithi170 ceph-mon[33394]: pgmap v965: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 902 op/s; 9433/169371 objects degraded (5.569%); 8126/169371 objects misplaced (4.798%); 7.4 MiB/s, 118 objects/s recovering 2024-02-10T07:52:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:47 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 9433/169371 objects degraded (5.569%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:47.968 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:47.997 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:47.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:48.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:48.127 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:48.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:48.229 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:48.269 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:48.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:48.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:48.372 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:48.401 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:48.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:48.502 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:48.543 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:48.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:48.645 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:48.687 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:48.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:48.788 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:48.833 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:52:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:52:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:52:48.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:48.836 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:48.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:48.938 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:48.988 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:52:48.991 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:48.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:48.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:49.094 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:49.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:49.122 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:49.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:49.223 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:49.273 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:49.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:49.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:49.376 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:49.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:49.407 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:49.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:49.426 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:52:49.426 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:52:49.427 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.427 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:157472 11.2 11:488012f6:::benchmark_data_smithi008_186804_object52710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-02-10T07:52:49.427 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:49.373089+0000", 2024-02-10T07:52:49.427 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.049424095000000001, 2024-02-10T07:52:49.427 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.049466304000000003, 2024-02-10T07:52:49.427 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:49.427 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-10T07:52:49.427 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:49.427 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:49.427 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:49.427 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 157472 2024-02-10T07:52:49.427 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.427 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:49.427 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.428 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:49.428 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.373089+0000", 2024-02-10T07:52:49.428 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:49.428 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.428 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.428 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:49.428 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.373089+0000", 2024-02-10T07:52:49.428 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:49.428 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.428 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.428 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:49.428 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.373091+0000", 2024-02-10T07:52:49.429 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6279999999999999e-06 2024-02-10T07:52:49.429 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.429 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.429 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:49.429 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.373096+0000", 2024-02-10T07:52:49.429 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.4240000000000001e-06 2024-02-10T07:52:49.429 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.430 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.430 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:49.430 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.373113+0000", 2024-02-10T07:52:49.430 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6498999999999999e-05 2024-02-10T07:52:49.430 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.430 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.430 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:49.430 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.373124+0000", 2024-02-10T07:52:49.430 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0845e-05 2024-02-10T07:52:49.430 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.430 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.430 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:49.430 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.373444+0000", 2024-02-10T07:52:49.430 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00032078599999999999 2024-02-10T07:52:49.431 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.431 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.431 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:49.431 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.373564+0000", 2024-02-10T07:52:49.431 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000119905 2024-02-10T07:52:49.431 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.431 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.431 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,3", 2024-02-10T07:52:49.431 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.373737+0000", 2024-02-10T07:52:49.431 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000172712 2024-02-10T07:52:49.431 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.431 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.431 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:49.431 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.393735+0000", 2024-02-10T07:52:49.432 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.019997653000000001 2024-02-10T07:52:49.432 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.432 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.432 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:49.432 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.393785+0000", 2024-02-10T07:52:49.432 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.0662999999999999e-05 2024-02-10T07:52:49.432 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.432 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.432 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:49.432 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.421729+0000", 2024-02-10T07:52:49.433 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.027944031000000001 2024-02-10T07:52:49.433 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.433 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.433 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "commit_sent", 2024-02-10T07:52:49.433 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.421782+0000", 2024-02-10T07:52:49.433 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.2976000000000002e-05 2024-02-10T07:52:49.433 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:49.433 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:49.433 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:49.433 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.433 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.433 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:157475 11.2 11:47479868:::benchmark_data_smithi008_186804_object52713:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-02-10T07:52:49.433 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:49.375290+0000", 2024-02-10T07:52:49.434 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.047222924999999999, 2024-02-10T07:52:49.434 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.047395637999999997, 2024-02-10T07:52:49.434 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:49.434 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:52:49.434 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:49.434 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:49.434 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:49.434 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 157475 2024-02-10T07:52:49.434 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.434 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:49.434 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.434 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:49.434 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.375290+0000", 2024-02-10T07:52:49.434 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:49.435 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.435 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.435 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:49.435 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.375290+0000", 2024-02-10T07:52:49.435 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:49.435 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.435 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.436 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:49.436 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.375292+0000", 2024-02-10T07:52:49.436 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.59e-06 2024-02-10T07:52:49.436 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.436 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.436 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:49.437 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.375297+0000", 2024-02-10T07:52:49.437 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.3000000000000001e-06 2024-02-10T07:52:49.437 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.437 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.437 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:49.437 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.375315+0000", 2024-02-10T07:52:49.438 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.8107000000000001e-05 2024-02-10T07:52:49.438 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.438 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.438 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:49.438 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.375325+0000", 2024-02-10T07:52:49.438 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.2110000000000004e-06 2024-02-10T07:52:49.439 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.439 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.439 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:49.439 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.393821+0000", 2024-02-10T07:52:49.439 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.018496622000000001 2024-02-10T07:52:49.440 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.440 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.443 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:49.443 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.393945+0000", 2024-02-10T07:52:49.443 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00012349299999999999 2024-02-10T07:52:49.444 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.444 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.444 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,3", 2024-02-10T07:52:49.444 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.394749+0000", 2024-02-10T07:52:49.444 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00080442899999999995 2024-02-10T07:52:49.444 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.444 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.444 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:52:49.444 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.416636+0000", 2024-02-10T07:52:49.444 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.021886985000000001 2024-02-10T07:52:49.444 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.445 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.445 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:52:49.445 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.422697+0000", 2024-02-10T07:52:49.445 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0060610220000000001 2024-02-10T07:52:49.445 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:49.445 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:49.445 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:49.445 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.445 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.445 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop_reply(client.25838.0:157475 11.2 e437/405 ondisk, result = 0)", 2024-02-10T07:52:49.445 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:49.410223+0000", 2024-02-10T07:52:49.445 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.012290011, 2024-02-10T07:52:49.445 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012568622999999999, 2024-02-10T07:52:49.445 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.410223+0000", 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.410223+0000", 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:49.446 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.410225+0000", 2024-02-10T07:52:49.447 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.285e-06 2024-02-10T07:52:49.447 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.447 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.447 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:49.447 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.410231+0000", 2024-02-10T07:52:49.447 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.8510000000000001e-06 2024-02-10T07:52:49.447 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.447 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.447 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:49.447 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.410248+0000", 2024-02-10T07:52:49.448 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.6288000000000001e-05 2024-02-10T07:52:49.448 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.448 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.448 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:49.448 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.410256+0000", 2024-02-10T07:52:49.448 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.721e-06 2024-02-10T07:52:49.448 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.448 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.448 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.422730+0000", 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.012473253 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.422739+0000", 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.7450000000000007e-06 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:157513 11.2 11.62e5f6d2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e437)", 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:52:49.422808+0000", 2024-02-10T07:52:49.449 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 4294967295.9997053, 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7627000000000002e-05, 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 157513 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.422808+0000", 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.450 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:52:49.451 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.422808+0000", 2024-02-10T07:52:49.451 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:52:49.451 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.451 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.451 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:52:49.451 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.422809+0000", 2024-02-10T07:52:49.451 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4079999999999999e-06 2024-02-10T07:52:49.451 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.452 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.452 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:52:49.452 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.422816+0000", 2024-02-10T07:52:49.452 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1970000000000001e-06 2024-02-10T07:52:49.452 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.452 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.452 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:52:49.452 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.422827+0000", 2024-02-10T07:52:49.452 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1549000000000001e-05 2024-02-10T07:52:49.452 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:52:49.452 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:52:49.452 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:52:49.453 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:52:49.422837+0000", 2024-02-10T07:52:49.453 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.9639999999999998e-06 2024-02-10T07:52:49.453 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:49.453 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:52:49.453 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:49.453 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:52:49.453 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:52:49.454 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 4 2024-02-10T07:52:49.454 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:52:49.508 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:49.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:49.537 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:49.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:49.574 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.6 to 1 (10000) 2024-02-10T07:52:49.638 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:49.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:49.672 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:49.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:49.773 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:49.821 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:49.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:49 smithi008 ceph-mon[29816]: pgmap v966: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 530 B/s rd, 0 B/s wr, 676 op/s; 9433/168639 objects degraded (5.594%); 8126/168639 objects misplaced (4.819%); 6.2 MiB/s, 98 objects/s recovering 2024-02-10T07:52:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:52:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:52:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:49 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2710704853' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:52:49.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:49 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:52:49.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:49 smithi008 ceph-mon[40743]: pgmap v966: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 530 B/s rd, 0 B/s wr, 676 op/s; 9433/168639 objects degraded (5.594%); 8126/168639 objects misplaced (4.819%); 6.2 MiB/s, 98 objects/s recovering 2024-02-10T07:52:49.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:52:49.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:52:49.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:49 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2710704853' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:52:49.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:49 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:52:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:49 smithi170 ceph-mon[33394]: pgmap v966: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 530 B/s rd, 0 B/s wr, 676 op/s; 9433/168639 objects degraded (5.594%); 8126/168639 objects misplaced (4.819%); 6.2 MiB/s, 98 objects/s recovering 2024-02-10T07:52:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:52:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:52:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:49 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2710704853' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:52:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:49 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:52:49.922 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:49.948 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:49.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:50.050 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:50.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:50.082 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:50.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:50.183 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:50.236 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:50.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:50.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:50.338 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:50.377 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:50.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:50.478 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:50.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:50.528 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:50.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:50.629 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:50.656 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:50.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:50.663 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:52:50.758 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:50.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:50.810 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:50.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:50.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:50 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:52:50.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:50 smithi008 ceph-mon[29816]: osdmap e438: 8 total, 8 up, 8 in 2024-02-10T07:52:50.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:50 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:52:50.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:50 smithi008 ceph-mon[40743]: osdmap e438: 8 total, 8 up, 8 in 2024-02-10T07:52:50.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:50 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:52:50.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:50 smithi170 ceph-mon[33394]: osdmap e438: 8 total, 8 up, 8 in 2024-02-10T07:52:50.912 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:50.941 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:50.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:51.043 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:51.085 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:51.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:51.187 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:51.213 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:51.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:51.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:51.220 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-10T07:52:51.315 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:51.390 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:51.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:51.492 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:51.548 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:51.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:51.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:51.575 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:51 smithi008 ceph-mon[29816]: pgmap v968: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 881 op/s; 9519/158994 objects degraded (5.987%); 7896/158994 objects misplaced (4.966%); 5.0 MiB/s, 79 objects/s recovering 2024-02-10T07:52:51.652 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:51.681 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:51.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:51.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:51.783 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:51.809 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:51.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:51.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:51.831 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:51 smithi008 ceph-mon[40743]: pgmap v968: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 881 op/s; 9519/158994 objects degraded (5.987%); 7896/158994 objects misplaced (4.966%); 5.0 MiB/s, 79 objects/s recovering 2024-02-10T07:52:51.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:51 smithi170 ceph-mon[33394]: pgmap v968: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 881 op/s; 9519/158994 objects degraded (5.987%); 7896/158994 objects misplaced (4.966%); 5.0 MiB/s, 79 objects/s recovering 2024-02-10T07:52:51.911 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:51.953 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:51.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:51.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:52.055 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:52.083 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:52.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:52.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:52.185 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:52.243 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:52.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:52.345 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:52.375 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:52.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:52.476 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:52.522 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:52.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:52.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:52.624 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:52.650 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:52.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:52.752 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:52.792 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:52.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:52.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:52 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 9519/158994 objects degraded (5.987%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:52.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:52 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:52:52.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:52 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:52:52.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:52 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 9519/158994 objects degraded (5.987%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:52.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:52 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:52:52.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:52 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:52:52.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:52 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 9519/158994 objects degraded (5.987%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:52.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:52 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:52:52.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:52 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:52:52.894 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:52.924 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:52.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:53.025 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:53.056 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:53.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:53.158 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:53.192 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:53.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:53.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:53.293 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:53.332 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:53.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:53.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:53.434 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:53.475 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:53.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:53.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:53.577 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:53.601 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:53.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:53.703 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:53.750 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:53.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:53.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:53.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:53 smithi008 ceph-mon[29816]: pgmap v969: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 859 op/s; 9549/156930 objects degraded (6.085%); 7452/156930 objects misplaced (4.749%); 7.3 MiB/s, 116 objects/s recovering 2024-02-10T07:52:53.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:53 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:52:53.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:53 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:52:53.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:53 smithi008 ceph-mon[29816]: osdmap e439: 8 total, 8 up, 8 in 2024-02-10T07:52:53.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:53 smithi008 ceph-mon[40743]: pgmap v969: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 859 op/s; 9549/156930 objects degraded (6.085%); 7452/156930 objects misplaced (4.749%); 7.3 MiB/s, 116 objects/s recovering 2024-02-10T07:52:53.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:53 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:52:53.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:53 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:52:53.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:53 smithi008 ceph-mon[40743]: osdmap e439: 8 total, 8 up, 8 in 2024-02-10T07:52:53.852 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:53.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:53 smithi170 ceph-mon[33394]: pgmap v969: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 859 op/s; 9549/156930 objects degraded (6.085%); 7452/156930 objects misplaced (4.749%); 7.3 MiB/s, 116 objects/s recovering 2024-02-10T07:52:53.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:53 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:52:53.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:53 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:52:53.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:53 smithi170 ceph-mon[33394]: osdmap e439: 8 total, 8 up, 8 in 2024-02-10T07:52:53.894 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:53.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:53.996 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:54.044 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:54.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:54.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:54.146 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:54.177 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:54.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:54.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:54.278 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:54.326 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:54.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:54.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:52:53 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:52:54.427 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:54.470 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:54.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:54.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:54.572 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:54.616 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:54.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:54.718 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:54.760 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:54.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:54.862 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:54.892 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:54.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:54.994 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:55.024 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:55.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:55.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:55.125 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:55.158 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:55.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:55.260 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:55.286 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:55.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:55.292 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 7 1 2024-02-10T07:52:55.388 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:55.414 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:55.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:55.516 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:55.543 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:55.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:55.645 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:55.694 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:55.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:55.796 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:55.825 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:55.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:55 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:55 smithi170 ceph-mon[33394]: pgmap v971: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 789 op/s; 9549/150435 objects degraded (6.348%); 7452/150435 objects misplaced (4.954%); 4.4 MiB/s, 69 objects/s recovering 2024-02-10T07:52:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:55 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:52:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:55 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:52:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:55 smithi170 ceph-mon[33394]: osdmap e440: 8 total, 8 up, 8 in 2024-02-10T07:52:55.927 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:55.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:55.956 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:55.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:56.057 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:56.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:55 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:56.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:55 smithi008 ceph-mon[29816]: pgmap v971: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 789 op/s; 9549/150435 objects degraded (6.348%); 7452/150435 objects misplaced (4.954%); 4.4 MiB/s, 69 objects/s recovering 2024-02-10T07:52:56.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:55 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:52:56.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:55 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:52:56.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:55 smithi008 ceph-mon[29816]: osdmap e440: 8 total, 8 up, 8 in 2024-02-10T07:52:56.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:55 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:52:56.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:55 smithi008 ceph-mon[40743]: pgmap v971: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 789 op/s; 9549/150435 objects degraded (6.348%); 7452/150435 objects misplaced (4.954%); 4.4 MiB/s, 69 objects/s recovering 2024-02-10T07:52:56.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:55 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:52:56.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:55 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:52:56.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:55 smithi008 ceph-mon[40743]: osdmap e440: 8 total, 8 up, 8 in 2024-02-10T07:52:56.084 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:56.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:56.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:56.186 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:56.221 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:56.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:56.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:56.323 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:56.350 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:56.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:56.452 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:56.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:56.510 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:56.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:56.611 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:56.675 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:56.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:56.776 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:56.777 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:52:56.807 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:56.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:56.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:56.909 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:56.972 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:56.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:57.073 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:57.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:57.137 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:57.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:57.238 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:57.264 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:57.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:57.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:57.366 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:57.397 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:57.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:57.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:57.499 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:57.525 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:57.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:57.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:57.627 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:52:57.634 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:52:57.654 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:57.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:52:57.670 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:52:57.714 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:52:57.755 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:52:57.756 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:57.781 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:57.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:57.808 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:52:57.845 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:57 smithi008 ceph-mon[29816]: pgmap v973: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 774 B/s rd, 0 B/s wr, 801 op/s; 9681/146001 objects degraded (6.631%); 6698/146001 objects misplaced (4.588%); 11 MiB/s, 178 objects/s recovering 2024-02-10T07:52:57.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:57 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 9681/146001 objects degraded (6.631%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:57.846 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:57 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/942934665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:52:57.846 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:57 smithi008 ceph-mon[40743]: pgmap v973: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 774 B/s rd, 0 B/s wr, 801 op/s; 9681/146001 objects degraded (6.631%); 6698/146001 objects misplaced (4.588%); 11 MiB/s, 178 objects/s recovering 2024-02-10T07:52:57.846 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:57 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 9681/146001 objects degraded (6.631%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:57.846 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:57 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/942934665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:52:57.846 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:52:57.883 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:57.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:57 smithi170 ceph-mon[33394]: pgmap v973: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 774 B/s rd, 0 B/s wr, 801 op/s; 9681/146001 objects degraded (6.631%); 6698/146001 objects misplaced (4.588%); 11 MiB/s, 178 objects/s recovering 2024-02-10T07:52:57.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:57 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 9681/146001 objects degraded (6.631%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:52:57.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:57 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/942934665' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:52:57.884 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:52:57.924 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:52:57.957 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:57.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:57.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:57.972 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:52:57.980 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:52:57.980 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:52:57.980 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:52:57.980 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:52:57.980 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:52:57.980 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:52:57.980 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:52:57.980 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:52:58.059 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:58.091 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:58.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:58.145 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:52:58.192 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:58.220 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:58.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:58.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:58.322 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:58.366 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:58.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:58.468 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:58.495 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:58.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:58.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:58.597 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:58.624 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:58.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:58.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:58.726 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:58.758 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:58.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:58.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:58 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/944674605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:52:58.790 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:58 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:52:58.791 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:58 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/942934665' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:52:58.791 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:58 smithi008 ceph-mon[29816]: osdmap e441: 8 total, 8 up, 8 in 2024-02-10T07:52:58.791 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:52:58 smithi008 ceph-mon[29816]: pgmap v975: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 650 op/s; 9681/145230 objects degraded (6.666%); 6698/145230 objects misplaced (4.612%); 6.5 MiB/s, 103 objects/s recovering 2024-02-10T07:52:58.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:58 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/944674605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:52:58.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:58 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:52:58.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:58 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/942934665' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:52:58.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:58 smithi008 ceph-mon[40743]: osdmap e441: 8 total, 8 up, 8 in 2024-02-10T07:52:58.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:52:58 smithi008 ceph-mon[40743]: pgmap v975: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 650 op/s; 9681/145230 objects degraded (6.666%); 6698/145230 objects misplaced (4.612%); 6.5 MiB/s, 103 objects/s recovering 2024-02-10T07:52:58.860 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:52:58.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:58 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/944674605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:52:58.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:58 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:52:58.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:58 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/942934665' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:52:58.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:58 smithi170 ceph-mon[33394]: osdmap e441: 8 total, 8 up, 8 in 2024-02-10T07:52:58.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:52:58 smithi170 ceph-mon[33394]: pgmap v975: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 650 op/s; 9681/145230 objects degraded (6.666%); 6698/145230 objects misplaced (4.612%); 6.5 MiB/s, 103 objects/s recovering 2024-02-10T07:52:58.907 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:58.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:58.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:52:59.008 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:52:59.036 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:59.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:52:59.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:52:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:52:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:52:59.138 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:52:59.183 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:59.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:59.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:52:59.285 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:52:59.330 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:59.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:59.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:52:59.432 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:52:59.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:59.460 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:59.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:52:59.561 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:52:59.591 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:52:59.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:59.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:52:59.643 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:52:59.643 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:52:59.693 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:59.733 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:59.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:59.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:59.835 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:52:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:52:59.877 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:52:59.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:52:59.977 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:00.009 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:00.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:00.111 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:00.139 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:00.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:00.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:00.241 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:00.277 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:00.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:00 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:53:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:00 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:53:00.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:00 smithi008 ceph-mon[29816]: osdmap e442: 8 total, 8 up, 8 in 2024-02-10T07:53:00.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:00 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:53:00.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:00 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:53:00.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:00 smithi008 ceph-mon[40743]: osdmap e442: 8 total, 8 up, 8 in 2024-02-10T07:53:00.378 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:00.409 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:00.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:00.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:00.511 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:00.538 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:00.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:00.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:00 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:53:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:00 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:53:00.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:00 smithi170 ceph-mon[33394]: osdmap e442: 8 total, 8 up, 8 in 2024-02-10T07:53:00.639 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:00.676 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:00.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:00.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:00.778 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:00.818 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:00.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:00.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:00.920 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:00.953 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:00.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:00.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:01.055 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:01.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:01.106 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:01.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:01.207 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:01.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:01.237 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:01.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:01.338 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:01.386 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:01.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:01.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:01.488 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:01.538 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:01.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:01.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:01 smithi008 ceph-mon[29816]: pgmap v977: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 820 op/s; 9782/135669 objects degraded (7.210%); 6436/135669 objects misplaced (4.744%); 8.2 MiB/s, 130 objects/s recovering 2024-02-10T07:53:01.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:01 smithi008 ceph-mon[40743]: pgmap v977: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 820 op/s; 9782/135669 objects degraded (7.210%); 6436/135669 objects misplaced (4.744%); 8.2 MiB/s, 130 objects/s recovering 2024-02-10T07:53:01.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:01 smithi170 ceph-mon[33394]: pgmap v977: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 820 op/s; 9782/135669 objects degraded (7.210%); 6436/135669 objects misplaced (4.744%); 8.2 MiB/s, 130 objects/s recovering 2024-02-10T07:53:01.639 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:01.667 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:01.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:01.769 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:01.800 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:01.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:01.902 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:01.929 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:01.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:01.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:01.943 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-10T07:53:02.030 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:02.099 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:02.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:02.160 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:53:02.200 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:02.240 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:02.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:02.341 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:02.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:02.419 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:02.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:02.504 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:02 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 9782/135669 objects degraded (7.210%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:02.504 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:02 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 9782/135669 objects degraded (7.210%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:02.523 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:02.553 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:02.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:02.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:02.571 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:53:02.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:02 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 9782/135669 objects degraded (7.210%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:02.655 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:02.683 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:02.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:02.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:02.785 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:02.813 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:02.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:02.914 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:02.940 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:02.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:02.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:03.041 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:03.232 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:03.303 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:03.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:03.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:03 smithi008 ceph-mon[29816]: pgmap v978: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 675 B/s rd, 0 B/s wr, 747 op/s; 9832/133452 objects degraded (7.367%); 5911/133452 objects misplaced (4.429%); 10 MiB/s, 166 objects/s recovering 2024-02-10T07:53:03.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:03 smithi008 ceph-mon[40743]: pgmap v978: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 675 B/s rd, 0 B/s wr, 747 op/s; 9832/133452 objects degraded (7.367%); 5911/133452 objects misplaced (4.429%); 10 MiB/s, 166 objects/s recovering 2024-02-10T07:53:03.405 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:03.463 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:03.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:03.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:03.566 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:03.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:03.617 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:03.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:03.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:03 smithi170 ceph-mon[33394]: pgmap v978: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 675 B/s rd, 0 B/s wr, 747 op/s; 9832/133452 objects degraded (7.367%); 5911/133452 objects misplaced (4.429%); 10 MiB/s, 166 objects/s recovering 2024-02-10T07:53:03.717 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:03.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:03.772 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:03.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:03.873 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:03.920 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:03.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:04.022 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:04.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:04.062 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:04.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:04.163 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:04.199 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.7 to 1 (10000) 2024-02-10T07:53:04.214 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:04.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:04.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:04.319 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:04.348 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:04.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:04.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:04.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:53:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:53:04.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:53:04.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:04 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1931131659' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:53:04.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:04 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:53:04.450 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:04.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:04.482 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:04.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:04.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:53:04.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:04 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1931131659' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:53:04.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:04 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:53:04.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:53:04.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:04 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1931131659' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:53:04.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:04 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:53:04.583 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:04.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:04.616 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:04.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:04.717 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:04.749 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:04.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:04.850 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:04.882 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:04.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:04.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:04.984 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:05.015 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:05.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:05.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:05.117 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:05.160 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:05.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:05.261 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:05.289 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:05.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:05.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:05.391 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:05.424 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:05.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:05.526 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:05.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:05.560 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:05.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:05.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:05 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:05.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:05 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:53:05.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:05 smithi008 ceph-mon[29816]: pgmap v980: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.03k op/s; 9832/126558 objects degraded (7.769%); 5911/126558 objects misplaced (4.671%); 6.6 MiB/s, 105 objects/s recovering 2024-02-10T07:53:05.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:05 smithi008 ceph-mon[29816]: osdmap e443: 8 total, 8 up, 8 in 2024-02-10T07:53:05.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:05 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:05.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:05 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:53:05.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:05 smithi008 ceph-mon[40743]: pgmap v980: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.03k op/s; 9832/126558 objects degraded (7.769%); 5911/126558 objects misplaced (4.671%); 6.6 MiB/s, 105 objects/s recovering 2024-02-10T07:53:05.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:05 smithi008 ceph-mon[40743]: osdmap e443: 8 total, 8 up, 8 in 2024-02-10T07:53:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:05 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:05 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:53:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:05 smithi170 ceph-mon[33394]: pgmap v980: 148 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.03k op/s; 9832/126558 objects degraded (7.769%); 5911/126558 objects misplaced (4.671%); 6.6 MiB/s, 105 objects/s recovering 2024-02-10T07:53:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:05 smithi170 ceph-mon[33394]: osdmap e443: 8 total, 8 up, 8 in 2024-02-10T07:53:05.661 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:05.702 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:05.717 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:05.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:05.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:05.758 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 2 1 2024-02-10T07:53:05.776 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:05.819 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:05.847 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:05.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:05.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:05.948 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:06.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:06.022 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:06.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:06.123 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:06.152 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:06.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:06.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:06.253 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:06.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:06.307 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:06.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:06.408 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:06.437 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:06.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:06.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:06.539 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:06.573 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:06.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:06.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:06.675 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:06.702 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:06.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:06.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:06.804 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:06.829 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:06.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:06.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:06.931 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:06.956 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:06.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:06.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:07.058 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:07.082 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:07.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:07.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:07.183 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:07.211 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:07.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:07.312 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:07.340 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:07.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:07.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:07.442 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:07.461 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:07 smithi008 ceph-mon[29816]: pgmap v981: 148 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 972 op/s; 9939/121902 objects degraded (8.153%); 5096/121902 objects misplaced (4.180%); 10 MiB/s, 167 objects/s recovering 2024-02-10T07:53:07.461 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:07 smithi008 ceph-mon[29816]: osdmap e444: 8 total, 8 up, 8 in 2024-02-10T07:53:07.461 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:07 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 9939/121902 objects degraded (8.153%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:07.462 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:07 smithi008 ceph-mon[40743]: pgmap v981: 148 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 972 op/s; 9939/121902 objects degraded (8.153%); 5096/121902 objects misplaced (4.180%); 10 MiB/s, 167 objects/s recovering 2024-02-10T07:53:07.462 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:07 smithi008 ceph-mon[40743]: osdmap e444: 8 total, 8 up, 8 in 2024-02-10T07:53:07.462 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:07 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 9939/121902 objects degraded (8.153%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:07.487 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:07.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:07.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:07.588 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:07.613 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:07.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:07.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:07 smithi170 ceph-mon[33394]: pgmap v981: 148 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 972 op/s; 9939/121902 objects degraded (8.153%); 5096/121902 objects misplaced (4.180%); 10 MiB/s, 167 objects/s recovering 2024-02-10T07:53:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:07 smithi170 ceph-mon[33394]: osdmap e444: 8 total, 8 up, 8 in 2024-02-10T07:53:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:07 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 9939/121902 objects degraded (8.153%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:07.715 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:07.758 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:07.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:07.860 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:07.890 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:07.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:07.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:07.992 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:08.022 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:08.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:08.124 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:08.163 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:08.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:08.264 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:08.306 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:08.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:08.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:08.408 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:08.437 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:08.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:08.538 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:08.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:08 smithi008 ceph-mon[29816]: osdmap e445: 8 total, 8 up, 8 in 2024-02-10T07:53:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:08 smithi008 ceph-mon[40743]: osdmap e445: 8 total, 8 up, 8 in 2024-02-10T07:53:08.582 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:08.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:08.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:08.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:08 smithi170 ceph-mon[33394]: osdmap e445: 8 total, 8 up, 8 in 2024-02-10T07:53:08.684 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:08.720 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:08.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:08.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:08.822 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:08.860 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:08.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:08.962 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:08.994 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:08.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:08.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:09.084 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:53:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:53:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:53:09.096 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:09.135 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:09.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:09.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:09.236 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:09.267 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:09.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:09.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:09.368 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:09.396 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:09.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:09.498 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:09.527 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:09.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:09.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:09 smithi008 ceph-mon[29816]: pgmap v984: 148 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 676 op/s; 9939/121293 objects degraded (8.194%); 5096/121293 objects misplaced (4.201%); 7.4 MiB/s, 117 objects/s recovering 2024-02-10T07:53:09.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:09 smithi008 ceph-mon[40743]: pgmap v984: 148 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 676 op/s; 9939/121293 objects degraded (8.194%); 5096/121293 objects misplaced (4.201%); 7.4 MiB/s, 117 objects/s recovering 2024-02-10T07:53:09.628 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:09 smithi170 ceph-mon[33394]: pgmap v984: 148 pgs: 1 active+recovery_wait+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 142 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 676 op/s; 9939/121293 objects degraded (8.194%); 5096/121293 objects misplaced (4.201%); 7.4 MiB/s, 117 objects/s recovering 2024-02-10T07:53:09.658 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:09.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:09.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:09.760 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:09.793 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:09.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:09.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:09.895 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:09.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:09.922 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:09.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:10.023 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:10.065 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:10.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:10.168 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:10.197 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:10.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:10.299 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:10.341 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:10.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:10.443 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:10.470 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:10.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:10.572 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:10.608 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:10.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:10.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:10.710 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:10.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:10.761 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:10.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:10.862 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:10.889 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:10.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:10.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:10.991 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:11.024 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:11.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:11.026 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:53:11.026 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:53:11.126 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:11.164 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:11.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:11.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:11.266 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:11.307 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:11.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:11.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:11.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:11 smithi008 ceph-mon[29816]: pgmap v985: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 776 op/s; 9649/112587 objects degraded (8.570%); 4942/112587 objects misplaced (4.389%); 12 MiB/s, 191 objects/s recovering 2024-02-10T07:53:11.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:11 smithi008 ceph-mon[40743]: pgmap v985: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 776 op/s; 9649/112587 objects degraded (8.570%); 4942/112587 objects misplaced (4.389%); 12 MiB/s, 191 objects/s recovering 2024-02-10T07:53:11.409 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:11.445 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:11.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:11.547 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:11.586 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:11.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:11.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:11 smithi170 ceph-mon[33394]: pgmap v985: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 776 op/s; 9649/112587 objects degraded (8.570%); 4942/112587 objects misplaced (4.389%); 12 MiB/s, 191 objects/s recovering 2024-02-10T07:53:11.687 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:11.720 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:11.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:11.822 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:11.857 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:11.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:11.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:11.959 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:12.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:12.028 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:12.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:12.129 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:12.181 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:12.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:12.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:12.283 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:12.315 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:12.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:12.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:12.417 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:12.448 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:12.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:12.550 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:12.558 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:12 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 9649/112587 objects degraded (8.570%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:12.558 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:12 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 9649/112587 objects degraded (8.570%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:12.606 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:12.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:12.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:12.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:12 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 9649/112587 objects degraded (8.570%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:12.708 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:12.745 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:12.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:12.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:12.847 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:12.877 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:12.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:12.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:12.979 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:13.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:13.009 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:13.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:13.110 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:13.136 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:13.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:13.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:13.238 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:13.271 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:13.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:13.373 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:13.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:13.423 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:13.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:13.524 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:13.559 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:13.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:13.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:13 smithi008 ceph-mon[29816]: pgmap v986: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 673 op/s; 9579/110415 objects degraded (8.675%); 4593/110415 objects misplaced (4.160%); 12 MiB/s, 196 objects/s recovering 2024-02-10T07:53:13.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:13 smithi008 ceph-mon[40743]: pgmap v986: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 673 op/s; 9579/110415 objects degraded (8.675%); 4593/110415 objects misplaced (4.160%); 12 MiB/s, 196 objects/s recovering 2024-02-10T07:53:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:13 smithi170 ceph-mon[33394]: pgmap v986: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 673 op/s; 9579/110415 objects degraded (8.675%); 4593/110415 objects misplaced (4.160%); 12 MiB/s, 196 objects/s recovering 2024-02-10T07:53:13.661 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:13.704 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:13.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:13.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:13.806 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:13.838 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:13.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:13.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:13.939 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:13.988 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:13.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:13.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:14.028 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-10T07:53:14.090 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:14.156 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:14.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:14.257 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:14.287 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:14.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:14.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:53:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:53:14.389 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:14.416 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:14.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:14.518 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:14.561 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:14.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:14.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:14.663 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:14.703 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:14.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:14.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:14.817 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:14.865 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:14.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:14.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:14.967 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:15.012 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:15.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:15.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:15.114 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:15.119 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:53:15.119 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:53:15.119 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:53:15.153 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:15.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:15.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:15.165 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:53:15.210 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:53:15.215 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:15.255 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:15.266 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:53:15.282 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:15.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:15.328 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:53:15.375 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:53:15.379 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:53:15.380 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:53:15.380 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:53:15.380 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:53:15.380 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:53:15.380 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:53:15.380 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:53:15.380 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:53:15.384 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:15.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:15.437 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:15.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:15.538 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:15.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:15 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:15.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:15 smithi008 ceph-mon[29816]: pgmap v987: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 793 op/s; 9598/102882 objects degraded (9.329%); 4427/102882 objects misplaced (4.303%); 7.1 MiB/s, 126 objects/s recovering 2024-02-10T07:53:15.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:15 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1056490668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:53:15.562 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:15 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:15.562 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:15 smithi008 ceph-mon[40743]: pgmap v987: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 793 op/s; 9598/102882 objects degraded (9.329%); 4427/102882 objects misplaced (4.303%); 7.1 MiB/s, 126 objects/s recovering 2024-02-10T07:53:15.562 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:15 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1056490668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:53:15.577 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:15.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:15 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:15 smithi170 ceph-mon[33394]: pgmap v987: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 793 op/s; 9598/102882 objects degraded (9.329%); 4427/102882 objects misplaced (4.303%); 7.1 MiB/s, 126 objects/s recovering 2024-02-10T07:53:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:15 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1056490668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:53:15.679 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:15.705 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:15.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:15.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:15.807 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:15.834 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:15.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:15.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:15.936 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:15.961 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:15.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:15.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:16.063 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:16.089 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:16.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:16.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:16.191 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:16.218 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:16.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:16.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:16.290 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:53:16.319 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:16.345 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:16.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:16.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:16.447 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:16.491 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:16.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:16.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:16 smithi008 ceph-mon[29816]: osdmap e446: 8 total, 8 up, 8 in 2024-02-10T07:53:16.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:16 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1352998103' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:53:16.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:16 smithi008 ceph-mon[40743]: osdmap e446: 8 total, 8 up, 8 in 2024-02-10T07:53:16.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:16 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1352998103' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:53:16.593 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:16.632 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:16 smithi170 ceph-mon[33394]: osdmap e446: 8 total, 8 up, 8 in 2024-02-10T07:53:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:16 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1352998103' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:53:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:16.634 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:16.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:16.735 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:16.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:16.768 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:16.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:16.869 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:16.908 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:16.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:16.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:17.010 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:17.044 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:17.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:17.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:17.146 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:17.170 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:17.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:17.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:17.272 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:17.317 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:17.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:17.419 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:17.461 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:17.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:17.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:17.563 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:17.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:17 smithi008 ceph-mon[29816]: pgmap v989: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 832 op/s; 9418/98910 objects degraded (9.522%); 4420/98910 objects misplaced (4.469%); 6.4 MiB/s, 133 objects/s recovering 2024-02-10T07:53:17.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:17 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:53:17.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:17 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1352998103' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:53:17.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:17 smithi008 ceph-mon[29816]: osdmap e447: 8 total, 8 up, 8 in 2024-02-10T07:53:17.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:17 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 9418/98910 objects degraded (9.522%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:17 smithi008 ceph-mon[40743]: pgmap v989: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 832 op/s; 9418/98910 objects degraded (9.522%); 4420/98910 objects misplaced (4.469%); 6.4 MiB/s, 133 objects/s recovering 2024-02-10T07:53:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:17 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:53:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:17 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1352998103' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:53:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:17 smithi008 ceph-mon[40743]: osdmap e447: 8 total, 8 up, 8 in 2024-02-10T07:53:17.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:17 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 9418/98910 objects degraded (9.522%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:17.609 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:17.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:17 smithi170 ceph-mon[33394]: pgmap v989: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 832 op/s; 9418/98910 objects degraded (9.522%); 4420/98910 objects misplaced (4.469%); 6.4 MiB/s, 133 objects/s recovering 2024-02-10T07:53:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:17 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:53:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:17 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1352998103' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:53:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:17 smithi170 ceph-mon[33394]: osdmap e447: 8 total, 8 up, 8 in 2024-02-10T07:53:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:17 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 9418/98910 objects degraded (9.522%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:17.710 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:17.741 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:17.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:17.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:17.843 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:17.874 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:17.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:17.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:17.976 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:18.018 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:18.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:18.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:18.120 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:18.151 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:18.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:18.253 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:18.280 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:18.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:18.382 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:18.416 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:18.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:18.517 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:18.549 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:18.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:18.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:18.650 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:18.684 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:18.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:18.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:18.785 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:18.826 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:18.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:18.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:18.928 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:18.969 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:18.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:19.071 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:19.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:53:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:53:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:53:19.098 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:19.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:19.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:19.199 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:19.236 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:19.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:19.338 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:19.371 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:19.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:19.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:19.472 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:19.521 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:19.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:19 smithi008 ceph-mon[29816]: pgmap v991: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 598 op/s; 9418/98238 objects degraded (9.587%); 4420/98238 objects misplaced (4.499%); 3.7 MiB/s, 94 objects/s recovering 2024-02-10T07:53:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:19 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:53:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:19 smithi008 ceph-mon[40743]: pgmap v991: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 598 op/s; 9418/98238 objects degraded (9.587%); 4420/98238 objects misplaced (4.499%); 3.7 MiB/s, 94 objects/s recovering 2024-02-10T07:53:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:19 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:53:19.622 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:19 smithi170 ceph-mon[33394]: pgmap v991: 148 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 598 op/s; 9418/98238 objects degraded (9.587%); 4420/98238 objects misplaced (4.499%); 3.7 MiB/s, 94 objects/s recovering 2024-02-10T07:53:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:19 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:53:19.655 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:19.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:19.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:19.757 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:19.798 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:19.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:19.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:19.899 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:19.929 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:19.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:19.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:20.031 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:20.065 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:20.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:20.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:20.167 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:20.197 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:20.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:20.299 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:20.332 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:20.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:20.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:20.433 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:20.463 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:20.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:20.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:20.565 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:20.602 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:20.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:20.704 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:20.739 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:20.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:20.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:20.766 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:20.842 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:20.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:20.892 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:20.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:20.993 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:21.029 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:21.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:21.130 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:21.161 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:21.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:21.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:21.263 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:21.293 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:21.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:21.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:21.395 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:21.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:21.432 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:21.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:21.533 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:21.565 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:21.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:21.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:21 smithi008 ceph-mon[29816]: 5.13 scrub starts 2024-02-10T07:53:21.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:21 smithi008 ceph-mon[29816]: 5.13 scrub ok 2024-02-10T07:53:21.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:21 smithi008 ceph-mon[29816]: pgmap v992: 148 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 942 op/s; 8352/87801 objects degraded (9.512%); 4420/87801 objects misplaced (5.034%); 991 KiB/s, 174 objects/s recovering 2024-02-10T07:53:21.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:21 smithi008 ceph-mon[40743]: 5.13 scrub starts 2024-02-10T07:53:21.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:21 smithi008 ceph-mon[40743]: 5.13 scrub ok 2024-02-10T07:53:21.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:21 smithi008 ceph-mon[40743]: pgmap v992: 148 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 942 op/s; 8352/87801 objects degraded (9.512%); 4420/87801 objects misplaced (5.034%); 991 KiB/s, 174 objects/s recovering 2024-02-10T07:53:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:21 smithi170 ceph-mon[33394]: 5.13 scrub starts 2024-02-10T07:53:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:21 smithi170 ceph-mon[33394]: 5.13 scrub ok 2024-02-10T07:53:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:21 smithi170 ceph-mon[33394]: pgmap v992: 148 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 942 op/s; 8352/87801 objects degraded (9.512%); 4420/87801 objects misplaced (5.034%); 991 KiB/s, 174 objects/s recovering 2024-02-10T07:53:21.667 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:21.704 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:21.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:21.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:21.806 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:21.834 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:21.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:21.936 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:21.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:21.976 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:21.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:22.077 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:22.109 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:22.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:22.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:22.211 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:22.241 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:22.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:22.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:22.343 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:22.395 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:22.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:22.413 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:53:22.498 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:22.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:22.540 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:22.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:22.549 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:22 smithi008 ceph-mon[29816]: 3.b scrub starts 2024-02-10T07:53:22.549 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:22 smithi008 ceph-mon[29816]: 3.b scrub ok 2024-02-10T07:53:22.549 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:22 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 8352/87801 objects degraded (9.512%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:22.549 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:22 smithi008 ceph-mon[40743]: 3.b scrub starts 2024-02-10T07:53:22.550 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:22 smithi008 ceph-mon[40743]: 3.b scrub ok 2024-02-10T07:53:22.550 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:22 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 8352/87801 objects degraded (9.512%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:22 smithi170 ceph-mon[33394]: 3.b scrub starts 2024-02-10T07:53:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:22 smithi170 ceph-mon[33394]: 3.b scrub ok 2024-02-10T07:53:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:22 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 8352/87801 objects degraded (9.512%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:22.640 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:22.669 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:22.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:22.771 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:22.799 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:22.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:22.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:22.901 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:22.941 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:22.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:22.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:23.043 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:23.074 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:23.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:23.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:23.176 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:23.203 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:23.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:23.305 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:23.316 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.2 to 1 (10000) 2024-02-10T07:53:23.338 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:23.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:23.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:23.439 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:23.481 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:23.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:23.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:23 smithi008 ceph-mon[29816]: 3.18 scrub starts 2024-02-10T07:53:23.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:23 smithi008 ceph-mon[29816]: 3.18 scrub ok 2024-02-10T07:53:23.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:23 smithi008 ceph-mon[29816]: pgmap v993: 148 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 693 op/s; 8352/86238 objects degraded (9.685%); 4420/86238 objects misplaced (5.125%); 56 KiB/s, 156 objects/s recovering 2024-02-10T07:53:23.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:23 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2091186604' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:53:23.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:23 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:53:23.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:23 smithi008 ceph-mon[40743]: 3.18 scrub starts 2024-02-10T07:53:23.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:23 smithi008 ceph-mon[40743]: 3.18 scrub ok 2024-02-10T07:53:23.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:23 smithi008 ceph-mon[40743]: pgmap v993: 148 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 693 op/s; 8352/86238 objects degraded (9.685%); 4420/86238 objects misplaced (5.125%); 56 KiB/s, 156 objects/s recovering 2024-02-10T07:53:23.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:23 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2091186604' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:53:23.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:23 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:53:23.583 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:23.619 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:23.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:23 smithi170 ceph-mon[33394]: 3.18 scrub starts 2024-02-10T07:53:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:23 smithi170 ceph-mon[33394]: 3.18 scrub ok 2024-02-10T07:53:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:23 smithi170 ceph-mon[33394]: pgmap v993: 148 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 693 op/s; 8352/86238 objects degraded (9.685%); 4420/86238 objects misplaced (5.125%); 56 KiB/s, 156 objects/s recovering 2024-02-10T07:53:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:23 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2091186604' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:53:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:23 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:53:23.721 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:23.759 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:23.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:23.798 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:53:23.860 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:23.907 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:23.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:23.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:24.009 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:24.080 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:24.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:24.182 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:24.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:24.221 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:24.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:24.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:53:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:53:24.322 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:24.353 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:24.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:24.455 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:24.484 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:24.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:24.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:24.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:24 smithi008 ceph-mon[29816]: 5.1b scrub starts 2024-02-10T07:53:24.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:24 smithi008 ceph-mon[29816]: 5.1b scrub ok 2024-02-10T07:53:24.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:24 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:53:24.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:24 smithi008 ceph-mon[29816]: osdmap e448: 8 total, 8 up, 8 in 2024-02-10T07:53:24.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:24 smithi008 ceph-mon[40743]: 5.1b scrub starts 2024-02-10T07:53:24.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:24 smithi008 ceph-mon[40743]: 5.1b scrub ok 2024-02-10T07:53:24.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:24 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:53:24.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:24 smithi008 ceph-mon[40743]: osdmap e448: 8 total, 8 up, 8 in 2024-02-10T07:53:24.585 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:24.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:24.619 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:24.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:24 smithi170 ceph-mon[33394]: 5.1b scrub starts 2024-02-10T07:53:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:24 smithi170 ceph-mon[33394]: 5.1b scrub ok 2024-02-10T07:53:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:24 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:53:24.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:24 smithi170 ceph-mon[33394]: osdmap e448: 8 total, 8 up, 8 in 2024-02-10T07:53:24.721 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:24.748 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:24.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:24.850 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:24.874 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:24.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:24.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:24.976 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:25.000 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:25.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:25.102 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:25.127 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:25.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:25.230 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-10T07:53:25.230 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:53:25.230 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:25.255 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:25.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:25.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:25.356 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:25.386 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:25.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:25.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:25.489 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:25.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:25.521 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:25.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:25.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:25 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:25.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:25 smithi008 ceph-mon[29816]: pgmap v995: 148 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 866 op/s; 8029/78105 objects degraded (10.280%); 4420/78105 objects misplaced (5.659%); 0 B/s, 173 objects/s recovering 2024-02-10T07:53:25.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:25 smithi008 ceph-mon[29816]: osdmap e449: 8 total, 8 up, 8 in 2024-02-10T07:53:25.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:25 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:25.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:25 smithi008 ceph-mon[40743]: pgmap v995: 148 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 866 op/s; 8029/78105 objects degraded (10.280%); 4420/78105 objects misplaced (5.659%); 0 B/s, 173 objects/s recovering 2024-02-10T07:53:25.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:25 smithi008 ceph-mon[40743]: osdmap e449: 8 total, 8 up, 8 in 2024-02-10T07:53:25.622 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:25 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:25 smithi170 ceph-mon[33394]: pgmap v995: 148 pgs: 1 active+recovery_wait+remapped, 3 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 866 op/s; 8029/78105 objects degraded (10.280%); 4420/78105 objects misplaced (5.659%); 0 B/s, 173 objects/s recovering 2024-02-10T07:53:25.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:25 smithi170 ceph-mon[33394]: osdmap e449: 8 total, 8 up, 8 in 2024-02-10T07:53:25.651 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:25.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:25.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:25.752 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:25.780 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:25.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:25.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:25.882 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:25.921 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:25.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:25.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:26.023 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:26.070 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:26.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:26.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:26.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:26.208 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:26.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:26.309 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:26.334 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:26.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:26.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:26.435 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:26.467 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:26.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:26.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:26.568 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:26.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:26 smithi008 ceph-mon[29816]: osdmap e450: 8 total, 8 up, 8 in 2024-02-10T07:53:26.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:26 smithi008 ceph-mon[40743]: osdmap e450: 8 total, 8 up, 8 in 2024-02-10T07:53:26.597 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:26.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:26 smithi170 ceph-mon[33394]: osdmap e450: 8 total, 8 up, 8 in 2024-02-10T07:53:26.699 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:26.739 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:26.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:26.771 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:53:26.771 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [ 2024-02-10T07:53:26.771 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.771 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:186359 11.0 11:0a38c1a2:::benchmark_data_smithi008_186804_object81597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-02-10T07:53:26.771 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:53:26.242592+0000", 2024-02-10T07:53:26.771 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.063273974999999996, 2024-02-10T07:53:26.771 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.063304526999999999, 2024-02-10T07:53:26.771 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:53:26.771 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:53:26.771 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 186359 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242592+0000", 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242592+0000", 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:53:26.772 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242593+0000", 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0750000000000001e-06 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242597+0000", 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.6030000000000001e-06 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242605+0000", 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.0209999999999999e-06 2024-02-10T07:53:26.773 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.774 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.774 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:53:26.774 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242611+0000", 2024-02-10T07:53:26.774 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.1720000000000004e-06 2024-02-10T07:53:26.774 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.774 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.774 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:53:26.774 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.245450+0000", 2024-02-10T07:53:26.774 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028391319999999999 2024-02-10T07:53:26.774 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.774 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.774 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:53:26.774 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.245598+0000", 2024-02-10T07:53:26.774 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014804999999999999 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 3,7", 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.245719+0000", 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.000121315 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:186360 11.d 11.5945d51d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:53:26.242622+0000", 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.063243595, 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.063325401000000003, 2024-02-10T07:53:26.775 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:53:26.776 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:53:26.776 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:53:26.776 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:53:26.776 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:53:26.776 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 186360 2024-02-10T07:53:26.776 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.776 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:53:26.776 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.776 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:53:26.776 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242622+0000", 2024-02-10T07:53:26.776 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:53:26.776 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.776 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.777 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:53:26.777 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242622+0000", 2024-02-10T07:53:26.777 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:53:26.777 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.777 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.777 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:53:26.777 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242623+0000", 2024-02-10T07:53:26.777 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.9799999999999999e-07 2024-02-10T07:53:26.777 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.777 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.777 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:53:26.777 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242625+0000", 2024-02-10T07:53:26.777 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.04e-06 2024-02-10T07:53:26.777 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242631+0000", 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.8499999999999999e-06 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242636+0000", 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.7589999999999997e-06 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.778 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:186361 11.c 11:3a40efd3:::benchmark_data_smithi008_186804_object81599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:53:26.242649+0000", 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.063217465, 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.063334693999999997, 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "waiting for sub ops", 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 186361 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242649+0000", 2024-02-10T07:53:26.779 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:53:26.780 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.780 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.780 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:53:26.780 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242649+0000", 2024-02-10T07:53:26.780 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:53:26.780 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.780 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.780 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:53:26.780 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242649+0000", 2024-02-10T07:53:26.780 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 3.8599999999999999e-07 2024-02-10T07:53:26.780 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.780 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.780 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:53:26.780 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242651+0000", 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0590000000000001e-06 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242657+0000", 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.84e-06 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242662+0000", 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.9559999999999996e-06 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:53:26.781 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242708+0000", 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.6516e-05 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242851+0000", 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00014295900000000001 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "waiting for subops from 2,4", 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.242988+0000", 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00013638999999999999 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "op_commit", 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.245884+0000", 2024-02-10T07:53:26.782 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.0028963460000000002 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "sub_op_commit_rec", 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.246254+0000", 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00036951600000000002 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:186365 11.d 11.4771165d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:53:26.243024+0000", 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.062842384000000001, 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.063008423999999993, 2024-02-10T07:53:26.783 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:53:26.784 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:53:26.784 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:53:26.784 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:53:26.784 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:53:26.784 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 186365 2024-02-10T07:53:26.784 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.784 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:53:26.784 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.784 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:53:26.784 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.243024+0000", 2024-02-10T07:53:26.784 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:53:26.784 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.784 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.784 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.243024+0000", 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.243025+0000", 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.8200000000000008e-07 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.243029+0000", 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.4519999999999999e-06 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.243040+0000", 2024-02-10T07:53:26.785 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1045e-05 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.243048+0000", 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.5610000000000001e-06 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:186368 11.7 11:eb1f2d55:::benchmark_data_smithi008_186804_object81606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:53:26.243736+0000", 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.062130157999999998, 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.062330301999999997, 2024-02-10T07:53:26.786 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 186368 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.243736+0000", 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:53:26.787 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.243736+0000", 2024-02-10T07:53:26.788 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:53:26.788 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.788 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.788 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:53:26.788 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.243737+0000", 2024-02-10T07:53:26.788 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 9.4300000000000001e-07 2024-02-10T07:53:26.788 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.788 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.788 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:53:26.788 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.243741+0000", 2024-02-10T07:53:26.788 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 4.2810000000000002e-06 2024-02-10T07:53:26.788 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.788 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.788 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.243752+0000", 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.0706e-05 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.243759+0000", 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.357e-06 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.246554+0000", 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00279489 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:53:26.789 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.246660+0000", 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00010624900000000001 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_repop(client.25838.0:186364 11.3 e450/391 11:cce1d132:::benchmark_data_smithi008_186804_object81602:head v 450'11845, mlcod=450'11843)", 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:53:26.244096+0000", 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.061770348000000003, 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.062018641999999999, 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "started", 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.790 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:53:26.791 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.244096+0000", 2024-02-10T07:53:26.791 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:53:26.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.791 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.791 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:53:26.791 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.244096+0000", 2024-02-10T07:53:26.791 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:53:26.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.791 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.791 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:53:26.791 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.244097+0000", 2024-02-10T07:53:26.791 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 8.5000000000000001e-07 2024-02-10T07:53:26.791 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.791 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.244103+0000", 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 6.5069999999999997e-06 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.244114+0000", 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1041e-05 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.244121+0000", 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.3329999999999999e-06 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.792 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "reached_pg", 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.244698+0000", 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.00057656899999999997 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "started", 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.244719+0000", 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 2.0772e-05 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: "description": "osd_op(client.25838.0:186369 11.0 11.70a08340 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e450)", 2024-02-10T07:53:26.793 INFO:teuthology.orchestra.run.smithi008.stdout: "initiated_at": "2024-02-10T07:53:26.245903+0000", 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: "age": 0.059963049999999997, 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0.060245951999999998, 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: "type_data": { 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: "flag_point": "queued for pg", 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: "client_info": { 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: "client": "client.25838", 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: "client_addr": "172.21.15.8:0/829963222", 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: "tid": 186369 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: "events": [ 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "initiated", 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.245903+0000", 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:53:26.794 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "header_read", 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.245903+0000", 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 0 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "throttled", 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.245904+0000", 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.4300000000000001e-06 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "all_read", 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.245910+0000", 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 5.7049999999999996e-06 2024-02-10T07:53:26.795 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "dispatched", 2024-02-10T07:53:26.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.245922+0000", 2024-02-10T07:53:26.796 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 1.1494999999999999e-05 2024-02-10T07:53:26.796 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2024-02-10T07:53:26.796 INFO:teuthology.orchestra.run.smithi008.stdout: { 2024-02-10T07:53:26.796 INFO:teuthology.orchestra.run.smithi008.stdout: "event": "queued_for_pg", 2024-02-10T07:53:26.796 INFO:teuthology.orchestra.run.smithi008.stdout: "time": "2024-02-10T07:53:26.245930+0000", 2024-02-10T07:53:26.796 INFO:teuthology.orchestra.run.smithi008.stdout: "duration": 7.9470000000000003e-06 2024-02-10T07:53:26.796 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.796 INFO:teuthology.orchestra.run.smithi008.stdout: ] 2024-02-10T07:53:26.796 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.796 INFO:teuthology.orchestra.run.smithi008.stdout: } 2024-02-10T07:53:26.796 INFO:teuthology.orchestra.run.smithi008.stdout: ], 2024-02-10T07:53:26.797 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 7 2024-02-10T07:53:26.797 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:53:26.840 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:26.874 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:26.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:26.976 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:27.012 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:27.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:27.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:27.065 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:27.114 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:27.146 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:27.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:27.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:27.248 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:27.296 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:27.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:27.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:27.398 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:27.430 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:27.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:27.476 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:27 smithi008 ceph-mon[29816]: 6.1b scrub starts 2024-02-10T07:53:27.477 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:27 smithi008 ceph-mon[29816]: pgmap v998: 148 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 738 op/s; 7859/74511 objects degraded (10.547%); 4357/74511 objects misplaced (5.847%); 0 B/s, 92 objects/s recovering 2024-02-10T07:53:27.477 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:27 smithi008 ceph-mon[29816]: 6.1b scrub ok 2024-02-10T07:53:27.477 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:27 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:53:27.477 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:27 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:53:27.477 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:27 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 7859/74511 objects degraded (10.547%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:27.477 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:27 smithi008 ceph-mon[40743]: 6.1b scrub starts 2024-02-10T07:53:27.477 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:27 smithi008 ceph-mon[40743]: pgmap v998: 148 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 738 op/s; 7859/74511 objects degraded (10.547%); 4357/74511 objects misplaced (5.847%); 0 B/s, 92 objects/s recovering 2024-02-10T07:53:27.477 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:27 smithi008 ceph-mon[40743]: 6.1b scrub ok 2024-02-10T07:53:27.478 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:27 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:53:27.478 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:27 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:53:27.478 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:27 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 7859/74511 objects degraded (10.547%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:27.531 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:27.571 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:27.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:27 smithi170 ceph-mon[33394]: 6.1b scrub starts 2024-02-10T07:53:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:27 smithi170 ceph-mon[33394]: pgmap v998: 148 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 738 op/s; 7859/74511 objects degraded (10.547%); 4357/74511 objects misplaced (5.847%); 0 B/s, 92 objects/s recovering 2024-02-10T07:53:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:27 smithi170 ceph-mon[33394]: 6.1b scrub ok 2024-02-10T07:53:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:27 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:53:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:27 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:53:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:27 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 7859/74511 objects degraded (10.547%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:27.673 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:27.718 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:27.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:27.820 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:27.853 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:27.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:27.955 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:27.998 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:27.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:28.100 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:28.146 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:28.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:28.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:28.248 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:28.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:28.281 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:28.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:28.382 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:28.421 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:28.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:28.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:28.523 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:28.564 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:28.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:28.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:28 smithi170 ceph-mon[33394]: osdmap e451: 8 total, 8 up, 8 in 2024-02-10T07:53:28.665 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:28.702 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:28.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:28.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:28.769 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:28 smithi008 ceph-mon[29816]: osdmap e451: 8 total, 8 up, 8 in 2024-02-10T07:53:28.769 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:28 smithi008 ceph-mon[40743]: osdmap e451: 8 total, 8 up, 8 in 2024-02-10T07:53:28.789 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:28.805 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:28.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:28.871 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:28.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:28.972 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:29.005 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:29.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:29.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:29.086 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:53:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:53:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:53:29.111 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:29.144 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:29.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:29.245 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:29.251 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:29.269 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-10T07:53:29.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:29.309 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:29.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:29.410 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:29.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:29.443 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:29.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:29.532 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:29 smithi008 ceph-mon[29816]: 4.10 scrub starts 2024-02-10T07:53:29.532 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:29 smithi008 ceph-mon[29816]: 4.10 scrub ok 2024-02-10T07:53:29.532 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:29 smithi008 ceph-mon[29816]: pgmap v1000: 148 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 846 op/s; 7859/73836 objects degraded (10.644%); 4357/73836 objects misplaced (5.901%); 0 B/s, 113 objects/s recovering 2024-02-10T07:53:29.532 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:29 smithi008 ceph-mon[40743]: 4.10 scrub starts 2024-02-10T07:53:29.532 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:29 smithi008 ceph-mon[40743]: 4.10 scrub ok 2024-02-10T07:53:29.533 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:29 smithi008 ceph-mon[40743]: pgmap v1000: 148 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 846 op/s; 7859/73836 objects degraded (10.644%); 4357/73836 objects misplaced (5.901%); 0 B/s, 113 objects/s recovering 2024-02-10T07:53:29.544 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:29.579 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:29.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:29.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:29 smithi170 ceph-mon[33394]: 4.10 scrub starts 2024-02-10T07:53:29.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:29 smithi170 ceph-mon[33394]: 4.10 scrub ok 2024-02-10T07:53:29.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:29 smithi170 ceph-mon[33394]: pgmap v1000: 148 pgs: 1 active+recovering+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 846 op/s; 7859/73836 objects degraded (10.644%); 4357/73836 objects misplaced (5.901%); 0 B/s, 113 objects/s recovering 2024-02-10T07:53:29.681 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:29.713 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:29.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:29.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:29.815 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:29.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:29.851 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:29.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:29.952 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:29.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:29.989 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:29.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:30.090 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:30.136 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:30.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:30.238 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:30.280 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:30.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:30.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:30.382 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:30.411 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:30.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:30.513 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:30.563 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:30.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:30.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:30 smithi170 ceph-mon[33394]: 4.17 scrub starts 2024-02-10T07:53:30.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:30 smithi170 ceph-mon[33394]: 4.17 scrub ok 2024-02-10T07:53:30.664 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:30.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:30.704 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:30.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:30.805 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:30.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:30 smithi008 ceph-mon[29816]: 4.17 scrub starts 2024-02-10T07:53:30.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:30 smithi008 ceph-mon[29816]: 4.17 scrub ok 2024-02-10T07:53:30.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:30 smithi008 ceph-mon[40743]: 4.17 scrub starts 2024-02-10T07:53:30.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:30 smithi008 ceph-mon[40743]: 4.17 scrub ok 2024-02-10T07:53:30.836 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:30.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:30.938 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:30.967 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:30.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:30.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:31.069 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:31.110 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:31.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:31.212 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:31.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:31.258 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:31.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:31.359 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:31.376 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:53:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a[64610]: logger=cleanup t=2024-02-10T07:53:30.892814658Z level=info msg="Completed cleanup jobs" duration=48.422894ms 2024-02-10T07:53:31.385 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:31.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:31.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:31.487 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:31.524 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:31.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:31.565 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:31 smithi008 ceph-mon[29816]: pgmap v1001: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 786 op/s; 7363/63957 objects degraded (11.512%); 4191/63957 objects misplaced (6.553%); 0 B/s, 149 objects/s recovering 2024-02-10T07:53:31.566 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:31 smithi008 ceph-mon[40743]: pgmap v1001: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 786 op/s; 7363/63957 objects degraded (11.512%); 4191/63957 objects misplaced (6.553%); 0 B/s, 149 objects/s recovering 2024-02-10T07:53:31.625 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:31 smithi170 ceph-mon[33394]: pgmap v1001: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 786 op/s; 7363/63957 objects degraded (11.512%); 4191/63957 objects misplaced (6.553%); 0 B/s, 149 objects/s recovering 2024-02-10T07:53:31.673 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:31.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:31.775 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:31.805 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:31.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:31.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:31.907 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:31.937 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:31.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:32.039 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:32.089 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:32.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:32.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:32.186 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:53:32.191 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:32.194 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:53:32.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:32.235 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:32.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:32.336 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:32.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:32.374 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:32.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:32.402 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:53:32.476 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:32.511 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:32.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:32.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:32.613 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:32.652 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:32.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:32.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:32.754 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:32.797 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:32.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:32.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:32 smithi008 ceph-mon[29816]: 11.b scrub starts 2024-02-10T07:53:32.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:32 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4264626462' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:53:32.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:32 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:53:32.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:32 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 7363/63957 objects degraded (11.512%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:32.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:32 smithi008 ceph-mon[40743]: 11.b scrub starts 2024-02-10T07:53:32.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:32 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4264626462' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:53:32.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:32 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:53:32.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:32 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 7363/63957 objects degraded (11.512%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:32.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:32 smithi170 ceph-mon[33394]: 11.b scrub starts 2024-02-10T07:53:32.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:32 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4264626462' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:53:32.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:32 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:53:32.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:32 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 7363/63957 objects degraded (11.512%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:32.899 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:32.933 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:32.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:33.035 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:33.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:33.065 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:33.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:33.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:33.194 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:33.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:33.295 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:33.331 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:33.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:33.432 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:33.478 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:33.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:33.580 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[29816]: 11.b scrub ok 2024-02-10T07:53:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[29816]: 6.b scrub starts 2024-02-10T07:53:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[29816]: from='client.16101 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:53:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[29816]: pgmap v1002: 148 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 650 B/s rd, 0 B/s wr, 668 op/s; 7363/62337 objects degraded (11.812%); 4191/62337 objects misplaced (6.723%); 0 B/s, 113 objects/s recovering 2024-02-10T07:53:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[29816]: 6.b scrub ok 2024-02-10T07:53:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:53:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:53:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[29816]: osdmap e452: 8 total, 8 up, 8 in 2024-02-10T07:53:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[40743]: 11.b scrub ok 2024-02-10T07:53:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[40743]: 6.b scrub starts 2024-02-10T07:53:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[40743]: from='client.16101 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:53:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[40743]: pgmap v1002: 148 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 650 B/s rd, 0 B/s wr, 668 op/s; 7363/62337 objects degraded (11.812%); 4191/62337 objects misplaced (6.723%); 0 B/s, 113 objects/s recovering 2024-02-10T07:53:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[40743]: 6.b scrub ok 2024-02-10T07:53:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:53:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:53:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:33 smithi008 ceph-mon[40743]: osdmap e452: 8 total, 8 up, 8 in 2024-02-10T07:53:33.610 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:33.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:33.712 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:33.742 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:33.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:33.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:33.844 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:33.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:33 smithi170 ceph-mon[33394]: 11.b scrub ok 2024-02-10T07:53:33.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:33 smithi170 ceph-mon[33394]: 6.b scrub starts 2024-02-10T07:53:33.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:33 smithi170 ceph-mon[33394]: from='client.16101 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:53:33.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:33 smithi170 ceph-mon[33394]: pgmap v1002: 148 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 650 B/s rd, 0 B/s wr, 668 op/s; 7363/62337 objects degraded (11.812%); 4191/62337 objects misplaced (6.723%); 0 B/s, 113 objects/s recovering 2024-02-10T07:53:33.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:33 smithi170 ceph-mon[33394]: 6.b scrub ok 2024-02-10T07:53:33.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:33 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:53:33.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:33 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:53:33.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:33 smithi170 ceph-mon[33394]: osdmap e452: 8 total, 8 up, 8 in 2024-02-10T07:53:33.891 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:33.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:33.993 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:34.049 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:34.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:34.150 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:34.177 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:34.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:34.279 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:34.372 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:34.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:34.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:34.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:53:34 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:53:34.474 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:34.500 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:34.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:34.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:34 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:53:34.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:34 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:53:34.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:34 smithi008 ceph-mon[29816]: osdmap e453: 8 total, 8 up, 8 in 2024-02-10T07:53:34.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:53:34.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:53:34.560 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:34 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:53:34.561 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:34 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:53:34.561 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:34 smithi008 ceph-mon[40743]: osdmap e453: 8 total, 8 up, 8 in 2024-02-10T07:53:34.561 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:53:34.561 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:53:34.602 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":1001,"stamp":"2024-02-10T07:53:30.192760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1384716660,"num_objects":21319,"num_object_clones":0,"num_object_copies":63957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7363,"num_objects_misplaced":4191,"num_objects_unfound":0,"num_objects_dirty":21319,"num_whiteouts":0,"num_read":3067,"num_read_kb":2913,"num_write":189586,"num_write_kb":6707038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15092,"num_bytes_recovered":803744373,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85596,"ondisk_log_size":85596,"up":444,"acting":442,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":449,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10787788,"kb_used_data":5408744,"kb_used_omap":19,"kb_used_meta":5378796,"kb_avail":739173428,"statfs":{"total":767960285184,"available":756913590272,"internally_reserved":0,"allocated":5538553856,"data_stored":5534662924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19634,"internal_metadata":5507887950},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,6,4,7,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-309067776,"num_objects":-4716,"num_object_clones":0,"num_object_copies":-14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-666,"num_objects_misplaced":-229,"num_objects_unfound":0,"num_objects_dirty":-4716,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4716,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":896,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001917"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":839,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.351605+0000","last_change":"2024-02-10T07:49:56.315126+0000","last_active":"2024-02-10T07:53:27.351605+0000","last_peered":"2024-02-10T07:53:27.351605+0000","last_clean":"2024-02-10T07:53:27.351605+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:53:27.351605+0000","last_undegraded":"2024-02-10T07:53:27.351605+0000","last_fullsized":"2024-02-10T07:53:27.351605+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.315029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.315029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":849,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.360687+0000","last_change":"2024-02-10T07:49:08.138553+0000","last_active":"2024-02-10T07:53:27.360687+0000","last_peered":"2024-02-10T07:53:27.360687+0000","last_clean":"2024-02-10T07:53:27.360687+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:53:27.360687+0000","last_undegraded":"2024-02-10T07:53:27.360687+0000","last_fullsized":"2024-02-10T07:53:27.360687+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:08.138442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.138442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":877,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.335180+0000","last_change":"2024-02-10T07:50:43.303349+0000","last_active":"2024-02-10T07:53:25.335180+0000","last_peered":"2024-02-10T07:53:25.335180+0000","last_clean":"2024-02-10T07:53:25.335180+0000","last_became_active":"2024-02-10T07:50:43.281905+0000","last_became_peered":"2024-02-10T07:50:43.281905+0000","last_unstale":"2024-02-10T07:53:25.335180+0000","last_undegraded":"2024-02-10T07:53:25.335180+0000","last_fullsized":"2024-02-10T07:53:25.335180+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:17.198701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:17.198701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"430'25","reported_seq":908,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.351764+0000","last_change":"2024-02-10T07:46:37.008978+0000","last_active":"2024-02-10T07:53:27.351764+0000","last_peered":"2024-02-10T07:53:27.351764+0000","last_clean":"2024-02-10T07:53:27.351764+0000","last_became_active":"2024-02-10T07:46:10.717475+0000","last_became_peered":"2024-02-10T07:46:10.717475+0000","last_unstale":"2024-02-10T07:53:27.351764+0000","last_undegraded":"2024-02-10T07:53:27.351764+0000","last_fullsized":"2024-02-10T07:53:27.351764+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'19","last_scrub_stamp":"2024-02-10T07:46:37.008841+0000","last_deep_scrub":"287'19","last_deep_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.008841+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":176,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:26.217976+0000","last_change":"2024-02-10T07:53:26.218018+0000","last_active":"2024-02-10T07:53:26.217976+0000","last_peered":"2024-02-10T07:53:26.217976+0000","last_clean":"2024-02-10T07:53:26.217976+0000","last_became_active":"2024-02-10T07:49:06.236235+0000","last_became_peered":"2024-02-10T07:49:06.236235+0000","last_unstale":"2024-02-10T07:53:26.217976+0000","last_undegraded":"2024-02-10T07:53:26.217976+0000","last_fullsized":"2024-02-10T07:53:26.217976+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:26.217749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:15.079428+0000","last_clean_scrub_stamp":"2024-02-10T07:53:26.217749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:52.906075+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":842,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.350549+0000","last_change":"2024-02-10T07:50:06.335390+0000","last_active":"2024-02-10T07:53:27.350549+0000","last_peered":"2024-02-10T07:53:27.350549+0000","last_clean":"2024-02-10T07:53:27.350549+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:53:27.350549+0000","last_undegraded":"2024-02-10T07:53:27.350549+0000","last_fullsized":"2024-02-10T07:53:27.350549+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:06.335262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:50:06.335262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":366,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.357843+0000","last_change":"2024-02-10T07:51:05.208836+0000","last_active":"2024-02-10T07:53:27.357843+0000","last_peered":"2024-02-10T07:53:27.357843+0000","last_clean":"2024-02-10T07:53:27.357843+0000","last_became_active":"2024-02-10T07:50:43.282449+0000","last_became_peered":"2024-02-10T07:50:43.282449+0000","last_unstale":"2024-02-10T07:53:27.357843+0000","last_undegraded":"2024-02-10T07:53:27.357843+0000","last_fullsized":"2024-02-10T07:53:27.357843+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:51:05.208755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:51:05.208755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"430'39","reported_seq":138,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.366760+0000","last_change":"2024-02-10T07:50:43.318008+0000","last_active":"2024-02-10T07:53:27.366760+0000","last_peered":"2024-02-10T07:53:27.366760+0000","last_clean":"2024-02-10T07:53:27.366760+0000","last_became_active":"2024-02-10T07:50:43.290681+0000","last_became_peered":"2024-02-10T07:50:43.290681+0000","last_unstale":"2024-02-10T07:53:27.366760+0000","last_undegraded":"2024-02-10T07:53:27.366760+0000","last_fullsized":"2024-02-10T07:53:27.366760+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:48:13.230908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.230908+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":836,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.334451+0000","last_change":"2024-02-10T07:50:05.620136+0000","last_active":"2024-02-10T07:53:25.334451+0000","last_peered":"2024-02-10T07:53:25.334451+0000","last_clean":"2024-02-10T07:53:25.334451+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:53:25.334451+0000","last_undegraded":"2024-02-10T07:53:25.334451+0000","last_fullsized":"2024-02-10T07:53:25.334451+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:05.619958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:05.619958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":827,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.379401+0000","last_change":"2024-02-10T07:53:23.117713+0000","last_active":"2024-02-10T07:53:27.379401+0000","last_peered":"2024-02-10T07:53:27.379401+0000","last_clean":"2024-02-10T07:53:27.379401+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:53:27.379401+0000","last_undegraded":"2024-02-10T07:53:27.379401+0000","last_fullsized":"2024-02-10T07:53:27.379401+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:23.117597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:23.117597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:27.492528+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":380,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.352999+0000","last_change":"2024-02-10T07:49:13.218634+0000","last_active":"2024-02-10T07:53:27.352999+0000","last_peered":"2024-02-10T07:53:27.352999+0000","last_clean":"2024-02-10T07:53:27.352999+0000","last_became_active":"2024-02-10T07:48:02.332960+0000","last_became_peered":"2024-02-10T07:48:02.332960+0000","last_unstale":"2024-02-10T07:53:27.352999+0000","last_undegraded":"2024-02-10T07:53:27.352999+0000","last_fullsized":"2024-02-10T07:53:27.352999+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.218549+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"430'21","reported_seq":873,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.351838+0000","last_change":"2024-02-10T07:47:49.553645+0000","last_active":"2024-02-10T07:53:27.351838+0000","last_peered":"2024-02-10T07:53:27.351838+0000","last_clean":"2024-02-10T07:53:27.351838+0000","last_became_active":"2024-02-10T07:47:47.965930+0000","last_became_peered":"2024-02-10T07:47:47.965930+0000","last_unstale":"2024-02-10T07:53:27.351838+0000","last_undegraded":"2024-02-10T07:53:27.351838+0000","last_fullsized":"2024-02-10T07:53:27.351838+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:49.553517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.553517+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":751,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.360858+0000","last_change":"2024-02-10T07:49:56.718482+0000","last_active":"2024-02-10T07:53:27.360858+0000","last_peered":"2024-02-10T07:53:27.360858+0000","last_clean":"2024-02-10T07:53:27.360858+0000","last_became_active":"2024-02-10T07:49:06.266689+0000","last_became_peered":"2024-02-10T07:49:06.266689+0000","last_unstale":"2024-02-10T07:53:27.360858+0000","last_undegraded":"2024-02-10T07:53:27.360858+0000","last_fullsized":"2024-02-10T07:53:27.360858+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.718382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.718382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":842,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353614+0000","last_change":"2024-02-10T07:48:18.069773+0000","last_active":"2024-02-10T07:53:27.353614+0000","last_peered":"2024-02-10T07:53:27.353614+0000","last_clean":"2024-02-10T07:53:27.353614+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:53:27.353614+0000","last_undegraded":"2024-02-10T07:53:27.353614+0000","last_fullsized":"2024-02-10T07:53:27.353614+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:18.069641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:18.069641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":368,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353370+0000","last_change":"2024-02-10T07:49:06.115547+0000","last_active":"2024-02-10T07:53:27.353370+0000","last_peered":"2024-02-10T07:53:27.353370+0000","last_clean":"2024-02-10T07:53:27.353370+0000","last_became_active":"2024-02-10T07:48:02.329859+0000","last_became_peered":"2024-02-10T07:48:02.329859+0000","last_unstale":"2024-02-10T07:53:27.353370+0000","last_undegraded":"2024-02-10T07:53:27.353370+0000","last_fullsized":"2024-02-10T07:53:27.353370+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.115466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.473686+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.115466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"430'9","reported_seq":855,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.352449+0000","last_change":"2024-02-10T07:48:11.611397+0000","last_active":"2024-02-10T07:53:27.352449+0000","last_peered":"2024-02-10T07:53:27.352449+0000","last_clean":"2024-02-10T07:53:27.352449+0000","last_became_active":"2024-02-10T07:47:47.966093+0000","last_became_peered":"2024-02-10T07:47:47.966093+0000","last_unstale":"2024-02-10T07:53:27.352449+0000","last_undegraded":"2024-02-10T07:53:27.352449+0000","last_fullsized":"2024-02-10T07:53:27.352449+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7","last_scrub_stamp":"2024-02-10T07:48:11.608499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.608499+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":841,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353471+0000","last_change":"2024-02-10T07:48:13.621135+0000","last_active":"2024-02-10T07:53:27.353471+0000","last_peered":"2024-02-10T07:53:27.353471+0000","last_clean":"2024-02-10T07:53:27.353471+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:53:27.353471+0000","last_undegraded":"2024-02-10T07:53:27.353471+0000","last_fullsized":"2024-02-10T07:53:27.353471+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:13.620938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.620938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":848,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.380763+0000","last_change":"2024-02-10T07:49:51.810158+0000","last_active":"2024-02-10T07:53:27.380763+0000","last_peered":"2024-02-10T07:53:27.380763+0000","last_clean":"2024-02-10T07:53:27.380763+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:53:27.380763+0000","last_undegraded":"2024-02-10T07:53:27.380763+0000","last_fullsized":"2024-02-10T07:53:27.380763+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:51.810062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:51.810062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":848,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.336110+0000","last_change":"2024-02-10T07:49:16.313655+0000","last_active":"2024-02-10T07:53:25.336110+0000","last_peered":"2024-02-10T07:53:25.336110+0000","last_clean":"2024-02-10T07:53:25.336110+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:53:25.336110+0000","last_undegraded":"2024-02-10T07:53:25.336110+0000","last_fullsized":"2024-02-10T07:53:25.336110+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:16.313573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.313573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"430'46","reported_seq":138,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.378418+0000","last_change":"2024-02-10T07:50:43.322322+0000","last_active":"2024-02-10T07:53:27.378418+0000","last_peered":"2024-02-10T07:53:27.378418+0000","last_clean":"2024-02-10T07:53:27.378418+0000","last_became_active":"2024-02-10T07:50:43.300153+0000","last_became_peered":"2024-02-10T07:50:43.300153+0000","last_unstale":"2024-02-10T07:53:27.378418+0000","last_undegraded":"2024-02-10T07:53:27.378418+0000","last_fullsized":"2024-02-10T07:53:27.378418+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:49:00.551605+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.551605+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":545,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.358135+0000","last_change":"2024-02-10T07:52:11.043473+0000","last_active":"2024-02-10T07:53:27.358135+0000","last_peered":"2024-02-10T07:53:27.358135+0000","last_clean":"2024-02-10T07:53:27.358135+0000","last_became_active":"2024-02-10T07:49:06.263790+0000","last_became_peered":"2024-02-10T07:49:06.263790+0000","last_unstale":"2024-02-10T07:53:27.358135+0000","last_undegraded":"2024-02-10T07:53:27.358135+0000","last_fullsized":"2024-02-10T07:53:27.358135+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:11.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:11.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":859,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353683+0000","last_change":"2024-02-10T07:50:43.286623+0000","last_active":"2024-02-10T07:53:27.353683+0000","last_peered":"2024-02-10T07:53:27.353683+0000","last_clean":"2024-02-10T07:53:27.353683+0000","last_became_active":"2024-02-10T07:50:43.285201+0000","last_became_peered":"2024-02-10T07:50:43.285201+0000","last_unstale":"2024-02-10T07:53:27.353683+0000","last_undegraded":"2024-02-10T07:53:27.353683+0000","last_fullsized":"2024-02-10T07:53:27.353683+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.680331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.680331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":862,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.351923+0000","last_change":"2024-02-10T07:50:43.282584+0000","last_active":"2024-02-10T07:53:27.351923+0000","last_peered":"2024-02-10T07:53:27.351923+0000","last_clean":"2024-02-10T07:53:27.351923+0000","last_became_active":"2024-02-10T07:50:43.281382+0000","last_became_peered":"2024-02-10T07:50:43.281382+0000","last_unstale":"2024-02-10T07:53:27.351923+0000","last_undegraded":"2024-02-10T07:53:27.351923+0000","last_fullsized":"2024-02-10T07:53:27.351923+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:04.396771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.396771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"430'24","reported_seq":818,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.377509+0000","last_change":"2024-02-10T07:49:58.618564+0000","last_active":"2024-02-10T07:53:27.377509+0000","last_peered":"2024-02-10T07:53:27.377509+0000","last_clean":"2024-02-10T07:53:27.377509+0000","last_became_active":"2024-02-10T07:49:06.263261+0000","last_became_peered":"2024-02-10T07:49:06.263261+0000","last_unstale":"2024-02-10T07:53:27.377509+0000","last_undegraded":"2024-02-10T07:53:27.377509+0000","last_fullsized":"2024-02-10T07:53:27.377509+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'18","last_scrub_stamp":"2024-02-10T07:49:58.618464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:58.618464+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":874,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.354582+0000","last_change":"2024-02-10T07:50:43.307775+0000","last_active":"2024-02-10T07:53:27.354582+0000","last_peered":"2024-02-10T07:53:27.354582+0000","last_clean":"2024-02-10T07:53:27.354582+0000","last_became_active":"2024-02-10T07:50:43.285254+0000","last_became_peered":"2024-02-10T07:50:43.285254+0000","last_unstale":"2024-02-10T07:53:27.354582+0000","last_undegraded":"2024-02-10T07:53:27.354582+0000","last_fullsized":"2024-02-10T07:53:27.354582+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:47:34.074746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:34.074746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"414'9","reported_seq":1147,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.357866+0000","last_change":"2024-02-10T07:51:13.307061+0000","last_active":"2024-02-10T07:53:27.357866+0000","last_peered":"2024-02-10T07:53:27.357866+0000","last_clean":"2024-02-10T07:53:27.357866+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:53:27.357866+0000","last_undegraded":"2024-02-10T07:53:27.357866+0000","last_fullsized":"2024-02-10T07:53:27.357866+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'9","last_scrub_stamp":"2024-02-10T07:51:13.306986+0000","last_deep_scrub":"414'9","last_deep_scrub_stamp":"2024-02-10T07:51:13.306986+0000","last_clean_scrub_stamp":"2024-02-10T07:51:13.306986+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":858,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.360587+0000","last_change":"2024-02-10T07:48:30.384635+0000","last_active":"2024-02-10T07:53:27.360587+0000","last_peered":"2024-02-10T07:53:27.360587+0000","last_clean":"2024-02-10T07:53:27.360587+0000","last_became_active":"2024-02-10T07:47:47.966127+0000","last_became_peered":"2024-02-10T07:47:47.966127+0000","last_unstale":"2024-02-10T07:53:27.360587+0000","last_undegraded":"2024-02-10T07:53:27.360587+0000","last_fullsized":"2024-02-10T07:53:27.360587+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:30.384546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:30.384546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"430'35","reported_seq":935,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.336693+0000","last_change":"2024-02-10T07:49:06.308245+0000","last_active":"2024-02-10T07:53:25.336693+0000","last_peered":"2024-02-10T07:53:25.336693+0000","last_clean":"2024-02-10T07:53:25.336693+0000","last_became_active":"2024-02-10T07:49:06.261487+0000","last_became_peered":"2024-02-10T07:49:06.261487+0000","last_unstale":"2024-02-10T07:53:25.336693+0000","last_undegraded":"2024-02-10T07:53:25.336693+0000","last_fullsized":"2024-02-10T07:53:25.336693+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:48:12.278982+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-02-10T07:42:17.820959+0000","last_clean_scrub_stamp":"2024-02-10T07:48:12.278982+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":845,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.380920+0000","last_change":"2024-02-10T07:49:11.231819+0000","last_active":"2024-02-10T07:53:27.380920+0000","last_peered":"2024-02-10T07:53:27.380920+0000","last_clean":"2024-02-10T07:53:27.380920+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:53:27.380920+0000","last_undegraded":"2024-02-10T07:53:27.380920+0000","last_fullsized":"2024-02-10T07:53:27.380920+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.231704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.231704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":846,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.336821+0000","last_change":"2024-02-10T07:50:43.289955+0000","last_active":"2024-02-10T07:53:25.336821+0000","last_peered":"2024-02-10T07:53:25.336821+0000","last_clean":"2024-02-10T07:53:25.336821+0000","last_became_active":"2024-02-10T07:50:43.289011+0000","last_became_peered":"2024-02-10T07:50:43.289011+0000","last_unstale":"2024-02-10T07:53:25.336821+0000","last_undegraded":"2024-02-10T07:53:25.336821+0000","last_fullsized":"2024-02-10T07:53:25.336821+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.208451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:06.200009+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.208451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":352,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.334530+0000","last_change":"2024-02-10T07:53:21.437757+0000","last_active":"2024-02-10T07:53:25.334530+0000","last_peered":"2024-02-10T07:53:25.334530+0000","last_clean":"2024-02-10T07:53:25.334530+0000","last_became_active":"2024-02-10T07:49:06.258789+0000","last_became_peered":"2024-02-10T07:49:06.258789+0000","last_unstale":"2024-02-10T07:53:25.334530+0000","last_undegraded":"2024-02-10T07:53:25.334530+0000","last_fullsized":"2024-02-10T07:53:25.334530+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:21.437653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:53:21.437653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.318460+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"430'27","reported_seq":915,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.357974+0000","last_change":"2024-02-10T07:52:11.834589+0000","last_active":"2024-02-10T07:53:27.357974+0000","last_peered":"2024-02-10T07:53:27.357974+0000","last_clean":"2024-02-10T07:53:27.357974+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:53:27.357974+0000","last_undegraded":"2024-02-10T07:53:27.357974+0000","last_fullsized":"2024-02-10T07:53:27.357974+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'27","last_scrub_stamp":"2024-02-10T07:52:11.834490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:52:11.834490+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:28.684473+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":861,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353532+0000","last_change":"2024-02-10T07:50:43.284049+0000","last_active":"2024-02-10T07:53:27.353532+0000","last_peered":"2024-02-10T07:53:27.353532+0000","last_clean":"2024-02-10T07:53:27.353532+0000","last_became_active":"2024-02-10T07:50:43.283688+0000","last_became_peered":"2024-02-10T07:50:43.283688+0000","last_unstale":"2024-02-10T07:53:27.353532+0000","last_undegraded":"2024-02-10T07:53:27.353532+0000","last_fullsized":"2024-02-10T07:53:27.353532+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.087614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.087614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":866,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.358319+0000","last_change":"2024-02-10T07:52:22.845994+0000","last_active":"2024-02-10T07:53:27.358319+0000","last_peered":"2024-02-10T07:53:27.358319+0000","last_clean":"2024-02-10T07:53:27.358319+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:53:27.358319+0000","last_undegraded":"2024-02-10T07:53:27.358319+0000","last_fullsized":"2024-02-10T07:53:27.358319+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:22.845773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:52:22.845773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":872,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.335431+0000","last_change":"2024-02-10T07:49:06.259100+0000","last_active":"2024-02-10T07:53:25.335431+0000","last_peered":"2024-02-10T07:53:25.335431+0000","last_clean":"2024-02-10T07:53:25.335431+0000","last_became_active":"2024-02-10T07:49:06.258763+0000","last_became_peered":"2024-02-10T07:49:06.258763+0000","last_unstale":"2024-02-10T07:53:25.335431+0000","last_undegraded":"2024-02-10T07:53:25.335431+0000","last_fullsized":"2024-02-10T07:53:25.335431+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:29.188832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:29.188832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"430'14","reported_seq":180,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.330350+0000","last_change":"2024-02-10T07:49:16.017097+0000","last_active":"2024-02-10T07:53:25.330350+0000","last_peered":"2024-02-10T07:53:25.330350+0000","last_clean":"2024-02-10T07:53:25.330350+0000","last_became_active":"2024-02-10T07:49:06.250577+0000","last_became_peered":"2024-02-10T07:49:06.250577+0000","last_unstale":"2024-02-10T07:53:25.330350+0000","last_undegraded":"2024-02-10T07:53:25.330350+0000","last_fullsized":"2024-02-10T07:53:25.330350+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'10","last_scrub_stamp":"2024-02-10T07:49:16.017003+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.017003+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":338,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.335631+0000","last_change":"2024-02-10T07:49:06.259895+0000","last_active":"2024-02-10T07:53:25.335631+0000","last_peered":"2024-02-10T07:53:25.335631+0000","last_clean":"2024-02-10T07:53:25.335631+0000","last_became_active":"2024-02-10T07:49:06.259341+0000","last_became_peered":"2024-02-10T07:49:06.259341+0000","last_unstale":"2024-02-10T07:53:25.335631+0000","last_undegraded":"2024-02-10T07:53:25.335631+0000","last_fullsized":"2024-02-10T07:53:25.335631+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:32.258321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:32.258321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":843,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353411+0000","last_change":"2024-02-10T07:48:24.171978+0000","last_active":"2024-02-10T07:53:27.353411+0000","last_peered":"2024-02-10T07:53:27.353411+0000","last_clean":"2024-02-10T07:53:27.353411+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:53:27.353411+0000","last_undegraded":"2024-02-10T07:53:27.353411+0000","last_fullsized":"2024-02-10T07:53:27.353411+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:24.171859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:24.171859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":800,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.380029+0000","last_change":"2024-02-10T07:52:27.104889+0000","last_active":"2024-02-10T07:53:27.380029+0000","last_peered":"2024-02-10T07:53:27.380029+0000","last_clean":"2024-02-10T07:53:27.380029+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:53:27.380029+0000","last_undegraded":"2024-02-10T07:53:27.380029+0000","last_fullsized":"2024-02-10T07:53:27.380029+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:27.104803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:52:27.104803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:49.184078+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"430'27","reported_seq":821,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353737+0000","last_change":"2024-02-10T07:49:53.245490+0000","last_active":"2024-02-10T07:53:27.353737+0000","last_peered":"2024-02-10T07:53:27.353737+0000","last_clean":"2024-02-10T07:53:27.353737+0000","last_became_active":"2024-02-10T07:49:06.239701+0000","last_became_peered":"2024-02-10T07:49:06.239701+0000","last_unstale":"2024-02-10T07:53:27.353737+0000","last_undegraded":"2024-02-10T07:53:27.353737+0000","last_fullsized":"2024-02-10T07:53:27.353737+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:53.245410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:53.245410+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":838,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.334338+0000","last_change":"2024-02-10T07:49:52.530010+0000","last_active":"2024-02-10T07:53:25.334338+0000","last_peered":"2024-02-10T07:53:25.334338+0000","last_clean":"2024-02-10T07:53:25.334338+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:53:25.334338+0000","last_undegraded":"2024-02-10T07:53:25.334338+0000","last_fullsized":"2024-02-10T07:53:25.334338+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.529908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:59.179342+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.529908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":346,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.335817+0000","last_change":"2024-02-10T07:53:19.493460+0000","last_active":"2024-02-10T07:53:25.335817+0000","last_peered":"2024-02-10T07:53:25.335817+0000","last_clean":"2024-02-10T07:53:25.335817+0000","last_became_active":"2024-02-10T07:49:06.265267+0000","last_became_peered":"2024-02-10T07:49:06.265267+0000","last_unstale":"2024-02-10T07:53:25.335817+0000","last_undegraded":"2024-02-10T07:53:25.335817+0000","last_fullsized":"2024-02-10T07:53:25.335817+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:19.492989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:19.492989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:32.822823+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":859,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.354041+0000","last_change":"2024-02-10T07:50:00.265308+0000","last_active":"2024-02-10T07:53:27.354041+0000","last_peered":"2024-02-10T07:53:27.354041+0000","last_clean":"2024-02-10T07:53:27.354041+0000","last_became_active":"2024-02-10T07:47:47.966472+0000","last_became_peered":"2024-02-10T07:47:47.966472+0000","last_unstale":"2024-02-10T07:53:27.354041+0000","last_undegraded":"2024-02-10T07:53:27.354041+0000","last_fullsized":"2024-02-10T07:53:27.354041+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.265208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.265208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"430'25","reported_seq":693,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.361006+0000","last_change":"2024-02-10T07:50:43.310533+0000","last_active":"2024-02-10T07:53:27.361006+0000","last_peered":"2024-02-10T07:53:27.361006+0000","last_clean":"2024-02-10T07:53:27.361006+0000","last_became_active":"2024-02-10T07:50:43.296447+0000","last_became_peered":"2024-02-10T07:50:43.296447+0000","last_unstale":"2024-02-10T07:53:27.361006+0000","last_undegraded":"2024-02-10T07:53:27.361006+0000","last_fullsized":"2024-02-10T07:53:27.361006+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:09.116615+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-02-10T07:39:32.934269+0000","last_clean_scrub_stamp":"2024-02-10T07:49:09.116615+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":526,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.334995+0000","last_change":"2024-02-10T07:49:06.260883+0000","last_active":"2024-02-10T07:53:25.334995+0000","last_peered":"2024-02-10T07:53:25.334995+0000","last_clean":"2024-02-10T07:53:25.334995+0000","last_became_active":"2024-02-10T07:49:06.260569+0000","last_became_peered":"2024-02-10T07:49:06.260569+0000","last_unstale":"2024-02-10T07:53:25.334995+0000","last_undegraded":"2024-02-10T07:53:25.334995+0000","last_fullsized":"2024-02-10T07:53:25.334995+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:31.117976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:31.117976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":873,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.361539+0000","last_change":"2024-02-10T07:50:02.685941+0000","last_active":"2024-02-10T07:53:27.361539+0000","last_peered":"2024-02-10T07:53:27.361539+0000","last_clean":"2024-02-10T07:53:27.361539+0000","last_became_active":"2024-02-10T07:49:06.252380+0000","last_became_peered":"2024-02-10T07:49:06.252380+0000","last_unstale":"2024-02-10T07:53:27.361539+0000","last_undegraded":"2024-02-10T07:53:27.361539+0000","last_fullsized":"2024-02-10T07:53:27.361539+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_clean_scrub_stamp":"2024-02-10T07:50:02.685831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":851,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353598+0000","last_change":"2024-02-10T07:50:00.345087+0000","last_active":"2024-02-10T07:53:27.353598+0000","last_peered":"2024-02-10T07:53:27.353598+0000","last_clean":"2024-02-10T07:53:27.353598+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:53:27.353598+0000","last_undegraded":"2024-02-10T07:53:27.353598+0000","last_fullsized":"2024-02-10T07:53:27.353598+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.344993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.344993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"430'32","reported_seq":923,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.352253+0000","last_change":"2024-02-10T07:47:43.494877+0000","last_active":"2024-02-10T07:53:27.352253+0000","last_peered":"2024-02-10T07:53:27.352253+0000","last_clean":"2024-02-10T07:53:27.352253+0000","last_became_active":"2024-02-10T07:46:10.721336+0000","last_became_peered":"2024-02-10T07:46:10.721336+0000","last_unstale":"2024-02-10T07:53:27.352253+0000","last_undegraded":"2024-02-10T07:53:27.352253+0000","last_fullsized":"2024-02-10T07:53:27.352253+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:47:43.494350+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.494350+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":843,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.336532+0000","last_change":"2024-02-10T07:49:06.262127+0000","last_active":"2024-02-10T07:53:25.336532+0000","last_peered":"2024-02-10T07:53:25.336532+0000","last_clean":"2024-02-10T07:53:25.336532+0000","last_became_active":"2024-02-10T07:49:06.261570+0000","last_became_peered":"2024-02-10T07:49:06.261570+0000","last_unstale":"2024-02-10T07:53:25.336532+0000","last_undegraded":"2024-02-10T07:53:25.336532+0000","last_fullsized":"2024-02-10T07:53:25.336532+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:43.186809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.186809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"356'11","reported_seq":347,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.380666+0000","last_change":"2024-02-10T07:48:25.339974+0000","last_active":"2024-02-10T07:53:27.380666+0000","last_peered":"2024-02-10T07:53:27.380666+0000","last_clean":"2024-02-10T07:53:27.380666+0000","last_became_active":"2024-02-10T07:48:02.371888+0000","last_became_peered":"2024-02-10T07:48:02.371888+0000","last_unstale":"2024-02-10T07:53:27.380666+0000","last_undegraded":"2024-02-10T07:53:27.380666+0000","last_fullsized":"2024-02-10T07:53:27.380666+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'11","last_scrub_stamp":"2024-02-10T07:48:25.339855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:25.339855+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":871,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353515+0000","last_change":"2024-02-10T07:50:43.283263+0000","last_active":"2024-02-10T07:53:27.353515+0000","last_peered":"2024-02-10T07:53:27.353515+0000","last_clean":"2024-02-10T07:53:27.353515+0000","last_became_active":"2024-02-10T07:50:43.282925+0000","last_became_peered":"2024-02-10T07:50:43.282925+0000","last_unstale":"2024-02-10T07:53:27.353515+0000","last_undegraded":"2024-02-10T07:53:27.353515+0000","last_fullsized":"2024-02-10T07:53:27.353515+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.335455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.335455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"430'35","reported_seq":487,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.361662+0000","last_change":"2024-02-10T07:49:12.150442+0000","last_active":"2024-02-10T07:53:27.361662+0000","last_peered":"2024-02-10T07:53:27.361662+0000","last_clean":"2024-02-10T07:53:27.361662+0000","last_became_active":"2024-02-10T07:48:02.329187+0000","last_became_peered":"2024-02-10T07:48:02.329187+0000","last_unstale":"2024-02-10T07:53:27.361662+0000","last_undegraded":"2024-02-10T07:53:27.361662+0000","last_fullsized":"2024-02-10T07:53:27.361662+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:49:12.150298+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:49:12.150298+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":851,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353263+0000","last_change":"2024-02-10T07:50:43.285390+0000","last_active":"2024-02-10T07:53:27.353263+0000","last_peered":"2024-02-10T07:53:27.353263+0000","last_clean":"2024-02-10T07:53:27.353263+0000","last_became_active":"2024-02-10T07:50:43.284846+0000","last_became_peered":"2024-02-10T07:50:43.284846+0000","last_unstale":"2024-02-10T07:53:27.353263+0000","last_undegraded":"2024-02-10T07:53:27.353263+0000","last_fullsized":"2024-02-10T07:53:27.353263+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:01.513972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:01.513972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"405'15","reported_seq":503,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.354502+0000","last_change":"2024-02-10T07:50:43.310228+0000","last_active":"2024-02-10T07:53:27.354502+0000","last_peered":"2024-02-10T07:53:27.354502+0000","last_clean":"2024-02-10T07:53:27.354502+0000","last_became_active":"2024-02-10T07:50:43.285213+0000","last_became_peered":"2024-02-10T07:50:43.285213+0000","last_unstale":"2024-02-10T07:53:27.354502+0000","last_undegraded":"2024-02-10T07:53:27.354502+0000","last_fullsized":"2024-02-10T07:53:27.354502+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'13","last_scrub_stamp":"2024-02-10T07:49:00.366610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.366610+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":864,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.335066+0000","last_change":"2024-02-10T07:50:07.501014+0000","last_active":"2024-02-10T07:53:25.335066+0000","last_peered":"2024-02-10T07:53:25.335066+0000","last_clean":"2024-02-10T07:53:25.335066+0000","last_became_active":"2024-02-10T07:49:06.257043+0000","last_became_peered":"2024-02-10T07:49:06.257043+0000","last_unstale":"2024-02-10T07:53:25.335066+0000","last_undegraded":"2024-02-10T07:53:25.335066+0000","last_fullsized":"2024-02-10T07:53:25.335066+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.500890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"430'21","reported_seq":613,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.354347+0000","last_change":"2024-02-10T07:49:06.277154+0000","last_active":"2024-02-10T07:53:27.354347+0000","last_peered":"2024-02-10T07:53:27.354347+0000","last_clean":"2024-02-10T07:53:27.354347+0000","last_became_active":"2024-02-10T07:49:06.255239+0000","last_became_peered":"2024-02-10T07:49:06.255239+0000","last_unstale":"2024-02-10T07:53:27.354347+0000","last_undegraded":"2024-02-10T07:53:27.354347+0000","last_fullsized":"2024-02-10T07:53:27.354347+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:39.610017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:39.610017+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":104,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.378557+0000","last_change":"2024-02-10T07:50:43.282730+0000","last_active":"2024-02-10T07:53:27.378557+0000","last_peered":"2024-02-10T07:53:27.378557+0000","last_clean":"2024-02-10T07:53:27.378557+0000","last_became_active":"2024-02-10T07:50:43.281549+0000","last_became_peered":"2024-02-10T07:50:43.281549+0000","last_unstale":"2024-02-10T07:53:27.378557+0000","last_undegraded":"2024-02-10T07:53:27.378557+0000","last_fullsized":"2024-02-10T07:53:27.378557+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:44.324046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:49:44.324046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":207,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.357813+0000","last_change":"2024-02-10T07:52:34.860375+0000","last_active":"2024-02-10T07:53:27.357813+0000","last_peered":"2024-02-10T07:53:27.357813+0000","last_clean":"2024-02-10T07:53:27.357813+0000","last_became_active":"2024-02-10T07:49:06.262933+0000","last_became_peered":"2024-02-10T07:49:06.262933+0000","last_unstale":"2024-02-10T07:53:27.357813+0000","last_undegraded":"2024-02-10T07:53:27.357813+0000","last_fullsized":"2024-02-10T07:53:27.357813+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:34.860273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:52:34.860273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:47.105853+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":859,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353233+0000","last_change":"2024-02-10T07:49:59.278666+0000","last_active":"2024-02-10T07:53:27.353233+0000","last_peered":"2024-02-10T07:53:27.353233+0000","last_clean":"2024-02-10T07:53:27.353233+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:53:27.353233+0000","last_undegraded":"2024-02-10T07:53:27.353233+0000","last_fullsized":"2024-02-10T07:53:27.353233+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.278574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.278574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"430'25","reported_seq":917,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.336718+0000","last_change":"2024-02-10T07:49:06.306965+0000","last_active":"2024-02-10T07:53:25.336718+0000","last_peered":"2024-02-10T07:53:25.336718+0000","last_clean":"2024-02-10T07:53:25.336718+0000","last_became_active":"2024-02-10T07:49:06.260962+0000","last_became_peered":"2024-02-10T07:49:06.260962+0000","last_unstale":"2024-02-10T07:53:25.336718+0000","last_undegraded":"2024-02-10T07:53:25.336718+0000","last_fullsized":"2024-02-10T07:53:25.336718+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:47:42.150605+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.150605+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":838,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.354135+0000","last_change":"2024-02-10T07:48:58.511630+0000","last_active":"2024-02-10T07:53:27.354135+0000","last_peered":"2024-02-10T07:53:27.354135+0000","last_clean":"2024-02-10T07:53:27.354135+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:53:27.354135+0000","last_undegraded":"2024-02-10T07:53:27.354135+0000","last_fullsized":"2024-02-10T07:53:27.354135+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:58.511251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.511251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":168,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.331347+0000","last_change":"2024-02-10T07:50:04.978177+0000","last_active":"2024-02-10T07:53:25.331347+0000","last_peered":"2024-02-10T07:53:25.331347+0000","last_clean":"2024-02-10T07:53:25.331347+0000","last_became_active":"2024-02-10T07:49:06.250832+0000","last_became_peered":"2024-02-10T07:49:06.250832+0000","last_unstale":"2024-02-10T07:53:25.331347+0000","last_undegraded":"2024-02-10T07:53:25.331347+0000","last_fullsized":"2024-02-10T07:53:25.331347+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:04.978097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.978097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":891,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.357503+0000","last_change":"2024-02-10T07:52:16.809094+0000","last_active":"2024-02-10T07:53:27.357503+0000","last_peered":"2024-02-10T07:53:27.357503+0000","last_clean":"2024-02-10T07:53:27.357503+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:53:27.357503+0000","last_undegraded":"2024-02-10T07:53:27.357503+0000","last_fullsized":"2024-02-10T07:53:27.357503+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:16.808978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:52:16.808978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:36.921179+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"430'18","reported_seq":185,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.331678+0000","last_change":"2024-02-10T07:49:43.919297+0000","last_active":"2024-02-10T07:53:25.331678+0000","last_peered":"2024-02-10T07:53:25.331678+0000","last_clean":"2024-02-10T07:53:25.331678+0000","last_became_active":"2024-02-10T07:49:06.258281+0000","last_became_peered":"2024-02-10T07:49:06.258281+0000","last_unstale":"2024-02-10T07:53:25.331678+0000","last_undegraded":"2024-02-10T07:53:25.331678+0000","last_fullsized":"2024-02-10T07:53:25.331678+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'12","last_scrub_stamp":"2024-02-10T07:49:43.919178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:43.919178+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"451'11939","reported_seq":12071,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:29.928309+0000","last_change":"2024-02-10T07:49:53.374457+0000","last_active":"2024-02-10T07:53:29.928309+0000","last_peered":"2024-02-10T07:53:29.928309+0000","last_clean":"2024-02-10T07:53:29.928309+0000","last_became_active":"2024-02-10T07:49:53.374185+0000","last_became_peered":"2024-02-10T07:49:53.374185+0000","last_unstale":"2024-02-10T07:53:29.928309+0000","last_undegraded":"2024-02-10T07:53:29.928309+0000","last_fullsized":"2024-02-10T07:53:29.928309+0000","mapping_epoch":391,"log_start":"426'7100","ondisk_log_start":"426'7100","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4839,"log_dups_size":0,"ondisk_log_size":4839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85524480,"num_objects":1305,"num_object_clones":0,"num_object_copies":3915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11939,"num_write_kb":423808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":380,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.357435+0000","last_change":"2024-02-10T07:52:21.875347+0000","last_active":"2024-02-10T07:53:27.357435+0000","last_peered":"2024-02-10T07:53:27.357435+0000","last_clean":"2024-02-10T07:53:27.357435+0000","last_became_active":"2024-02-10T07:48:02.366002+0000","last_became_peered":"2024-02-10T07:48:02.366002+0000","last_unstale":"2024-02-10T07:53:27.357435+0000","last_undegraded":"2024-02-10T07:53:27.357435+0000","last_fullsized":"2024-02-10T07:53:27.357435+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:21.875258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:21.875258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:47.914931+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"245'4","reported_seq":873,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.354621+0000","last_change":"2024-02-10T07:49:08.311056+0000","last_active":"2024-02-10T07:53:27.354621+0000","last_peered":"2024-02-10T07:53:27.354621+0000","last_clean":"2024-02-10T07:53:27.354621+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:53:27.354621+0000","last_undegraded":"2024-02-10T07:53:27.354621+0000","last_fullsized":"2024-02-10T07:53:27.354621+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'4","last_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_deep_scrub":"245'4","last_deep_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.310917+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"430'46","reported_seq":617,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353236+0000","last_change":"2024-02-10T07:48:11.967227+0000","last_active":"2024-02-10T07:53:27.353236+0000","last_peered":"2024-02-10T07:53:27.353236+0000","last_clean":"2024-02-10T07:53:27.353236+0000","last_became_active":"2024-02-10T07:48:02.330130+0000","last_became_peered":"2024-02-10T07:48:02.330130+0000","last_unstale":"2024-02-10T07:53:27.353236+0000","last_undegraded":"2024-02-10T07:53:27.353236+0000","last_fullsized":"2024-02-10T07:53:27.353236+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:48:11.966957+0000","last_deep_scrub":"246'22","last_deep_scrub_stamp":"2024-02-10T07:41:26.771112+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.966957+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":1119,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353833+0000","last_change":"2024-02-10T07:48:53.423840+0000","last_active":"2024-02-10T07:53:27.353833+0000","last_peered":"2024-02-10T07:53:27.353833+0000","last_clean":"2024-02-10T07:53:27.353833+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:53:27.353833+0000","last_undegraded":"2024-02-10T07:53:27.353833+0000","last_fullsized":"2024-02-10T07:53:27.353833+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:48:53.423613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:53.423613+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"450'11358","reported_seq":20186,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.422640+0000","last_change":"2024-02-10T07:53:07.243749+0000","last_active":"2024-02-10T07:53:25.422640+0000","last_peered":"2024-02-10T07:53:25.422640+0000","last_clean":"2024-02-10T07:53:25.422640+0000","last_became_active":"2024-02-10T07:53:07.218448+0000","last_became_peered":"2024-02-10T07:53:07.218448+0000","last_unstale":"2024-02-10T07:53:25.422640+0000","last_undegraded":"2024-02-10T07:53:25.422640+0000","last_fullsized":"2024-02-10T07:53:25.422640+0000","mapping_epoch":444,"log_start":"414'5500","ondisk_log_start":"414'5500","created":391,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92930048,"num_objects":1418,"num_object_clones":0,"num_object_copies":4254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11358,"num_write_kb":408832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4344,"num_bytes_recovered":284688384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":675,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.330735+0000","last_change":"2024-02-10T07:49:42.990229+0000","last_active":"2024-02-10T07:53:25.330735+0000","last_peered":"2024-02-10T07:53:25.330735+0000","last_clean":"2024-02-10T07:53:25.330735+0000","last_became_active":"2024-02-10T07:47:47.966183+0000","last_became_peered":"2024-02-10T07:47:47.966183+0000","last_unstale":"2024-02-10T07:53:25.330735+0000","last_undegraded":"2024-02-10T07:53:25.330735+0000","last_fullsized":"2024-02-10T07:53:25.330735+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:42.990098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:42.990098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":851,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.354227+0000","last_change":"2024-02-10T07:48:56.416159+0000","last_active":"2024-02-10T07:53:27.354227+0000","last_peered":"2024-02-10T07:53:27.354227+0000","last_clean":"2024-02-10T07:53:27.354227+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:53:27.354227+0000","last_undegraded":"2024-02-10T07:53:27.354227+0000","last_fullsized":"2024-02-10T07:53:27.354227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.416037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.416037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"430'29","reported_seq":966,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.351434+0000","last_change":"2024-02-10T07:49:43.339027+0000","last_active":"2024-02-10T07:53:27.351434+0000","last_peered":"2024-02-10T07:53:27.351434+0000","last_clean":"2024-02-10T07:53:27.351434+0000","last_became_active":"2024-02-10T07:49:06.243569+0000","last_became_peered":"2024-02-10T07:49:06.243569+0000","last_unstale":"2024-02-10T07:53:27.351434+0000","last_undegraded":"2024-02-10T07:53:27.351434+0000","last_fullsized":"2024-02-10T07:53:27.351434+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-02-10T07:49:43.338928+0000","last_deep_scrub":"245'17","last_deep_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_clean_scrub_stamp":"2024-02-10T07:49:43.338928+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":104,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.366455+0000","last_change":"2024-02-10T07:50:43.285105+0000","last_active":"2024-02-10T07:53:27.366455+0000","last_peered":"2024-02-10T07:53:27.366455+0000","last_clean":"2024-02-10T07:53:27.366455+0000","last_became_active":"2024-02-10T07:50:43.284749+0000","last_became_peered":"2024-02-10T07:50:43.284749+0000","last_unstale":"2024-02-10T07:53:27.366455+0000","last_undegraded":"2024-02-10T07:53:27.366455+0000","last_fullsized":"2024-02-10T07:53:27.366455+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:04.308807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.308807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"451'11945","reported_seq":12077,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:28.456234+0000","last_change":"2024-02-10T07:49:53.374953+0000","last_active":"2024-02-10T07:53:28.456234+0000","last_peered":"2024-02-10T07:53:28.456234+0000","last_clean":"2024-02-10T07:53:28.456234+0000","last_became_active":"2024-02-10T07:49:53.374631+0000","last_became_peered":"2024-02-10T07:49:53.374631+0000","last_unstale":"2024-02-10T07:53:28.456234+0000","last_undegraded":"2024-02-10T07:53:28.456234+0000","last_fullsized":"2024-02-10T07:53:28.456234+0000","mapping_epoch":391,"log_start":"426'7100","ondisk_log_start":"426'7100","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4845,"log_dups_size":0,"ondisk_log_size":4845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88276992,"num_objects":1347,"num_object_clones":0,"num_object_copies":4041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11945,"num_write_kb":425344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":872,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.352867+0000","last_change":"2024-02-10T07:50:43.281637+0000","last_active":"2024-02-10T07:53:27.352867+0000","last_peered":"2024-02-10T07:53:27.352867+0000","last_clean":"2024-02-10T07:53:27.352867+0000","last_became_active":"2024-02-10T07:50:43.281262+0000","last_became_peered":"2024-02-10T07:50:43.281262+0000","last_unstale":"2024-02-10T07:53:27.352867+0000","last_undegraded":"2024-02-10T07:53:27.352867+0000","last_fullsized":"2024-02-10T07:53:27.352867+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.361139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":104,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.366948+0000","last_change":"2024-02-10T07:50:43.281022+0000","last_active":"2024-02-10T07:53:27.366948+0000","last_peered":"2024-02-10T07:53:27.366948+0000","last_clean":"2024-02-10T07:53:27.366948+0000","last_became_active":"2024-02-10T07:50:43.280617+0000","last_became_peered":"2024-02-10T07:50:43.280617+0000","last_unstale":"2024-02-10T07:53:27.366948+0000","last_undegraded":"2024-02-10T07:53:27.366948+0000","last_fullsized":"2024-02-10T07:53:27.366948+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:21.172601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:21.172601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":868,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.354250+0000","last_change":"2024-02-10T07:50:07.252577+0000","last_active":"2024-02-10T07:53:27.354250+0000","last_peered":"2024-02-10T07:53:27.354250+0000","last_clean":"2024-02-10T07:53:27.354250+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:53:27.354250+0000","last_undegraded":"2024-02-10T07:53:27.354250+0000","last_fullsized":"2024-02-10T07:53:27.354250+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.252458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.252458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"430'51","reported_seq":949,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.357531+0000","last_change":"2024-02-10T07:51:10.348915+0000","last_active":"2024-02-10T07:53:27.357531+0000","last_peered":"2024-02-10T07:53:27.357531+0000","last_clean":"2024-02-10T07:53:27.357531+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:53:27.357531+0000","last_undegraded":"2024-02-10T07:53:27.357531+0000","last_fullsized":"2024-02-10T07:53:27.357531+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'35","last_scrub_stamp":"2024-02-10T07:51:10.348752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:51:10.348752+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"451'11995","reported_seq":13878,"reported_epoch":451,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-10T07:53:28.444327+0000","last_change":"2024-02-10T07:53:05.273271+0000","last_active":"2024-02-10T07:53:28.444327+0000","last_peered":"2024-02-10T07:53:28.444327+0000","last_clean":"2024-02-10T07:50:42.227612+0000","last_became_active":"2024-02-10T07:50:44.671664+0000","last_became_peered":"2024-02-10T07:50:44.671664+0000","last_unstale":"2024-02-10T07:53:28.444327+0000","last_undegraded":"2024-02-10T07:50:44.289079+0000","last_fullsized":"2024-02-10T07:50:44.263721+0000","mapping_epoch":405,"log_start":"426'7100","ondisk_log_start":"426'7100","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84475904,"num_objects":1289,"num_object_clones":0,"num_object_copies":3867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3622,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11995,"num_write_kb":425088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":875,"num_bytes_recovered":17498112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6],"avail_no_missing":["7","1","6"],"object_location_counts":[{"shards":"6,7","objects":-2333}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":859,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.336706+0000","last_change":"2024-02-10T07:50:43.284817+0000","last_active":"2024-02-10T07:53:25.336706+0000","last_peered":"2024-02-10T07:53:25.336706+0000","last_clean":"2024-02-10T07:53:25.336706+0000","last_became_active":"2024-02-10T07:50:43.284289+0000","last_became_peered":"2024-02-10T07:50:43.284289+0000","last_unstale":"2024-02-10T07:53:25.336706+0000","last_undegraded":"2024-02-10T07:53:25.336706+0000","last_fullsized":"2024-02-10T07:53:25.336706+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:00.336175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.336175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":104,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.367391+0000","last_change":"2024-02-10T07:50:43.295279+0000","last_active":"2024-02-10T07:53:27.367391+0000","last_peered":"2024-02-10T07:53:27.367391+0000","last_clean":"2024-02-10T07:53:27.367391+0000","last_became_active":"2024-02-10T07:50:43.294972+0000","last_became_peered":"2024-02-10T07:50:43.294972+0000","last_unstale":"2024-02-10T07:53:27.367391+0000","last_undegraded":"2024-02-10T07:53:27.367391+0000","last_fullsized":"2024-02-10T07:53:27.367391+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:28.822700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:28.822700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":168,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.330841+0000","last_change":"2024-02-10T07:49:52.968676+0000","last_active":"2024-02-10T07:53:25.330841+0000","last_peered":"2024-02-10T07:53:25.330841+0000","last_clean":"2024-02-10T07:53:25.330841+0000","last_became_active":"2024-02-10T07:49:06.249431+0000","last_became_peered":"2024-02-10T07:49:06.249431+0000","last_unstale":"2024-02-10T07:53:25.330841+0000","last_undegraded":"2024-02-10T07:53:25.330841+0000","last_fullsized":"2024-02-10T07:53:25.330841+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.968567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.968567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"430'21","reported_seq":879,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.336275+0000","last_change":"2024-02-10T07:48:59.304277+0000","last_active":"2024-02-10T07:53:25.336275+0000","last_peered":"2024-02-10T07:53:25.336275+0000","last_clean":"2024-02-10T07:53:25.336275+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:53:25.336275+0000","last_undegraded":"2024-02-10T07:53:25.336275+0000","last_fullsized":"2024-02-10T07:53:25.336275+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:48:59.304199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.304199+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"450'11638","reported_seq":11768,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:26.484567+0000","last_change":"2024-02-10T07:49:53.284745+0000","last_active":"2024-02-10T07:53:26.484567+0000","last_peered":"2024-02-10T07:53:26.484567+0000","last_clean":"2024-02-10T07:53:26.484567+0000","last_became_active":"2024-02-10T07:49:53.283642+0000","last_became_peered":"2024-02-10T07:49:53.283642+0000","last_unstale":"2024-02-10T07:53:26.484567+0000","last_undegraded":"2024-02-10T07:53:26.484567+0000","last_fullsized":"2024-02-10T07:53:26.484567+0000","mapping_epoch":391,"log_start":"422'6700","ondisk_log_start":"422'6700","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4938,"log_dups_size":0,"ondisk_log_size":4938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95158272,"num_objects":1452,"num_object_clones":0,"num_object_copies":4356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11638,"num_write_kb":418880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":169,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.331487+0000","last_change":"2024-02-10T07:49:50.032577+0000","last_active":"2024-02-10T07:53:25.331487+0000","last_peered":"2024-02-10T07:53:25.331487+0000","last_clean":"2024-02-10T07:53:25.331487+0000","last_became_active":"2024-02-10T07:49:06.238367+0000","last_became_peered":"2024-02-10T07:49:06.238367+0000","last_unstale":"2024-02-10T07:53:25.331487+0000","last_undegraded":"2024-02-10T07:53:25.331487+0000","last_fullsized":"2024-02-10T07:53:25.331487+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:50.032505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.032505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":271,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.361117+0000","last_change":"2024-02-10T07:49:10.087939+0000","last_active":"2024-02-10T07:53:27.361117+0000","last_peered":"2024-02-10T07:53:27.361117+0000","last_clean":"2024-02-10T07:53:27.361117+0000","last_became_active":"2024-02-10T07:48:02.328316+0000","last_became_peered":"2024-02-10T07:48:02.328316+0000","last_unstale":"2024-02-10T07:53:27.361117+0000","last_undegraded":"2024-02-10T07:53:27.361117+0000","last_fullsized":"2024-02-10T07:53:27.361117+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:49:10.087795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:49:10.087795+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":844,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.352121+0000","last_change":"2024-02-10T07:49:19.198419+0000","last_active":"2024-02-10T07:53:27.352121+0000","last_peered":"2024-02-10T07:53:27.352121+0000","last_clean":"2024-02-10T07:53:27.352121+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:53:27.352121+0000","last_undegraded":"2024-02-10T07:53:27.352121+0000","last_fullsized":"2024-02-10T07:53:27.352121+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:19.198312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:04.220328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.198312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"430'59","reported_seq":1308,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.361779+0000","last_change":"2024-02-10T07:50:43.327112+0000","last_active":"2024-02-10T07:53:27.361779+0000","last_peered":"2024-02-10T07:53:27.361779+0000","last_clean":"2024-02-10T07:53:27.361779+0000","last_became_active":"2024-02-10T07:50:43.297262+0000","last_became_peered":"2024-02-10T07:50:43.297262+0000","last_unstale":"2024-02-10T07:53:27.361779+0000","last_undegraded":"2024-02-10T07:53:27.361779+0000","last_fullsized":"2024-02-10T07:53:27.361779+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'46","last_scrub_stamp":"2024-02-10T07:49:55.714323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:55.714323+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":850,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.334824+0000","last_change":"2024-02-10T07:49:17.357664+0000","last_active":"2024-02-10T07:53:25.334824+0000","last_peered":"2024-02-10T07:53:25.334824+0000","last_clean":"2024-02-10T07:53:25.334824+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:53:25.334824+0000","last_undegraded":"2024-02-10T07:53:25.334824+0000","last_fullsized":"2024-02-10T07:53:25.334824+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.357533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:22.786949+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.357533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"451'11888","reported_seq":12081,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:28.526670+0000","last_change":"2024-02-10T07:51:07.529876+0000","last_active":"2024-02-10T07:53:28.526670+0000","last_peered":"2024-02-10T07:53:28.526670+0000","last_clean":"2024-02-10T07:53:28.526670+0000","last_became_active":"2024-02-10T07:49:53.374133+0000","last_became_peered":"2024-02-10T07:49:53.374133+0000","last_unstale":"2024-02-10T07:53:28.526670+0000","last_undegraded":"2024-02-10T07:53:28.526670+0000","last_fullsized":"2024-02-10T07:53:28.526670+0000","mapping_epoch":391,"log_start":"418'6300","ondisk_log_start":"418'6300","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'5453","last_scrub_stamp":"2024-02-10T07:51:07.529708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:51:07.529708+0000","objects_scrubbed":5421,"log_size":5588,"log_dups_size":0,"ondisk_log_size":5588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85327872,"num_objects":1302,"num_object_clones":0,"num_object_copies":3906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11888,"num_write_kb":422080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":841,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.354346+0000","last_change":"2024-02-10T07:47:50.081222+0000","last_active":"2024-02-10T07:53:27.354346+0000","last_peered":"2024-02-10T07:53:27.354346+0000","last_clean":"2024-02-10T07:53:27.354346+0000","last_became_active":"2024-02-10T07:47:48.149256+0000","last_became_peered":"2024-02-10T07:47:48.149256+0000","last_unstale":"2024-02-10T07:53:27.354346+0000","last_undegraded":"2024-02-10T07:53:27.354346+0000","last_fullsized":"2024-02-10T07:53:27.354346+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.080714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:42.730875+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.080714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":870,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.360189+0000","last_change":"2024-02-10T07:50:43.288780+0000","last_active":"2024-02-10T07:53:27.360189+0000","last_peered":"2024-02-10T07:53:27.360189+0000","last_clean":"2024-02-10T07:53:27.360189+0000","last_became_active":"2024-02-10T07:50:43.288388+0000","last_became_peered":"2024-02-10T07:50:43.288388+0000","last_unstale":"2024-02-10T07:53:27.360189+0000","last_undegraded":"2024-02-10T07:53:27.360189+0000","last_fullsized":"2024-02-10T07:53:27.360189+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:49.886354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:49.886354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":104,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.366683+0000","last_change":"2024-02-10T07:50:43.287356+0000","last_active":"2024-02-10T07:53:27.366683+0000","last_peered":"2024-02-10T07:53:27.366683+0000","last_clean":"2024-02-10T07:53:27.366683+0000","last_became_active":"2024-02-10T07:50:43.287043+0000","last_became_peered":"2024-02-10T07:50:43.287043+0000","last_unstale":"2024-02-10T07:53:27.366683+0000","last_undegraded":"2024-02-10T07:53:27.366683+0000","last_fullsized":"2024-02-10T07:53:27.366683+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:15.234112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.234112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"430'21","reported_seq":211,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.354461+0000","last_change":"2024-02-10T07:49:04.340445+0000","last_active":"2024-02-10T07:53:27.354461+0000","last_peered":"2024-02-10T07:53:27.354461+0000","last_clean":"2024-02-10T07:53:27.354461+0000","last_became_active":"2024-02-10T07:48:35.057662+0000","last_became_peered":"2024-02-10T07:48:35.057662+0000","last_unstale":"2024-02-10T07:53:27.354461+0000","last_undegraded":"2024-02-10T07:53:27.354461+0000","last_fullsized":"2024-02-10T07:53:27.354461+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'17","last_scrub_stamp":"2024-02-10T07:49:04.340366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.340366+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":104,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.367496+0000","last_change":"2024-02-10T07:50:43.280896+0000","last_active":"2024-02-10T07:53:27.367496+0000","last_peered":"2024-02-10T07:53:27.367496+0000","last_clean":"2024-02-10T07:53:27.367496+0000","last_became_active":"2024-02-10T07:50:43.279125+0000","last_became_peered":"2024-02-10T07:50:43.279125+0000","last_unstale":"2024-02-10T07:53:27.367496+0000","last_undegraded":"2024-02-10T07:53:27.367496+0000","last_fullsized":"2024-02-10T07:53:27.367496+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:50.232544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.232544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"451'11523","reported_seq":11715,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:28.535376+0000","last_change":"2024-02-10T07:51:04.295699+0000","last_active":"2024-02-10T07:53:28.535376+0000","last_peered":"2024-02-10T07:53:28.535376+0000","last_clean":"2024-02-10T07:53:28.535376+0000","last_became_active":"2024-02-10T07:49:53.374185+0000","last_became_peered":"2024-02-10T07:49:53.374185+0000","last_unstale":"2024-02-10T07:53:28.535376+0000","last_undegraded":"2024-02-10T07:53:28.535376+0000","last_fullsized":"2024-02-10T07:53:28.535376+0000","mapping_epoch":391,"log_start":"418'5900","ondisk_log_start":"418'5900","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'5121","last_scrub_stamp":"2024-02-10T07:51:04.295599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:51:04.295599+0000","objects_scrubbed":5106,"log_size":5623,"log_dups_size":0,"ondisk_log_size":5623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88408064,"num_objects":1349,"num_object_clones":0,"num_object_copies":4047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11523,"num_write_kb":411904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":260,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.352813+0000","last_change":"2024-02-10T07:48:02.330253+0000","last_active":"2024-02-10T07:53:27.352813+0000","last_peered":"2024-02-10T07:53:27.352813+0000","last_clean":"2024-02-10T07:53:27.352813+0000","last_became_active":"2024-02-10T07:48:02.329687+0000","last_became_peered":"2024-02-10T07:48:02.329687+0000","last_unstale":"2024-02-10T07:53:27.352813+0000","last_undegraded":"2024-02-10T07:53:27.352813+0000","last_fullsized":"2024-02-10T07:53:27.352813+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:47:49.750508+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.750508+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1097,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353036+0000","last_change":"2024-02-10T07:48:15.033077+0000","last_active":"2024-02-10T07:53:27.353036+0000","last_peered":"2024-02-10T07:53:27.353036+0000","last_clean":"2024-02-10T07:53:27.353036+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:53:27.353036+0000","last_undegraded":"2024-02-10T07:53:27.353036+0000","last_fullsized":"2024-02-10T07:53:27.353036+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:48:15.032944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.032944+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":489,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.358004+0000","last_change":"2024-02-10T07:50:58.025300+0000","last_active":"2024-02-10T07:53:27.358004+0000","last_peered":"2024-02-10T07:53:27.358004+0000","last_clean":"2024-02-10T07:53:27.358004+0000","last_became_active":"2024-02-10T07:49:06.256223+0000","last_became_peered":"2024-02-10T07:49:06.256223+0000","last_unstale":"2024-02-10T07:53:27.358004+0000","last_undegraded":"2024-02-10T07:53:27.358004+0000","last_fullsized":"2024-02-10T07:53:27.358004+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:58.025193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:58.025193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"430'48","reported_seq":1197,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.358033+0000","last_change":"2024-02-10T07:52:03.839258+0000","last_active":"2024-02-10T07:53:27.358033+0000","last_peered":"2024-02-10T07:53:27.358033+0000","last_clean":"2024-02-10T07:53:27.358033+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:53:27.358033+0000","last_undegraded":"2024-02-10T07:53:27.358033+0000","last_fullsized":"2024-02-10T07:53:27.358033+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'48","last_scrub_stamp":"2024-02-10T07:52:03.839174+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:52:03.839174+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":873,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.354660+0000","last_change":"2024-02-10T07:50:43.285613+0000","last_active":"2024-02-10T07:53:27.354660+0000","last_peered":"2024-02-10T07:53:27.354660+0000","last_clean":"2024-02-10T07:53:27.354660+0000","last_became_active":"2024-02-10T07:50:43.284708+0000","last_became_peered":"2024-02-10T07:50:43.284708+0000","last_unstale":"2024-02-10T07:53:27.354660+0000","last_undegraded":"2024-02-10T07:53:27.354660+0000","last_fullsized":"2024-02-10T07:53:27.354660+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:13.369607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.736901+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.369607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"451'11647","reported_seq":11838,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:28.507662+0000","last_change":"2024-02-10T07:52:10.380821+0000","last_active":"2024-02-10T07:53:28.507662+0000","last_peered":"2024-02-10T07:53:28.507662+0000","last_clean":"2024-02-10T07:53:28.507662+0000","last_became_active":"2024-02-10T07:49:53.283492+0000","last_became_peered":"2024-02-10T07:49:53.283492+0000","last_unstale":"2024-02-10T07:53:28.507662+0000","last_undegraded":"2024-02-10T07:53:28.507662+0000","last_fullsized":"2024-02-10T07:53:28.507662+0000","mapping_epoch":391,"log_start":"418'6000","ondisk_log_start":"418'6000","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'7934","last_scrub_stamp":"2024-02-10T07:52:10.380570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:52:10.380570+0000","objects_scrubbed":5089,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:38.547867+0000","scrub_duration":1510,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87097344,"num_objects":1329,"num_object_clones":0,"num_object_copies":3987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11647,"num_write_kb":415232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":644,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.361593+0000","last_change":"2024-02-10T07:50:07.589020+0000","last_active":"2024-02-10T07:53:27.361593+0000","last_peered":"2024-02-10T07:53:27.361593+0000","last_clean":"2024-02-10T07:53:27.361593+0000","last_became_active":"2024-02-10T07:49:06.270457+0000","last_became_peered":"2024-02-10T07:49:06.270457+0000","last_unstale":"2024-02-10T07:53:27.361593+0000","last_undegraded":"2024-02-10T07:53:27.361593+0000","last_fullsized":"2024-02-10T07:53:27.361593+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.588888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.588888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":857,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.334882+0000","last_change":"2024-02-10T07:49:06.267039+0000","last_active":"2024-02-10T07:53:25.334882+0000","last_peered":"2024-02-10T07:53:25.334882+0000","last_clean":"2024-02-10T07:53:25.334882+0000","last_became_active":"2024-02-10T07:49:06.266625+0000","last_became_peered":"2024-02-10T07:49:06.266625+0000","last_unstale":"2024-02-10T07:53:25.334882+0000","last_undegraded":"2024-02-10T07:53:25.334882+0000","last_fullsized":"2024-02-10T07:53:25.334882+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_clean_scrub_stamp":"2024-02-10T07:47:41.181770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2162,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.352990+0000","last_change":"2024-02-10T07:50:04.159154+0000","last_active":"2024-02-10T07:53:27.352990+0000","last_peered":"2024-02-10T07:53:27.352990+0000","last_clean":"2024-02-10T07:53:27.352990+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:53:27.352990+0000","last_undegraded":"2024-02-10T07:53:27.352990+0000","last_fullsized":"2024-02-10T07:53:27.352990+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:50:04.159046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.159046+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1290,"num_read_kb":1285,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"430'42","reported_seq":945,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.336565+0000","last_change":"2024-02-10T07:50:43.324808+0000","last_active":"2024-02-10T07:53:25.336565+0000","last_peered":"2024-02-10T07:53:25.336565+0000","last_clean":"2024-02-10T07:53:25.336565+0000","last_became_active":"2024-02-10T07:50:43.291771+0000","last_became_peered":"2024-02-10T07:50:43.291771+0000","last_unstale":"2024-02-10T07:53:25.336565+0000","last_undegraded":"2024-02-10T07:53:25.336565+0000","last_fullsized":"2024-02-10T07:53:25.336565+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:47.417096+0000","last_deep_scrub":"237'18","last_deep_scrub_stamp":"2024-02-10T07:40:31.005376+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.417096+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":814,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.379881+0000","last_change":"2024-02-10T07:52:10.066440+0000","last_active":"2024-02-10T07:53:27.379881+0000","last_peered":"2024-02-10T07:53:27.379881+0000","last_clean":"2024-02-10T07:53:27.379881+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:53:27.379881+0000","last_undegraded":"2024-02-10T07:53:27.379881+0000","last_fullsized":"2024-02-10T07:53:27.379881+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:10.066284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:52:10.066284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:30.002155+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"451'11642","reported_seq":20435,"reported_epoch":451,"state":"active+recovery_wait","last_fresh":"2024-02-10T07:53:29.921941+0000","last_change":"2024-02-10T07:53:25.400523+0000","last_active":"2024-02-10T07:53:29.921941+0000","last_peered":"2024-02-10T07:53:29.921941+0000","last_clean":"2024-02-10T07:50:42.244817+0000","last_became_active":"2024-02-10T07:53:25.355044+0000","last_became_peered":"2024-02-10T07:53:25.355044+0000","last_unstale":"2024-02-10T07:53:29.921941+0000","last_undegraded":"2024-02-10T07:53:29.921941+0000","last_fullsized":"2024-02-10T07:53:29.921941+0000","mapping_epoch":449,"log_start":"425'6800","ondisk_log_start":"425'6800","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4842,"log_dups_size":0,"ondisk_log_size":4842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83230720,"num_objects":1270,"num_object_clones":0,"num_object_copies":3810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11642,"num_write_kb":413184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4326,"num_bytes_recovered":234225664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":["1","2","4","7"],"object_location_counts":[{"shards":"1,2,4","objects":1270}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":866,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.360771+0000","last_change":"2024-02-10T07:50:43.296823+0000","last_active":"2024-02-10T07:53:27.360771+0000","last_peered":"2024-02-10T07:53:27.360771+0000","last_clean":"2024-02-10T07:53:27.360771+0000","last_became_active":"2024-02-10T07:50:43.296488+0000","last_became_peered":"2024-02-10T07:50:43.296488+0000","last_unstale":"2024-02-10T07:53:27.360771+0000","last_undegraded":"2024-02-10T07:53:27.360771+0000","last_fullsized":"2024-02-10T07:53:27.360771+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.630083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.630083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":871,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.363319+0000","last_change":"2024-02-10T07:50:43.298468+0000","last_active":"2024-02-10T07:53:27.363319+0000","last_peered":"2024-02-10T07:53:27.363319+0000","last_clean":"2024-02-10T07:53:27.363319+0000","last_became_active":"2024-02-10T07:50:43.297861+0000","last_became_peered":"2024-02-10T07:50:43.297861+0000","last_unstale":"2024-02-10T07:53:27.363319+0000","last_undegraded":"2024-02-10T07:53:27.363319+0000","last_fullsized":"2024-02-10T07:53:27.363319+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:05.193158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:49:05.193158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"430'47","reported_seq":929,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.336013+0000","last_change":"2024-02-10T07:49:57.522503+0000","last_active":"2024-02-10T07:53:25.336013+0000","last_peered":"2024-02-10T07:53:25.336013+0000","last_clean":"2024-02-10T07:53:25.336013+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:53:25.336013+0000","last_undegraded":"2024-02-10T07:53:25.336013+0000","last_fullsized":"2024-02-10T07:53:25.336013+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'37","last_scrub_stamp":"2024-02-10T07:49:57.522391+0000","last_deep_scrub":"271'37","last_deep_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.522391+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":882,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.357621+0000","last_change":"2024-02-10T07:51:02.173651+0000","last_active":"2024-02-10T07:53:27.357621+0000","last_peered":"2024-02-10T07:53:27.357621+0000","last_clean":"2024-02-10T07:53:27.357621+0000","last_became_active":"2024-02-10T07:48:35.062118+0000","last_became_peered":"2024-02-10T07:48:35.062118+0000","last_unstale":"2024-02-10T07:53:27.357621+0000","last_undegraded":"2024-02-10T07:53:27.357621+0000","last_fullsized":"2024-02-10T07:53:27.357621+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:51:02.173535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:00.292320+0000","last_clean_scrub_stamp":"2024-02-10T07:51:02.173535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"451'11816","reported_seq":12411,"reported_epoch":451,"state":"active+recovering+remapped","last_fresh":"2024-02-10T07:53:29.912670+0000","last_change":"2024-02-10T07:53:23.626288+0000","last_active":"2024-02-10T07:53:29.912670+0000","last_peered":"2024-02-10T07:53:29.912670+0000","last_clean":"2024-02-10T07:50:42.239377+0000","last_became_active":"2024-02-10T07:50:44.668375+0000","last_became_peered":"2024-02-10T07:50:44.668375+0000","last_unstale":"2024-02-10T07:53:29.912670+0000","last_undegraded":"2024-02-10T07:53:29.912670+0000","last_fullsized":"2024-02-10T07:53:29.912670+0000","mapping_epoch":405,"log_start":"426'6900","ondisk_log_start":"426'6900","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4916,"log_dups_size":0,"ondisk_log_size":4916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80150552,"num_objects":1224,"num_object_clones":0,"num_object_copies":3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4191,"num_objects_unfound":0,"num_objects_dirty":1224,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11816,"num_write_kb":417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":-2967}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":743,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.357236+0000","last_change":"2024-02-10T07:52:23.855559+0000","last_active":"2024-02-10T07:53:27.357236+0000","last_peered":"2024-02-10T07:53:27.357236+0000","last_clean":"2024-02-10T07:53:27.357236+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:53:27.357236+0000","last_undegraded":"2024-02-10T07:53:27.357236+0000","last_fullsized":"2024-02-10T07:53:27.357236+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:23.855477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:23.855477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:31.864541+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":104,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.378609+0000","last_change":"2024-02-10T07:50:43.289736+0000","last_active":"2024-02-10T07:53:27.378609+0000","last_peered":"2024-02-10T07:53:27.378609+0000","last_clean":"2024-02-10T07:53:27.378609+0000","last_became_active":"2024-02-10T07:50:43.289458+0000","last_became_peered":"2024-02-10T07:50:43.289458+0000","last_unstale":"2024-02-10T07:53:27.378609+0000","last_undegraded":"2024-02-10T07:53:27.378609+0000","last_fullsized":"2024-02-10T07:53:27.378609+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:48.809513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:48.809513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"430'35","reported_seq":676,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.361443+0000","last_change":"2024-02-10T07:49:19.084010+0000","last_active":"2024-02-10T07:53:27.361443+0000","last_peered":"2024-02-10T07:53:27.361443+0000","last_clean":"2024-02-10T07:53:27.361443+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:53:27.361443+0000","last_undegraded":"2024-02-10T07:53:27.361443+0000","last_fullsized":"2024-02-10T07:53:27.361443+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'25","last_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_deep_scrub":"382'25","last_deep_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.083856+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"158'10","reported_seq":1119,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.336374+0000","last_change":"2024-02-10T07:49:50.396822+0000","last_active":"2024-02-10T07:53:25.336374+0000","last_peered":"2024-02-10T07:53:25.336374+0000","last_clean":"2024-02-10T07:53:25.336374+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:53:25.336374+0000","last_undegraded":"2024-02-10T07:53:25.336374+0000","last_fullsized":"2024-02-10T07:53:25.336374+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:49:50.396730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.396730+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.e","version":"451'11764","reported_seq":11896,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:28.478359+0000","last_change":"2024-02-10T07:49:53.281229+0000","last_active":"2024-02-10T07:53:28.478359+0000","last_peered":"2024-02-10T07:53:28.478359+0000","last_clean":"2024-02-10T07:53:28.478359+0000","last_became_active":"2024-02-10T07:49:53.280920+0000","last_became_peered":"2024-02-10T07:49:53.280920+0000","last_unstale":"2024-02-10T07:53:28.478359+0000","last_undegraded":"2024-02-10T07:53:28.478359+0000","last_fullsized":"2024-02-10T07:53:28.478359+0000","mapping_epoch":391,"log_start":"426'6900","ondisk_log_start":"426'6900","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87556096,"num_objects":1336,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11764,"num_write_kb":419200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":852,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353701+0000","last_change":"2024-02-10T07:50:43.283505+0000","last_active":"2024-02-10T07:53:27.353701+0000","last_peered":"2024-02-10T07:53:27.353701+0000","last_clean":"2024-02-10T07:53:27.353701+0000","last_became_active":"2024-02-10T07:50:43.283161+0000","last_became_peered":"2024-02-10T07:50:43.283161+0000","last_unstale":"2024-02-10T07:53:27.353701+0000","last_undegraded":"2024-02-10T07:53:27.353701+0000","last_fullsized":"2024-02-10T07:53:27.353701+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_clean_scrub_stamp":"2024-02-10T07:48:23.271454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"296'2","reported_seq":859,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.336628+0000","last_change":"2024-02-10T07:50:03.643473+0000","last_active":"2024-02-10T07:53:25.336628+0000","last_peered":"2024-02-10T07:53:25.336628+0000","last_clean":"2024-02-10T07:53:25.336628+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:53:25.336628+0000","last_undegraded":"2024-02-10T07:53:25.336628+0000","last_fullsized":"2024-02-10T07:53:25.336628+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'2","last_scrub_stamp":"2024-02-10T07:50:03.643367+0000","last_deep_scrub":"296'2","last_deep_scrub_stamp":"2024-02-10T07:45:03.233375+0000","last_clean_scrub_stamp":"2024-02-10T07:50:03.643367+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"430'42","reported_seq":912,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.351237+0000","last_change":"2024-02-10T07:49:20.216865+0000","last_active":"2024-02-10T07:53:27.351237+0000","last_peered":"2024-02-10T07:53:27.351237+0000","last_clean":"2024-02-10T07:53:27.351237+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:53:27.351237+0000","last_undegraded":"2024-02-10T07:53:27.351237+0000","last_fullsized":"2024-02-10T07:53:27.351237+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-02-10T07:49:20.216750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:20.216750+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"376'12","reported_seq":909,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.359924+0000","last_change":"2024-02-10T07:49:06.282456+0000","last_active":"2024-02-10T07:53:27.359924+0000","last_peered":"2024-02-10T07:53:27.359924+0000","last_clean":"2024-02-10T07:53:27.359924+0000","last_became_active":"2024-02-10T07:49:06.255324+0000","last_became_peered":"2024-02-10T07:49:06.255324+0000","last_unstale":"2024-02-10T07:53:27.359924+0000","last_undegraded":"2024-02-10T07:53:27.359924+0000","last_fullsized":"2024-02-10T07:53:27.359924+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'11","last_scrub_stamp":"2024-02-10T07:48:58.368663+0000","last_deep_scrub":"227'9","last_deep_scrub_stamp":"2024-02-10T07:39:53.440221+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.368663+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"451'11636","reported_seq":13018,"reported_epoch":451,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-10T07:53:28.469214+0000","last_change":"2024-02-10T07:53:14.380137+0000","last_active":"2024-02-10T07:53:28.469214+0000","last_peered":"2024-02-10T07:53:28.469214+0000","last_clean":"2024-02-10T07:50:42.258059+0000","last_became_active":"2024-02-10T07:50:44.666535+0000","last_became_peered":"2024-02-10T07:50:44.666535+0000","last_unstale":"2024-02-10T07:53:28.469214+0000","last_undegraded":"2024-02-10T07:50:44.288061+0000","last_fullsized":"2024-02-10T07:50:44.263325+0000","mapping_epoch":405,"log_start":"423'6700","ondisk_log_start":"423'6700","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84279296,"num_objects":1286,"num_object_clones":0,"num_object_copies":3858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11636,"num_write_kb":413504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":623,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"1,7","objects":-2455}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":849,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.351693+0000","last_change":"2024-02-10T07:50:43.282245+0000","last_active":"2024-02-10T07:53:27.351693+0000","last_peered":"2024-02-10T07:53:27.351693+0000","last_clean":"2024-02-10T07:53:27.351693+0000","last_became_active":"2024-02-10T07:50:43.281465+0000","last_became_peered":"2024-02-10T07:50:43.281465+0000","last_unstale":"2024-02-10T07:53:27.351693+0000","last_undegraded":"2024-02-10T07:53:27.351693+0000","last_fullsized":"2024-02-10T07:53:27.351693+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:27.819612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:46:27.819612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":333,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.354529+0000","last_change":"2024-02-10T07:47:50.948710+0000","last_active":"2024-02-10T07:53:27.354529+0000","last_peered":"2024-02-10T07:53:27.354529+0000","last_clean":"2024-02-10T07:53:27.354529+0000","last_became_active":"2024-02-10T07:47:47.966443+0000","last_became_peered":"2024-02-10T07:47:47.966443+0000","last_unstale":"2024-02-10T07:53:27.354529+0000","last_undegraded":"2024-02-10T07:53:27.354529+0000","last_fullsized":"2024-02-10T07:53:27.354529+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.948624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.948624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"430'42","reported_seq":851,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353325+0000","last_change":"2024-02-10T07:49:57.303310+0000","last_active":"2024-02-10T07:53:27.353325+0000","last_peered":"2024-02-10T07:53:27.353325+0000","last_clean":"2024-02-10T07:53:27.353325+0000","last_became_active":"2024-02-10T07:49:06.252935+0000","last_became_peered":"2024-02-10T07:49:06.252935+0000","last_unstale":"2024-02-10T07:53:27.353325+0000","last_undegraded":"2024-02-10T07:53:27.353325+0000","last_fullsized":"2024-02-10T07:53:27.353325+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:57.303200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.303200+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":851,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.350979+0000","last_change":"2024-02-10T07:49:06.239655+0000","last_active":"2024-02-10T07:53:27.350979+0000","last_peered":"2024-02-10T07:53:27.350979+0000","last_clean":"2024-02-10T07:53:27.350979+0000","last_became_active":"2024-02-10T07:49:06.239190+0000","last_became_peered":"2024-02-10T07:49:06.239190+0000","last_unstale":"2024-02-10T07:53:27.350979+0000","last_undegraded":"2024-02-10T07:53:27.350979+0000","last_fullsized":"2024-02-10T07:53:27.350979+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:42.549018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.549018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"451'11860","reported_seq":11991,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:29.902543+0000","last_change":"2024-02-10T07:49:53.282174+0000","last_active":"2024-02-10T07:53:29.902543+0000","last_peered":"2024-02-10T07:53:29.902543+0000","last_clean":"2024-02-10T07:53:29.902543+0000","last_became_active":"2024-02-10T07:49:53.281832+0000","last_became_peered":"2024-02-10T07:49:53.281832+0000","last_unstale":"2024-02-10T07:53:29.902543+0000","last_undegraded":"2024-02-10T07:53:29.902543+0000","last_fullsized":"2024-02-10T07:53:29.902543+0000","mapping_epoch":391,"log_start":"426'7000","ondisk_log_start":"426'7000","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4860,"log_dups_size":0,"ondisk_log_size":4860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83230720,"num_objects":1270,"num_object_clones":0,"num_object_copies":3810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11860,"num_write_kb":420160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":358,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.377418+0000","last_change":"2024-02-10T07:49:06.274374+0000","last_active":"2024-02-10T07:53:27.377418+0000","last_peered":"2024-02-10T07:53:27.377418+0000","last_clean":"2024-02-10T07:53:27.377418+0000","last_became_active":"2024-02-10T07:48:02.371926+0000","last_became_peered":"2024-02-10T07:48:02.371926+0000","last_unstale":"2024-02-10T07:53:27.377418+0000","last_undegraded":"2024-02-10T07:53:27.377418+0000","last_fullsized":"2024-02-10T07:53:27.377418+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.274263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:49.704024+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.274263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"395'12","reported_seq":265,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.357611+0000","last_change":"2024-02-10T07:52:24.850402+0000","last_active":"2024-02-10T07:53:27.357611+0000","last_peered":"2024-02-10T07:53:27.357611+0000","last_clean":"2024-02-10T07:53:27.357611+0000","last_became_active":"2024-02-10T07:48:35.071916+0000","last_became_peered":"2024-02-10T07:48:35.071916+0000","last_unstale":"2024-02-10T07:53:27.357611+0000","last_undegraded":"2024-02-10T07:53:27.357611+0000","last_fullsized":"2024-02-10T07:53:27.357611+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'12","last_scrub_stamp":"2024-02-10T07:52:24.850269+0000","last_deep_scrub":"395'12","last_deep_scrub_stamp":"2024-02-10T07:52:24.850269+0000","last_clean_scrub_stamp":"2024-02-10T07:52:24.850269+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:39.041346+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":169,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.332122+0000","last_change":"2024-02-10T07:49:44.962126+0000","last_active":"2024-02-10T07:53:25.332122+0000","last_peered":"2024-02-10T07:53:25.332122+0000","last_clean":"2024-02-10T07:53:25.332122+0000","last_became_active":"2024-02-10T07:49:06.251038+0000","last_became_peered":"2024-02-10T07:49:06.251038+0000","last_unstale":"2024-02-10T07:53:25.332122+0000","last_undegraded":"2024-02-10T07:53:25.332122+0000","last_fullsized":"2024-02-10T07:53:25.332122+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:44.962041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:44.962041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"430'39","reported_seq":923,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.362291+0000","last_change":"2024-02-10T07:50:00.655466+0000","last_active":"2024-02-10T07:53:27.362291+0000","last_peered":"2024-02-10T07:53:27.362291+0000","last_clean":"2024-02-10T07:53:27.362291+0000","last_became_active":"2024-02-10T07:47:47.967427+0000","last_became_peered":"2024-02-10T07:47:47.967427+0000","last_unstale":"2024-02-10T07:53:27.362291+0000","last_undegraded":"2024-02-10T07:53:27.362291+0000","last_fullsized":"2024-02-10T07:53:27.362291+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:50:00.655377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.655377+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"451'12034","reported_seq":12232,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:28.541374+0000","last_change":"2024-02-10T07:51:08.956873+0000","last_active":"2024-02-10T07:53:28.541374+0000","last_peered":"2024-02-10T07:53:28.541374+0000","last_clean":"2024-02-10T07:53:28.541374+0000","last_became_active":"2024-02-10T07:49:53.283415+0000","last_became_peered":"2024-02-10T07:49:53.283415+0000","last_unstale":"2024-02-10T07:53:28.541374+0000","last_undegraded":"2024-02-10T07:53:28.541374+0000","last_fullsized":"2024-02-10T07:53:28.541374+0000","mapping_epoch":391,"log_start":"418'6400","ondisk_log_start":"418'6400","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'5587","last_scrub_stamp":"2024-02-10T07:51:08.956707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:51:08.956707+0000","objects_scrubbed":5548,"log_size":5634,"log_dups_size":0,"ondisk_log_size":5634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86114304,"num_objects":1314,"num_object_clones":0,"num_object_copies":3942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12034,"num_write_kb":427136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":552,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.357739+0000","last_change":"2024-02-10T07:51:59.948039+0000","last_active":"2024-02-10T07:53:27.357739+0000","last_peered":"2024-02-10T07:53:27.357739+0000","last_clean":"2024-02-10T07:53:27.357739+0000","last_became_active":"2024-02-10T07:49:06.268874+0000","last_became_peered":"2024-02-10T07:49:06.268874+0000","last_unstale":"2024-02-10T07:53:27.357739+0000","last_undegraded":"2024-02-10T07:53:27.357739+0000","last_fullsized":"2024-02-10T07:53:27.357739+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:51:59.947908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_clean_scrub_stamp":"2024-02-10T07:51:59.947908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":332,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353174+0000","last_change":"2024-02-10T07:49:57.310447+0000","last_active":"2024-02-10T07:53:27.353174+0000","last_peered":"2024-02-10T07:53:27.353174+0000","last_clean":"2024-02-10T07:53:27.353174+0000","last_became_active":"2024-02-10T07:48:02.333689+0000","last_became_peered":"2024-02-10T07:48:02.333689+0000","last_unstale":"2024-02-10T07:53:27.353174+0000","last_undegraded":"2024-02-10T07:53:27.353174+0000","last_fullsized":"2024-02-10T07:53:27.353174+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:57.310333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.310333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":855,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.351070+0000","last_change":"2024-02-10T07:50:43.283476+0000","last_active":"2024-02-10T07:53:27.351070+0000","last_peered":"2024-02-10T07:53:27.351070+0000","last_clean":"2024-02-10T07:53:27.351070+0000","last_became_active":"2024-02-10T07:50:43.283104+0000","last_became_peered":"2024-02-10T07:50:43.283104+0000","last_unstale":"2024-02-10T07:53:27.351070+0000","last_undegraded":"2024-02-10T07:53:27.351070+0000","last_fullsized":"2024-02-10T07:53:27.351070+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:37.995362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.995362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"430'32","reported_seq":941,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.352934+0000","last_change":"2024-02-10T07:49:16.249257+0000","last_active":"2024-02-10T07:53:27.352934+0000","last_peered":"2024-02-10T07:53:27.352934+0000","last_clean":"2024-02-10T07:53:27.352934+0000","last_became_active":"2024-02-10T07:48:35.062573+0000","last_became_peered":"2024-02-10T07:48:35.062573+0000","last_unstale":"2024-02-10T07:53:27.352934+0000","last_undegraded":"2024-02-10T07:53:27.352934+0000","last_fullsized":"2024-02-10T07:53:27.352934+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:49:16.248936+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.248936+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"451'11739","reported_seq":20693,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:29.039724+0000","last_change":"2024-02-10T07:53:16.355834+0000","last_active":"2024-02-10T07:53:29.039724+0000","last_peered":"2024-02-10T07:53:29.039724+0000","last_clean":"2024-02-10T07:53:29.039724+0000","last_became_active":"2024-02-10T07:53:16.292628+0000","last_became_peered":"2024-02-10T07:53:16.292628+0000","last_unstale":"2024-02-10T07:53:29.039724+0000","last_undegraded":"2024-02-10T07:53:29.039724+0000","last_fullsized":"2024-02-10T07:53:29.039724+0000","mapping_epoch":446,"log_start":"410'5300","ondisk_log_start":"410'5300","created":391,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":6439,"log_dups_size":0,"ondisk_log_size":6439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85917696,"num_objects":1311,"num_object_clones":0,"num_object_copies":3933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11739,"num_write_kb":417600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4405,"num_bytes_recovered":265551872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":104,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.368607+0000","last_change":"2024-02-10T07:50:43.295417+0000","last_active":"2024-02-10T07:53:27.368607+0000","last_peered":"2024-02-10T07:53:27.368607+0000","last_clean":"2024-02-10T07:53:27.368607+0000","last_became_active":"2024-02-10T07:50:43.290295+0000","last_became_peered":"2024-02-10T07:50:43.290295+0000","last_unstale":"2024-02-10T07:53:27.368607+0000","last_undegraded":"2024-02-10T07:53:27.368607+0000","last_fullsized":"2024-02-10T07:53:27.368607+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.153368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.153368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":841,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.361879+0000","last_change":"2024-02-10T07:49:07.170105+0000","last_active":"2024-02-10T07:53:27.361879+0000","last_peered":"2024-02-10T07:53:27.361879+0000","last_clean":"2024-02-10T07:53:27.361879+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:53:27.361879+0000","last_undegraded":"2024-02-10T07:53:27.361879+0000","last_fullsized":"2024-02-10T07:53:27.361879+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:07.169962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:07.169962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":882,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.357753+0000","last_change":"2024-02-10T07:52:26.772420+0000","last_active":"2024-02-10T07:53:27.357753+0000","last_peered":"2024-02-10T07:53:27.357753+0000","last_clean":"2024-02-10T07:53:27.357753+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:53:27.357753+0000","last_undegraded":"2024-02-10T07:53:27.357753+0000","last_fullsized":"2024-02-10T07:53:27.357753+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:26.772322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:03.803361+0000","last_clean_scrub_stamp":"2024-02-10T07:52:26.772322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:55.097614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"430'53","reported_seq":989,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353092+0000","last_change":"2024-02-10T07:50:43.316983+0000","last_active":"2024-02-10T07:53:27.353092+0000","last_peered":"2024-02-10T07:53:27.353092+0000","last_clean":"2024-02-10T07:53:27.353092+0000","last_became_active":"2024-02-10T07:50:43.281879+0000","last_became_peered":"2024-02-10T07:50:43.281879+0000","last_unstale":"2024-02-10T07:53:27.353092+0000","last_undegraded":"2024-02-10T07:53:27.353092+0000","last_fullsized":"2024-02-10T07:53:27.353092+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T07:48:59.066256+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.066256+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"451'11975","reported_seq":12169,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:28.536292+0000","last_change":"2024-02-10T07:51:05.662504+0000","last_active":"2024-02-10T07:53:28.536292+0000","last_peered":"2024-02-10T07:53:28.536292+0000","last_clean":"2024-02-10T07:53:28.536292+0000","last_became_active":"2024-02-10T07:49:53.280975+0000","last_became_peered":"2024-02-10T07:49:53.280975+0000","last_unstale":"2024-02-10T07:53:28.536292+0000","last_undegraded":"2024-02-10T07:53:28.536292+0000","last_fullsized":"2024-02-10T07:53:28.536292+0000","mapping_epoch":391,"log_start":"419'6400","ondisk_log_start":"419'6400","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'5413","last_scrub_stamp":"2024-02-10T07:51:05.662310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:51:05.662310+0000","objects_scrubbed":5394,"log_size":5575,"log_dups_size":0,"ondisk_log_size":5575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86441984,"num_objects":1319,"num_object_clones":0,"num_object_copies":3957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":425408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":843,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353526+0000","last_change":"2024-02-10T07:50:43.290883+0000","last_active":"2024-02-10T07:53:27.353526+0000","last_peered":"2024-02-10T07:53:27.353526+0000","last_clean":"2024-02-10T07:53:27.353526+0000","last_became_active":"2024-02-10T07:50:43.290334+0000","last_became_peered":"2024-02-10T07:50:43.290334+0000","last_unstale":"2024-02-10T07:53:27.353526+0000","last_undegraded":"2024-02-10T07:53:27.353526+0000","last_fullsized":"2024-02-10T07:53:27.353526+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:47.275882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.275882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"405'12","reported_seq":142,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.378752+0000","last_change":"2024-02-10T07:50:43.294666+0000","last_active":"2024-02-10T07:53:27.378752+0000","last_peered":"2024-02-10T07:53:27.378752+0000","last_clean":"2024-02-10T07:53:27.378752+0000","last_became_active":"2024-02-10T07:50:43.280142+0000","last_became_peered":"2024-02-10T07:50:43.280142+0000","last_unstale":"2024-02-10T07:53:27.378752+0000","last_undegraded":"2024-02-10T07:53:27.378752+0000","last_fullsized":"2024-02-10T07:53:27.378752+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'11","last_scrub_stamp":"2024-02-10T07:49:14.335064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:14.335064+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":346,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-02-10T07:53:25.336191+0000","last_change":"2024-02-10T07:53:20.474601+0000","last_active":"2024-02-10T07:53:25.336191+0000","last_peered":"2024-02-10T07:53:25.336191+0000","last_clean":"2024-02-10T07:53:25.336191+0000","last_became_active":"2024-02-10T07:49:06.258926+0000","last_became_peered":"2024-02-10T07:49:06.258926+0000","last_unstale":"2024-02-10T07:53:25.336191+0000","last_undegraded":"2024-02-10T07:53:25.336191+0000","last_fullsized":"2024-02-10T07:53:25.336191+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:20.474497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_clean_scrub_stamp":"2024-02-10T07:53:20.474497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:48.442368+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"430'30","reported_seq":923,"reported_epoch":451,"state":"active+clean","last_fresh":"2024-02-10T07:53:27.353665+0000","last_change":"2024-02-10T07:49:06.289632+0000","last_active":"2024-02-10T07:53:27.353665+0000","last_peered":"2024-02-10T07:53:27.353665+0000","last_clean":"2024-02-10T07:53:27.353665+0000","last_became_active":"2024-02-10T07:49:06.238858+0000","last_became_peered":"2024-02-10T07:49:06.238858+0000","last_unstale":"2024-02-10T07:53:27.353665+0000","last_undegraded":"2024-02-10T07:53:27.353665+0000","last_fullsized":"2024-02-10T07:53:27.353665+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:48:15.588199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.588199+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1384120344,"num_objects":21121,"num_object_clones":0,"num_object_copies":63363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7363,"num_objects_misplaced":4191,"num_objects_unfound":0,"num_objects_dirty":21121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188399,"num_write_kb":6704577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14802,"num_bytes_recovered":801964032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5444091904,"data_stored":5444075616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84299,"ondisk_log_size":84299,"up":48,"acting":46,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1290,"num_read_kb":1285,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1540,"num_read_kb":1297,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":6382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1060,"ondisk_log_size":1060,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038024,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1854708,"kb_used_data":1057192,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":91890444,"statfs":{"total":95995035648,"available":94095814656,"internally_reserved":0,"allocated":1082564608,"data_stored":1082070668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":816642506},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":3.0030000000000001,"5min":1.8480000000000001,"15min":1.6899999999999999},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":18.202999999999999,"5min":39.494,"15min":39.494},"last":0.48499999999999999},{"interface":"front","average":{"1min":2.968,"5min":1.8959999999999999,"15min":1.7030000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.23000000000000001},"max":{"1min":19.745000000000001,"5min":38.524000000000001,"15min":38.524000000000001},"last":0.442}]},{"osd":1,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":6.6909999999999998,"5min":2.4009999999999998,"15min":2.069},"min":{"1min":0.33400000000000002,"5min":0.251,"15min":0.24399999999999999},"max":{"1min":50.149000000000001,"5min":50.149000000000001,"15min":50.149000000000001},"last":1.105},{"interface":"front","average":{"1min":6.5720000000000001,"5min":2.431,"15min":2.0190000000000001},"min":{"1min":0.33100000000000002,"5min":0.26300000000000001,"15min":0.24399999999999999},"max":{"1min":54.048000000000002,"5min":54.048000000000002,"15min":54.048000000000002},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":1.25,"15min":1.141},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.255},"max":{"1min":7.0019999999999998,"5min":7.0019999999999998,"15min":18.57},"last":0.53200000000000003},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.4510000000000001,"15min":1.2989999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":7.8410000000000002,"5min":8.3949999999999996,"15min":18.521000000000001},"last":1.071}]},{"osd":3,"last update":"Sat Feb 10 07:53:02 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.264,"15min":1.1220000000000001},"min":{"1min":0.379,"5min":0.307,"15min":0.307},"max":{"1min":4.7590000000000003,"5min":4.7590000000000003,"15min":4.7590000000000003},"last":0.499},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.198,"15min":1.014},"min":{"1min":0.40799999999999997,"5min":0.313,"15min":0.313},"max":{"1min":6.2839999999999998,"5min":6.2839999999999998,"15min":6.2839999999999998},"last":2.1219999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.347,"15min":1.1719999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.22},"max":{"1min":1.5960000000000001,"5min":26.001000000000001,"15min":26.001000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.45700000000000002,"5min":1.3089999999999999,"15min":1.1539999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.94199999999999995,"5min":27.09,"15min":27.09},"last":0.34499999999999997}]},{"osd":5,"last update":"Sat Feb 10 07:53:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.91700000000000004,"15min":1.0389999999999999},"min":{"1min":0.29299999999999998,"5min":0.26600000000000001,"15min":0.245},"max":{"1min":1.6830000000000001,"5min":6.1820000000000004,"15min":18.126999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.1279999999999999,"15min":1.1659999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.9510000000000001,"5min":6.9219999999999997,"15min":19.548999999999999},"last":0.71499999999999997}]},{"osd":6,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":1.1599999999999999,"15min":1.159},"min":{"1min":0.30299999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.0409999999999999,"5min":7.4329999999999998,"15min":17.832000000000001},"last":0.75},{"interface":"front","average":{"1min":0.56000000000000005,"5min":1.077,"15min":1.179},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.29799999999999999},"max":{"1min":1.1719999999999999,"5min":6.101,"15min":20.449999999999999},"last":0.42499999999999999}]}]},{"osd":6,"up_from":50,"seq":214748365069,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1342428,"kb_used_data":689296,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":92402724,"statfs":{"total":95995035648,"available":94620389376,"internally_reserved":0,"allocated":705839104,"data_stored":705368781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":668793286},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":2.4039999999999999,"5min":2.0030000000000001,"15min":1.6850000000000001},"min":{"1min":0.249,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.166,"5min":19.484999999999999,"15min":24.727},"last":0.249},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.395,"15min":1.373},"min":{"1min":0.27000000000000002,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":5.1870000000000003,"5min":14.808,"15min":29.084},"last":1.232}]},{"osd":1,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":1.669,"15min":1.8520000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23799999999999999},"max":{"1min":9.923,"5min":15.007999999999999,"15min":37.659999999999997},"last":0.375},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.599,"15min":1.887},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.253},"max":{"1min":4,"5min":19.524999999999999,"15min":37.75},"last":0.311}]},{"osd":2,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":1.6579999999999999,"15min":1.3320000000000001},"min":{"1min":0.30199999999999999,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":6.25,"5min":15.768000000000001,"15min":15.768000000000001},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.714,"5min":1.456,"15min":1.167},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.28399999999999997},"max":{"1min":9.4540000000000006,"5min":15.728999999999999,"15min":15.728999999999999},"last":1.9990000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.399,"15min":1.637},"min":{"1min":0.33100000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.6500000000000004,"5min":13.279,"15min":13.279},"last":1.8069999999999999},{"interface":"front","average":{"1min":1.9850000000000001,"5min":1.2889999999999999,"15min":1.2589999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":7.7949999999999999,"5min":7.7949999999999999,"15min":7.7949999999999999},"last":0.41199999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.84299999999999997,"15min":0.84899999999999998},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.95999999999999996,"5min":7.6509999999999998,"15min":7.6509999999999998},"last":1.6259999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.997,"15min":0.88600000000000001},"min":{"1min":0.28699999999999998,"5min":0.27700000000000002,"15min":0.26000000000000001},"max":{"1min":2.198,"5min":5.7050000000000001,"15min":7.0300000000000002},"last":0.63800000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:53:03 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.70199999999999996,"15min":1.0640000000000001},"min":{"1min":0.32600000000000001,"5min":0.28799999999999998,"15min":0.24099999999999999},"max":{"1min":0.91500000000000004,"5min":3.5699999999999998,"15min":44.689},"last":1.6719999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.1439999999999999,"15min":1.2170000000000001},"min":{"1min":0.32100000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.129,"5min":8.9420000000000002,"15min":44.465000000000003},"last":0.32700000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:53:15 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.121,"15min":1.3089999999999999},"min":{"1min":0.314,"5min":0.29999999999999999,"15min":0.28100000000000003},"max":{"1min":2.9449999999999998,"5min":4.9379999999999997,"15min":42.631999999999998},"last":0.624},{"interface":"front","average":{"1min":0.65700000000000003,"5min":1.0589999999999999,"15min":1.224},"min":{"1min":0.376,"5min":0.32500000000000001,"15min":0.315},"max":{"1min":2.5990000000000002,"5min":6.5419999999999998,"15min":44.792999999999999},"last":0.39800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575143,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1848056,"kb_used_data":1022768,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":91897096,"statfs":{"total":95995035648,"available":94102626304,"internally_reserved":0,"allocated":1047314432,"data_stored":1046811524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":845085126},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:52:48 2024","interfaces":[{"interface":"back","average":{"1min":3.4660000000000002,"5min":1.7629999999999999,"15min":1.9410000000000001},"min":{"1min":0.24199999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":14.382999999999999,"5min":14.382999999999999,"15min":114.27},"last":1.1060000000000001},{"interface":"front","average":{"1min":3.222,"5min":1.5609999999999999,"15min":1.8859999999999999},"min":{"1min":0.48299999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.16,"5min":14.16,"15min":119.499},"last":0.95699999999999996}]},{"osd":2,"last update":"Sat Feb 10 07:53:19 2024","interfaces":[{"interface":"back","average":{"1min":2.8029999999999999,"5min":2.1200000000000001,"15min":1.4570000000000001},"min":{"1min":0.246,"5min":0.23400000000000001,"15min":0.20399999999999999},"max":{"1min":6.2510000000000003,"5min":14.343,"15min":14.343},"last":2.2709999999999999},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.8129999999999999,"15min":1.45},"min":{"1min":0.58999999999999997,"5min":0.24099999999999999,"15min":0.224},"max":{"1min":7.5279999999999996,"5min":14.204000000000001,"15min":14.204000000000001},"last":0.92400000000000004}]},{"osd":3,"last update":"Sat Feb 10 07:52:57 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":1.7430000000000001,"15min":0.90600000000000003},"min":{"1min":0.49299999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":14.452999999999999,"5min":22.367999999999999,"15min":22.367999999999999},"last":1.3839999999999999},{"interface":"front","average":{"1min":2.23,"5min":1.8129999999999999,"15min":0.86199999999999999},"min":{"1min":0.33200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":5.9429999999999996,"5min":22.199999999999999,"15min":22.199999999999999},"last":1.4239999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:53:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.379,"15min":2.1560000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28399999999999997},"max":{"1min":7.617,"5min":7.617,"15min":207.45699999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.3,"15min":1.3069999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.28899999999999998},"max":{"1min":4.0099999999999998,"5min":10.327999999999999,"15min":32.201000000000001},"last":0.54700000000000004}]},{"osd":5,"last update":"Sat Feb 10 07:53:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7609999999999999,"5min":1.7909999999999999,"15min":2.0449999999999999},"min":{"1min":0.49399999999999999,"5min":0.34200000000000003,"15min":0.308},"max":{"1min":7.5490000000000004,"5min":23.978999999999999,"15min":98.703000000000003},"last":1.139},{"interface":"front","average":{"1min":1.401,"5min":1.5609999999999999,"15min":1.9850000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.30199999999999999},"max":{"1min":5.8399999999999999,"5min":24.024999999999999,"15min":98.664000000000001},"last":0.98199999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:52:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.4039999999999999,"15min":1.45},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":4.0709999999999997,"5min":10.291,"15min":36.808},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.661,"15min":1.6890000000000001},"min":{"1min":0.45000000000000001,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":7.5839999999999996,"5min":7.5839999999999996,"15min":33.774999999999999},"last":1.1639999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:53:19 2024","interfaces":[{"interface":"back","average":{"1min":2.048,"5min":1.7709999999999999,"15min":1.5489999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":7.649,"5min":24.099,"15min":24.099},"last":1.02},{"interface":"front","average":{"1min":1.639,"5min":1.331,"15min":1.288},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.34300000000000003},"max":{"1min":4.0439999999999996,"5min":10.356999999999999,"15min":10.356999999999999},"last":0.64200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738667,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1525740,"kb_used_data":758748,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92219412,"statfs":{"total":95995035648,"available":94432677888,"internally_reserved":0,"allocated":776957952,"data_stored":776468273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":785381829},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:52:42 2024","interfaces":[{"interface":"back","average":{"1min":3.1499999999999999,"5min":1.8500000000000001,"15min":2.444},"min":{"1min":0.52400000000000002,"5min":0.20499999999999999,"15min":0.182},"max":{"1min":7.4500000000000002,"5min":16.942,"15min":156.16200000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":2.863,"5min":1.4119999999999999,"15min":2.29},"min":{"1min":0.55900000000000005,"5min":0.192,"15min":0.183},"max":{"1min":8.3780000000000001,"5min":8.4550000000000001,"15min":156.941},"last":1.6459999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:53:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5739999999999998,"5min":2.0409999999999999,"15min":1.6499999999999999},"min":{"1min":0.65700000000000003,"5min":0.221,"15min":0.20000000000000001},"max":{"1min":19.733000000000001,"5min":19.733000000000001,"15min":34.603000000000002},"last":0.80400000000000005},{"interface":"front","average":{"1min":2.9100000000000001,"5min":1.9370000000000001,"15min":1.444},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.22600000000000001},"max":{"1min":8.4060000000000006,"5min":10.116,"15min":34.718000000000004},"last":0.42599999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:52:57 2024","interfaces":[{"interface":"back","average":{"1min":2.5870000000000002,"5min":1.496,"15min":0.92000000000000004},"min":{"1min":0.30099999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":8.3659999999999997,"5min":8.6359999999999992,"15min":8.6359999999999992},"last":0.45900000000000002},{"interface":"front","average":{"1min":2.6829999999999998,"5min":1.73,"15min":0.99199999999999999},"min":{"1min":0.24099999999999999,"5min":0.214,"15min":0.214},"max":{"1min":10.029,"5min":17.172999999999998,"15min":17.172999999999998},"last":0.55700000000000005}]},{"osd":4,"last update":"Sat Feb 10 07:53:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1.79,"15min":1.4450000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.27600000000000002},"max":{"1min":4.6369999999999996,"5min":16.991,"15min":37.286000000000001},"last":0.313},{"interface":"front","average":{"1min":2.1720000000000002,"5min":1.6359999999999999,"15min":1.3999999999999999},"min":{"1min":0.34100000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":8.5939999999999994,"5min":8.5939999999999994,"15min":34.683},"last":0.40899999999999997}]},{"osd":5,"last update":"Sat Feb 10 07:53:00 2024","interfaces":[{"interface":"back","average":{"1min":2.1179999999999999,"5min":1.8580000000000001,"15min":1.998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32300000000000001},"max":{"1min":5.992,"5min":17.062999999999999,"15min":41.429000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":2.202,"5min":1.5580000000000001,"15min":1.8660000000000001},"min":{"1min":0.379,"5min":0.378,"15min":0.308},"max":{"1min":8.6460000000000008,"5min":8.6460000000000008,"15min":41.456000000000003},"last":0.44700000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:52:42 2024","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.3979999999999999,"15min":1.552},"min":{"1min":0.41899999999999998,"5min":0.34599999999999997,"15min":0.33600000000000002},"max":{"1min":5.0030000000000001,"5min":8.5630000000000006,"15min":41.587000000000003},"last":1.7110000000000001},{"interface":"front","average":{"1min":2.0350000000000001,"5min":1.7669999999999999,"15min":1.794},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":7.4219999999999997,"5min":17.027999999999999,"15min":41.256},"last":0.38700000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:53:22 2024","interfaces":[{"interface":"back","average":{"1min":2.6680000000000001,"5min":1.7669999999999999,"15min":1.6160000000000001},"min":{"1min":0.41199999999999998,"5min":0.39600000000000002,"15min":0.33300000000000002},"max":{"1min":8.6809999999999992,"5min":8.6809999999999992,"15min":34.710999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.831,"5min":1.589,"15min":1.4039999999999999},"min":{"1min":0.45400000000000001,"5min":0.39200000000000002,"15min":0.379},"max":{"1min":8.3230000000000004,"5min":8.5999999999999996,"15min":34.997999999999998},"last":0.5}]}]},{"osd":2,"up_from":18,"seq":77309411617,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1172820,"kb_used_data":501576,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92572332,"statfs":{"total":95995035648,"available":94794067968,"internally_reserved":0,"allocated":513613824,"data_stored":513151739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:23 2024","interfaces":[{"interface":"back","average":{"1min":3.5510000000000002,"5min":2.46,"15min":1.8029999999999999},"min":{"1min":0.47499999999999998,"5min":0.20699999999999999,"15min":0.191},"max":{"1min":32.371000000000002,"5min":32.371000000000002,"15min":65.430999999999997},"last":0.373},{"interface":"front","average":{"1min":4.0869999999999997,"5min":2.6749999999999998,"15min":1.6990000000000001},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.20100000000000001},"max":{"1min":33.280000000000001,"5min":38.173000000000002,"15min":38.173000000000002},"last":0.47099999999999997}]},{"osd":1,"last update":"Sat Feb 10 07:53:23 2024","interfaces":[{"interface":"back","average":{"1min":3.2490000000000001,"5min":1.9690000000000001,"15min":1.736},"min":{"1min":0.46700000000000003,"5min":0.21299999999999999,"15min":0.20599999999999999},"max":{"1min":13.071999999999999,"5min":27.606000000000002,"15min":65.795000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":4.0030000000000001,"5min":2.456,"15min":1.681},"min":{"1min":0.38500000000000001,"5min":0.219,"15min":0.20499999999999999},"max":{"1min":12.17,"5min":29.709,"15min":29.709},"last":0.68100000000000005}]},{"osd":3,"last update":"Sat Feb 10 07:52:59 2024","interfaces":[{"interface":"back","average":{"1min":1.984,"5min":1.163,"15min":0.76800000000000002},"min":{"1min":0.39000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":4.9009999999999998,"5min":7.0819999999999999,"15min":7.0819999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":2.3999999999999999,"5min":1.391,"15min":0.78900000000000003},"min":{"1min":0.41199999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":6.4269999999999996,"5min":7.7530000000000001,"15min":7.7530000000000001},"last":1.0509999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:53:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.2569999999999999,"15min":1.3799999999999999},"min":{"1min":0.51000000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.5249999999999999,"5min":11.395,"15min":65.494},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.585,"15min":1.2450000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":6.4800000000000004,"5min":7.7149999999999999,"15min":7.7149999999999999},"last":0.624}]},{"osd":5,"last update":"Sat Feb 10 07:53:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3849999999999998,"5min":1.337,"15min":1.456},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.32600000000000001},"max":{"1min":10.807,"5min":10.807,"15min":25.533000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.0840000000000001,"15min":1.534},"min":{"1min":0.58299999999999996,"5min":0.35599999999999998,"15min":0.34499999999999997},"max":{"1min":4.7789999999999999,"5min":4.9749999999999996,"15min":65.834000000000003},"last":0.432}]},{"osd":6,"last update":"Sat Feb 10 07:52:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.2589999999999999,"15min":1.6459999999999999},"min":{"1min":0.51900000000000002,"5min":0.39300000000000002,"15min":0.35199999999999998},"max":{"1min":3.637,"5min":5.069,"15min":65.748000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":1.7,"5min":1.276,"15min":1.347},"min":{"1min":0.36299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.6200000000000001,"5min":6.6280000000000001,"15min":25.901},"last":0.58599999999999997}]},{"osd":7,"last update":"Sat Feb 10 07:53:23 2024","interfaces":[{"interface":"back","average":{"1min":2.5030000000000001,"5min":1.702,"15min":1.3979999999999999},"min":{"1min":0.42299999999999999,"5min":0.36899999999999999,"15min":0.36699999999999999},"max":{"1min":10.859999999999999,"5min":10.859999999999999,"15min":10.859999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.2210000000000001,"15min":1.444},"min":{"1min":0.42799999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.609,"5min":5.0979999999999999,"15min":65.539000000000001},"last":0.504}]}]},{"osd":3,"up_from":347,"seq":1490353651781,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905008,"kb_used_data":659360,"kb_used_omap":5,"kb_used_meta":245626,"kb_avail":92840144,"statfs":{"total":95995035648,"available":95068307456,"internally_reserved":0,"allocated":675184640,"data_stored":674689447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5625,"internal_metadata":251521543},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":3.5259999999999998,"5min":1.764,"15min":0.93799999999999994},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":28.247,"5min":28.247,"15min":28.247},"last":0.23899999999999999},{"interface":"front","average":{"1min":3.7149999999999999,"5min":1.774,"15min":1.008},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":27.361999999999998,"5min":27.361999999999998,"15min":27.361999999999998},"last":0.35999999999999999}]},{"osd":1,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":6.3849999999999998,"5min":2.3500000000000001,"15min":1.0469999999999999},"min":{"1min":0.23000000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":30.718,"5min":30.718,"15min":30.718},"last":15.547000000000001},{"interface":"front","average":{"1min":6.0140000000000002,"5min":2.181,"15min":1.012},"min":{"1min":0.38200000000000001,"5min":0.22,"15min":0.22},"max":{"1min":32.927999999999997,"5min":32.927999999999997,"15min":32.927999999999997},"last":12.583}]},{"osd":2,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":2.165,"5min":1.3979999999999999,"15min":0.77400000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.8949999999999996,"5min":10.686,"15min":10.686},"last":0.51100000000000001},{"interface":"front","average":{"1min":3.395,"5min":1.6739999999999999,"15min":0.874},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":11.25,"5min":11.25,"15min":11.25},"last":0.436}]},{"osd":4,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.8580000000000001,"15min":2.4569999999999999},"min":{"1min":0.29899999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.1589999999999998,"5min":28.707999999999998,"15min":28.707999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.544,"5min":1.772,"15min":2.6389999999999998},"min":{"1min":0.379,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":7.8220000000000001,"5min":28.760999999999999,"15min":28.760999999999999},"last":0.41599999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":2.1299999999999999,"5min":1.476,"15min":1.056},"min":{"1min":0.40100000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.2799999999999994,"5min":15.525,"15min":15.525},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.3169999999999999,"15min":0.93700000000000006},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":5.109,"5min":10.298,"15min":10.298},"last":0.52900000000000003}]},{"osd":6,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":2.1789999999999998,"5min":2.3519999999999999,"15min":2.5},"min":{"1min":0.45000000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":8.3219999999999992,"5min":55.750999999999998,"15min":55.750999999999998},"last":0.497},{"interface":"front","average":{"1min":2.141,"5min":2.653,"15min":3.0369999999999999},"min":{"1min":0.40999999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":7.7380000000000004,"5min":55.807000000000002,"15min":55.807000000000002},"last":0.78100000000000003}]},{"osd":7,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":2.0600000000000001,"5min":1.722,"15min":1.3169999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.6790000000000003,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.706,"5min":1.4299999999999999,"15min":1.3220000000000001},"min":{"1min":0.433,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":7.8650000000000002,"5min":7.8650000000000002,"15min":7.8650000000000002},"last":0.55000000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953751,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1083900,"kb_used_data":352940,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92661252,"statfs":{"total":95995035648,"available":94885122048,"internally_reserved":0,"allocated":361410560,"data_stored":360911639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4510000000000001,"5min":1.4079999999999999,"15min":1.7050000000000001},"min":{"1min":0.29699999999999999,"5min":0.21299999999999999,"15min":0.20499999999999999},"max":{"1min":20.004999999999999,"5min":20.004999999999999,"15min":74.533000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":3.1200000000000001,"5min":1.881,"15min":1.827},"min":{"1min":0.252,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":20.050000000000001,"5min":20.050000000000001,"15min":74.340999999999994},"last":0.497}]},{"osd":1,"last update":"Sat Feb 10 07:53:03 2024","interfaces":[{"interface":"back","average":{"1min":3.0899999999999999,"5min":2.6080000000000001,"15min":2.448},"min":{"1min":0.25800000000000001,"5min":0.245,"15min":0.23200000000000001},"max":{"1min":34.348999999999997,"5min":66.343999999999994,"15min":69.852000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":3.5939999999999999,"5min":2.2650000000000001,"15min":2.411},"min":{"1min":0.27800000000000002,"5min":0.27700000000000002,"15min":0.26000000000000001},"max":{"1min":34.837000000000003,"5min":35.375,"15min":70.692999999999998},"last":3.2759999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:53:03 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.399,"15min":1.1890000000000001},"min":{"1min":0.28599999999999998,"5min":0.28399999999999997,"15min":0.26600000000000001},"max":{"1min":3.3460000000000001,"5min":7.1379999999999999,"15min":15.351000000000001},"last":2.2010000000000001},{"interface":"front","average":{"1min":1.54,"5min":1.28,"15min":1.1080000000000001},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":7.1870000000000003,"5min":7.335,"15min":8.5129999999999999},"last":2.2869999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:52:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":1.9930000000000001,"15min":3.827},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.1299999999999999,"5min":66.379999999999995,"15min":66.379999999999995},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.45,"5min":1.335,"15min":1.627},"min":{"1min":0.34999999999999998,"5min":0.309,"15min":0.309},"max":{"1min":6.6429999999999998,"5min":6.6429999999999998,"15min":6.6429999999999998},"last":0.317}]},{"osd":5,"last update":"Sat Feb 10 07:53:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":1.0029999999999999,"15min":0.995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.23100000000000001},"max":{"1min":2.1389999999999998,"5min":7.3639999999999999,"15min":10.502000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.746,"15min":0.98399999999999999},"min":{"1min":0.312,"5min":0.30199999999999999,"15min":0.247},"max":{"1min":1.613,"5min":3.6219999999999999,"15min":9.2460000000000004},"last":0.47699999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":1.8799999999999999,"15min":1.21},"min":{"1min":0.29199999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.1439999999999999,"5min":67.433999999999997,"15min":67.433999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":1.837,"15min":1.2769999999999999},"min":{"1min":0.307,"5min":0.29199999999999998,"15min":0.24399999999999999},"max":{"1min":1.7989999999999999,"5min":66.876000000000005,"15min":66.876000000000005},"last":0.46000000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:53:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":1.0189999999999999,"15min":0.97899999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.307},"max":{"1min":1.879,"5min":7.4930000000000003,"15min":7.4930000000000003},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.86099999999999999,"15min":1.006},"min":{"1min":0.36699999999999999,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":1.8420000000000001,"5min":3.8479999999999999,"15min":10.919},"last":0.51400000000000001}]}]},{"osd":5,"up_from":156,"seq":670014898380,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1055128,"kb_used_data":366864,"kb_used_omap":4,"kb_used_meta":688251,"kb_avail":92690024,"statfs":{"total":95995035648,"available":94914584576,"internally_reserved":0,"allocated":375668736,"data_stored":375190853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":704769698},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":2.052,"15min":1.778},"min":{"1min":0.27000000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.7290000000000001,"5min":24.292999999999999,"15min":52.938000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.752,"5min":1.899,"15min":1.9419999999999999},"min":{"1min":0.27800000000000002,"5min":0.23899999999999999,"15min":0.22900000000000001},"max":{"1min":5.9939999999999998,"5min":23.908000000000001,"15min":52.805},"last":0.39500000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":2.6890000000000001,"5min":2.2069999999999999,"15min":2.0339999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.245},"max":{"1min":23.405000000000001,"5min":28.780999999999999,"15min":54.298000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":2.9649999999999999,"5min":2.3260000000000001,"15min":1.98},"min":{"1min":0.254,"5min":0.248,"15min":0.246},"max":{"1min":23.98,"5min":28.356000000000002,"15min":52.969999999999999},"last":5.1340000000000003}]},{"osd":2,"last update":"Sat Feb 10 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":2.3050000000000002,"5min":1.413,"15min":1.1899999999999999},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":10.039999999999999,"5min":10.039999999999999,"15min":10.039999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.9770000000000001,"5min":1.446,"15min":1.234},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28100000000000003},"max":{"1min":9.6069999999999993,"5min":9.6069999999999993,"15min":13.662000000000001},"last":0.65900000000000003}]},{"osd":3,"last update":"Sat Feb 10 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":2.2719999999999998,"15min":1.9099999999999999},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.7060000000000004,"5min":97.119,"15min":97.119},"last":0.442},{"interface":"front","average":{"1min":2.2050000000000001,"5min":2.488,"15min":1.8120000000000001},"min":{"1min":0.39100000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.6180000000000003,"5min":97.141999999999996,"15min":97.141999999999996},"last":0.60499999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":1.2450000000000001,"15min":1.137},"min":{"1min":0.27200000000000002,"5min":0.214,"15min":0.214},"max":{"1min":3.0760000000000001,"5min":8.5890000000000004,"15min":8.952},"last":0.31},{"interface":"front","average":{"1min":0.70199999999999996,"5min":1.127,"15min":1.0580000000000001},"min":{"1min":0.25,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.5019999999999998,"5min":6.202,"15min":10.180999999999999},"last":0.36699999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":1.4870000000000001,"15min":1.139},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.23999999999999999},"max":{"1min":3.048,"5min":21.748000000000001,"15min":21.748000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":1.4339999999999999,"15min":1.085},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.26300000000000001},"max":{"1min":3.9399999999999999,"5min":22.187999999999999,"15min":22.187999999999999},"last":0.46100000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":1.375,"15min":1.2529999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.044,"5min":22.302,"15min":22.302},"last":0.505},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.4419999999999999,"15min":1.1439999999999999},"min":{"1min":0.34999999999999998,"5min":0.33400000000000002,"15min":0.26300000000000001},"max":{"1min":3.004,"5min":21.779,"15min":21.779},"last":0.58099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":775094272,"data_stored":775094272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1029836800,"data_stored":1029832728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":499257344,"data_stored":499253272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":664207360,"data_stored":664207360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":350949376,"data_stored":350945304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":368640000,"data_stored":368640000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":689639424,"data_stored":689635352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1066467328,"data_stored":1066467328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:53:34.610 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:53:34.610 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:34.673 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:34.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:34.775 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:34.852 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:34.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:34.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:34.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:34 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:53:34.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:34 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:53:34.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:34 smithi170 ceph-mon[33394]: osdmap e453: 8 total, 8 up, 8 in 2024-02-10T07:53:34.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:53:34.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:53:34.955 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:35.032 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:35.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:35.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:35.134 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:35.188 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:35.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:35.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:35.290 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:35.331 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:35.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:35.433 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:35.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:35.464 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:35.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:35.470 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:35.565 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:35.572 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:53:35.572 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:35 smithi008 ceph-mon[40743]: 6.13 scrub starts 2024-02-10T07:53:35.573 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:35 smithi008 ceph-mon[40743]: 6.13 scrub ok 2024-02-10T07:53:35.573 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:35 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:35.573 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:35 smithi008 ceph-mon[40743]: 5.7 scrub starts 2024-02-10T07:53:35.573 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:35 smithi008 ceph-mon[40743]: 5.7 scrub ok 2024-02-10T07:53:35.573 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:35 smithi008 ceph-mon[40743]: pgmap v1005: 148 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 954 op/s; 7005/54264 objects degraded (12.909%); 4191/54264 objects misplaced (7.723%); 0 B/s, 149 objects/s recovering 2024-02-10T07:53:35.573 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:35 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:53:35.574 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:35 smithi008 ceph-mon[29816]: 6.13 scrub starts 2024-02-10T07:53:35.574 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:35 smithi008 ceph-mon[29816]: 6.13 scrub ok 2024-02-10T07:53:35.574 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:35 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:35.574 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:35 smithi008 ceph-mon[29816]: 5.7 scrub starts 2024-02-10T07:53:35.574 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:35 smithi008 ceph-mon[29816]: 5.7 scrub ok 2024-02-10T07:53:35.574 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:35 smithi008 ceph-mon[29816]: pgmap v1005: 148 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 954 op/s; 7005/54264 objects degraded (12.909%); 4191/54264 objects misplaced (7.723%); 0 B/s, 149 objects/s recovering 2024-02-10T07:53:35.574 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:35 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:53:35.613 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:35.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:35.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:35.627 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:53:35.669 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:53:35.709 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:53:35.715 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:35.752 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:53:35.775 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:35.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:35.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:35.793 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:53:35.835 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:53:35.877 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:35.881 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:53:35.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:35 smithi170 ceph-mon[33394]: 6.13 scrub starts 2024-02-10T07:53:35.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:35 smithi170 ceph-mon[33394]: 6.13 scrub ok 2024-02-10T07:53:35.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:35 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:35.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:35 smithi170 ceph-mon[33394]: 5.7 scrub starts 2024-02-10T07:53:35.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:35 smithi170 ceph-mon[33394]: 5.7 scrub ok 2024-02-10T07:53:35.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:35 smithi170 ceph-mon[33394]: pgmap v1005: 148 pgs: 1 active+clean+scrubbing, 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 954 op/s; 7005/54264 objects degraded (12.909%); 4191/54264 objects misplaced (7.723%); 0 B/s, 149 objects/s recovering 2024-02-10T07:53:35.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:35 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:53:35.897 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:53:35.897 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:53:35.897 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:53:35.897 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:53:35.897 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:53:35.897 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:53:35.897 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:53:35.897 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:53:35.929 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:35.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:36.031 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:36.064 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:36.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:36.166 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:36.197 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:36.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:36.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:36.299 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:36.334 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:36.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:36.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:36.436 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:36.466 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:36.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:36.568 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:36.607 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:36.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:36.640 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:36 smithi008 ceph-mon[29816]: 3.c scrub starts 2024-02-10T07:53:36.640 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:36 smithi008 ceph-mon[29816]: 3.c scrub ok 2024-02-10T07:53:36.640 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:36 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3571233586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:53:36.640 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:36 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:53:36.640 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:36 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:53:36.640 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:36 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:53:36.640 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:36 smithi008 ceph-mon[40743]: 3.c scrub starts 2024-02-10T07:53:36.640 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:36 smithi008 ceph-mon[40743]: 3.c scrub ok 2024-02-10T07:53:36.641 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:36 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3571233586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:53:36.641 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:36 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:53:36.641 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:36 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:53:36.641 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:36 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:53:36.709 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:36.746 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:36.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:36.848 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:36.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:36 smithi170 ceph-mon[33394]: 3.c scrub starts 2024-02-10T07:53:36.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:36 smithi170 ceph-mon[33394]: 3.c scrub ok 2024-02-10T07:53:36.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:36 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3571233586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:53:36.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:36 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:53:36.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:36 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:53:36.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:36 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:53:36.885 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:36.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:36.987 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:37.029 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:37.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:37.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:37.132 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:37.159 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:37.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:37.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:37.171 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:37.261 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:37.265 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set noscrub 2024-02-10T07:53:37.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:37.309 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:37.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:37.410 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:37.440 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:37.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:37.542 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:37.571 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:37.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:37.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:37.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:37 smithi008 ceph-mon[29816]: 6.11 scrub starts 2024-02-10T07:53:37.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:37 smithi008 ceph-mon[29816]: 6.11 scrub ok 2024-02-10T07:53:37.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:37 smithi008 ceph-mon[29816]: pgmap v1006: 148 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s; 7005/50367 objects degraded (13.908%); 4054/50367 objects misplaced (8.049%); 0 B/s, 144 objects/s recovering 2024-02-10T07:53:37.584 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:37 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 7005/50367 objects degraded (13.908%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:37.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:37 smithi008 ceph-mon[40743]: 6.11 scrub starts 2024-02-10T07:53:37.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:37 smithi008 ceph-mon[40743]: 6.11 scrub ok 2024-02-10T07:53:37.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:37 smithi008 ceph-mon[40743]: pgmap v1006: 148 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s; 7005/50367 objects degraded (13.908%); 4054/50367 objects misplaced (8.049%); 0 B/s, 144 objects/s recovering 2024-02-10T07:53:37.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:37 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 7005/50367 objects degraded (13.908%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:37.673 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:37.739 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:37.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:37.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:37.841 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:37.876 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:37.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:37.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:37 smithi170 ceph-mon[33394]: 6.11 scrub starts 2024-02-10T07:53:37.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:37 smithi170 ceph-mon[33394]: 6.11 scrub ok 2024-02-10T07:53:37.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:37 smithi170 ceph-mon[33394]: pgmap v1006: 148 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 977 op/s; 7005/50367 objects degraded (13.908%); 4054/50367 objects misplaced (8.049%); 0 B/s, 144 objects/s recovering 2024-02-10T07:53:37.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:37 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 7005/50367 objects degraded (13.908%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:37.979 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:38.008 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:38.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:38.110 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:38.140 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:38.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:38.242 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:38.289 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:38.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:38.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:38.391 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:38.432 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:38.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:38.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:38.533 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:38.573 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:38.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:38.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:38.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:38 smithi008 ceph-mon[29816]: 3.8 scrub starts 2024-02-10T07:53:38.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:38 smithi008 ceph-mon[29816]: 3.8 scrub ok 2024-02-10T07:53:38.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:38 smithi008 ceph-mon[40743]: 3.8 scrub starts 2024-02-10T07:53:38.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:38 smithi008 ceph-mon[40743]: 3.8 scrub ok 2024-02-10T07:53:38.675 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:38.729 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:38.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:38.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:38.830 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:38.869 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:38.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:38.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:38.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:38 smithi170 ceph-mon[33394]: 3.8 scrub starts 2024-02-10T07:53:38.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:38 smithi170 ceph-mon[33394]: 3.8 scrub ok 2024-02-10T07:53:38.971 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:39.005 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:39.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:39.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:39.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:53:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:53:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:53:39.107 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:39.130 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-10T07:53:39.136 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:39.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:39.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:39.238 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:39.267 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:39.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:39.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:39.369 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:39.430 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:39.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:39.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:39.532 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:39.577 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:39.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:39.675 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:39 smithi008 ceph-mon[29816]: pgmap v1007: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 1021 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 599 op/s; 7005/49563 objects degraded (14.134%); 4054/49563 objects misplaced (8.179%); 0 B/s, 61 objects/s recovering 2024-02-10T07:53:39.676 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:39 smithi008 ceph-mon[40743]: pgmap v1007: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 1021 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 599 op/s; 7005/49563 objects degraded (14.134%); 4054/49563 objects misplaced (8.179%); 0 B/s, 61 objects/s recovering 2024-02-10T07:53:39.678 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:39.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:39.709 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:39.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:39.810 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:39.841 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:39.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:39.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:39 smithi170 ceph-mon[33394]: pgmap v1007: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 1021 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 599 op/s; 7005/49563 objects degraded (14.134%); 4054/49563 objects misplaced (8.179%); 0 B/s, 61 objects/s recovering 2024-02-10T07:53:39.943 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:39.947 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:39.971 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:39.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:40.031 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:53:40.031 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:53:40.073 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:40.104 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:40.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:40.206 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:40.237 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:40.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:40.338 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:40.386 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:40.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:40.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:40.487 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:40.531 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:40.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:40.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:40.633 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:40.668 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:40.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:40.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:40.770 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:40.797 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:40.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:40.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:40.899 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:40.936 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:40.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:40.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:41.038 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:41.085 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:41.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:41.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:41.187 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:41.228 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:41.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:41.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:41.330 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:41.340 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-10T07:53:41.360 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:41.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:41.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:41.461 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:41.487 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:41.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:41.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:41.563 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:41 smithi008 ceph-mon[29816]: pgmap v1008: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 790 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 993 op/s; 6644/38502 objects degraded (17.256%); 3916/38502 objects misplaced (10.171%); 0 B/s, 124 objects/s recovering 2024-02-10T07:53:41.563 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:41 smithi008 ceph-mon[40743]: pgmap v1008: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 790 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 993 op/s; 6644/38502 objects degraded (17.256%); 3916/38502 objects misplaced (10.171%); 0 B/s, 124 objects/s recovering 2024-02-10T07:53:41.589 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:41.614 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:41.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:41.716 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:41.765 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:41.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:41.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:41.867 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:41.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:41 smithi170 ceph-mon[33394]: pgmap v1008: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 790 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 993 op/s; 6644/38502 objects degraded (17.256%); 3916/38502 objects misplaced (10.171%); 0 B/s, 124 objects/s recovering 2024-02-10T07:53:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:41.918 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:41.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:42.019 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:42.051 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:42.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:42.152 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:42.180 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:42.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:42.281 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:42.325 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:42.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:42.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:42.427 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:42.471 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:42.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:42.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:42.572 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:42.602 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:42.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:42.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:42.704 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:42.711 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:53:42.712 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:42 smithi008 ceph-mon[29816]: 6.6 scrub starts 2024-02-10T07:53:42.712 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:42 smithi008 ceph-mon[29816]: 6.6 scrub ok 2024-02-10T07:53:42.712 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:42 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 6644/38502 objects degraded (17.256%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:42.713 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:42 smithi008 ceph-mon[40743]: 6.6 scrub starts 2024-02-10T07:53:42.713 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:42 smithi008 ceph-mon[40743]: 6.6 scrub ok 2024-02-10T07:53:42.713 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:42 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 6644/38502 objects degraded (17.256%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:42.721 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:53:42.731 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:42.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:42.832 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:42.874 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:42.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:42.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:42.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:42 smithi170 ceph-mon[33394]: 6.6 scrub starts 2024-02-10T07:53:42.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:42 smithi170 ceph-mon[33394]: 6.6 scrub ok 2024-02-10T07:53:42.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:42 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 6644/38502 objects degraded (17.256%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:42.976 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:43.009 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:43.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:43.110 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:43.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:43.165 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:43.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:43.265 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:43.296 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:43.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:43.397 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:43.428 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:43.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:43.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:43.529 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:43.568 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:43.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:43.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:43.669 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:43.702 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:43 smithi008 ceph-mon[29816]: 3.1 scrub starts 2024-02-10T07:53:43.702 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:43 smithi008 ceph-mon[29816]: 3.1 scrub ok 2024-02-10T07:53:43.702 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:43 smithi008 ceph-mon[29816]: 5.16 scrub starts 2024-02-10T07:53:43.702 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:43 smithi008 ceph-mon[29816]: pgmap v1009: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 756 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 865 op/s; 6644/36873 objects degraded (18.019%); 3916/36873 objects misplaced (10.620%); 0 B/s, 101 objects/s recovering 2024-02-10T07:53:43.702 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:43 smithi008 ceph-mon[29816]: 5.16 scrub ok 2024-02-10T07:53:43.702 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:43 smithi008 ceph-mon[40743]: 3.1 scrub starts 2024-02-10T07:53:43.702 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:43 smithi008 ceph-mon[40743]: 3.1 scrub ok 2024-02-10T07:53:43.702 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:43 smithi008 ceph-mon[40743]: 5.16 scrub starts 2024-02-10T07:53:43.703 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:43 smithi008 ceph-mon[40743]: pgmap v1009: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 756 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 865 op/s; 6644/36873 objects degraded (18.019%); 3916/36873 objects misplaced (10.620%); 0 B/s, 101 objects/s recovering 2024-02-10T07:53:43.703 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:43 smithi008 ceph-mon[40743]: 5.16 scrub ok 2024-02-10T07:53:43.713 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:43.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:43.815 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:43.861 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:43.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:43.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:43 smithi170 ceph-mon[33394]: 3.1 scrub starts 2024-02-10T07:53:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:43 smithi170 ceph-mon[33394]: 3.1 scrub ok 2024-02-10T07:53:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:43 smithi170 ceph-mon[33394]: 5.16 scrub starts 2024-02-10T07:53:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:43 smithi170 ceph-mon[33394]: pgmap v1009: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 756 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 865 op/s; 6644/36873 objects degraded (18.019%); 3916/36873 objects misplaced (10.620%); 0 B/s, 101 objects/s recovering 2024-02-10T07:53:43.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:43 smithi170 ceph-mon[33394]: 5.16 scrub ok 2024-02-10T07:53:43.947 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":1009,"stamp":"2024-02-10T07:53:42.203951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":793057652,"num_objects":12291,"num_object_clones":0,"num_object_copies":36873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6644,"num_objects_misplaced":3916,"num_objects_unfound":0,"num_objects_dirty":12291,"num_whiteouts":0,"num_read":3077,"num_read_kb":2923,"num_write":198614,"num_write_kb":6707038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16086,"num_bytes_recovered":803744373,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86024,"ondisk_log_size":86024,"up":444,"acting":442,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":449,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8998924,"kb_used_data":3532492,"kb_used_omap":19,"kb_used_meta":5466220,"kb_avail":740962292,"statfs":{"total":767960285184,"available":758745387008,"internally_reserved":0,"allocated":3617271808,"data_stored":3613307060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19634,"internal_metadata":5597410126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-556269568,"num_objects":-8488,"num_object_clones":0,"num_object_copies":-25464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-719,"num_objects_misplaced":-275,"num_objects_unfound":0,"num_objects_dirty":-8488,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8488,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":994,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.821536"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":843,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.401858+0000","last_change":"2024-02-10T07:49:56.315126+0000","last_active":"2024-02-10T07:53:33.401858+0000","last_peered":"2024-02-10T07:53:33.401858+0000","last_clean":"2024-02-10T07:53:33.401858+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:53:33.401858+0000","last_undegraded":"2024-02-10T07:53:33.401858+0000","last_fullsized":"2024-02-10T07:53:33.401858+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.315029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.315029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":853,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397249+0000","last_change":"2024-02-10T07:49:08.138553+0000","last_active":"2024-02-10T07:53:33.397249+0000","last_peered":"2024-02-10T07:53:33.397249+0000","last_clean":"2024-02-10T07:53:33.397249+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:53:33.397249+0000","last_undegraded":"2024-02-10T07:53:33.397249+0000","last_fullsized":"2024-02-10T07:53:33.397249+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:08.138442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.138442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":883,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.407834+0000","last_change":"2024-02-10T07:50:43.303349+0000","last_active":"2024-02-10T07:53:33.407834+0000","last_peered":"2024-02-10T07:53:33.407834+0000","last_clean":"2024-02-10T07:53:33.407834+0000","last_became_active":"2024-02-10T07:50:43.281905+0000","last_became_peered":"2024-02-10T07:50:43.281905+0000","last_unstale":"2024-02-10T07:53:33.407834+0000","last_undegraded":"2024-02-10T07:53:33.407834+0000","last_fullsized":"2024-02-10T07:53:33.407834+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:17.198701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:17.198701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"430'25","reported_seq":912,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.401941+0000","last_change":"2024-02-10T07:46:37.008978+0000","last_active":"2024-02-10T07:53:33.401941+0000","last_peered":"2024-02-10T07:53:33.401941+0000","last_clean":"2024-02-10T07:53:33.401941+0000","last_became_active":"2024-02-10T07:46:10.717475+0000","last_became_peered":"2024-02-10T07:46:10.717475+0000","last_unstale":"2024-02-10T07:53:33.401941+0000","last_undegraded":"2024-02-10T07:53:33.401941+0000","last_fullsized":"2024-02-10T07:53:33.401941+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'19","last_scrub_stamp":"2024-02-10T07:46:37.008841+0000","last_deep_scrub":"287'19","last_deep_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.008841+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":182,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.407632+0000","last_change":"2024-02-10T07:53:26.218018+0000","last_active":"2024-02-10T07:53:33.407632+0000","last_peered":"2024-02-10T07:53:33.407632+0000","last_clean":"2024-02-10T07:53:33.407632+0000","last_became_active":"2024-02-10T07:49:06.236235+0000","last_became_peered":"2024-02-10T07:49:06.236235+0000","last_unstale":"2024-02-10T07:53:33.407632+0000","last_undegraded":"2024-02-10T07:53:33.407632+0000","last_fullsized":"2024-02-10T07:53:33.407632+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:26.217749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:15.079428+0000","last_clean_scrub_stamp":"2024-02-10T07:53:26.217749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:52.906075+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":846,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.401837+0000","last_change":"2024-02-10T07:50:06.335390+0000","last_active":"2024-02-10T07:53:33.401837+0000","last_peered":"2024-02-10T07:53:33.401837+0000","last_clean":"2024-02-10T07:53:33.401837+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:53:33.401837+0000","last_undegraded":"2024-02-10T07:53:33.401837+0000","last_fullsized":"2024-02-10T07:53:33.401837+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:06.335262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:50:06.335262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":370,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.396474+0000","last_change":"2024-02-10T07:51:05.208836+0000","last_active":"2024-02-10T07:53:33.396474+0000","last_peered":"2024-02-10T07:53:33.396474+0000","last_clean":"2024-02-10T07:53:33.396474+0000","last_became_active":"2024-02-10T07:50:43.282449+0000","last_became_peered":"2024-02-10T07:50:43.282449+0000","last_unstale":"2024-02-10T07:53:33.396474+0000","last_undegraded":"2024-02-10T07:53:33.396474+0000","last_fullsized":"2024-02-10T07:53:33.396474+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:51:05.208755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:51:05.208755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"430'39","reported_seq":142,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.404638+0000","last_change":"2024-02-10T07:50:43.318008+0000","last_active":"2024-02-10T07:53:33.404638+0000","last_peered":"2024-02-10T07:53:33.404638+0000","last_clean":"2024-02-10T07:53:33.404638+0000","last_became_active":"2024-02-10T07:50:43.290681+0000","last_became_peered":"2024-02-10T07:50:43.290681+0000","last_unstale":"2024-02-10T07:53:33.404638+0000","last_undegraded":"2024-02-10T07:53:33.404638+0000","last_fullsized":"2024-02-10T07:53:33.404638+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:48:13.230908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.230908+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":842,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.406706+0000","last_change":"2024-02-10T07:50:05.620136+0000","last_active":"2024-02-10T07:53:33.406706+0000","last_peered":"2024-02-10T07:53:33.406706+0000","last_clean":"2024-02-10T07:53:33.406706+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:53:33.406706+0000","last_undegraded":"2024-02-10T07:53:33.406706+0000","last_fullsized":"2024-02-10T07:53:33.406706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:05.619958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:05.619958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":831,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.392660+0000","last_change":"2024-02-10T07:53:23.117713+0000","last_active":"2024-02-10T07:53:33.392660+0000","last_peered":"2024-02-10T07:53:33.392660+0000","last_clean":"2024-02-10T07:53:33.392660+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:53:33.392660+0000","last_undegraded":"2024-02-10T07:53:33.392660+0000","last_fullsized":"2024-02-10T07:53:33.392660+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:23.117597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:23.117597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:27.492528+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":384,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402401+0000","last_change":"2024-02-10T07:49:13.218634+0000","last_active":"2024-02-10T07:53:33.402401+0000","last_peered":"2024-02-10T07:53:33.402401+0000","last_clean":"2024-02-10T07:53:33.402401+0000","last_became_active":"2024-02-10T07:48:02.332960+0000","last_became_peered":"2024-02-10T07:48:02.332960+0000","last_unstale":"2024-02-10T07:53:33.402401+0000","last_undegraded":"2024-02-10T07:53:33.402401+0000","last_fullsized":"2024-02-10T07:53:33.402401+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.218549+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"430'21","reported_seq":877,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.401997+0000","last_change":"2024-02-10T07:47:49.553645+0000","last_active":"2024-02-10T07:53:33.401997+0000","last_peered":"2024-02-10T07:53:33.401997+0000","last_clean":"2024-02-10T07:53:33.401997+0000","last_became_active":"2024-02-10T07:47:47.965930+0000","last_became_peered":"2024-02-10T07:47:47.965930+0000","last_unstale":"2024-02-10T07:53:33.401997+0000","last_undegraded":"2024-02-10T07:53:33.401997+0000","last_fullsized":"2024-02-10T07:53:33.401997+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:49.553517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.553517+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":755,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397435+0000","last_change":"2024-02-10T07:49:56.718482+0000","last_active":"2024-02-10T07:53:33.397435+0000","last_peered":"2024-02-10T07:53:33.397435+0000","last_clean":"2024-02-10T07:53:33.397435+0000","last_became_active":"2024-02-10T07:49:06.266689+0000","last_became_peered":"2024-02-10T07:49:06.266689+0000","last_unstale":"2024-02-10T07:53:33.397435+0000","last_undegraded":"2024-02-10T07:53:33.397435+0000","last_fullsized":"2024-02-10T07:53:33.397435+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.718382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.718382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":846,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.394361+0000","last_change":"2024-02-10T07:48:18.069773+0000","last_active":"2024-02-10T07:53:33.394361+0000","last_peered":"2024-02-10T07:53:33.394361+0000","last_clean":"2024-02-10T07:53:33.394361+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:53:33.394361+0000","last_undegraded":"2024-02-10T07:53:33.394361+0000","last_fullsized":"2024-02-10T07:53:33.394361+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:18.069641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:18.069641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":372,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402680+0000","last_change":"2024-02-10T07:49:06.115547+0000","last_active":"2024-02-10T07:53:33.402680+0000","last_peered":"2024-02-10T07:53:33.402680+0000","last_clean":"2024-02-10T07:53:33.402680+0000","last_became_active":"2024-02-10T07:48:02.329859+0000","last_became_peered":"2024-02-10T07:48:02.329859+0000","last_unstale":"2024-02-10T07:53:33.402680+0000","last_undegraded":"2024-02-10T07:53:33.402680+0000","last_fullsized":"2024-02-10T07:53:33.402680+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.115466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.473686+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.115466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"430'9","reported_seq":859,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402345+0000","last_change":"2024-02-10T07:48:11.611397+0000","last_active":"2024-02-10T07:53:33.402345+0000","last_peered":"2024-02-10T07:53:33.402345+0000","last_clean":"2024-02-10T07:53:33.402345+0000","last_became_active":"2024-02-10T07:47:47.966093+0000","last_became_peered":"2024-02-10T07:47:47.966093+0000","last_unstale":"2024-02-10T07:53:33.402345+0000","last_undegraded":"2024-02-10T07:53:33.402345+0000","last_fullsized":"2024-02-10T07:53:33.402345+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7","last_scrub_stamp":"2024-02-10T07:48:11.608499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.608499+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":845,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402693+0000","last_change":"2024-02-10T07:48:13.621135+0000","last_active":"2024-02-10T07:53:33.402693+0000","last_peered":"2024-02-10T07:53:33.402693+0000","last_clean":"2024-02-10T07:53:33.402693+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:53:33.402693+0000","last_undegraded":"2024-02-10T07:53:33.402693+0000","last_fullsized":"2024-02-10T07:53:33.402693+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:13.620938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.620938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":852,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.399665+0000","last_change":"2024-02-10T07:49:51.810158+0000","last_active":"2024-02-10T07:53:33.399665+0000","last_peered":"2024-02-10T07:53:33.399665+0000","last_clean":"2024-02-10T07:53:33.399665+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:53:33.399665+0000","last_undegraded":"2024-02-10T07:53:33.399665+0000","last_fullsized":"2024-02-10T07:53:33.399665+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:51.810062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:51.810062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":854,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.407730+0000","last_change":"2024-02-10T07:49:16.313655+0000","last_active":"2024-02-10T07:53:33.407730+0000","last_peered":"2024-02-10T07:53:33.407730+0000","last_clean":"2024-02-10T07:53:33.407730+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:53:33.407730+0000","last_undegraded":"2024-02-10T07:53:33.407730+0000","last_fullsized":"2024-02-10T07:53:33.407730+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:16.313573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.313573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"430'46","reported_seq":142,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.404840+0000","last_change":"2024-02-10T07:50:43.322322+0000","last_active":"2024-02-10T07:53:33.404840+0000","last_peered":"2024-02-10T07:53:33.404840+0000","last_clean":"2024-02-10T07:53:33.404840+0000","last_became_active":"2024-02-10T07:50:43.300153+0000","last_became_peered":"2024-02-10T07:50:43.300153+0000","last_unstale":"2024-02-10T07:53:33.404840+0000","last_undegraded":"2024-02-10T07:53:33.404840+0000","last_fullsized":"2024-02-10T07:53:33.404840+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:49:00.551605+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.551605+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":549,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.396321+0000","last_change":"2024-02-10T07:52:11.043473+0000","last_active":"2024-02-10T07:53:33.396321+0000","last_peered":"2024-02-10T07:53:33.396321+0000","last_clean":"2024-02-10T07:53:33.396321+0000","last_became_active":"2024-02-10T07:49:06.263790+0000","last_became_peered":"2024-02-10T07:49:06.263790+0000","last_unstale":"2024-02-10T07:53:33.396321+0000","last_undegraded":"2024-02-10T07:53:33.396321+0000","last_fullsized":"2024-02-10T07:53:33.396321+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:11.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:11.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":863,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402963+0000","last_change":"2024-02-10T07:50:43.286623+0000","last_active":"2024-02-10T07:53:33.402963+0000","last_peered":"2024-02-10T07:53:33.402963+0000","last_clean":"2024-02-10T07:53:33.402963+0000","last_became_active":"2024-02-10T07:50:43.285201+0000","last_became_peered":"2024-02-10T07:50:43.285201+0000","last_unstale":"2024-02-10T07:53:33.402963+0000","last_undegraded":"2024-02-10T07:53:33.402963+0000","last_fullsized":"2024-02-10T07:53:33.402963+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.680331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.680331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":866,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402018+0000","last_change":"2024-02-10T07:50:43.282584+0000","last_active":"2024-02-10T07:53:33.402018+0000","last_peered":"2024-02-10T07:53:33.402018+0000","last_clean":"2024-02-10T07:53:33.402018+0000","last_became_active":"2024-02-10T07:50:43.281382+0000","last_became_peered":"2024-02-10T07:50:43.281382+0000","last_unstale":"2024-02-10T07:53:33.402018+0000","last_undegraded":"2024-02-10T07:53:33.402018+0000","last_fullsized":"2024-02-10T07:53:33.402018+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:04.396771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.396771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"430'24","reported_seq":822,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.399490+0000","last_change":"2024-02-10T07:49:58.618564+0000","last_active":"2024-02-10T07:53:33.399490+0000","last_peered":"2024-02-10T07:53:33.399490+0000","last_clean":"2024-02-10T07:53:33.399490+0000","last_became_active":"2024-02-10T07:49:06.263261+0000","last_became_peered":"2024-02-10T07:49:06.263261+0000","last_unstale":"2024-02-10T07:53:33.399490+0000","last_undegraded":"2024-02-10T07:53:33.399490+0000","last_fullsized":"2024-02-10T07:53:33.399490+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'18","last_scrub_stamp":"2024-02-10T07:49:58.618464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:58.618464+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":878,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.395185+0000","last_change":"2024-02-10T07:50:43.307775+0000","last_active":"2024-02-10T07:53:33.395185+0000","last_peered":"2024-02-10T07:53:33.395185+0000","last_clean":"2024-02-10T07:53:33.395185+0000","last_became_active":"2024-02-10T07:50:43.285254+0000","last_became_peered":"2024-02-10T07:50:43.285254+0000","last_unstale":"2024-02-10T07:53:33.395185+0000","last_undegraded":"2024-02-10T07:53:33.395185+0000","last_fullsized":"2024-02-10T07:53:33.395185+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:47:34.074746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:34.074746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"414'9","reported_seq":1153,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.396153+0000","last_change":"2024-02-10T07:51:13.307061+0000","last_active":"2024-02-10T07:53:33.396153+0000","last_peered":"2024-02-10T07:53:33.396153+0000","last_clean":"2024-02-10T07:53:33.396153+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:53:33.396153+0000","last_undegraded":"2024-02-10T07:53:33.396153+0000","last_fullsized":"2024-02-10T07:53:33.396153+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'9","last_scrub_stamp":"2024-02-10T07:51:13.306986+0000","last_deep_scrub":"414'9","last_deep_scrub_stamp":"2024-02-10T07:51:13.306986+0000","last_clean_scrub_stamp":"2024-02-10T07:51:13.306986+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":863,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397194+0000","last_change":"2024-02-10T07:48:30.384635+0000","last_active":"2024-02-10T07:53:33.397194+0000","last_peered":"2024-02-10T07:53:33.397194+0000","last_clean":"2024-02-10T07:53:33.397194+0000","last_became_active":"2024-02-10T07:47:47.966127+0000","last_became_peered":"2024-02-10T07:47:47.966127+0000","last_unstale":"2024-02-10T07:53:33.397194+0000","last_undegraded":"2024-02-10T07:53:33.397194+0000","last_fullsized":"2024-02-10T07:53:33.397194+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:30.384546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:30.384546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"430'35","reported_seq":941,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.408555+0000","last_change":"2024-02-10T07:49:06.308245+0000","last_active":"2024-02-10T07:53:33.408555+0000","last_peered":"2024-02-10T07:53:33.408555+0000","last_clean":"2024-02-10T07:53:33.408555+0000","last_became_active":"2024-02-10T07:49:06.261487+0000","last_became_peered":"2024-02-10T07:49:06.261487+0000","last_unstale":"2024-02-10T07:53:33.408555+0000","last_undegraded":"2024-02-10T07:53:33.408555+0000","last_fullsized":"2024-02-10T07:53:33.408555+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:48:12.278982+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-02-10T07:42:17.820959+0000","last_clean_scrub_stamp":"2024-02-10T07:48:12.278982+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":849,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.399787+0000","last_change":"2024-02-10T07:49:11.231819+0000","last_active":"2024-02-10T07:53:33.399787+0000","last_peered":"2024-02-10T07:53:33.399787+0000","last_clean":"2024-02-10T07:53:33.399787+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:53:33.399787+0000","last_undegraded":"2024-02-10T07:53:33.399787+0000","last_fullsized":"2024-02-10T07:53:33.399787+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.231704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.231704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":852,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.408797+0000","last_change":"2024-02-10T07:50:43.289955+0000","last_active":"2024-02-10T07:53:33.408797+0000","last_peered":"2024-02-10T07:53:33.408797+0000","last_clean":"2024-02-10T07:53:33.408797+0000","last_became_active":"2024-02-10T07:50:43.289011+0000","last_became_peered":"2024-02-10T07:50:43.289011+0000","last_unstale":"2024-02-10T07:53:33.408797+0000","last_undegraded":"2024-02-10T07:53:33.408797+0000","last_fullsized":"2024-02-10T07:53:33.408797+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.208451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:06.200009+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.208451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":358,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.406618+0000","last_change":"2024-02-10T07:53:21.437757+0000","last_active":"2024-02-10T07:53:33.406618+0000","last_peered":"2024-02-10T07:53:33.406618+0000","last_clean":"2024-02-10T07:53:33.406618+0000","last_became_active":"2024-02-10T07:49:06.258789+0000","last_became_peered":"2024-02-10T07:49:06.258789+0000","last_unstale":"2024-02-10T07:53:33.406618+0000","last_undegraded":"2024-02-10T07:53:33.406618+0000","last_fullsized":"2024-02-10T07:53:33.406618+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:21.437653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:53:21.437653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.318460+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"430'27","reported_seq":919,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.396197+0000","last_change":"2024-02-10T07:52:11.834589+0000","last_active":"2024-02-10T07:53:33.396197+0000","last_peered":"2024-02-10T07:53:33.396197+0000","last_clean":"2024-02-10T07:53:33.396197+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:53:33.396197+0000","last_undegraded":"2024-02-10T07:53:33.396197+0000","last_fullsized":"2024-02-10T07:53:33.396197+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'27","last_scrub_stamp":"2024-02-10T07:52:11.834490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:52:11.834490+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":865,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.394328+0000","last_change":"2024-02-10T07:50:43.284049+0000","last_active":"2024-02-10T07:53:33.394328+0000","last_peered":"2024-02-10T07:53:33.394328+0000","last_clean":"2024-02-10T07:53:33.394328+0000","last_became_active":"2024-02-10T07:50:43.283688+0000","last_became_peered":"2024-02-10T07:50:43.283688+0000","last_unstale":"2024-02-10T07:53:33.394328+0000","last_undegraded":"2024-02-10T07:53:33.394328+0000","last_fullsized":"2024-02-10T07:53:33.394328+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.087614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.087614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":870,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.396011+0000","last_change":"2024-02-10T07:52:22.845994+0000","last_active":"2024-02-10T07:53:33.396011+0000","last_peered":"2024-02-10T07:53:33.396011+0000","last_clean":"2024-02-10T07:53:33.396011+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:53:33.396011+0000","last_undegraded":"2024-02-10T07:53:33.396011+0000","last_fullsized":"2024-02-10T07:53:33.396011+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:22.845773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:52:22.845773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":879,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.408317+0000","last_change":"2024-02-10T07:49:06.259100+0000","last_active":"2024-02-10T07:53:33.408317+0000","last_peered":"2024-02-10T07:53:33.408317+0000","last_clean":"2024-02-10T07:53:33.408317+0000","last_became_active":"2024-02-10T07:49:06.258763+0000","last_became_peered":"2024-02-10T07:49:06.258763+0000","last_unstale":"2024-02-10T07:53:33.408317+0000","last_undegraded":"2024-02-10T07:53:33.408317+0000","last_fullsized":"2024-02-10T07:53:33.408317+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:29.188832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:29.188832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"430'14","reported_seq":194,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.409615+0000","last_change":"2024-02-10T07:53:28.121930+0000","last_active":"2024-02-10T07:53:33.409615+0000","last_peered":"2024-02-10T07:53:33.409615+0000","last_clean":"2024-02-10T07:53:33.409615+0000","last_became_active":"2024-02-10T07:49:06.250577+0000","last_became_peered":"2024-02-10T07:49:06.250577+0000","last_unstale":"2024-02-10T07:53:33.409615+0000","last_undegraded":"2024-02-10T07:53:33.409615+0000","last_fullsized":"2024-02-10T07:53:33.409615+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'14","last_scrub_stamp":"2024-02-10T07:53:28.117356+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:53:28.117356+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:28.261781+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":352,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.873820+0000","last_change":"2024-02-10T07:53:33.873865+0000","last_active":"2024-02-10T07:53:33.873820+0000","last_peered":"2024-02-10T07:53:33.873820+0000","last_clean":"2024-02-10T07:53:33.873820+0000","last_became_active":"2024-02-10T07:49:06.259341+0000","last_became_peered":"2024-02-10T07:49:06.259341+0000","last_unstale":"2024-02-10T07:53:33.873820+0000","last_undegraded":"2024-02-10T07:53:33.873820+0000","last_fullsized":"2024-02-10T07:53:33.873820+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:33.873724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:53:33.873724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:51.425277+0000","scrub_duration":324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":847,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.394627+0000","last_change":"2024-02-10T07:48:24.171978+0000","last_active":"2024-02-10T07:53:33.394627+0000","last_peered":"2024-02-10T07:53:33.394627+0000","last_clean":"2024-02-10T07:53:33.394627+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:53:33.394627+0000","last_undegraded":"2024-02-10T07:53:33.394627+0000","last_fullsized":"2024-02-10T07:53:33.394627+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:24.171859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:24.171859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":804,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.393189+0000","last_change":"2024-02-10T07:52:27.104889+0000","last_active":"2024-02-10T07:53:33.393189+0000","last_peered":"2024-02-10T07:53:33.393189+0000","last_clean":"2024-02-10T07:53:33.393189+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:53:33.393189+0000","last_undegraded":"2024-02-10T07:53:33.393189+0000","last_fullsized":"2024-02-10T07:53:33.393189+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:27.104803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:52:27.104803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:49.184078+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"430'27","reported_seq":825,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.394575+0000","last_change":"2024-02-10T07:49:53.245490+0000","last_active":"2024-02-10T07:53:33.394575+0000","last_peered":"2024-02-10T07:53:33.394575+0000","last_clean":"2024-02-10T07:53:33.394575+0000","last_became_active":"2024-02-10T07:49:06.239701+0000","last_became_peered":"2024-02-10T07:49:06.239701+0000","last_unstale":"2024-02-10T07:53:33.394575+0000","last_undegraded":"2024-02-10T07:53:33.394575+0000","last_fullsized":"2024-02-10T07:53:33.394575+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:53.245410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:53.245410+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":844,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.407030+0000","last_change":"2024-02-10T07:49:52.530010+0000","last_active":"2024-02-10T07:53:33.407030+0000","last_peered":"2024-02-10T07:53:33.407030+0000","last_clean":"2024-02-10T07:53:33.407030+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:53:33.407030+0000","last_undegraded":"2024-02-10T07:53:33.407030+0000","last_fullsized":"2024-02-10T07:53:33.407030+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.529908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:59.179342+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.529908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":352,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.407428+0000","last_change":"2024-02-10T07:53:19.493460+0000","last_active":"2024-02-10T07:53:33.407428+0000","last_peered":"2024-02-10T07:53:33.407428+0000","last_clean":"2024-02-10T07:53:33.407428+0000","last_became_active":"2024-02-10T07:49:06.265267+0000","last_became_peered":"2024-02-10T07:49:06.265267+0000","last_unstale":"2024-02-10T07:53:33.407428+0000","last_undegraded":"2024-02-10T07:53:33.407428+0000","last_fullsized":"2024-02-10T07:53:33.407428+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:19.492989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:19.492989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:32.822823+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":864,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.394961+0000","last_change":"2024-02-10T07:50:00.265308+0000","last_active":"2024-02-10T07:53:33.394961+0000","last_peered":"2024-02-10T07:53:33.394961+0000","last_clean":"2024-02-10T07:53:33.394961+0000","last_became_active":"2024-02-10T07:47:47.966472+0000","last_became_peered":"2024-02-10T07:47:47.966472+0000","last_unstale":"2024-02-10T07:53:33.394961+0000","last_undegraded":"2024-02-10T07:53:33.394961+0000","last_fullsized":"2024-02-10T07:53:33.394961+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.265208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.265208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"430'25","reported_seq":697,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397566+0000","last_change":"2024-02-10T07:50:43.310533+0000","last_active":"2024-02-10T07:53:33.397566+0000","last_peered":"2024-02-10T07:53:33.397566+0000","last_clean":"2024-02-10T07:53:33.397566+0000","last_became_active":"2024-02-10T07:50:43.296447+0000","last_became_peered":"2024-02-10T07:50:43.296447+0000","last_unstale":"2024-02-10T07:53:33.397566+0000","last_undegraded":"2024-02-10T07:53:33.397566+0000","last_fullsized":"2024-02-10T07:53:33.397566+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:09.116615+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-02-10T07:39:32.934269+0000","last_clean_scrub_stamp":"2024-02-10T07:49:09.116615+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":540,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:35.716265+0000","last_change":"2024-02-10T07:53:35.716416+0000","last_active":"2024-02-10T07:53:35.716265+0000","last_peered":"2024-02-10T07:53:35.716265+0000","last_clean":"2024-02-10T07:53:35.716265+0000","last_became_active":"2024-02-10T07:49:06.260569+0000","last_became_peered":"2024-02-10T07:49:06.260569+0000","last_unstale":"2024-02-10T07:53:35.716265+0000","last_undegraded":"2024-02-10T07:53:35.716265+0000","last_fullsized":"2024-02-10T07:53:35.716265+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:35.715987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:53:35.715987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:50.372195+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":877,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.398067+0000","last_change":"2024-02-10T07:50:02.685941+0000","last_active":"2024-02-10T07:53:33.398067+0000","last_peered":"2024-02-10T07:53:33.398067+0000","last_clean":"2024-02-10T07:53:33.398067+0000","last_became_active":"2024-02-10T07:49:06.252380+0000","last_became_peered":"2024-02-10T07:49:06.252380+0000","last_unstale":"2024-02-10T07:53:33.398067+0000","last_undegraded":"2024-02-10T07:53:33.398067+0000","last_fullsized":"2024-02-10T07:53:33.398067+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_clean_scrub_stamp":"2024-02-10T07:50:02.685831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":855,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402769+0000","last_change":"2024-02-10T07:50:00.345087+0000","last_active":"2024-02-10T07:53:33.402769+0000","last_peered":"2024-02-10T07:53:33.402769+0000","last_clean":"2024-02-10T07:53:33.402769+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:53:33.402769+0000","last_undegraded":"2024-02-10T07:53:33.402769+0000","last_fullsized":"2024-02-10T07:53:33.402769+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.344993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.344993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"430'32","reported_seq":927,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402234+0000","last_change":"2024-02-10T07:47:43.494877+0000","last_active":"2024-02-10T07:53:33.402234+0000","last_peered":"2024-02-10T07:53:33.402234+0000","last_clean":"2024-02-10T07:53:33.402234+0000","last_became_active":"2024-02-10T07:46:10.721336+0000","last_became_peered":"2024-02-10T07:46:10.721336+0000","last_unstale":"2024-02-10T07:53:33.402234+0000","last_undegraded":"2024-02-10T07:53:33.402234+0000","last_fullsized":"2024-02-10T07:53:33.402234+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:47:43.494350+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.494350+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":849,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.408357+0000","last_change":"2024-02-10T07:49:06.262127+0000","last_active":"2024-02-10T07:53:33.408357+0000","last_peered":"2024-02-10T07:53:33.408357+0000","last_clean":"2024-02-10T07:53:33.408357+0000","last_became_active":"2024-02-10T07:49:06.261570+0000","last_became_peered":"2024-02-10T07:49:06.261570+0000","last_unstale":"2024-02-10T07:53:33.408357+0000","last_undegraded":"2024-02-10T07:53:33.408357+0000","last_fullsized":"2024-02-10T07:53:33.408357+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:43.186809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.186809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"356'11","reported_seq":353,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.399618+0000","last_change":"2024-02-10T07:48:25.339974+0000","last_active":"2024-02-10T07:53:33.399618+0000","last_peered":"2024-02-10T07:53:33.399618+0000","last_clean":"2024-02-10T07:53:33.399618+0000","last_became_active":"2024-02-10T07:48:02.371888+0000","last_became_peered":"2024-02-10T07:48:02.371888+0000","last_unstale":"2024-02-10T07:53:33.399618+0000","last_undegraded":"2024-02-10T07:53:33.399618+0000","last_fullsized":"2024-02-10T07:53:33.399618+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'11","last_scrub_stamp":"2024-02-10T07:48:25.339855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:25.339855+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":875,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.394550+0000","last_change":"2024-02-10T07:50:43.283263+0000","last_active":"2024-02-10T07:53:33.394550+0000","last_peered":"2024-02-10T07:53:33.394550+0000","last_clean":"2024-02-10T07:53:33.394550+0000","last_became_active":"2024-02-10T07:50:43.282925+0000","last_became_peered":"2024-02-10T07:50:43.282925+0000","last_unstale":"2024-02-10T07:53:33.394550+0000","last_undegraded":"2024-02-10T07:53:33.394550+0000","last_fullsized":"2024-02-10T07:53:33.394550+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.335455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.335455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"430'35","reported_seq":491,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397639+0000","last_change":"2024-02-10T07:49:12.150442+0000","last_active":"2024-02-10T07:53:33.397639+0000","last_peered":"2024-02-10T07:53:33.397639+0000","last_clean":"2024-02-10T07:53:33.397639+0000","last_became_active":"2024-02-10T07:48:02.329187+0000","last_became_peered":"2024-02-10T07:48:02.329187+0000","last_unstale":"2024-02-10T07:53:33.397639+0000","last_undegraded":"2024-02-10T07:53:33.397639+0000","last_fullsized":"2024-02-10T07:53:33.397639+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:49:12.150298+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:49:12.150298+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":855,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402889+0000","last_change":"2024-02-10T07:50:43.285390+0000","last_active":"2024-02-10T07:53:33.402889+0000","last_peered":"2024-02-10T07:53:33.402889+0000","last_clean":"2024-02-10T07:53:33.402889+0000","last_became_active":"2024-02-10T07:50:43.284846+0000","last_became_peered":"2024-02-10T07:50:43.284846+0000","last_unstale":"2024-02-10T07:53:33.402889+0000","last_undegraded":"2024-02-10T07:53:33.402889+0000","last_fullsized":"2024-02-10T07:53:33.402889+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:01.513972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:01.513972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"405'15","reported_seq":509,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.395132+0000","last_change":"2024-02-10T07:50:43.310228+0000","last_active":"2024-02-10T07:53:33.395132+0000","last_peered":"2024-02-10T07:53:33.395132+0000","last_clean":"2024-02-10T07:53:33.395132+0000","last_became_active":"2024-02-10T07:50:43.285213+0000","last_became_peered":"2024-02-10T07:50:43.285213+0000","last_unstale":"2024-02-10T07:53:33.395132+0000","last_undegraded":"2024-02-10T07:53:33.395132+0000","last_fullsized":"2024-02-10T07:53:33.395132+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'13","last_scrub_stamp":"2024-02-10T07:49:00.366610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.366610+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":870,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.407485+0000","last_change":"2024-02-10T07:50:07.501014+0000","last_active":"2024-02-10T07:53:33.407485+0000","last_peered":"2024-02-10T07:53:33.407485+0000","last_clean":"2024-02-10T07:53:33.407485+0000","last_became_active":"2024-02-10T07:49:06.257043+0000","last_became_peered":"2024-02-10T07:49:06.257043+0000","last_unstale":"2024-02-10T07:53:33.407485+0000","last_undegraded":"2024-02-10T07:53:33.407485+0000","last_fullsized":"2024-02-10T07:53:33.407485+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.500890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"430'21","reported_seq":617,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.395271+0000","last_change":"2024-02-10T07:49:06.277154+0000","last_active":"2024-02-10T07:53:33.395271+0000","last_peered":"2024-02-10T07:53:33.395271+0000","last_clean":"2024-02-10T07:53:33.395271+0000","last_became_active":"2024-02-10T07:49:06.255239+0000","last_became_peered":"2024-02-10T07:49:06.255239+0000","last_unstale":"2024-02-10T07:53:33.395271+0000","last_undegraded":"2024-02-10T07:53:33.395271+0000","last_fullsized":"2024-02-10T07:53:33.395271+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:39.610017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:39.610017+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":108,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.404995+0000","last_change":"2024-02-10T07:50:43.282730+0000","last_active":"2024-02-10T07:53:33.404995+0000","last_peered":"2024-02-10T07:53:33.404995+0000","last_clean":"2024-02-10T07:53:33.404995+0000","last_became_active":"2024-02-10T07:50:43.281549+0000","last_became_peered":"2024-02-10T07:50:43.281549+0000","last_unstale":"2024-02-10T07:53:33.404995+0000","last_undegraded":"2024-02-10T07:53:33.404995+0000","last_fullsized":"2024-02-10T07:53:33.404995+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:44.324046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:49:44.324046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":211,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.395455+0000","last_change":"2024-02-10T07:52:34.860375+0000","last_active":"2024-02-10T07:53:33.395455+0000","last_peered":"2024-02-10T07:53:33.395455+0000","last_clean":"2024-02-10T07:53:33.395455+0000","last_became_active":"2024-02-10T07:49:06.262933+0000","last_became_peered":"2024-02-10T07:49:06.262933+0000","last_unstale":"2024-02-10T07:53:33.395455+0000","last_undegraded":"2024-02-10T07:53:33.395455+0000","last_fullsized":"2024-02-10T07:53:33.395455+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:34.860273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:52:34.860273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:47.105853+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":863,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.394442+0000","last_change":"2024-02-10T07:49:59.278666+0000","last_active":"2024-02-10T07:53:33.394442+0000","last_peered":"2024-02-10T07:53:33.394442+0000","last_clean":"2024-02-10T07:53:33.394442+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:53:33.394442+0000","last_undegraded":"2024-02-10T07:53:33.394442+0000","last_fullsized":"2024-02-10T07:53:33.394442+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.278574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.278574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"430'25","reported_seq":923,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.408647+0000","last_change":"2024-02-10T07:49:06.306965+0000","last_active":"2024-02-10T07:53:33.408647+0000","last_peered":"2024-02-10T07:53:33.408647+0000","last_clean":"2024-02-10T07:53:33.408647+0000","last_became_active":"2024-02-10T07:49:06.260962+0000","last_became_peered":"2024-02-10T07:49:06.260962+0000","last_unstale":"2024-02-10T07:53:33.408647+0000","last_undegraded":"2024-02-10T07:53:33.408647+0000","last_fullsized":"2024-02-10T07:53:33.408647+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:47:42.150605+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.150605+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":842,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.394852+0000","last_change":"2024-02-10T07:48:58.511630+0000","last_active":"2024-02-10T07:53:33.394852+0000","last_peered":"2024-02-10T07:53:33.394852+0000","last_clean":"2024-02-10T07:53:33.394852+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:53:33.394852+0000","last_undegraded":"2024-02-10T07:53:33.394852+0000","last_fullsized":"2024-02-10T07:53:33.394852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:58.511251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.511251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":174,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.408624+0000","last_change":"2024-02-10T07:50:04.978177+0000","last_active":"2024-02-10T07:53:33.408624+0000","last_peered":"2024-02-10T07:53:33.408624+0000","last_clean":"2024-02-10T07:53:33.408624+0000","last_became_active":"2024-02-10T07:49:06.250832+0000","last_became_peered":"2024-02-10T07:49:06.250832+0000","last_unstale":"2024-02-10T07:53:33.408624+0000","last_undegraded":"2024-02-10T07:53:33.408624+0000","last_fullsized":"2024-02-10T07:53:33.408624+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:04.978097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.978097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":895,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.395162+0000","last_change":"2024-02-10T07:52:16.809094+0000","last_active":"2024-02-10T07:53:33.395162+0000","last_peered":"2024-02-10T07:53:33.395162+0000","last_clean":"2024-02-10T07:53:33.395162+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:53:33.395162+0000","last_undegraded":"2024-02-10T07:53:33.395162+0000","last_fullsized":"2024-02-10T07:53:33.395162+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:16.808978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:52:16.808978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:36.921179+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"430'18","reported_seq":199,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.408932+0000","last_change":"2024-02-10T07:53:29.079571+0000","last_active":"2024-02-10T07:53:33.408932+0000","last_peered":"2024-02-10T07:53:33.408932+0000","last_clean":"2024-02-10T07:53:33.408932+0000","last_became_active":"2024-02-10T07:49:06.258281+0000","last_became_peered":"2024-02-10T07:49:06.258281+0000","last_unstale":"2024-02-10T07:53:33.408932+0000","last_undegraded":"2024-02-10T07:53:33.408932+0000","last_fullsized":"2024-02-10T07:53:33.408932+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'18","last_scrub_stamp":"2024-02-10T07:53:29.076952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:53:29.076952+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:30.380886+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"453'12495","reported_seq":12630,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:39.960657+0000","last_change":"2024-02-10T07:49:53.374457+0000","last_active":"2024-02-10T07:53:39.960657+0000","last_peered":"2024-02-10T07:53:39.960657+0000","last_clean":"2024-02-10T07:53:39.960657+0000","last_became_active":"2024-02-10T07:49:53.374185+0000","last_became_peered":"2024-02-10T07:49:53.374185+0000","last_unstale":"2024-02-10T07:53:39.960657+0000","last_undegraded":"2024-02-10T07:53:39.960657+0000","last_fullsized":"2024-02-10T07:53:39.960657+0000","mapping_epoch":391,"log_start":"430'7600","ondisk_log_start":"430'7600","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49086464,"num_objects":749,"num_object_clones":0,"num_object_copies":2247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12495,"num_write_kb":423808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":384,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.396754+0000","last_change":"2024-02-10T07:52:21.875347+0000","last_active":"2024-02-10T07:53:33.396754+0000","last_peered":"2024-02-10T07:53:33.396754+0000","last_clean":"2024-02-10T07:53:33.396754+0000","last_became_active":"2024-02-10T07:48:02.366002+0000","last_became_peered":"2024-02-10T07:48:02.366002+0000","last_unstale":"2024-02-10T07:53:33.396754+0000","last_undegraded":"2024-02-10T07:53:33.396754+0000","last_fullsized":"2024-02-10T07:53:33.396754+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:21.875258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:21.875258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:47.914931+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"245'4","reported_seq":877,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.403225+0000","last_change":"2024-02-10T07:49:08.311056+0000","last_active":"2024-02-10T07:53:33.403225+0000","last_peered":"2024-02-10T07:53:33.403225+0000","last_clean":"2024-02-10T07:53:33.403225+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:53:33.403225+0000","last_undegraded":"2024-02-10T07:53:33.403225+0000","last_fullsized":"2024-02-10T07:53:33.403225+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'4","last_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_deep_scrub":"245'4","last_deep_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.310917+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"430'46","reported_seq":621,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.394522+0000","last_change":"2024-02-10T07:48:11.967227+0000","last_active":"2024-02-10T07:53:33.394522+0000","last_peered":"2024-02-10T07:53:33.394522+0000","last_clean":"2024-02-10T07:53:33.394522+0000","last_became_active":"2024-02-10T07:48:02.330130+0000","last_became_peered":"2024-02-10T07:48:02.330130+0000","last_unstale":"2024-02-10T07:53:33.394522+0000","last_undegraded":"2024-02-10T07:53:33.394522+0000","last_fullsized":"2024-02-10T07:53:33.394522+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:48:11.966957+0000","last_deep_scrub":"246'22","last_deep_scrub_stamp":"2024-02-10T07:41:26.771112+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.966957+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":1126,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:36.644013+0000","last_change":"2024-02-10T07:48:53.423840+0000","last_active":"2024-02-10T07:53:36.644013+0000","last_peered":"2024-02-10T07:53:36.644013+0000","last_clean":"2024-02-10T07:53:36.644013+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:53:36.644013+0000","last_undegraded":"2024-02-10T07:53:36.644013+0000","last_fullsized":"2024-02-10T07:53:36.644013+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:48:53.423613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:53.423613+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"453'12119","reported_seq":20953,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:40.327871+0000","last_change":"2024-02-10T07:53:07.243749+0000","last_active":"2024-02-10T07:53:40.327871+0000","last_peered":"2024-02-10T07:53:40.327871+0000","last_clean":"2024-02-10T07:53:40.327871+0000","last_became_active":"2024-02-10T07:53:07.218448+0000","last_became_peered":"2024-02-10T07:53:07.218448+0000","last_unstale":"2024-02-10T07:53:40.327871+0000","last_undegraded":"2024-02-10T07:53:40.327871+0000","last_fullsized":"2024-02-10T07:53:40.327871+0000","mapping_epoch":444,"log_start":"421'6300","ondisk_log_start":"421'6300","created":391,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":5819,"log_dups_size":0,"ondisk_log_size":5819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43057152,"num_objects":657,"num_object_clones":0,"num_object_copies":1971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12119,"num_write_kb":408832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4344,"num_bytes_recovered":284688384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":689,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.407784+0000","last_change":"2024-02-10T07:53:32.243227+0000","last_active":"2024-02-10T07:53:33.407784+0000","last_peered":"2024-02-10T07:53:33.407784+0000","last_clean":"2024-02-10T07:53:33.407784+0000","last_became_active":"2024-02-10T07:47:47.966183+0000","last_became_peered":"2024-02-10T07:47:47.966183+0000","last_unstale":"2024-02-10T07:53:33.407784+0000","last_undegraded":"2024-02-10T07:53:33.407784+0000","last_fullsized":"2024-02-10T07:53:33.407784+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:32.242307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:53:32.242307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:47.454633+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":855,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397840+0000","last_change":"2024-02-10T07:48:56.416159+0000","last_active":"2024-02-10T07:53:33.397840+0000","last_peered":"2024-02-10T07:53:33.397840+0000","last_clean":"2024-02-10T07:53:33.397840+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:53:33.397840+0000","last_undegraded":"2024-02-10T07:53:33.397840+0000","last_fullsized":"2024-02-10T07:53:33.397840+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.416037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.416037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"430'29","reported_seq":971,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402794+0000","last_change":"2024-02-10T07:49:43.339027+0000","last_active":"2024-02-10T07:53:33.402794+0000","last_peered":"2024-02-10T07:53:33.402794+0000","last_clean":"2024-02-10T07:53:33.402794+0000","last_became_active":"2024-02-10T07:49:06.243569+0000","last_became_peered":"2024-02-10T07:49:06.243569+0000","last_unstale":"2024-02-10T07:53:33.402794+0000","last_undegraded":"2024-02-10T07:53:33.402794+0000","last_fullsized":"2024-02-10T07:53:33.402794+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-02-10T07:49:43.338928+0000","last_deep_scrub":"245'17","last_deep_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_clean_scrub_stamp":"2024-02-10T07:49:43.338928+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":108,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.405294+0000","last_change":"2024-02-10T07:50:43.285105+0000","last_active":"2024-02-10T07:53:33.405294+0000","last_peered":"2024-02-10T07:53:33.405294+0000","last_clean":"2024-02-10T07:53:33.405294+0000","last_became_active":"2024-02-10T07:50:43.284749+0000","last_became_peered":"2024-02-10T07:50:43.284749+0000","last_unstale":"2024-02-10T07:53:33.405294+0000","last_undegraded":"2024-02-10T07:53:33.405294+0000","last_fullsized":"2024-02-10T07:53:33.405294+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:04.308807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.308807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"453'12473","reported_seq":12608,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:38.483602+0000","last_change":"2024-02-10T07:49:53.374953+0000","last_active":"2024-02-10T07:53:38.483602+0000","last_peered":"2024-02-10T07:53:38.483602+0000","last_clean":"2024-02-10T07:53:38.483602+0000","last_became_active":"2024-02-10T07:49:53.374631+0000","last_became_peered":"2024-02-10T07:49:53.374631+0000","last_unstale":"2024-02-10T07:53:38.483602+0000","last_undegraded":"2024-02-10T07:53:38.483602+0000","last_fullsized":"2024-02-10T07:53:38.483602+0000","mapping_epoch":391,"log_start":"430'7600","ondisk_log_start":"430'7600","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53673984,"num_objects":819,"num_object_clones":0,"num_object_copies":2457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12473,"num_write_kb":425344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":877,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.394124+0000","last_change":"2024-02-10T07:50:43.281637+0000","last_active":"2024-02-10T07:53:33.394124+0000","last_peered":"2024-02-10T07:53:33.394124+0000","last_clean":"2024-02-10T07:53:33.394124+0000","last_became_active":"2024-02-10T07:50:43.281262+0000","last_became_peered":"2024-02-10T07:50:43.281262+0000","last_unstale":"2024-02-10T07:53:33.394124+0000","last_undegraded":"2024-02-10T07:53:33.394124+0000","last_fullsized":"2024-02-10T07:53:33.394124+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.361139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":108,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.405198+0000","last_change":"2024-02-10T07:50:43.281022+0000","last_active":"2024-02-10T07:53:33.405198+0000","last_peered":"2024-02-10T07:53:33.405198+0000","last_clean":"2024-02-10T07:53:33.405198+0000","last_became_active":"2024-02-10T07:50:43.280617+0000","last_became_peered":"2024-02-10T07:50:43.280617+0000","last_unstale":"2024-02-10T07:53:33.405198+0000","last_undegraded":"2024-02-10T07:53:33.405198+0000","last_fullsized":"2024-02-10T07:53:33.405198+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:21.172601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:21.172601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":872,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.395196+0000","last_change":"2024-02-10T07:50:07.252577+0000","last_active":"2024-02-10T07:53:33.395196+0000","last_peered":"2024-02-10T07:53:33.395196+0000","last_clean":"2024-02-10T07:53:33.395196+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:53:33.395196+0000","last_undegraded":"2024-02-10T07:53:33.395196+0000","last_fullsized":"2024-02-10T07:53:33.395196+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.252458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.252458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"430'51","reported_seq":954,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.396929+0000","last_change":"2024-02-10T07:51:10.348915+0000","last_active":"2024-02-10T07:53:33.396929+0000","last_peered":"2024-02-10T07:53:33.396929+0000","last_clean":"2024-02-10T07:53:33.396929+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:53:33.396929+0000","last_undegraded":"2024-02-10T07:53:33.396929+0000","last_fullsized":"2024-02-10T07:53:33.396929+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'35","last_scrub_stamp":"2024-02-10T07:51:10.348752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:51:10.348752+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"453'12497","reported_seq":15084,"reported_epoch":453,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-10T07:53:38.467539+0000","last_change":"2024-02-10T07:53:05.273271+0000","last_active":"2024-02-10T07:53:38.467539+0000","last_peered":"2024-02-10T07:53:38.467539+0000","last_clean":"2024-02-10T07:50:42.227612+0000","last_became_active":"2024-02-10T07:50:44.671664+0000","last_became_peered":"2024-02-10T07:50:44.671664+0000","last_unstale":"2024-02-10T07:53:38.467539+0000","last_undegraded":"2024-02-10T07:50:44.289079+0000","last_fullsized":"2024-02-10T07:50:44.263721+0000","mapping_epoch":405,"log_start":"430'7600","ondisk_log_start":"430'7600","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4897,"log_dups_size":0,"ondisk_log_size":4897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51576832,"num_objects":787,"num_object_clones":0,"num_object_copies":2361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12497,"num_write_kb":425088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1225,"num_bytes_recovered":17498112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6],"avail_no_missing":["7","1","6"],"object_location_counts":[{"shards":"6,7","objects":-2485}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":866,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.411763+0000","last_change":"2024-02-10T07:50:43.284817+0000","last_active":"2024-02-10T07:53:33.411763+0000","last_peered":"2024-02-10T07:53:33.411763+0000","last_clean":"2024-02-10T07:53:33.411763+0000","last_became_active":"2024-02-10T07:50:43.284289+0000","last_became_peered":"2024-02-10T07:50:43.284289+0000","last_unstale":"2024-02-10T07:53:33.411763+0000","last_undegraded":"2024-02-10T07:53:33.411763+0000","last_fullsized":"2024-02-10T07:53:33.411763+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:00.336175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.336175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":108,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.404513+0000","last_change":"2024-02-10T07:50:43.295279+0000","last_active":"2024-02-10T07:53:33.404513+0000","last_peered":"2024-02-10T07:53:33.404513+0000","last_clean":"2024-02-10T07:53:33.404513+0000","last_became_active":"2024-02-10T07:50:43.294972+0000","last_became_peered":"2024-02-10T07:50:43.294972+0000","last_unstale":"2024-02-10T07:53:33.404513+0000","last_undegraded":"2024-02-10T07:53:33.404513+0000","last_fullsized":"2024-02-10T07:53:33.404513+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:28.822700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:28.822700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":183,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:35.090294+0000","last_change":"2024-02-10T07:53:35.090294+0000","last_active":"2024-02-10T07:53:35.090294+0000","last_peered":"2024-02-10T07:53:35.090294+0000","last_clean":"2024-02-10T07:53:35.090294+0000","last_became_active":"2024-02-10T07:49:06.249431+0000","last_became_peered":"2024-02-10T07:49:06.249431+0000","last_unstale":"2024-02-10T07:53:35.090294+0000","last_undegraded":"2024-02-10T07:53:35.090294+0000","last_fullsized":"2024-02-10T07:53:35.090294+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:35.090175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:53:35.090175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:50.135166+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"430'21","reported_seq":885,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.408201+0000","last_change":"2024-02-10T07:48:59.304277+0000","last_active":"2024-02-10T07:53:33.408201+0000","last_peered":"2024-02-10T07:53:33.408201+0000","last_clean":"2024-02-10T07:53:33.408201+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:53:33.408201+0000","last_undegraded":"2024-02-10T07:53:33.408201+0000","last_fullsized":"2024-02-10T07:53:33.408201+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:48:59.304199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.304199+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"453'12454","reported_seq":12611,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:41.534114+0000","last_change":"2024-02-10T07:53:31.672874+0000","last_active":"2024-02-10T07:53:41.534114+0000","last_peered":"2024-02-10T07:53:41.534114+0000","last_clean":"2024-02-10T07:53:41.534114+0000","last_became_active":"2024-02-10T07:49:53.283642+0000","last_became_peered":"2024-02-10T07:49:53.283642+0000","last_unstale":"2024-02-10T07:53:41.534114+0000","last_undegraded":"2024-02-10T07:53:41.534114+0000","last_fullsized":"2024-02-10T07:53:41.534114+0000","mapping_epoch":391,"log_start":"430'7600","ondisk_log_start":"430'7600","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'11916","last_scrub_stamp":"2024-02-10T07:53:31.672693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:53:31.672693+0000","objects_scrubbed":1187,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:54.159495+0000","scrub_duration":623,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41680896,"num_objects":636,"num_object_clones":0,"num_object_copies":1908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12454,"num_write_kb":418880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":183,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:41.173049+0000","last_change":"2024-02-10T07:53:41.173096+0000","last_active":"2024-02-10T07:53:41.173049+0000","last_peered":"2024-02-10T07:53:41.173049+0000","last_clean":"2024-02-10T07:53:41.173049+0000","last_became_active":"2024-02-10T07:49:06.238367+0000","last_became_peered":"2024-02-10T07:49:06.238367+0000","last_unstale":"2024-02-10T07:53:41.173049+0000","last_undegraded":"2024-02-10T07:53:41.173049+0000","last_fullsized":"2024-02-10T07:53:41.173049+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:41.172967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:53:41.172967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:48.222331+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":275,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397884+0000","last_change":"2024-02-10T07:49:10.087939+0000","last_active":"2024-02-10T07:53:33.397884+0000","last_peered":"2024-02-10T07:53:33.397884+0000","last_clean":"2024-02-10T07:53:33.397884+0000","last_became_active":"2024-02-10T07:48:02.328316+0000","last_became_peered":"2024-02-10T07:48:02.328316+0000","last_unstale":"2024-02-10T07:53:33.397884+0000","last_undegraded":"2024-02-10T07:53:33.397884+0000","last_fullsized":"2024-02-10T07:53:33.397884+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:49:10.087795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:49:10.087795+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":848,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402063+0000","last_change":"2024-02-10T07:49:19.198419+0000","last_active":"2024-02-10T07:53:33.402063+0000","last_peered":"2024-02-10T07:53:33.402063+0000","last_clean":"2024-02-10T07:53:33.402063+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:53:33.402063+0000","last_undegraded":"2024-02-10T07:53:33.402063+0000","last_fullsized":"2024-02-10T07:53:33.402063+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:19.198312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:04.220328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.198312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"430'59","reported_seq":1314,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397870+0000","last_change":"2024-02-10T07:50:43.327112+0000","last_active":"2024-02-10T07:53:33.397870+0000","last_peered":"2024-02-10T07:53:33.397870+0000","last_clean":"2024-02-10T07:53:33.397870+0000","last_became_active":"2024-02-10T07:50:43.297262+0000","last_became_peered":"2024-02-10T07:50:43.297262+0000","last_unstale":"2024-02-10T07:53:33.397870+0000","last_undegraded":"2024-02-10T07:53:33.397870+0000","last_fullsized":"2024-02-10T07:53:33.397870+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'46","last_scrub_stamp":"2024-02-10T07:49:55.714323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:55.714323+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":856,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.407552+0000","last_change":"2024-02-10T07:49:17.357664+0000","last_active":"2024-02-10T07:53:33.407552+0000","last_peered":"2024-02-10T07:53:33.407552+0000","last_clean":"2024-02-10T07:53:33.407552+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:53:33.407552+0000","last_undegraded":"2024-02-10T07:53:33.407552+0000","last_fullsized":"2024-02-10T07:53:33.407552+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.357533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:22.786949+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.357533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"453'12383","reported_seq":12581,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:38.538820+0000","last_change":"2024-02-10T07:51:07.529876+0000","last_active":"2024-02-10T07:53:38.538820+0000","last_peered":"2024-02-10T07:53:38.538820+0000","last_clean":"2024-02-10T07:53:38.538820+0000","last_became_active":"2024-02-10T07:49:53.374133+0000","last_became_peered":"2024-02-10T07:49:53.374133+0000","last_unstale":"2024-02-10T07:53:38.538820+0000","last_undegraded":"2024-02-10T07:53:38.538820+0000","last_fullsized":"2024-02-10T07:53:38.538820+0000","mapping_epoch":391,"log_start":"423'6800","ondisk_log_start":"423'6800","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'5453","last_scrub_stamp":"2024-02-10T07:51:07.529708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:51:07.529708+0000","objects_scrubbed":5421,"log_size":5583,"log_dups_size":0,"ondisk_log_size":5583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1297,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52887552,"num_objects":807,"num_object_clones":0,"num_object_copies":2421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12383,"num_write_kb":422080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":845,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.395353+0000","last_change":"2024-02-10T07:47:50.081222+0000","last_active":"2024-02-10T07:53:33.395353+0000","last_peered":"2024-02-10T07:53:33.395353+0000","last_clean":"2024-02-10T07:53:33.395353+0000","last_became_active":"2024-02-10T07:47:48.149256+0000","last_became_peered":"2024-02-10T07:47:48.149256+0000","last_unstale":"2024-02-10T07:53:33.395353+0000","last_undegraded":"2024-02-10T07:53:33.395353+0000","last_fullsized":"2024-02-10T07:53:33.395353+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.080714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:42.730875+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.080714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":874,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.398651+0000","last_change":"2024-02-10T07:50:43.288780+0000","last_active":"2024-02-10T07:53:33.398651+0000","last_peered":"2024-02-10T07:53:33.398651+0000","last_clean":"2024-02-10T07:53:33.398651+0000","last_became_active":"2024-02-10T07:50:43.288388+0000","last_became_peered":"2024-02-10T07:50:43.288388+0000","last_unstale":"2024-02-10T07:53:33.398651+0000","last_undegraded":"2024-02-10T07:53:33.398651+0000","last_fullsized":"2024-02-10T07:53:33.398651+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:49.886354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:49.886354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":108,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.404544+0000","last_change":"2024-02-10T07:50:43.287356+0000","last_active":"2024-02-10T07:53:33.404544+0000","last_peered":"2024-02-10T07:53:33.404544+0000","last_clean":"2024-02-10T07:53:33.404544+0000","last_became_active":"2024-02-10T07:50:43.287043+0000","last_became_peered":"2024-02-10T07:50:43.287043+0000","last_unstale":"2024-02-10T07:53:33.404544+0000","last_undegraded":"2024-02-10T07:53:33.404544+0000","last_fullsized":"2024-02-10T07:53:33.404544+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:15.234112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.234112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"430'21","reported_seq":215,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.399274+0000","last_change":"2024-02-10T07:49:04.340445+0000","last_active":"2024-02-10T07:53:33.399274+0000","last_peered":"2024-02-10T07:53:33.399274+0000","last_clean":"2024-02-10T07:53:33.399274+0000","last_became_active":"2024-02-10T07:48:35.057662+0000","last_became_peered":"2024-02-10T07:48:35.057662+0000","last_unstale":"2024-02-10T07:53:33.399274+0000","last_undegraded":"2024-02-10T07:53:33.399274+0000","last_fullsized":"2024-02-10T07:53:33.399274+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'17","last_scrub_stamp":"2024-02-10T07:49:04.340366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.340366+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":108,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.405065+0000","last_change":"2024-02-10T07:50:43.280896+0000","last_active":"2024-02-10T07:53:33.405065+0000","last_peered":"2024-02-10T07:53:33.405065+0000","last_clean":"2024-02-10T07:53:33.405065+0000","last_became_active":"2024-02-10T07:50:43.279125+0000","last_became_peered":"2024-02-10T07:50:43.279125+0000","last_unstale":"2024-02-10T07:53:33.405065+0000","last_undegraded":"2024-02-10T07:53:33.405065+0000","last_fullsized":"2024-02-10T07:53:33.405065+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:50.232544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.232544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"453'12059","reported_seq":12255,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:38.542025+0000","last_change":"2024-02-10T07:51:04.295699+0000","last_active":"2024-02-10T07:53:38.542025+0000","last_peered":"2024-02-10T07:53:38.542025+0000","last_clean":"2024-02-10T07:53:38.542025+0000","last_became_active":"2024-02-10T07:49:53.374185+0000","last_became_peered":"2024-02-10T07:49:53.374185+0000","last_unstale":"2024-02-10T07:53:38.542025+0000","last_undegraded":"2024-02-10T07:53:38.542025+0000","last_fullsized":"2024-02-10T07:53:38.542025+0000","mapping_epoch":391,"log_start":"422'6500","ondisk_log_start":"422'6500","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'5121","last_scrub_stamp":"2024-02-10T07:51:04.295599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:51:04.295599+0000","objects_scrubbed":5106,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53280768,"num_objects":813,"num_object_clones":0,"num_object_copies":2439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12059,"num_write_kb":411904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":264,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402423+0000","last_change":"2024-02-10T07:48:02.330253+0000","last_active":"2024-02-10T07:53:33.402423+0000","last_peered":"2024-02-10T07:53:33.402423+0000","last_clean":"2024-02-10T07:53:33.402423+0000","last_became_active":"2024-02-10T07:48:02.329687+0000","last_became_peered":"2024-02-10T07:48:02.329687+0000","last_unstale":"2024-02-10T07:53:33.402423+0000","last_undegraded":"2024-02-10T07:53:33.402423+0000","last_fullsized":"2024-02-10T07:53:33.402423+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:47:49.750508+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.750508+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1101,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.394282+0000","last_change":"2024-02-10T07:48:15.033077+0000","last_active":"2024-02-10T07:53:33.394282+0000","last_peered":"2024-02-10T07:53:33.394282+0000","last_clean":"2024-02-10T07:53:33.394282+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:53:33.394282+0000","last_undegraded":"2024-02-10T07:53:33.394282+0000","last_fullsized":"2024-02-10T07:53:33.394282+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:48:15.032944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.032944+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":493,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397431+0000","last_change":"2024-02-10T07:50:58.025300+0000","last_active":"2024-02-10T07:53:33.397431+0000","last_peered":"2024-02-10T07:53:33.397431+0000","last_clean":"2024-02-10T07:53:33.397431+0000","last_became_active":"2024-02-10T07:49:06.256223+0000","last_became_peered":"2024-02-10T07:49:06.256223+0000","last_unstale":"2024-02-10T07:53:33.397431+0000","last_undegraded":"2024-02-10T07:53:33.397431+0000","last_fullsized":"2024-02-10T07:53:33.397431+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:58.025193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:58.025193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"430'48","reported_seq":1203,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.396530+0000","last_change":"2024-02-10T07:52:03.839258+0000","last_active":"2024-02-10T07:53:33.396530+0000","last_peered":"2024-02-10T07:53:33.396530+0000","last_clean":"2024-02-10T07:53:33.396530+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:53:33.396530+0000","last_undegraded":"2024-02-10T07:53:33.396530+0000","last_fullsized":"2024-02-10T07:53:33.396530+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'48","last_scrub_stamp":"2024-02-10T07:52:03.839174+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:52:03.839174+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":877,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.395284+0000","last_change":"2024-02-10T07:50:43.285613+0000","last_active":"2024-02-10T07:53:33.395284+0000","last_peered":"2024-02-10T07:53:33.395284+0000","last_clean":"2024-02-10T07:53:33.395284+0000","last_became_active":"2024-02-10T07:50:43.284708+0000","last_became_peered":"2024-02-10T07:50:43.284708+0000","last_unstale":"2024-02-10T07:53:33.395284+0000","last_undegraded":"2024-02-10T07:53:33.395284+0000","last_fullsized":"2024-02-10T07:53:33.395284+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:13.369607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.736901+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.369607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"453'12186","reported_seq":12381,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:38.532629+0000","last_change":"2024-02-10T07:52:10.380821+0000","last_active":"2024-02-10T07:53:38.532629+0000","last_peered":"2024-02-10T07:53:38.532629+0000","last_clean":"2024-02-10T07:53:38.532629+0000","last_became_active":"2024-02-10T07:49:53.283492+0000","last_became_peered":"2024-02-10T07:49:53.283492+0000","last_unstale":"2024-02-10T07:53:38.532629+0000","last_undegraded":"2024-02-10T07:53:38.532629+0000","last_fullsized":"2024-02-10T07:53:38.532629+0000","mapping_epoch":391,"log_start":"422'6600","ondisk_log_start":"422'6600","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"431'7934","last_scrub_stamp":"2024-02-10T07:52:10.380570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:52:10.380570+0000","objects_scrubbed":5089,"log_size":5586,"log_dups_size":0,"ondisk_log_size":5586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:38.547867+0000","scrub_duration":1510,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51773440,"num_objects":790,"num_object_clones":0,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12186,"num_write_kb":415232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":648,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397542+0000","last_change":"2024-02-10T07:50:07.589020+0000","last_active":"2024-02-10T07:53:33.397542+0000","last_peered":"2024-02-10T07:53:33.397542+0000","last_clean":"2024-02-10T07:53:33.397542+0000","last_became_active":"2024-02-10T07:49:06.270457+0000","last_became_peered":"2024-02-10T07:49:06.270457+0000","last_unstale":"2024-02-10T07:53:33.397542+0000","last_undegraded":"2024-02-10T07:53:33.397542+0000","last_fullsized":"2024-02-10T07:53:33.397542+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.588888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.588888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":863,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.410420+0000","last_change":"2024-02-10T07:49:06.267039+0000","last_active":"2024-02-10T07:53:33.410420+0000","last_peered":"2024-02-10T07:53:33.410420+0000","last_clean":"2024-02-10T07:53:33.410420+0000","last_became_active":"2024-02-10T07:49:06.266625+0000","last_became_peered":"2024-02-10T07:49:06.266625+0000","last_unstale":"2024-02-10T07:53:33.410420+0000","last_undegraded":"2024-02-10T07:53:33.410420+0000","last_fullsized":"2024-02-10T07:53:33.410420+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_clean_scrub_stamp":"2024-02-10T07:47:41.181770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2176,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:37.286716+0000","last_change":"2024-02-10T07:50:04.159154+0000","last_active":"2024-02-10T07:53:37.286716+0000","last_peered":"2024-02-10T07:53:37.286716+0000","last_clean":"2024-02-10T07:53:37.286716+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:53:37.286716+0000","last_undegraded":"2024-02-10T07:53:37.286716+0000","last_fullsized":"2024-02-10T07:53:37.286716+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:50:04.159046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.159046+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1300,"num_read_kb":1295,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"430'42","reported_seq":952,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.409336+0000","last_change":"2024-02-10T07:50:43.324808+0000","last_active":"2024-02-10T07:53:33.409336+0000","last_peered":"2024-02-10T07:53:33.409336+0000","last_clean":"2024-02-10T07:53:33.409336+0000","last_became_active":"2024-02-10T07:50:43.291771+0000","last_became_peered":"2024-02-10T07:50:43.291771+0000","last_unstale":"2024-02-10T07:53:33.409336+0000","last_undegraded":"2024-02-10T07:53:33.409336+0000","last_fullsized":"2024-02-10T07:53:33.409336+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:47.417096+0000","last_deep_scrub":"237'18","last_deep_scrub_stamp":"2024-02-10T07:40:31.005376+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.417096+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":826,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:34.104461+0000","last_change":"2024-02-10T07:53:34.104503+0000","last_active":"2024-02-10T07:53:34.104461+0000","last_peered":"2024-02-10T07:53:34.104461+0000","last_clean":"2024-02-10T07:53:34.104461+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:53:34.104461+0000","last_undegraded":"2024-02-10T07:53:34.104461+0000","last_fullsized":"2024-02-10T07:53:34.104461+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:34.104386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:34.104386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:46.902238+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"453'12185","reported_seq":20982,"reported_epoch":453,"state":"active+recovery_wait","last_fresh":"2024-02-10T07:53:39.931173+0000","last_change":"2024-02-10T07:53:25.400523+0000","last_active":"2024-02-10T07:53:39.931173+0000","last_peered":"2024-02-10T07:53:39.931173+0000","last_clean":"2024-02-10T07:50:42.244817+0000","last_became_active":"2024-02-10T07:53:25.355044+0000","last_became_peered":"2024-02-10T07:53:25.355044+0000","last_unstale":"2024-02-10T07:53:39.931173+0000","last_undegraded":"2024-02-10T07:53:39.931173+0000","last_fullsized":"2024-02-10T07:53:39.931173+0000","mapping_epoch":449,"log_start":"429'7300","ondisk_log_start":"429'7300","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4885,"log_dups_size":0,"ondisk_log_size":4885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47644672,"num_objects":727,"num_object_clones":0,"num_object_copies":2181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12185,"num_write_kb":413184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4326,"num_bytes_recovered":234225664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":["1","2","4","7"],"object_location_counts":[{"shards":"1,2,4","objects":727}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":870,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397340+0000","last_change":"2024-02-10T07:50:43.296823+0000","last_active":"2024-02-10T07:53:33.397340+0000","last_peered":"2024-02-10T07:53:33.397340+0000","last_clean":"2024-02-10T07:53:33.397340+0000","last_became_active":"2024-02-10T07:50:43.296488+0000","last_became_peered":"2024-02-10T07:50:43.296488+0000","last_unstale":"2024-02-10T07:53:33.397340+0000","last_undegraded":"2024-02-10T07:53:33.397340+0000","last_fullsized":"2024-02-10T07:53:33.397340+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.630083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.630083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":875,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397822+0000","last_change":"2024-02-10T07:50:43.298468+0000","last_active":"2024-02-10T07:53:33.397822+0000","last_peered":"2024-02-10T07:53:33.397822+0000","last_clean":"2024-02-10T07:53:33.397822+0000","last_became_active":"2024-02-10T07:50:43.297861+0000","last_became_peered":"2024-02-10T07:50:43.297861+0000","last_unstale":"2024-02-10T07:53:33.397822+0000","last_undegraded":"2024-02-10T07:53:33.397822+0000","last_fullsized":"2024-02-10T07:53:33.397822+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:05.193158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:49:05.193158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"430'47","reported_seq":935,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.407640+0000","last_change":"2024-02-10T07:49:57.522503+0000","last_active":"2024-02-10T07:53:33.407640+0000","last_peered":"2024-02-10T07:53:33.407640+0000","last_clean":"2024-02-10T07:53:33.407640+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:53:33.407640+0000","last_undegraded":"2024-02-10T07:53:33.407640+0000","last_fullsized":"2024-02-10T07:53:33.407640+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'37","last_scrub_stamp":"2024-02-10T07:49:57.522391+0000","last_deep_scrub":"271'37","last_deep_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.522391+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":886,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.396970+0000","last_change":"2024-02-10T07:51:02.173651+0000","last_active":"2024-02-10T07:53:33.396970+0000","last_peered":"2024-02-10T07:53:33.396970+0000","last_clean":"2024-02-10T07:53:33.396970+0000","last_became_active":"2024-02-10T07:48:35.062118+0000","last_became_peered":"2024-02-10T07:48:35.062118+0000","last_unstale":"2024-02-10T07:53:33.396970+0000","last_undegraded":"2024-02-10T07:53:33.396970+0000","last_fullsized":"2024-02-10T07:53:33.396970+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:51:02.173535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:00.292320+0000","last_clean_scrub_stamp":"2024-02-10T07:51:02.173535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"453'12318","reported_seq":13467,"reported_epoch":453,"state":"active+recovering+remapped","last_fresh":"2024-02-10T07:53:39.907704+0000","last_change":"2024-02-10T07:53:23.626288+0000","last_active":"2024-02-10T07:53:39.907704+0000","last_peered":"2024-02-10T07:53:39.907704+0000","last_clean":"2024-02-10T07:50:42.239377+0000","last_became_active":"2024-02-10T07:50:44.668375+0000","last_became_peered":"2024-02-10T07:50:44.668375+0000","last_unstale":"2024-02-10T07:53:39.907704+0000","last_undegraded":"2024-02-10T07:53:39.907704+0000","last_fullsized":"2024-02-10T07:53:39.907704+0000","mapping_epoch":405,"log_start":"430'7400","ondisk_log_start":"430'7400","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4918,"log_dups_size":0,"ondisk_log_size":4918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47251480,"num_objects":722,"num_object_clones":0,"num_object_copies":2166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3916,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12318,"num_write_kb":417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":504,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[1,4,6],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":-3194}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":747,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.394997+0000","last_change":"2024-02-10T07:52:23.855559+0000","last_active":"2024-02-10T07:53:33.394997+0000","last_peered":"2024-02-10T07:53:33.394997+0000","last_clean":"2024-02-10T07:53:33.394997+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:53:33.394997+0000","last_undegraded":"2024-02-10T07:53:33.394997+0000","last_fullsized":"2024-02-10T07:53:33.394997+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:23.855477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:23.855477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":108,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.405226+0000","last_change":"2024-02-10T07:50:43.289736+0000","last_active":"2024-02-10T07:53:33.405226+0000","last_peered":"2024-02-10T07:53:33.405226+0000","last_clean":"2024-02-10T07:53:33.405226+0000","last_became_active":"2024-02-10T07:50:43.289458+0000","last_became_peered":"2024-02-10T07:50:43.289458+0000","last_unstale":"2024-02-10T07:53:33.405226+0000","last_undegraded":"2024-02-10T07:53:33.405226+0000","last_fullsized":"2024-02-10T07:53:33.405226+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:48.809513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:48.809513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"430'35","reported_seq":680,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397978+0000","last_change":"2024-02-10T07:49:19.084010+0000","last_active":"2024-02-10T07:53:33.397978+0000","last_peered":"2024-02-10T07:53:33.397978+0000","last_clean":"2024-02-10T07:53:33.397978+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:53:33.397978+0000","last_undegraded":"2024-02-10T07:53:33.397978+0000","last_fullsized":"2024-02-10T07:53:33.397978+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'25","last_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_deep_scrub":"382'25","last_deep_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.083856+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"158'10","reported_seq":1128,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.408164+0000","last_change":"2024-02-10T07:49:50.396822+0000","last_active":"2024-02-10T07:53:33.408164+0000","last_peered":"2024-02-10T07:53:33.408164+0000","last_clean":"2024-02-10T07:53:33.408164+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:53:33.408164+0000","last_undegraded":"2024-02-10T07:53:33.408164+0000","last_fullsized":"2024-02-10T07:53:33.408164+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:49:50.396730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.396730+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.e","version":"453'12300","reported_seq":12436,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:38.476197+0000","last_change":"2024-02-10T07:49:53.281229+0000","last_active":"2024-02-10T07:53:38.476197+0000","last_peered":"2024-02-10T07:53:38.476197+0000","last_clean":"2024-02-10T07:53:38.476197+0000","last_became_active":"2024-02-10T07:49:53.280920+0000","last_became_peered":"2024-02-10T07:49:53.280920+0000","last_unstale":"2024-02-10T07:53:38.476197+0000","last_undegraded":"2024-02-10T07:53:38.476197+0000","last_fullsized":"2024-02-10T07:53:38.476197+0000","mapping_epoch":391,"log_start":"430'7400","ondisk_log_start":"430'7400","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4900,"log_dups_size":0,"ondisk_log_size":4900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52428800,"num_objects":800,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12300,"num_write_kb":419200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":856,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.394838+0000","last_change":"2024-02-10T07:50:43.283505+0000","last_active":"2024-02-10T07:53:33.394838+0000","last_peered":"2024-02-10T07:53:33.394838+0000","last_clean":"2024-02-10T07:53:33.394838+0000","last_became_active":"2024-02-10T07:50:43.283161+0000","last_became_peered":"2024-02-10T07:50:43.283161+0000","last_unstale":"2024-02-10T07:53:33.394838+0000","last_undegraded":"2024-02-10T07:53:33.394838+0000","last_fullsized":"2024-02-10T07:53:33.394838+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_clean_scrub_stamp":"2024-02-10T07:48:23.271454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"296'2","reported_seq":866,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.408503+0000","last_change":"2024-02-10T07:50:03.643473+0000","last_active":"2024-02-10T07:53:33.408503+0000","last_peered":"2024-02-10T07:53:33.408503+0000","last_clean":"2024-02-10T07:53:33.408503+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:53:33.408503+0000","last_undegraded":"2024-02-10T07:53:33.408503+0000","last_fullsized":"2024-02-10T07:53:33.408503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'2","last_scrub_stamp":"2024-02-10T07:50:03.643367+0000","last_deep_scrub":"296'2","last_deep_scrub_stamp":"2024-02-10T07:45:03.233375+0000","last_clean_scrub_stamp":"2024-02-10T07:50:03.643367+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"430'42","reported_seq":916,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402469+0000","last_change":"2024-02-10T07:49:20.216865+0000","last_active":"2024-02-10T07:53:33.402469+0000","last_peered":"2024-02-10T07:53:33.402469+0000","last_clean":"2024-02-10T07:53:33.402469+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:53:33.402469+0000","last_undegraded":"2024-02-10T07:53:33.402469+0000","last_fullsized":"2024-02-10T07:53:33.402469+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-02-10T07:49:20.216750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:20.216750+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"376'12","reported_seq":915,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.399105+0000","last_change":"2024-02-10T07:49:06.282456+0000","last_active":"2024-02-10T07:53:33.399105+0000","last_peered":"2024-02-10T07:53:33.399105+0000","last_clean":"2024-02-10T07:53:33.399105+0000","last_became_active":"2024-02-10T07:49:06.255324+0000","last_became_peered":"2024-02-10T07:49:06.255324+0000","last_unstale":"2024-02-10T07:53:33.399105+0000","last_undegraded":"2024-02-10T07:53:33.399105+0000","last_fullsized":"2024-02-10T07:53:33.399105+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'11","last_scrub_stamp":"2024-02-10T07:48:58.368663+0000","last_deep_scrub":"227'9","last_deep_scrub_stamp":"2024-02-10T07:39:53.440221+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.368663+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"453'12158","reported_seq":14282,"reported_epoch":453,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-10T07:53:38.483088+0000","last_change":"2024-02-10T07:53:14.380137+0000","last_active":"2024-02-10T07:53:38.483088+0000","last_peered":"2024-02-10T07:53:38.483088+0000","last_clean":"2024-02-10T07:50:42.258059+0000","last_became_active":"2024-02-10T07:50:44.666535+0000","last_became_peered":"2024-02-10T07:50:44.666535+0000","last_unstale":"2024-02-10T07:53:38.483088+0000","last_undegraded":"2024-02-10T07:50:44.288061+0000","last_fullsized":"2024-02-10T07:50:44.263325+0000","mapping_epoch":405,"log_start":"429'7300","ondisk_log_start":"429'7300","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50069504,"num_objects":764,"num_object_clones":0,"num_object_copies":2292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3372,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12158,"num_write_kb":413504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":992,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"1,7","objects":-2608}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":853,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.401907+0000","last_change":"2024-02-10T07:50:43.282245+0000","last_active":"2024-02-10T07:53:33.401907+0000","last_peered":"2024-02-10T07:53:33.401907+0000","last_clean":"2024-02-10T07:53:33.401907+0000","last_became_active":"2024-02-10T07:50:43.281465+0000","last_became_peered":"2024-02-10T07:50:43.281465+0000","last_unstale":"2024-02-10T07:53:33.401907+0000","last_undegraded":"2024-02-10T07:53:33.401907+0000","last_fullsized":"2024-02-10T07:53:33.401907+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:27.819612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:46:27.819612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":337,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.403074+0000","last_change":"2024-02-10T07:47:50.948710+0000","last_active":"2024-02-10T07:53:33.403074+0000","last_peered":"2024-02-10T07:53:33.403074+0000","last_clean":"2024-02-10T07:53:33.403074+0000","last_became_active":"2024-02-10T07:47:47.966443+0000","last_became_peered":"2024-02-10T07:47:47.966443+0000","last_unstale":"2024-02-10T07:53:33.403074+0000","last_undegraded":"2024-02-10T07:53:33.403074+0000","last_fullsized":"2024-02-10T07:53:33.403074+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.948624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.948624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"430'42","reported_seq":855,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.394520+0000","last_change":"2024-02-10T07:49:57.303310+0000","last_active":"2024-02-10T07:53:33.394520+0000","last_peered":"2024-02-10T07:53:33.394520+0000","last_clean":"2024-02-10T07:53:33.394520+0000","last_became_active":"2024-02-10T07:49:06.252935+0000","last_became_peered":"2024-02-10T07:49:06.252935+0000","last_unstale":"2024-02-10T07:53:33.394520+0000","last_undegraded":"2024-02-10T07:53:33.394520+0000","last_fullsized":"2024-02-10T07:53:33.394520+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:57.303200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.303200+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":855,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402233+0000","last_change":"2024-02-10T07:49:06.239655+0000","last_active":"2024-02-10T07:53:33.402233+0000","last_peered":"2024-02-10T07:53:33.402233+0000","last_clean":"2024-02-10T07:53:33.402233+0000","last_became_active":"2024-02-10T07:49:06.239190+0000","last_became_peered":"2024-02-10T07:49:06.239190+0000","last_unstale":"2024-02-10T07:53:33.402233+0000","last_undegraded":"2024-02-10T07:53:33.402233+0000","last_fullsized":"2024-02-10T07:53:33.402233+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:42.549018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.549018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"453'12447","reported_seq":12583,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:39.921036+0000","last_change":"2024-02-10T07:49:53.282174+0000","last_active":"2024-02-10T07:53:39.921036+0000","last_peered":"2024-02-10T07:53:39.921036+0000","last_clean":"2024-02-10T07:53:39.921036+0000","last_became_active":"2024-02-10T07:49:53.281832+0000","last_became_peered":"2024-02-10T07:49:53.281832+0000","last_unstale":"2024-02-10T07:53:39.921036+0000","last_undegraded":"2024-02-10T07:53:39.921036+0000","last_fullsized":"2024-02-10T07:53:39.921036+0000","mapping_epoch":391,"log_start":"430'7600","ondisk_log_start":"430'7600","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44761088,"num_objects":683,"num_object_clones":0,"num_object_copies":2049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12447,"num_write_kb":420160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":362,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.399400+0000","last_change":"2024-02-10T07:49:06.274374+0000","last_active":"2024-02-10T07:53:33.399400+0000","last_peered":"2024-02-10T07:53:33.399400+0000","last_clean":"2024-02-10T07:53:33.399400+0000","last_became_active":"2024-02-10T07:48:02.371926+0000","last_became_peered":"2024-02-10T07:48:02.371926+0000","last_unstale":"2024-02-10T07:53:33.399400+0000","last_undegraded":"2024-02-10T07:53:33.399400+0000","last_fullsized":"2024-02-10T07:53:33.399400+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.274263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:49.704024+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.274263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"395'12","reported_seq":271,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.395333+0000","last_change":"2024-02-10T07:52:24.850402+0000","last_active":"2024-02-10T07:53:33.395333+0000","last_peered":"2024-02-10T07:53:33.395333+0000","last_clean":"2024-02-10T07:53:33.395333+0000","last_became_active":"2024-02-10T07:48:35.071916+0000","last_became_peered":"2024-02-10T07:48:35.071916+0000","last_unstale":"2024-02-10T07:53:33.395333+0000","last_undegraded":"2024-02-10T07:53:33.395333+0000","last_fullsized":"2024-02-10T07:53:33.395333+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'12","last_scrub_stamp":"2024-02-10T07:52:24.850269+0000","last_deep_scrub":"395'12","last_deep_scrub_stamp":"2024-02-10T07:52:24.850269+0000","last_clean_scrub_stamp":"2024-02-10T07:52:24.850269+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:39.041346+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":183,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:37.144234+0000","last_change":"2024-02-10T07:53:37.144276+0000","last_active":"2024-02-10T07:53:37.144234+0000","last_peered":"2024-02-10T07:53:37.144234+0000","last_clean":"2024-02-10T07:53:37.144234+0000","last_became_active":"2024-02-10T07:49:06.251038+0000","last_became_peered":"2024-02-10T07:49:06.251038+0000","last_unstale":"2024-02-10T07:53:37.144234+0000","last_undegraded":"2024-02-10T07:53:37.144234+0000","last_fullsized":"2024-02-10T07:53:37.144234+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:37.144142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:53:37.144142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:01.122427+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"430'39","reported_seq":927,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.401998+0000","last_change":"2024-02-10T07:50:00.655466+0000","last_active":"2024-02-10T07:53:33.401998+0000","last_peered":"2024-02-10T07:53:33.401998+0000","last_clean":"2024-02-10T07:53:33.401998+0000","last_became_active":"2024-02-10T07:47:47.967427+0000","last_became_peered":"2024-02-10T07:47:47.967427+0000","last_unstale":"2024-02-10T07:53:33.401998+0000","last_undegraded":"2024-02-10T07:53:33.401998+0000","last_fullsized":"2024-02-10T07:53:33.401998+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:50:00.655377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.655377+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"453'12546","reported_seq":12748,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:38.532569+0000","last_change":"2024-02-10T07:51:08.956873+0000","last_active":"2024-02-10T07:53:38.532569+0000","last_peered":"2024-02-10T07:53:38.532569+0000","last_clean":"2024-02-10T07:53:38.532569+0000","last_became_active":"2024-02-10T07:49:53.283415+0000","last_became_peered":"2024-02-10T07:49:53.283415+0000","last_unstale":"2024-02-10T07:53:38.532569+0000","last_undegraded":"2024-02-10T07:53:38.532569+0000","last_fullsized":"2024-02-10T07:53:38.532569+0000","mapping_epoch":391,"log_start":"423'6900","ondisk_log_start":"423'6900","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'5587","last_scrub_stamp":"2024-02-10T07:51:08.956707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:51:08.956707+0000","objects_scrubbed":5548,"log_size":5646,"log_dups_size":0,"ondisk_log_size":5646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52559872,"num_objects":802,"num_object_clones":0,"num_object_copies":2406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12546,"num_write_kb":427136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":556,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.395545+0000","last_change":"2024-02-10T07:51:59.948039+0000","last_active":"2024-02-10T07:53:33.395545+0000","last_peered":"2024-02-10T07:53:33.395545+0000","last_clean":"2024-02-10T07:53:33.395545+0000","last_became_active":"2024-02-10T07:49:06.268874+0000","last_became_peered":"2024-02-10T07:49:06.268874+0000","last_unstale":"2024-02-10T07:53:33.395545+0000","last_undegraded":"2024-02-10T07:53:33.395545+0000","last_fullsized":"2024-02-10T07:53:33.395545+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:51:59.947908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_clean_scrub_stamp":"2024-02-10T07:51:59.947908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":336,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402718+0000","last_change":"2024-02-10T07:49:57.310447+0000","last_active":"2024-02-10T07:53:33.402718+0000","last_peered":"2024-02-10T07:53:33.402718+0000","last_clean":"2024-02-10T07:53:33.402718+0000","last_became_active":"2024-02-10T07:48:02.333689+0000","last_became_peered":"2024-02-10T07:48:02.333689+0000","last_unstale":"2024-02-10T07:53:33.402718+0000","last_undegraded":"2024-02-10T07:53:33.402718+0000","last_fullsized":"2024-02-10T07:53:33.402718+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:57.310333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.310333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":860,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402360+0000","last_change":"2024-02-10T07:50:43.283476+0000","last_active":"2024-02-10T07:53:33.402360+0000","last_peered":"2024-02-10T07:53:33.402360+0000","last_clean":"2024-02-10T07:53:33.402360+0000","last_became_active":"2024-02-10T07:50:43.283104+0000","last_became_peered":"2024-02-10T07:50:43.283104+0000","last_unstale":"2024-02-10T07:53:33.402360+0000","last_undegraded":"2024-02-10T07:53:33.402360+0000","last_fullsized":"2024-02-10T07:53:33.402360+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:37.995362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.995362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"430'32","reported_seq":945,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402598+0000","last_change":"2024-02-10T07:49:16.249257+0000","last_active":"2024-02-10T07:53:33.402598+0000","last_peered":"2024-02-10T07:53:33.402598+0000","last_clean":"2024-02-10T07:53:33.402598+0000","last_became_active":"2024-02-10T07:48:35.062573+0000","last_became_peered":"2024-02-10T07:48:35.062573+0000","last_unstale":"2024-02-10T07:53:33.402598+0000","last_undegraded":"2024-02-10T07:53:33.402598+0000","last_fullsized":"2024-02-10T07:53:33.402598+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:49:16.248936+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.248936+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"453'12294","reported_seq":21251,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:39.097564+0000","last_change":"2024-02-10T07:53:16.355834+0000","last_active":"2024-02-10T07:53:39.097564+0000","last_peered":"2024-02-10T07:53:39.097564+0000","last_clean":"2024-02-10T07:53:39.097564+0000","last_became_active":"2024-02-10T07:53:16.292628+0000","last_became_peered":"2024-02-10T07:53:16.292628+0000","last_unstale":"2024-02-10T07:53:39.097564+0000","last_undegraded":"2024-02-10T07:53:39.097564+0000","last_fullsized":"2024-02-10T07:53:39.097564+0000","mapping_epoch":446,"log_start":"410'5300","ondisk_log_start":"410'5300","created":391,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.261205+0000","objects_scrubbed":0,"log_size":6994,"log_dups_size":0,"ondisk_log_size":6994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49545216,"num_objects":756,"num_object_clones":0,"num_object_copies":2268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12294,"num_write_kb":417600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4405,"num_bytes_recovered":265551872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":108,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.406351+0000","last_change":"2024-02-10T07:50:43.295417+0000","last_active":"2024-02-10T07:53:33.406351+0000","last_peered":"2024-02-10T07:53:33.406351+0000","last_clean":"2024-02-10T07:53:33.406351+0000","last_became_active":"2024-02-10T07:50:43.290295+0000","last_became_peered":"2024-02-10T07:50:43.290295+0000","last_unstale":"2024-02-10T07:53:33.406351+0000","last_undegraded":"2024-02-10T07:53:33.406351+0000","last_fullsized":"2024-02-10T07:53:33.406351+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.153368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.153368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":845,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397916+0000","last_change":"2024-02-10T07:49:07.170105+0000","last_active":"2024-02-10T07:53:33.397916+0000","last_peered":"2024-02-10T07:53:33.397916+0000","last_clean":"2024-02-10T07:53:33.397916+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:53:33.397916+0000","last_undegraded":"2024-02-10T07:53:33.397916+0000","last_fullsized":"2024-02-10T07:53:33.397916+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:07.169962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:07.169962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":887,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.397066+0000","last_change":"2024-02-10T07:52:26.772420+0000","last_active":"2024-02-10T07:53:33.397066+0000","last_peered":"2024-02-10T07:53:33.397066+0000","last_clean":"2024-02-10T07:53:33.397066+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:53:33.397066+0000","last_undegraded":"2024-02-10T07:53:33.397066+0000","last_fullsized":"2024-02-10T07:53:33.397066+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:26.772322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:03.803361+0000","last_clean_scrub_stamp":"2024-02-10T07:52:26.772322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:53:55.097614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"430'53","reported_seq":993,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402415+0000","last_change":"2024-02-10T07:50:43.316983+0000","last_active":"2024-02-10T07:53:33.402415+0000","last_peered":"2024-02-10T07:53:33.402415+0000","last_clean":"2024-02-10T07:53:33.402415+0000","last_became_active":"2024-02-10T07:50:43.281879+0000","last_became_peered":"2024-02-10T07:50:43.281879+0000","last_unstale":"2024-02-10T07:53:33.402415+0000","last_undegraded":"2024-02-10T07:53:33.402415+0000","last_fullsized":"2024-02-10T07:53:33.402415+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T07:48:59.066256+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.066256+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"453'12513","reported_seq":12711,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:38.511401+0000","last_change":"2024-02-10T07:51:05.662504+0000","last_active":"2024-02-10T07:53:38.511401+0000","last_peered":"2024-02-10T07:53:38.511401+0000","last_clean":"2024-02-10T07:53:38.511401+0000","last_became_active":"2024-02-10T07:49:53.280975+0000","last_became_peered":"2024-02-10T07:49:53.280975+0000","last_unstale":"2024-02-10T07:53:38.511401+0000","last_undegraded":"2024-02-10T07:53:38.511401+0000","last_fullsized":"2024-02-10T07:53:38.511401+0000","mapping_epoch":391,"log_start":"424'6900","ondisk_log_start":"424'6900","created":391,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'5413","last_scrub_stamp":"2024-02-10T07:51:05.662310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:52.261205+0000","last_clean_scrub_stamp":"2024-02-10T07:51:05.662310+0000","objects_scrubbed":5394,"log_size":5613,"log_dups_size":0,"ondisk_log_size":5613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51183616,"num_objects":781,"num_object_clones":0,"num_object_copies":2343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12513,"num_write_kb":425408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":847,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402790+0000","last_change":"2024-02-10T07:50:43.290883+0000","last_active":"2024-02-10T07:53:33.402790+0000","last_peered":"2024-02-10T07:53:33.402790+0000","last_clean":"2024-02-10T07:53:33.402790+0000","last_became_active":"2024-02-10T07:50:43.290334+0000","last_became_peered":"2024-02-10T07:50:43.290334+0000","last_unstale":"2024-02-10T07:53:33.402790+0000","last_undegraded":"2024-02-10T07:53:33.402790+0000","last_fullsized":"2024-02-10T07:53:33.402790+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:47.275882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.275882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"405'12","reported_seq":148,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.405426+0000","last_change":"2024-02-10T07:50:43.294666+0000","last_active":"2024-02-10T07:53:33.405426+0000","last_peered":"2024-02-10T07:53:33.405426+0000","last_clean":"2024-02-10T07:53:33.405426+0000","last_became_active":"2024-02-10T07:50:43.280142+0000","last_became_peered":"2024-02-10T07:50:43.280142+0000","last_unstale":"2024-02-10T07:53:33.405426+0000","last_undegraded":"2024-02-10T07:53:33.405426+0000","last_fullsized":"2024-02-10T07:53:33.405426+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'11","last_scrub_stamp":"2024-02-10T07:49:14.335064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:14.335064+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":352,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.407815+0000","last_change":"2024-02-10T07:53:20.474601+0000","last_active":"2024-02-10T07:53:33.407815+0000","last_peered":"2024-02-10T07:53:33.407815+0000","last_clean":"2024-02-10T07:53:33.407815+0000","last_became_active":"2024-02-10T07:49:06.258926+0000","last_became_peered":"2024-02-10T07:49:06.258926+0000","last_unstale":"2024-02-10T07:53:33.407815+0000","last_undegraded":"2024-02-10T07:53:33.407815+0000","last_fullsized":"2024-02-10T07:53:33.407815+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:20.474497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_clean_scrub_stamp":"2024-02-10T07:53:20.474497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:48.442368+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"430'30","reported_seq":927,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-10T07:53:33.402867+0000","last_change":"2024-02-10T07:49:06.289632+0000","last_active":"2024-02-10T07:53:33.402867+0000","last_peered":"2024-02-10T07:53:33.402867+0000","last_clean":"2024-02-10T07:53:33.402867+0000","last_became_active":"2024-02-10T07:49:06.238858+0000","last_became_peered":"2024-02-10T07:49:06.238858+0000","last_unstale":"2024-02-10T07:53:33.402867+0000","last_undegraded":"2024-02-10T07:53:33.402867+0000","last_fullsized":"2024-02-10T07:53:33.402867+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:48:15.588199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.588199+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":792461336,"num_objects":12093,"num_object_clones":0,"num_object_copies":36279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6644,"num_objects_misplaced":3916,"num_objects_unfound":0,"num_objects_dirty":12093,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197427,"num_write_kb":6704577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15796,"num_bytes_recovered":801964032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3501342720,"data_stored":3501326432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84727,"ondisk_log_size":84727,"up":48,"acting":46,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1300,"num_read_kb":1295,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1540,"num_read_kb":1297,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":6382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1060,"ondisk_log_size":1060,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038026,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1628680,"kb_used_data":835264,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92116472,"statfs":{"total":95995035648,"available":94327267328,"internally_reserved":0,"allocated":855310336,"data_stored":854808618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":812382666},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":3.0030000000000001,"5min":1.8480000000000001,"15min":1.6899999999999999},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":18.202999999999999,"5min":39.494,"15min":39.494},"last":0.26200000000000001},{"interface":"front","average":{"1min":2.968,"5min":1.8959999999999999,"15min":1.7030000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.23000000000000001},"max":{"1min":19.745000000000001,"5min":38.524000000000001,"15min":38.524000000000001},"last":1.397}]},{"osd":1,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":6.6909999999999998,"5min":2.4009999999999998,"15min":2.069},"min":{"1min":0.33400000000000002,"5min":0.251,"15min":0.24399999999999999},"max":{"1min":50.149000000000001,"5min":50.149000000000001,"15min":50.149000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":6.5720000000000001,"5min":2.431,"15min":2.0190000000000001},"min":{"1min":0.33100000000000002,"5min":0.26300000000000001,"15min":0.24399999999999999},"max":{"1min":54.048000000000002,"5min":54.048000000000002,"15min":54.048000000000002},"last":1.3440000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":1.25,"15min":1.141},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.255},"max":{"1min":7.0019999999999998,"5min":7.0019999999999998,"15min":18.57},"last":1.109},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.4510000000000001,"15min":1.2989999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":7.8410000000000002,"5min":8.3949999999999996,"15min":18.521000000000001},"last":3.0819999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:53:02 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.264,"15min":1.1220000000000001},"min":{"1min":0.379,"5min":0.307,"15min":0.307},"max":{"1min":4.7590000000000003,"5min":4.7590000000000003,"15min":4.7590000000000003},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.198,"15min":1.014},"min":{"1min":0.40799999999999997,"5min":0.313,"15min":0.313},"max":{"1min":6.2839999999999998,"5min":6.2839999999999998,"15min":6.2839999999999998},"last":1.6180000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.347,"15min":1.1719999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.22},"max":{"1min":1.5960000000000001,"5min":26.001000000000001,"15min":26.001000000000001},"last":1.3080000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":1.3089999999999999,"15min":1.1539999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.94199999999999995,"5min":27.09,"15min":27.09},"last":0.38400000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:53:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.91700000000000004,"15min":1.0389999999999999},"min":{"1min":0.29299999999999998,"5min":0.26600000000000001,"15min":0.245},"max":{"1min":1.6830000000000001,"5min":6.1820000000000004,"15min":18.126999999999999},"last":1.5940000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.1279999999999999,"15min":1.1659999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.9510000000000001,"5min":6.9219999999999997,"15min":19.548999999999999},"last":1.4239999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":1.1599999999999999,"15min":1.159},"min":{"1min":0.30299999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.0409999999999999,"5min":7.4329999999999998,"15min":17.832000000000001},"last":1.446},{"interface":"front","average":{"1min":0.56000000000000005,"5min":1.077,"15min":1.179},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.29799999999999999},"max":{"1min":1.1719999999999999,"5min":6.101,"15min":20.449999999999999},"last":0.84399999999999997}]}]},{"osd":6,"up_from":50,"seq":214748365071,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1077680,"kb_used_data":424552,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":92667472,"statfs":{"total":95995035648,"available":94891491328,"internally_reserved":0,"allocated":434741248,"data_stored":434263147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":668793286},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":2.4039999999999999,"5min":2.0030000000000001,"15min":1.6850000000000001},"min":{"1min":0.249,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":11.166,"5min":19.484999999999999,"15min":24.727},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.395,"15min":1.373},"min":{"1min":0.27000000000000002,"5min":0.23899999999999999,"15min":0.23100000000000001},"max":{"1min":5.1870000000000003,"5min":14.808,"15min":29.084},"last":0.36799999999999999}]},{"osd":1,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":1.669,"15min":1.8520000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23799999999999999},"max":{"1min":9.923,"5min":15.007999999999999,"15min":37.659999999999997},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.599,"15min":1.887},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.253},"max":{"1min":4,"5min":19.524999999999999,"15min":37.75},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":1.6579999999999999,"15min":1.3320000000000001},"min":{"1min":0.30199999999999999,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":6.25,"5min":15.768000000000001,"15min":15.768000000000001},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.714,"5min":1.456,"15min":1.167},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.28399999999999997},"max":{"1min":9.4540000000000006,"5min":15.728999999999999,"15min":15.728999999999999},"last":0.39800000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":1.77,"5min":1.399,"15min":1.637},"min":{"1min":0.33100000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.6500000000000004,"5min":13.279,"15min":13.279},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.9850000000000001,"5min":1.2889999999999999,"15min":1.2589999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":7.7949999999999999,"5min":7.7949999999999999,"15min":7.7949999999999999},"last":0.82599999999999996}]},{"osd":4,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.84299999999999997,"15min":0.84899999999999998},"min":{"1min":0.26900000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.95999999999999996,"5min":7.6509999999999998,"15min":7.6509999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.997,"15min":0.88600000000000001},"min":{"1min":0.28699999999999998,"5min":0.27700000000000002,"15min":0.26000000000000001},"max":{"1min":2.198,"5min":5.7050000000000001,"15min":7.0300000000000002},"last":0.35699999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:53:03 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.70199999999999996,"15min":1.0640000000000001},"min":{"1min":0.32600000000000001,"5min":0.28799999999999998,"15min":0.24099999999999999},"max":{"1min":0.91500000000000004,"5min":3.5699999999999998,"15min":44.689},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.1439999999999999,"15min":1.2170000000000001},"min":{"1min":0.32100000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.129,"5min":8.9420000000000002,"15min":44.465000000000003},"last":0.32800000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:53:15 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.121,"15min":1.3089999999999999},"min":{"1min":0.314,"5min":0.29999999999999999,"15min":0.28100000000000003},"max":{"1min":2.9449999999999998,"5min":4.9379999999999997,"15min":42.631999999999998},"last":1.5860000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":1.0589999999999999,"15min":1.224},"min":{"1min":0.376,"5min":0.32500000000000001,"15min":0.315},"max":{"1min":2.5990000000000002,"5min":6.5419999999999998,"15min":44.792999999999999},"last":1.3360000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575145,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1556560,"kb_used_data":713224,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92188592,"statfs":{"total":95995035648,"available":94401118208,"internally_reserved":0,"allocated":730341376,"data_stored":729830690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":863566278},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:52:48 2024","interfaces":[{"interface":"back","average":{"1min":3.4660000000000002,"5min":1.7629999999999999,"15min":1.9410000000000001},"min":{"1min":0.24199999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":14.382999999999999,"5min":14.382999999999999,"15min":114.27},"last":1.7649999999999999},{"interface":"front","average":{"1min":3.222,"5min":1.5609999999999999,"15min":1.8859999999999999},"min":{"1min":0.48299999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.16,"5min":14.16,"15min":119.499},"last":1.0900000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:53:19 2024","interfaces":[{"interface":"back","average":{"1min":2.8029999999999999,"5min":2.1200000000000001,"15min":1.4570000000000001},"min":{"1min":0.246,"5min":0.23400000000000001,"15min":0.20399999999999999},"max":{"1min":6.2510000000000003,"5min":14.343,"15min":14.343},"last":0.27800000000000002},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.8129999999999999,"15min":1.45},"min":{"1min":0.58999999999999997,"5min":0.24099999999999999,"15min":0.224},"max":{"1min":7.5279999999999996,"5min":14.204000000000001,"15min":14.204000000000001},"last":1.0529999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:52:57 2024","interfaces":[{"interface":"back","average":{"1min":2.613,"5min":1.7430000000000001,"15min":0.90600000000000003},"min":{"1min":0.49299999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":14.452999999999999,"5min":22.367999999999999,"15min":22.367999999999999},"last":1.738},{"interface":"front","average":{"1min":2.23,"5min":1.8129999999999999,"15min":0.86199999999999999},"min":{"1min":0.33200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":5.9429999999999996,"5min":22.199999999999999,"15min":22.199999999999999},"last":0.52500000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:53:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.379,"15min":2.1560000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28399999999999997},"max":{"1min":7.617,"5min":7.617,"15min":207.45699999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.3,"15min":1.3069999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.28899999999999998},"max":{"1min":4.0099999999999998,"5min":10.327999999999999,"15min":32.201000000000001},"last":1.5329999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:53:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7609999999999999,"5min":1.7909999999999999,"15min":2.0449999999999999},"min":{"1min":0.49399999999999999,"5min":0.34200000000000003,"15min":0.308},"max":{"1min":7.5490000000000004,"5min":23.978999999999999,"15min":98.703000000000003},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.5609999999999999,"15min":1.9850000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.30199999999999999},"max":{"1min":5.8399999999999999,"5min":24.024999999999999,"15min":98.664000000000001},"last":0.56200000000000006}]},{"osd":6,"last update":"Sat Feb 10 07:52:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.4039999999999999,"15min":1.45},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":4.0709999999999997,"5min":10.291,"15min":36.808},"last":1.502},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.661,"15min":1.6890000000000001},"min":{"1min":0.45000000000000001,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":7.5839999999999996,"5min":7.5839999999999996,"15min":33.774999999999999},"last":1.71}]},{"osd":7,"last update":"Sat Feb 10 07:53:19 2024","interfaces":[{"interface":"back","average":{"1min":2.048,"5min":1.7709999999999999,"15min":1.5489999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":7.649,"5min":24.099,"15min":24.099},"last":1.635},{"interface":"front","average":{"1min":1.639,"5min":1.331,"15min":1.288},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.34300000000000003},"max":{"1min":4.0439999999999996,"5min":10.356999999999999,"15min":10.356999999999999},"last":1.5640000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738670,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1315656,"kb_used_data":513920,"kb_used_omap":1,"kb_used_meta":801662,"kb_avail":92429496,"statfs":{"total":95995035648,"available":94647803904,"internally_reserved":0,"allocated":526254080,"data_stored":525752759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":820902341},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:52:42 2024","interfaces":[{"interface":"back","average":{"1min":3.1499999999999999,"5min":1.8500000000000001,"15min":2.444},"min":{"1min":0.52400000000000002,"5min":0.20499999999999999,"15min":0.182},"max":{"1min":7.4500000000000002,"5min":16.942,"15min":156.16200000000001},"last":2.302},{"interface":"front","average":{"1min":2.863,"5min":1.4119999999999999,"15min":2.29},"min":{"1min":0.55900000000000005,"5min":0.192,"15min":0.183},"max":{"1min":8.3780000000000001,"5min":8.4550000000000001,"15min":156.941},"last":2.2730000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:53:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5739999999999998,"5min":2.0409999999999999,"15min":1.6499999999999999},"min":{"1min":0.65700000000000003,"5min":0.221,"15min":0.20000000000000001},"max":{"1min":19.733000000000001,"5min":19.733000000000001,"15min":34.603000000000002},"last":2.3650000000000002},{"interface":"front","average":{"1min":2.9100000000000001,"5min":1.9370000000000001,"15min":1.444},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.22600000000000001},"max":{"1min":8.4060000000000006,"5min":10.116,"15min":34.718000000000004},"last":5.7800000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:52:57 2024","interfaces":[{"interface":"back","average":{"1min":2.5870000000000002,"5min":1.496,"15min":0.92000000000000004},"min":{"1min":0.30099999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":8.3659999999999997,"5min":8.6359999999999992,"15min":8.6359999999999992},"last":10.372},{"interface":"front","average":{"1min":2.6829999999999998,"5min":1.73,"15min":0.99199999999999999},"min":{"1min":0.24099999999999999,"5min":0.214,"15min":0.214},"max":{"1min":10.029,"5min":17.172999999999998,"15min":17.172999999999998},"last":2.0310000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:53:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7529999999999999,"5min":1.79,"15min":1.4450000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.27600000000000002},"max":{"1min":4.6369999999999996,"5min":16.991,"15min":37.286000000000001},"last":1.9279999999999999},{"interface":"front","average":{"1min":2.1720000000000002,"5min":1.6359999999999999,"15min":1.3999999999999999},"min":{"1min":0.34100000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":8.5939999999999994,"5min":8.5939999999999994,"15min":34.683},"last":0.41299999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:53:00 2024","interfaces":[{"interface":"back","average":{"1min":2.1179999999999999,"5min":1.8580000000000001,"15min":1.998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32300000000000001},"max":{"1min":5.992,"5min":17.062999999999999,"15min":41.429000000000002},"last":1.9730000000000001},{"interface":"front","average":{"1min":2.202,"5min":1.5580000000000001,"15min":1.8660000000000001},"min":{"1min":0.379,"5min":0.378,"15min":0.308},"max":{"1min":8.6460000000000008,"5min":8.6460000000000008,"15min":41.456000000000003},"last":0.45700000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:52:42 2024","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.3979999999999999,"15min":1.552},"min":{"1min":0.41899999999999998,"5min":0.34599999999999997,"15min":0.33600000000000002},"max":{"1min":5.0030000000000001,"5min":8.5630000000000006,"15min":41.587000000000003},"last":5.6669999999999998},{"interface":"front","average":{"1min":2.0350000000000001,"5min":1.7669999999999999,"15min":1.794},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":7.4219999999999997,"5min":17.027999999999999,"15min":41.256},"last":2.0030000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:53:22 2024","interfaces":[{"interface":"back","average":{"1min":2.6680000000000001,"5min":1.7669999999999999,"15min":1.6160000000000001},"min":{"1min":0.41199999999999998,"5min":0.39600000000000002,"15min":0.33300000000000002},"max":{"1min":8.6809999999999992,"5min":8.6809999999999992,"15min":34.710999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.831,"5min":1.589,"15min":1.4039999999999999},"min":{"1min":0.45400000000000001,"5min":0.39200000000000002,"15min":0.379},"max":{"1min":8.3230000000000004,"5min":8.5999999999999996,"15min":34.997999999999998},"last":5.742}]}]},{"osd":2,"up_from":18,"seq":77309411619,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":976552,"kb_used_data":305376,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92768600,"statfs":{"total":95995035648,"available":94995046400,"internally_reserved":0,"allocated":312705024,"data_stored":312235161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,8,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:23 2024","interfaces":[{"interface":"back","average":{"1min":3.5510000000000002,"5min":2.46,"15min":1.8029999999999999},"min":{"1min":0.47499999999999998,"5min":0.20699999999999999,"15min":0.191},"max":{"1min":32.371000000000002,"5min":32.371000000000002,"15min":65.430999999999997},"last":0.56599999999999995},{"interface":"front","average":{"1min":4.0869999999999997,"5min":2.6749999999999998,"15min":1.6990000000000001},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.20100000000000001},"max":{"1min":33.280000000000001,"5min":38.173000000000002,"15min":38.173000000000002},"last":0.24299999999999999}]},{"osd":1,"last update":"Sat Feb 10 07:53:23 2024","interfaces":[{"interface":"back","average":{"1min":3.2490000000000001,"5min":1.9690000000000001,"15min":1.736},"min":{"1min":0.46700000000000003,"5min":0.21299999999999999,"15min":0.20599999999999999},"max":{"1min":13.071999999999999,"5min":27.606000000000002,"15min":65.795000000000002},"last":1.2070000000000001},{"interface":"front","average":{"1min":4.0030000000000001,"5min":2.456,"15min":1.681},"min":{"1min":0.38500000000000001,"5min":0.219,"15min":0.20499999999999999},"max":{"1min":12.17,"5min":29.709,"15min":29.709},"last":0.83699999999999997}]},{"osd":3,"last update":"Sat Feb 10 07:52:59 2024","interfaces":[{"interface":"back","average":{"1min":1.984,"5min":1.163,"15min":0.76800000000000002},"min":{"1min":0.39000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":4.9009999999999998,"5min":7.0819999999999999,"15min":7.0819999999999999},"last":1.3360000000000001},{"interface":"front","average":{"1min":2.3999999999999999,"5min":1.391,"15min":0.78900000000000003},"min":{"1min":0.41199999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":6.4269999999999996,"5min":7.7530000000000001,"15min":7.7530000000000001},"last":0.66500000000000004}]},{"osd":4,"last update":"Sat Feb 10 07:53:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.2569999999999999,"15min":1.3799999999999999},"min":{"1min":0.51000000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.5249999999999999,"5min":11.395,"15min":65.494},"last":1.179},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.585,"15min":1.2450000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":6.4800000000000004,"5min":7.7149999999999999,"15min":7.7149999999999999},"last":0.86699999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:53:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3849999999999998,"5min":1.337,"15min":1.456},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.32600000000000001},"max":{"1min":10.807,"5min":10.807,"15min":25.533000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.0840000000000001,"15min":1.534},"min":{"1min":0.58299999999999996,"5min":0.35599999999999998,"15min":0.34499999999999997},"max":{"1min":4.7789999999999999,"5min":4.9749999999999996,"15min":65.834000000000003},"last":1.0680000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:52:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.2589999999999999,"15min":1.6459999999999999},"min":{"1min":0.51900000000000002,"5min":0.39300000000000002,"15min":0.35199999999999998},"max":{"1min":3.637,"5min":5.069,"15min":65.748000000000005},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.7,"5min":1.276,"15min":1.347},"min":{"1min":0.36299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.6200000000000001,"5min":6.6280000000000001,"15min":25.901},"last":0.93899999999999995}]},{"osd":7,"last update":"Sat Feb 10 07:53:23 2024","interfaces":[{"interface":"back","average":{"1min":2.5030000000000001,"5min":1.702,"15min":1.3979999999999999},"min":{"1min":0.42299999999999999,"5min":0.36899999999999999,"15min":0.36699999999999999},"max":{"1min":10.859999999999999,"5min":10.859999999999999,"15min":10.859999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.2210000000000001,"15min":1.444},"min":{"1min":0.42799999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.609,"5min":5.0979999999999999,"15min":65.539000000000001},"last":1.101}]}]},{"osd":3,"up_from":347,"seq":1490353651784,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":540172,"kb_used_data":290372,"kb_used_omap":5,"kb_used_meta":249786,"kb_avail":93204980,"statfs":{"total":95995035648,"available":95441899520,"internally_reserved":0,"allocated":297340928,"data_stored":296834093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5625,"internal_metadata":255781383},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":3.5259999999999998,"5min":1.764,"15min":0.93799999999999994},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":28.247,"5min":28.247,"15min":28.247},"last":1.0249999999999999},{"interface":"front","average":{"1min":3.7149999999999999,"5min":1.774,"15min":1.008},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":27.361999999999998,"5min":27.361999999999998,"15min":27.361999999999998},"last":3.2599999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":6.3849999999999998,"5min":2.3500000000000001,"15min":1.0469999999999999},"min":{"1min":0.23000000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":30.718,"5min":30.718,"15min":30.718},"last":0.372},{"interface":"front","average":{"1min":6.0140000000000002,"5min":2.181,"15min":1.012},"min":{"1min":0.38200000000000001,"5min":0.22,"15min":0.22},"max":{"1min":32.927999999999997,"5min":32.927999999999997,"15min":32.927999999999997},"last":0.33400000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":2.165,"5min":1.3979999999999999,"15min":0.77400000000000002},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.8949999999999996,"5min":10.686,"15min":10.686},"last":1.6279999999999999},{"interface":"front","average":{"1min":3.395,"5min":1.6739999999999999,"15min":0.874},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":11.25,"5min":11.25,"15min":11.25},"last":3.4369999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.8580000000000001,"15min":2.4569999999999999},"min":{"1min":0.29899999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.1589999999999998,"5min":28.707999999999998,"15min":28.707999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.544,"5min":1.772,"15min":2.6389999999999998},"min":{"1min":0.379,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":7.8220000000000001,"5min":28.760999999999999,"15min":28.760999999999999},"last":0.96299999999999997}]},{"osd":5,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":2.1299999999999999,"5min":1.476,"15min":1.056},"min":{"1min":0.40100000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.2799999999999994,"5min":15.525,"15min":15.525},"last":3.3580000000000001},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.3169999999999999,"15min":0.93700000000000006},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":5.109,"5min":10.298,"15min":10.298},"last":0.72099999999999997}]},{"osd":6,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":2.1789999999999998,"5min":2.3519999999999999,"15min":2.5},"min":{"1min":0.45000000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":8.3219999999999992,"5min":55.750999999999998,"15min":55.750999999999998},"last":4.923},{"interface":"front","average":{"1min":2.141,"5min":2.653,"15min":3.0369999999999999},"min":{"1min":0.40999999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":7.7380000000000004,"5min":55.807000000000002,"15min":55.807000000000002},"last":1.2110000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:52:53 2024","interfaces":[{"interface":"back","average":{"1min":2.0600000000000001,"5min":1.722,"15min":1.3169999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.6790000000000003,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.706,"5min":1.4299999999999999,"15min":1.3220000000000001},"min":{"1min":0.433,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":7.8650000000000002,"5min":7.8650000000000002,"15min":7.8650000000000002},"last":0.998}]}]},{"osd":4,"up_from":32,"seq":137438953753,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":974668,"kb_used_data":217412,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92770484,"statfs":{"total":95995035648,"available":94996975616,"internally_reserved":0,"allocated":222629888,"data_stored":222123189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4510000000000001,"5min":1.4079999999999999,"15min":1.7050000000000001},"min":{"1min":0.29699999999999999,"5min":0.21299999999999999,"15min":0.20499999999999999},"max":{"1min":20.004999999999999,"5min":20.004999999999999,"15min":74.533000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":3.1200000000000001,"5min":1.881,"15min":1.827},"min":{"1min":0.252,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":20.050000000000001,"5min":20.050000000000001,"15min":74.340999999999994},"last":0.34799999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:53:03 2024","interfaces":[{"interface":"back","average":{"1min":3.0899999999999999,"5min":2.6080000000000001,"15min":2.448},"min":{"1min":0.25800000000000001,"5min":0.245,"15min":0.23200000000000001},"max":{"1min":34.348999999999997,"5min":66.343999999999994,"15min":69.852000000000004},"last":0.66400000000000003},{"interface":"front","average":{"1min":3.5939999999999999,"5min":2.2650000000000001,"15min":2.411},"min":{"1min":0.27800000000000002,"5min":0.27700000000000002,"15min":0.26000000000000001},"max":{"1min":34.837000000000003,"5min":35.375,"15min":70.692999999999998},"last":1.028}]},{"osd":2,"last update":"Sat Feb 10 07:53:03 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.399,"15min":1.1890000000000001},"min":{"1min":0.28599999999999998,"5min":0.28399999999999997,"15min":0.26600000000000001},"max":{"1min":3.3460000000000001,"5min":7.1379999999999999,"15min":15.351000000000001},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.54,"5min":1.28,"15min":1.1080000000000001},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":7.1870000000000003,"5min":7.335,"15min":8.5129999999999999},"last":1.6120000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:52:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":1.9930000000000001,"15min":3.827},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.1299999999999999,"5min":66.379999999999995,"15min":66.379999999999995},"last":1.714},{"interface":"front","average":{"1min":1.45,"5min":1.335,"15min":1.627},"min":{"1min":0.34999999999999998,"5min":0.309,"15min":0.309},"max":{"1min":6.6429999999999998,"5min":6.6429999999999998,"15min":6.6429999999999998},"last":1.798}]},{"osd":5,"last update":"Sat Feb 10 07:53:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":1.0029999999999999,"15min":0.995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.23100000000000001},"max":{"1min":2.1389999999999998,"5min":7.3639999999999999,"15min":10.502000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.746,"15min":0.98399999999999999},"min":{"1min":0.312,"5min":0.30199999999999999,"15min":0.247},"max":{"1min":1.613,"5min":3.6219999999999999,"15min":9.2460000000000004},"last":0.36299999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:52:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":1.8799999999999999,"15min":1.21},"min":{"1min":0.29199999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.1439999999999999,"5min":67.433999999999997,"15min":67.433999999999997},"last":0.315},{"interface":"front","average":{"1min":0.57799999999999996,"5min":1.837,"15min":1.2769999999999999},"min":{"1min":0.307,"5min":0.29199999999999998,"15min":0.24399999999999999},"max":{"1min":1.7989999999999999,"5min":66.876000000000005,"15min":66.876000000000005},"last":0.40400000000000003}]},{"osd":7,"last update":"Sat Feb 10 07:53:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":1.0189999999999999,"15min":0.97899999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.307},"max":{"1min":1.879,"5min":7.4930000000000003,"15min":7.4930000000000003},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.86099999999999999,"15min":1.006},"min":{"1min":0.36699999999999999,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":1.8420000000000001,"5min":3.8479999999999999,"15min":10.919},"last":0.32900000000000001}]}]},{"osd":5,"up_from":156,"seq":670014898382,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":928956,"kb_used_data":232372,"kb_used_omap":4,"kb_used_meta":696571,"kb_avail":92816196,"statfs":{"total":95995035648,"available":95043784704,"internally_reserved":0,"allocated":237948928,"data_stored":237459403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":713289378},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":2.052,"15min":1.778},"min":{"1min":0.27000000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.7290000000000001,"5min":24.292999999999999,"15min":52.938000000000002},"last":0.27400000000000002},{"interface":"front","average":{"1min":1.752,"5min":1.899,"15min":1.9419999999999999},"min":{"1min":0.27800000000000002,"5min":0.23899999999999999,"15min":0.22900000000000001},"max":{"1min":5.9939999999999998,"5min":23.908000000000001,"15min":52.805},"last":0.69199999999999995}]},{"osd":1,"last update":"Sat Feb 10 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":2.6890000000000001,"5min":2.2069999999999999,"15min":2.0339999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.245},"max":{"1min":23.405000000000001,"5min":28.780999999999999,"15min":54.298000000000002},"last":3.1520000000000001},{"interface":"front","average":{"1min":2.9649999999999999,"5min":2.3260000000000001,"15min":1.98},"min":{"1min":0.254,"5min":0.248,"15min":0.246},"max":{"1min":23.98,"5min":28.356000000000002,"15min":52.969999999999999},"last":0.83199999999999996}]},{"osd":2,"last update":"Sat Feb 10 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":2.3050000000000002,"5min":1.413,"15min":1.1899999999999999},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":10.039999999999999,"5min":10.039999999999999,"15min":10.039999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.9770000000000001,"5min":1.446,"15min":1.234},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28100000000000003},"max":{"1min":9.6069999999999993,"5min":9.6069999999999993,"15min":13.662000000000001},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Feb 10 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":2.2719999999999998,"15min":1.9099999999999999},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.7060000000000004,"5min":97.119,"15min":97.119},"last":2.3500000000000001},{"interface":"front","average":{"1min":2.2050000000000001,"5min":2.488,"15min":1.8120000000000001},"min":{"1min":0.39100000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":4.6180000000000003,"5min":97.141999999999996,"15min":97.141999999999996},"last":0.34899999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":1.2450000000000001,"15min":1.137},"min":{"1min":0.27200000000000002,"5min":0.214,"15min":0.214},"max":{"1min":3.0760000000000001,"5min":8.5890000000000004,"15min":8.952},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":1.127,"15min":1.0580000000000001},"min":{"1min":0.25,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.5019999999999998,"5min":6.202,"15min":10.180999999999999},"last":0.315}]},{"osd":6,"last update":"Sat Feb 10 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":1.4870000000000001,"15min":1.139},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.23999999999999999},"max":{"1min":3.048,"5min":21.748000000000001,"15min":21.748000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":1.4339999999999999,"15min":1.085},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.26300000000000001},"max":{"1min":3.9399999999999999,"5min":22.187999999999999,"15min":22.187999999999999},"last":0.38100000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:52:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":1.375,"15min":1.2529999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.044,"5min":22.302,"15min":22.302},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.4419999999999999,"15min":1.1439999999999999},"min":{"1min":0.34999999999999998,"5min":0.33400000000000002,"15min":0.26300000000000001},"max":{"1min":3.004,"5min":21.779,"15min":21.779},"last":0.498}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":508231680,"data_stored":508231680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":714936320,"data_stored":714932248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294785024,"data_stored":294780952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285081600,"data_stored":285081600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":213389312,"data_stored":213385240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":230359040,"data_stored":230359040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":418451456,"data_stored":418447384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":836108288,"data_stored":836108288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:53:43.954 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph status --format=json 2024-02-10T07:53:43.962 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:44.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:44.055 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:44.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:44.156 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:44.186 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:44.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:44.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:44.288 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:44.345 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:44.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:44.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:44.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:53:44 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:53:44.447 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:44.486 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:44.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:44.545 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:44.550 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:44.588 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:44.612 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:44 smithi008 ceph-mon[29816]: 6.16 scrub starts 2024-02-10T07:53:44.612 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:44 smithi008 ceph-mon[29816]: from='client.26024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:53:44.612 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:44 smithi008 ceph-mon[29816]: 6.16 scrub ok 2024-02-10T07:53:44.612 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:44 smithi008 ceph-mon[40743]: 6.16 scrub starts 2024-02-10T07:53:44.612 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:44 smithi008 ceph-mon[40743]: from='client.26024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:53:44.612 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:44 smithi008 ceph-mon[40743]: 6.16 scrub ok 2024-02-10T07:53:44.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:44.653 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:44.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:44.754 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:44.795 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:44.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:44.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:44 smithi170 ceph-mon[33394]: 6.16 scrub starts 2024-02-10T07:53:44.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:44 smithi170 ceph-mon[33394]: from='client.26024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:53:44.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:44 smithi170 ceph-mon[33394]: 6.16 scrub ok 2024-02-10T07:53:44.897 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:44.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:44.943 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:44.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:45.044 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:45.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:45.088 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:45.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:45.189 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:45.232 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:45.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:45.334 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:45.385 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:45.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:45.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:45.472 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:45 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:45.472 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:45 smithi008 ceph-mon[29816]: pgmap v1010: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 586 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.04k op/s; 6247/28683 objects degraded (21.779%); 3916/28683 objects misplaced (13.653%); 0 B/s, 128 objects/s recovering 2024-02-10T07:53:45.472 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:45 smithi008 ceph-mon[29816]: 4.16 scrub starts 2024-02-10T07:53:45.472 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:45 smithi008 ceph-mon[29816]: 4.16 scrub ok 2024-02-10T07:53:45.473 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:45 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1768008339' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:53:45.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:45 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:45.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:45 smithi008 ceph-mon[40743]: pgmap v1010: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 586 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.04k op/s; 6247/28683 objects degraded (21.779%); 3916/28683 objects misplaced (13.653%); 0 B/s, 128 objects/s recovering 2024-02-10T07:53:45.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:45 smithi008 ceph-mon[40743]: 4.16 scrub starts 2024-02-10T07:53:45.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:45 smithi008 ceph-mon[40743]: 4.16 scrub ok 2024-02-10T07:53:45.473 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:45 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1768008339' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:53:45.473 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is set 2024-02-10T07:53:45.488 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:45.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:45.521 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:45.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:45.622 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:45.668 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:45.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:45.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:45.769 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:45.797 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:45.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:45.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:45.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:45 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:45.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:45 smithi170 ceph-mon[33394]: pgmap v1010: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 586 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.04k op/s; 6247/28683 objects degraded (21.779%); 3916/28683 objects misplaced (13.653%); 0 B/s, 128 objects/s recovering 2024-02-10T07:53:45.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:45 smithi170 ceph-mon[33394]: 4.16 scrub starts 2024-02-10T07:53:45.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:45 smithi170 ceph-mon[33394]: 4.16 scrub ok 2024-02-10T07:53:45.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:45 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1768008339' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-10T07:53:45.898 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:45.937 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:45.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:45.959 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:46.038 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:46.084 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:46.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:46.186 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:46.225 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:46.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:46.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:46.326 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:46.367 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:46.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:46.468 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:46.497 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:46.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:46.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:46.599 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:46.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:46.656 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:46.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:46.741 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:46 smithi008 ceph-mon[29816]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:53:46.741 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:46 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1768008339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:53:46.741 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:46 smithi008 ceph-mon[29816]: osdmap e454: 8 total, 8 up, 8 in 2024-02-10T07:53:46.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:46 smithi008 ceph-mon[40743]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:53:46.741 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:46 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1768008339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:53:46.742 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:46 smithi008 ceph-mon[40743]: osdmap e454: 8 total, 8 up, 8 in 2024-02-10T07:53:46.757 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:46.789 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:46.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:46.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:46.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:46 smithi170 ceph-mon[33394]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:53:46.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:46 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1768008339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-10T07:53:46.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:46 smithi170 ceph-mon[33394]: osdmap e454: 8 total, 8 up, 8 in 2024-02-10T07:53:46.890 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:46.932 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:46.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:46.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:47.033 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:47.074 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:47.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:47.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:47.176 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:47.211 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:47.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:47.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:47.313 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:47.344 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:47.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:47.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:47.446 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:47.477 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:47.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:47.579 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:47.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:47.626 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:47.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:47.727 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:47.758 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:47.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:47.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:47.807 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:47 smithi008 ceph-mon[29816]: pgmap v1012: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 502 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 857 op/s; 6247/24678 objects degraded (25.314%); 3783/24678 objects misplaced (15.329%); 0 B/s, 102 objects/s recovering 2024-02-10T07:53:47.807 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:47 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 6247/24678 objects degraded (25.314%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:47.807 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:47 smithi008 ceph-mon[40743]: pgmap v1012: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 502 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 857 op/s; 6247/24678 objects degraded (25.314%); 3783/24678 objects misplaced (15.329%); 0 B/s, 102 objects/s recovering 2024-02-10T07:53:47.807 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:47 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 6247/24678 objects degraded (25.314%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:47.859 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:47 smithi170 ceph-mon[33394]: pgmap v1012: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 502 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 857 op/s; 6247/24678 objects degraded (25.314%); 3783/24678 objects misplaced (15.329%); 0 B/s, 102 objects/s recovering 2024-02-10T07:53:47.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:47 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 6247/24678 objects degraded (25.314%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:47.887 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:47.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:47.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:47.989 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:48.021 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:48.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:48.122 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:48.159 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:48.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:48.261 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:48.294 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:48.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:48.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:48.396 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:48.434 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:48.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:48.535 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:48.568 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:48.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:48.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:48.670 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:48.709 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:48.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:48.810 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:48.848 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:48.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:48.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:48.950 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:48.981 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:48.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:48.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:49.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:53:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:53:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:53:49.083 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:49.125 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:49.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:49.226 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:49.257 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:49.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:49.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:49.358 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:49.389 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:49.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:49.491 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:49.527 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:49.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:49.629 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:49.665 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:49.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:49.767 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:49.793 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:49.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:49.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:49 smithi170 ceph-mon[33394]: pgmap v1013: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 487 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 854 op/s; 6247/23943 objects degraded (26.091%); 3783/23943 objects misplaced (15.800%); 0 B/s, 102 objects/s recovering 2024-02-10T07:53:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:53:49.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:53:49.895 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:49.934 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:49.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:49.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:50.035 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:50.069 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:50.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:50.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:50.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:49 smithi008 ceph-mon[29816]: pgmap v1013: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 487 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 854 op/s; 6247/23943 objects degraded (26.091%); 3783/23943 objects misplaced (15.800%); 0 B/s, 102 objects/s recovering 2024-02-10T07:53:50.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:53:50.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:53:50.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:49 smithi008 ceph-mon[40743]: pgmap v1013: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 487 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 854 op/s; 6247/23943 objects degraded (26.091%); 3783/23943 objects misplaced (15.800%); 0 B/s, 102 objects/s recovering 2024-02-10T07:53:50.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:53:50.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:53:50.170 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:50.198 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:50.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:50.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:50.300 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:50.335 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:50.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:50.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:50.437 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:50.468 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:50.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:50.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:50.570 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:50.604 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:50.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:50.705 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:50.734 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:50.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:50.835 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:50.872 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:50.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:50.974 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:51.008 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:51.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:51.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:51.110 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:51.145 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:51.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:51.247 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:51.285 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:51.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:51.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:51.387 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:51.423 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:51.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:51.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:51.525 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:51.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:51.566 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:51.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:51.666 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:51.713 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:51.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:51.817 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:51.852 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:51.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:51.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:51 smithi170 ceph-mon[33394]: pgmap v1014: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 255 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 857 op/s; 5854/12804 objects degraded (45.720%); 3642/12804 objects misplaced (28.444%); 0 B/s, 106 objects/s recovering 2024-02-10T07:53:51.954 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:51.985 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:51.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:51.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:52.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:51 smithi008 ceph-mon[29816]: pgmap v1014: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 255 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 857 op/s; 5854/12804 objects degraded (45.720%); 3642/12804 objects misplaced (28.444%); 0 B/s, 106 objects/s recovering 2024-02-10T07:53:52.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:51 smithi008 ceph-mon[40743]: pgmap v1014: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 255 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 857 op/s; 5854/12804 objects degraded (45.720%); 3642/12804 objects misplaced (28.444%); 0 B/s, 106 objects/s recovering 2024-02-10T07:53:52.086 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:52.123 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:52.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:52.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:52.225 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:52.261 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:52.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:52.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:52.362 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:52.400 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:52.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:52.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:52.502 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:52.535 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:52.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:52.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:52.637 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:52.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:52.680 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:52.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:52.781 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:52.806 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:52.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:52.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:52.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:52 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 5854/12804 objects degraded (45.720%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:52.908 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:52.939 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:52.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:52.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:53.041 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:53.065 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:53.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:53.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:53.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:52 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 5854/12804 objects degraded (45.720%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:53.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:52 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 5854/12804 objects degraded (45.720%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:53.163 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Removed 104759 objects 2024-02-10T07:53:53.163 INFO:tasks.radosbench.radosbench.0.smithi008.stdout:Clean up completed and total clean up time :136.484 2024-02-10T07:53:53.167 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:53.192 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:53.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:53.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:53.294 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:53.318 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:53.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:53.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:53.420 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:53.445 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:53.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:53.547 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:53.571 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:53.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:53.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:53.673 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:53.698 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:53.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:53.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:53.799 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:53.824 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:53.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:53.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:53.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:53 smithi170 ceph-mon[33394]: pgmap v1015: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 220 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 858 op/s; 5854/11127 objects degraded (52.611%); 3642/11127 objects misplaced (32.731%); 0 B/s, 106 objects/s recovering 2024-02-10T07:53:53.926 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:53.954 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:53.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:54.056 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:54.080 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:54.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:54.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:54.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:53 smithi008 ceph-mon[29816]: pgmap v1015: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 220 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 858 op/s; 5854/11127 objects degraded (52.611%); 3642/11127 objects misplaced (32.731%); 0 B/s, 106 objects/s recovering 2024-02-10T07:53:54.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:53 smithi008 ceph-mon[40743]: pgmap v1015: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 220 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 858 op/s; 5854/11127 objects degraded (52.611%); 3642/11127 objects misplaced (32.731%); 0 B/s, 106 objects/s recovering 2024-02-10T07:53:54.182 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:54.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:54.209 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:54.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:54.310 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:54.335 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:54.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:54.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:54.340 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:53:54.340 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:53:54.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:53:54 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:53:54.438 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:54.462 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:54.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:54.564 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:54.589 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:54.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:54.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:54.691 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:54.717 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:54.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:54.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:54.818 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:54.845 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:54.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:54.947 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:54.971 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:54.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:54.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:55.073 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:55.098 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:55.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:55.200 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:55.224 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:55.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:55.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:55.326 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:55.353 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:55.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:55.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:55.454 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:55.477 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:55.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:55.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:55.579 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:55.605 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:55.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:55.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:55.707 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:55.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:55.735 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:55.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:55.774 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:55 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:55.774 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:55 smithi008 ceph-mon[29816]: pgmap v1016: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 58 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 845 op/s; 5330/3354 objects degraded (158.915%); 3642/3354 objects misplaced (108.587%); 0 B/s, 119 objects/s recovering 2024-02-10T07:53:55.775 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:55 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:55.775 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:55 smithi008 ceph-mon[40743]: pgmap v1016: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 58 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 845 op/s; 5330/3354 objects degraded (158.915%); 3642/3354 objects misplaced (108.587%); 0 B/s, 119 objects/s recovering 2024-02-10T07:53:55.836 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:55.860 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:55.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:55.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:55.882 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:55 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:53:55.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:55 smithi170 ceph-mon[33394]: pgmap v1016: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 58 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 845 op/s; 5330/3354 objects degraded (158.915%); 3642/3354 objects misplaced (108.587%); 0 B/s, 119 objects/s recovering 2024-02-10T07:53:55.962 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:55.968 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-10T07:53:55.988 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:55.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:55.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:56.089 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:56.113 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:56.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:56.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:56.215 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:56.240 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:56.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:56.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:56.342 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:56.367 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:56.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:56.469 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:56.493 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:56.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:56.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:56.595 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:56.620 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:56.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:56.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:56.722 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:56.745 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:56.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:56.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:56.847 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:56.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:56.875 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:56.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:56.916 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:53:56.916 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:56 smithi008 ceph-mon[29816]: pgmap v1017: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 6.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 953 B/s rd, 0 B/s wr, 739 op/s; 5330/882 objects degraded (604.308%); 3330/882 objects misplaced (377.551%); 0 B/s, 127 objects/s recovering 2024-02-10T07:53:56.917 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:56 smithi008 ceph-mon[40743]: pgmap v1017: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 6.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 953 B/s rd, 0 B/s wr, 739 op/s; 5330/882 objects degraded (604.308%); 3330/882 objects misplaced (377.551%); 0 B/s, 127 objects/s recovering 2024-02-10T07:53:56.940 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:53:56.965 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:53:56.976 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:56.988 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:53:57.002 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:57.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:57.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:57.010 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:53:57.033 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:53:57.056 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:53:57.081 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:53:57.083 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:53:57.083 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:53:57.083 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:53:57.083 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:53:57.083 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:53:57.083 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:53:57.083 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:53:57.083 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:53:57.103 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:57.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:56 smithi170 ceph-mon[33394]: pgmap v1017: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 6.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 953 B/s rd, 0 B/s wr, 739 op/s; 5330/882 objects degraded (604.308%); 3330/882 objects misplaced (377.551%); 0 B/s, 127 objects/s recovering 2024-02-10T07:53:57.134 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:57.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:57.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:57.236 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:57.260 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:57.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:57.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:57.362 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:57.386 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:57.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:57.478 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-02-10T07:53:57.478 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-02-10T07:53:57.488 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:57.513 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:57.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:57.615 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:57.640 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:57.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:57.742 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:57.767 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:57.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:57.808 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-10T07:53:57.831 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:57 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2384748704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:53:57.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:57 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 5330/882 objects degraded (604.308%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:57.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:57 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2384748704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:53:57.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:57 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 5330/882 objects degraded (604.308%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:57.869 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:57.897 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:57.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:57.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:57.999 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:53:58.029 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:58.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:58.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:53:58.131 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:58.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:57 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2384748704' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:53:58.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:57 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 5330/882 objects degraded (604.308%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:53:58.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:58.169 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:58.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:58.195 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:58.218 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:58.245 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:53:58.270 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:58.295 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:58.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:58.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:58.397 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:58.421 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:58.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:58.523 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:53:58.547 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:58.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:53:58.549 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:53:58.549 INFO:teuthology.orchestra.run.smithi008.stdout:{"fsid":"8c8244e6-c7e5-11ee-95b8-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5330/882 objects degraded (604.308%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1553,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":454,"num_osds":8,"num_up_osds":8,"osd_up_since":1707551266,"num_in_osds":8,"osd_in_since":1707551442,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":144},{"state_name":"active+recovering+undersized+degraded+remapped","count":2},{"state_name":"active+recovery_wait","count":1},{"state_name":"active+recovering+remapped","count":1}],"num_pgs":148,"num_pools":7,"num_objects":294,"data_bytes":6887772,"bytes_used":6778945536,"bytes_avail":761181339648,"bytes_total":767960285184,"degraded_objects":5330,"degraded_total":882,"degraded_ratio":6.0430839002267582,"misplaced_objects":3330,"misplaced_total":882,"misplaced_ratio":3.7755102040816326,"recovering_objects_per_sec":127,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1370,"num_bytes_recovered":0,"num_keys_recovered":0,"read_bytes_sec":953,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":738},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.8:8443/","prometheus":"http://172.21.15.8:9283/"}},"servicemap":{"epoch":17,"modified":"2024-02-10T07:48:48.075100+0000","services":{"rgw":{"daemons":{"summary":"","24356":{"start_epoch":12,"start_stamp":"2024-02-10T07:31:58.404558+0000","gid":24356,"addr":"172.21.15.8:0/1933562162","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)","ceph_version_short":"19.0.0-1361-g290212a1","container_hostname":"smithi008","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi008","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024","kernel_version":"5.14.0-419.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32494648","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"e13f4d73-e6ee-4c04-9e1f-ab0b8dc42c1d","zone_name":"default","zonegroup_id":"3e6c94ee-8628-4131-87db-41eaa46e75bf","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-02-10T07:53:58.650 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:58.680 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:58.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:58.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:58.782 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:53:58.812 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:58.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:53:58.914 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:53:58.939 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:58.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:58.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:53:59.041 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:59.066 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:59.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:59.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:53:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:58 smithi008 ceph-mon[29816]: pgmap v1018: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 669 op/s; 5330/594 objects degraded (897.306%); 3330/594 objects misplaced (560.606%); 0 B/s, 114 objects/s recovering 2024-02-10T07:53:59.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:58 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/966568834' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-10T07:53:59.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:53:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:53:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:53:59.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:58 smithi008 ceph-mon[40743]: pgmap v1018: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 669 op/s; 5330/594 objects degraded (897.306%); 3330/594 objects misplaced (560.606%); 0 B/s, 114 objects/s recovering 2024-02-10T07:53:59.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:58 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/966568834' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-10T07:53:59.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:58 smithi170 ceph-mon[33394]: pgmap v1018: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 669 op/s; 5330/594 objects degraded (897.306%); 3330/594 objects misplaced (560.606%); 0 B/s, 114 objects/s recovering 2024-02-10T07:53:59.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:58 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/966568834' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-02-10T07:53:59.168 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:59.194 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:59.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:59.295 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:59.321 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:59.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:59.423 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:59.448 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:59.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:59.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:59.549 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:53:59.574 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:59.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:53:59.676 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:53:59.700 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:53:59.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:59.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:53:59.802 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:53:59.828 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:59.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:53:59.922 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-10T07:53:59.930 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:53:59.948 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:59 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:53:59.948 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:59 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:53:59.948 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:53:59 smithi008 ceph-mon[29816]: osdmap e455: 8 total, 8 up, 8 in 2024-02-10T07:53:59.948 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:59 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:53:59.948 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:59 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:53:59.949 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:53:59 smithi008 ceph-mon[40743]: osdmap e455: 8 total, 8 up, 8 in 2024-02-10T07:53:59.975 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:53:59.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:53:59.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:54:00.033 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:59 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:54:00.033 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:59 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:54:00.033 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:53:59 smithi170 ceph-mon[33394]: osdmap e455: 8 total, 8 up, 8 in 2024-02-10T07:54:00.077 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:54:00.101 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:00.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:00.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:54:00.203 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:54:00.227 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:00.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:00.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:54:00.329 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:54:00.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:00.355 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:00.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:54:00.456 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:54:00.481 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:00.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:00.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:54:00.583 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:54:00.608 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:00.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:00.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:54:00.710 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:54:00.737 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:00.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:00.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:54:00.839 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:54:00.866 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:00.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:54:00.967 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:54:00.995 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:00.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:00.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:54:01.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:00 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:54:01.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:00 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:54:01.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:00 smithi008 ceph-mon[29816]: pgmap v1020: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 407 op/s; 3706/594 objects degraded (623.906%); 2690/594 objects misplaced (452.862%); 0 B/s, 309 objects/s recovering 2024-02-10T07:54:01.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:00 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:54:01.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:00 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:54:01.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:00 smithi008 ceph-mon[40743]: pgmap v1020: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 407 op/s; 3706/594 objects degraded (623.906%); 2690/594 objects misplaced (452.862%); 0 B/s, 309 objects/s recovering 2024-02-10T07:54:01.097 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:54:01.122 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:01.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:01.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:54:01.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:00 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:54:01.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:00 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:54:01.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:00 smithi170 ceph-mon[33394]: pgmap v1020: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 407 op/s; 3706/594 objects degraded (623.906%); 2690/594 objects misplaced (452.862%); 0 B/s, 309 objects/s recovering 2024-02-10T07:54:01.225 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:54:01.250 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:01.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:01.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:54:01.352 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:54:01.377 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:01.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:01.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:54:01.479 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:54:01.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:01.505 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:01.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:54:01.606 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:54:01.630 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:01.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:01.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:54:01.732 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:54:01.756 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:01.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:54:01.858 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:54:01.884 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:01.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:01.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:54:01.985 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:54:02.013 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:02.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:02.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:54:02.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:01 smithi008 ceph-mon[29816]: osdmap e456: 8 total, 8 up, 8 in 2024-02-10T07:54:02.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:01 smithi008 ceph-mon[40743]: osdmap e456: 8 total, 8 up, 8 in 2024-02-10T07:54:02.115 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:54:02.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:01 smithi170 ceph-mon[33394]: osdmap e456: 8 total, 8 up, 8 in 2024-02-10T07:54:02.141 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:02.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:54:02.243 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:54:02.267 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:02.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:02.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:54:02.369 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:54:02.398 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:02.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:02.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:54:02.499 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:54:02.524 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:02.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:02.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:54:02.626 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:54:02.655 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:02.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:02.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:54:02.756 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:54:02.783 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:02.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:54:02.884 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:54:02.911 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:02.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:54:03.013 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:54:03.039 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:03.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:54:03.081 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:02 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 3706/594 objects degraded (623.906%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:54:03.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:02 smithi008 ceph-mon[29816]: osdmap e457: 8 total, 8 up, 8 in 2024-02-10T07:54:03.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:02 smithi008 ceph-mon[29816]: pgmap v1023: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 16 op/s; 3706/594 objects degraded (623.906%); 2690/594 objects misplaced (452.862%); 0 B/s, 377 objects/s recovering 2024-02-10T07:54:03.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:02 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:54:03.082 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:02 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:54:03.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:02 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 3706/594 objects degraded (623.906%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:54:03.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:02 smithi008 ceph-mon[40743]: osdmap e457: 8 total, 8 up, 8 in 2024-02-10T07:54:03.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:02 smithi008 ceph-mon[40743]: pgmap v1023: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 16 op/s; 3706/594 objects degraded (623.906%); 2690/594 objects misplaced (452.862%); 0 B/s, 377 objects/s recovering 2024-02-10T07:54:03.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:02 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:54:03.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:02 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:54:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:02 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 3706/594 objects degraded (623.906%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:54:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:02 smithi170 ceph-mon[33394]: osdmap e457: 8 total, 8 up, 8 in 2024-02-10T07:54:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:02 smithi170 ceph-mon[33394]: pgmap v1023: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 16 op/s; 3706/594 objects degraded (623.906%); 2690/594 objects misplaced (452.862%); 0 B/s, 377 objects/s recovering 2024-02-10T07:54:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:02 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:54:03.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:02 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:54:03.141 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:54:03.172 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:03.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:03.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:54:03.274 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:54:03.299 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:03.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:03.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:54:03.401 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:54:03.427 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:03.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:03.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:54:03.529 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:54:03.557 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:03.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:03.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:54:03.659 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:54:03.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:03.683 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:03.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:54:03.784 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:54:03.812 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:03.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:54:03.914 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:54:03.941 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:03.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:54:04.045 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:54:04.070 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:04.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:04.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:54:04.172 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:54:04.198 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:04.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:04.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:54:04.300 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:54:04.323 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:04.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:04.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:54:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:04 smithi008 ceph-mon[29816]: osdmap e458: 8 total, 8 up, 8 in 2024-02-10T07:54:04.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:54:04.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:04 smithi008 ceph-mon[40743]: osdmap e458: 8 total, 8 up, 8 in 2024-02-10T07:54:04.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:54:04.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:54:04 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:54:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:04 smithi170 ceph-mon[33394]: osdmap e458: 8 total, 8 up, 8 in 2024-02-10T07:54:04.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:54:04.426 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:54:04.451 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:04.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:04.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:54:04.553 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:54:04.577 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:04.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:54:04.679 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:54:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:04.703 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:04.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:54:04.804 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:54:04.828 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:04.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:04.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:54:04.930 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:54:04.957 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:04.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:04.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:54:05.059 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:54:05.087 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:05.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:05.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:54:05.189 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:54:05.213 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:05.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:54:05.315 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:54:05.341 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:05.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:05.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:54:05.443 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:54:05.468 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:05.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:05.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:54:05.570 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:54:05.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:05 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:05.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:05 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:54:05.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:05 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:54:05.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:05 smithi008 ceph-mon[29816]: osdmap e459: 8 total, 8 up, 8 in 2024-02-10T07:54:05.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:05 smithi008 ceph-mon[29816]: pgmap v1026: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 616/594 objects degraded (103.704%); 2690/594 objects misplaced (452.862%); 0 B/s, 772 objects/s recovering 2024-02-10T07:54:05.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:05 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:05.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:05 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:54:05.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:05 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:54:05.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:05 smithi008 ceph-mon[40743]: osdmap e459: 8 total, 8 up, 8 in 2024-02-10T07:54:05.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:05 smithi008 ceph-mon[40743]: pgmap v1026: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 616/594 objects degraded (103.704%); 2690/594 objects misplaced (452.862%); 0 B/s, 772 objects/s recovering 2024-02-10T07:54:05.598 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:05.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:05.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:54:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:05 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:05 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:54:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:05 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:54:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:05 smithi170 ceph-mon[33394]: osdmap e459: 8 total, 8 up, 8 in 2024-02-10T07:54:05.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:05 smithi170 ceph-mon[33394]: pgmap v1026: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 144 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 616/594 objects degraded (103.704%); 2690/594 objects misplaced (452.862%); 0 B/s, 772 objects/s recovering 2024-02-10T07:54:05.700 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:54:05.728 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:05.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:05.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:54:05.830 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:54:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:05.857 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:05.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:54:05.957 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:54:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:05.984 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:05.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:54:06.085 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:54:06.113 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:06.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:06.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:54:06.214 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:54:06.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:06.242 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:06.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:54:06.343 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:54:06.369 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:06.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:06.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:54:06.471 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:54:06.497 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:06.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:06.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:54:06.565 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:06 smithi008 ceph-mon[29816]: osdmap e460: 8 total, 8 up, 8 in 2024-02-10T07:54:06.565 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:06 smithi008 ceph-mon[40743]: osdmap e460: 8 total, 8 up, 8 in 2024-02-10T07:54:06.599 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:54:06.625 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:06.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:06.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:54:06.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:06 smithi170 ceph-mon[33394]: osdmap e460: 8 total, 8 up, 8 in 2024-02-10T07:54:06.727 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:54:06.753 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:06.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:54:06.855 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:54:06.883 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:06.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:06.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:54:06.985 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:54:07.012 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:07.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:07.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:54:07.036 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-02-10T07:54:07.036 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:54:07.114 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:54:07.153 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:07.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:07.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:54:07.255 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5 2024-02-10T07:54:07.283 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:07.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:07.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.5' 2024-02-10T07:54:07.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:07 smithi008 ceph-mon[29816]: osdmap e461: 8 total, 8 up, 8 in 2024-02-10T07:54:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:07 smithi008 ceph-mon[29816]: pgmap v1029: 148 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 616/594 objects degraded (103.704%); 1868/594 objects misplaced (314.478%); 0 B/s, 977 objects/s recovering 2024-02-10T07:54:07.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:07 smithi008 ceph-mon[29816]: Health check update: Degraded data redundancy: 616/594 objects degraded (103.704%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:54:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:07 smithi008 ceph-mon[40743]: osdmap e461: 8 total, 8 up, 8 in 2024-02-10T07:54:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:07 smithi008 ceph-mon[40743]: pgmap v1029: 148 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 616/594 objects degraded (103.704%); 1868/594 objects misplaced (314.478%); 0 B/s, 977 objects/s recovering 2024-02-10T07:54:07.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:07 smithi008 ceph-mon[40743]: Health check update: Degraded data redundancy: 616/594 objects degraded (103.704%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:54:07.384 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6 2024-02-10T07:54:07.412 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:07.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.6' 2024-02-10T07:54:07.517 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:54:07.545 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:07.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:54:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:07 smithi170 ceph-mon[33394]: osdmap e461: 8 total, 8 up, 8 in 2024-02-10T07:54:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:07 smithi170 ceph-mon[33394]: pgmap v1029: 148 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 616/594 objects degraded (103.704%); 1868/594 objects misplaced (314.478%); 0 B/s, 977 objects/s recovering 2024-02-10T07:54:07.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:07 smithi170 ceph-mon[33394]: Health check update: Degraded data redundancy: 616/594 objects degraded (103.704%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-02-10T07:54:07.647 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:54:07.674 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:07.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:07.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:54:07.776 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:54:07.802 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:07.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:07.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:54:07.904 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:54:07.929 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:07.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:54:08.031 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:54:08.059 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:08.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:08.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:54:08.160 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:54:08.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:08.189 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:08.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:54:08.290 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:54:08.318 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:08.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:08.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:54:08.420 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:54:08.444 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:08.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:54:08.546 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:54:08.574 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:08.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:08.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:54:08.675 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:54:08.701 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:08.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:08.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:54:08.802 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:54:08.830 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:08.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:54:08.932 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:54:08.959 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:08.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:08.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:54:09.061 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:54:09.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:54:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:54:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:54:09.084 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:09.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:54:09.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7 2024-02-10T07:54:09.214 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:09.214 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:09.215 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:09.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:09.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.7' 2024-02-10T07:54:09.294 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'unique_pool_4' removed 2024-02-10T07:54:09.316 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3 2024-02-10T07:54:09.335 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:09 smithi008 ceph-mon[29816]: pgmap v1030: 148 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s; 616/594 objects degraded (103.704%); 1868/594 objects misplaced (314.478%); 0 B/s, 771 objects/s recovering 2024-02-10T07:54:09.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:09 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/944269385' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:54:09.336 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:09 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:54:09.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:09 smithi008 ceph-mon[40743]: pgmap v1030: 148 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s; 616/594 objects degraded (103.704%); 1868/594 objects misplaced (314.478%); 0 B/s, 771 objects/s recovering 2024-02-10T07:54:09.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:09 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/944269385' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:54:09.336 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:09 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:54:09.397 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:09.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:09.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.3' 2024-02-10T07:54:09.499 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:54:09.523 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:09.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:54:09.625 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2 2024-02-10T07:54:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:09 smithi170 ceph-mon[33394]: pgmap v1030: 148 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s; 616/594 objects degraded (103.704%); 1868/594 objects misplaced (314.478%); 0 B/s, 771 objects/s recovering 2024-02-10T07:54:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:09 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/944269385' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:54:09.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:09 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-10T07:54:09.647 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:09.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:09.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.2' 2024-02-10T07:54:09.749 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0 2024-02-10T07:54:09.772 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:09.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.0' 2024-02-10T07:54:09.874 DEBUG:teuthology.orchestra.run.smithi170:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4 2024-02-10T07:54:09.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:09.900 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: docker: command not found 2024-02-10T07:54:09.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi170 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.4' 2024-02-10T07:54:10.001 DEBUG:teuthology.orchestra.run.smithi008:> sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1 2024-02-10T07:54:10.025 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: docker: command not found 2024-02-10T07:54:10.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T07:54:10.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi008 with status 1: 'sudo docker kill -s 1 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd.1' 2024-02-10T07:54:10.124 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-02-10T07:54:10.134 INFO:tasks.thrashosds:joining thrashosds 2024-02-10T07:54:10.284 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is set 2024-02-10T07:54:10.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:10 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:54:10.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:10 smithi008 ceph-mon[29816]: osdmap e462: 8 total, 8 up, 8 in 2024-02-10T07:54:10.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:10 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3383801523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:54:10.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:10 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:54:10.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:10 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:54:10.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:10 smithi008 ceph-mon[40743]: osdmap e462: 8 total, 8 up, 8 in 2024-02-10T07:54:10.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:10 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3383801523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:54:10.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:10 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:54:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:10 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-10T07:54:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:10 smithi170 ceph-mon[33394]: osdmap e462: 8 total, 8 up, 8 in 2024-02-10T07:54:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:10 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3383801523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:54:10.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:10 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:54:11.318 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:11 smithi008 ceph-mon[29816]: pgmap v1032: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:54:11.318 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:11 smithi008 ceph-mon[29816]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 616/594 objects degraded (103.704%), 2 pgs degraded, 2 pgs undersized) 2024-02-10T07:54:11.318 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:11 smithi008 ceph-mon[29816]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:54:11.318 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:11 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:54:11.318 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:11 smithi008 ceph-mon[29816]: osdmap e463: 8 total, 8 up, 8 in 2024-02-10T07:54:11.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:11 smithi008 ceph-mon[40743]: pgmap v1032: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:54:11.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:11 smithi008 ceph-mon[40743]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 616/594 objects degraded (103.704%), 2 pgs degraded, 2 pgs undersized) 2024-02-10T07:54:11.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:11 smithi008 ceph-mon[40743]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:54:11.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:11 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:54:11.318 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:11 smithi008 ceph-mon[40743]: osdmap e463: 8 total, 8 up, 8 in 2024-02-10T07:54:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:11 smithi170 ceph-mon[33394]: pgmap v1032: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-02-10T07:54:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:11 smithi170 ceph-mon[33394]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 616/594 objects degraded (103.704%), 2 pgs degraded, 2 pgs undersized) 2024-02-10T07:54:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:11 smithi170 ceph-mon[33394]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:54:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:11 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-10T07:54:11.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:11 smithi170 ceph-mon[33394]: osdmap e463: 8 total, 8 up, 8 in 2024-02-10T07:54:13.201 INFO:teuthology.orchestra.run.smithi008.stdout:osd.0: {} 2024-02-10T07:54:13.220 INFO:teuthology.orchestra.run.smithi008.stdout:osd.1: {} 2024-02-10T07:54:13.240 INFO:teuthology.orchestra.run.smithi008.stdout:osd.2: {} 2024-02-10T07:54:13.259 INFO:teuthology.orchestra.run.smithi008.stdout:osd.3: {} 2024-02-10T07:54:13.280 INFO:teuthology.orchestra.run.smithi008.stdout:osd.4: {} 2024-02-10T07:54:13.301 INFO:teuthology.orchestra.run.smithi008.stdout:osd.5: {} 2024-02-10T07:54:13.322 INFO:teuthology.orchestra.run.smithi008.stdout:osd.6: {} 2024-02-10T07:54:13.343 INFO:teuthology.orchestra.run.smithi008.stdout:osd.7: {} 2024-02-10T07:54:13.344 INFO:teuthology.orchestra.run.smithi008.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-10T07:54:13.344 INFO:teuthology.orchestra.run.smithi008.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-10T07:54:13.344 INFO:teuthology.orchestra.run.smithi008.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-10T07:54:13.344 INFO:teuthology.orchestra.run.smithi008.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-10T07:54:13.344 INFO:teuthology.orchestra.run.smithi008.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-10T07:54:13.344 INFO:teuthology.orchestra.run.smithi008.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-10T07:54:13.344 INFO:teuthology.orchestra.run.smithi008.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-10T07:54:13.345 INFO:teuthology.orchestra.run.smithi008.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-10T07:54:13.345 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset noscrub 2024-02-10T07:54:13.372 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:13 smithi008 ceph-mon[29816]: pgmap v1034: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-02-10T07:54:13.372 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:13 smithi008 ceph-mon[40743]: pgmap v1034: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-02-10T07:54:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:13 smithi170 ceph-mon[33394]: pgmap v1034: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-02-10T07:54:14.325 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2024-02-10T07:54:14.326 INFO:teuthology.orchestra.run.smithi008.stdout: "ops": [], 2024-02-10T07:54:14.326 INFO:teuthology.orchestra.run.smithi008.stdout: "num_ops": 0 2024-02-10T07:54:14.326 INFO:teuthology.orchestra.run.smithi008.stdout:} 2024-02-10T07:54:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:14 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1702191919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:54:14.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:54:14 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:54:14.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:14 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1702191919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:54:14.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:14 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1702191919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-10T07:54:15.211 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-10T07:54:15.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:15 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:15.528 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:15 smithi008 ceph-mon[29816]: pgmap v1035: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:54:15.528 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:15 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:15.528 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:15 smithi008 ceph-mon[40743]: pgmap v1035: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:54:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:15 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:15 smithi170 ceph-mon[33394]: pgmap v1035: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:54:16.376 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:16.406 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:17.532 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:17 smithi008 ceph-mon[29816]: pgmap v1036: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:54:17.533 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:17 smithi008 ceph-mon[40743]: pgmap v1036: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:54:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:17 smithi170 ceph-mon[33394]: pgmap v1036: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:54:17.862 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:54:17.864 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:54:18.533 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":1036,"stamp":"2024-02-10T07:54:16.213977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3111,"num_read_kb":2957,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":1780341,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1297,"ondisk_log_size":1297,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5969608,"kb_used_data":444956,"kb_used_omap":19,"kb_used_meta":5524524,"kb_avail":743991608,"statfs":{"total":767960285184,"available":761847406592,"internally_reserved":0,"allocated":455634944,"data_stored":451337484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19634,"internal_metadata":5657113422},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":84,"apply_latency_ms":84,"commit_latency_ns":84000000,"apply_latency_ns":84000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-616,"num_objects_misplaced":-1868,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":-209520,"num_write_kb":-6704577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-23872,"num_bytes_recovered":-801964032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002304"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":863,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278797+0000","last_change":"2024-02-10T07:49:56.315126+0000","last_active":"2024-02-10T07:54:10.278797+0000","last_peered":"2024-02-10T07:54:10.278797+0000","last_clean":"2024-02-10T07:54:10.278797+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:54:10.278797+0000","last_undegraded":"2024-02-10T07:54:10.278797+0000","last_fullsized":"2024-02-10T07:54:10.278797+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.315029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.315029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876881+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":874,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.686984+0000","last_change":"2024-02-10T07:49:08.138553+0000","last_active":"2024-02-10T07:54:10.686984+0000","last_peered":"2024-02-10T07:54:10.686984+0000","last_clean":"2024-02-10T07:54:10.686984+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:54:10.686984+0000","last_undegraded":"2024-02-10T07:54:10.686984+0000","last_fullsized":"2024-02-10T07:54:10.686984+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:08.138442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.138442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651825+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":903,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.473585+0000","last_change":"2024-02-10T07:50:43.303349+0000","last_active":"2024-02-10T07:54:10.473585+0000","last_peered":"2024-02-10T07:54:10.473585+0000","last_clean":"2024-02-10T07:54:10.473585+0000","last_became_active":"2024-02-10T07:50:43.281905+0000","last_became_peered":"2024-02-10T07:50:43.281905+0000","last_unstale":"2024-02-10T07:54:10.473585+0000","last_undegraded":"2024-02-10T07:54:10.473585+0000","last_fullsized":"2024-02-10T07:54:10.473585+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:17.198701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:17.198701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.477521+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"430'25","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278460+0000","last_change":"2024-02-10T07:46:37.008978+0000","last_active":"2024-02-10T07:54:10.278460+0000","last_peered":"2024-02-10T07:54:10.278460+0000","last_clean":"2024-02-10T07:54:10.278460+0000","last_became_active":"2024-02-10T07:46:10.717475+0000","last_became_peered":"2024-02-10T07:46:10.717475+0000","last_unstale":"2024-02-10T07:54:10.278460+0000","last_undegraded":"2024-02-10T07:54:10.278460+0000","last_fullsized":"2024-02-10T07:54:10.278460+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'19","last_scrub_stamp":"2024-02-10T07:46:37.008841+0000","last_deep_scrub":"287'19","last_deep_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.008841+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876499+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":203,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278286+0000","last_change":"2024-02-10T07:53:26.218018+0000","last_active":"2024-02-10T07:54:10.278286+0000","last_peered":"2024-02-10T07:54:10.278286+0000","last_clean":"2024-02-10T07:54:10.278286+0000","last_became_active":"2024-02-10T07:49:06.236235+0000","last_became_peered":"2024-02-10T07:49:06.236235+0000","last_unstale":"2024-02-10T07:54:10.278286+0000","last_undegraded":"2024-02-10T07:54:10.278286+0000","last_fullsized":"2024-02-10T07:54:10.278286+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:26.217749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:15.079428+0000","last_clean_scrub_stamp":"2024-02-10T07:53:26.217749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:52.906075+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":867,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278347+0000","last_change":"2024-02-10T07:50:06.335390+0000","last_active":"2024-02-10T07:54:10.278347+0000","last_peered":"2024-02-10T07:54:10.278347+0000","last_clean":"2024-02-10T07:54:10.278347+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:54:10.278347+0000","last_undegraded":"2024-02-10T07:54:10.278347+0000","last_fullsized":"2024-02-10T07:54:10.278347+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:06.335262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:50:06.335262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876939+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":390,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.971906+0000","last_change":"2024-02-10T07:51:05.208836+0000","last_active":"2024-02-10T07:54:10.971906+0000","last_peered":"2024-02-10T07:54:10.971906+0000","last_clean":"2024-02-10T07:54:10.971906+0000","last_became_active":"2024-02-10T07:50:43.282449+0000","last_became_peered":"2024-02-10T07:50:43.282449+0000","last_unstale":"2024-02-10T07:54:10.971906+0000","last_undegraded":"2024-02-10T07:54:10.971906+0000","last_fullsized":"2024-02-10T07:54:10.971906+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:51:05.208755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:51:05.208755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"430'39","reported_seq":162,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.969532+0000","last_change":"2024-02-10T07:50:43.318008+0000","last_active":"2024-02-10T07:54:10.969532+0000","last_peered":"2024-02-10T07:54:10.969532+0000","last_clean":"2024-02-10T07:54:10.969532+0000","last_became_active":"2024-02-10T07:50:43.290681+0000","last_became_peered":"2024-02-10T07:50:43.290681+0000","last_unstale":"2024-02-10T07:54:10.969532+0000","last_undegraded":"2024-02-10T07:54:10.969532+0000","last_fullsized":"2024-02-10T07:54:10.969532+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:48:13.230908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.230908+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.506511+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":863,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.473123+0000","last_change":"2024-02-10T07:50:05.620136+0000","last_active":"2024-02-10T07:54:10.473123+0000","last_peered":"2024-02-10T07:54:10.473123+0000","last_clean":"2024-02-10T07:54:10.473123+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:54:10.473123+0000","last_undegraded":"2024-02-10T07:54:10.473123+0000","last_fullsized":"2024-02-10T07:54:10.473123+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:05.619958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:05.619958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.477932+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":852,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.970848+0000","last_change":"2024-02-10T07:53:23.117713+0000","last_active":"2024-02-10T07:54:10.970848+0000","last_peered":"2024-02-10T07:54:10.970848+0000","last_clean":"2024-02-10T07:54:10.970848+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:54:10.970848+0000","last_undegraded":"2024-02-10T07:54:10.970848+0000","last_fullsized":"2024-02-10T07:54:10.970848+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:23.117597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:23.117597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:27.492528+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":404,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279237+0000","last_change":"2024-02-10T07:49:13.218634+0000","last_active":"2024-02-10T07:54:10.279237+0000","last_peered":"2024-02-10T07:54:10.279237+0000","last_clean":"2024-02-10T07:54:10.279237+0000","last_became_active":"2024-02-10T07:48:02.332960+0000","last_became_peered":"2024-02-10T07:48:02.332960+0000","last_unstale":"2024-02-10T07:54:10.279237+0000","last_undegraded":"2024-02-10T07:54:10.279237+0000","last_fullsized":"2024-02-10T07:54:10.279237+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.218549+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"430'21","reported_seq":897,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279102+0000","last_change":"2024-02-10T07:47:49.553645+0000","last_active":"2024-02-10T07:54:10.279102+0000","last_peered":"2024-02-10T07:54:10.279102+0000","last_clean":"2024-02-10T07:54:10.279102+0000","last_became_active":"2024-02-10T07:47:47.965930+0000","last_became_peered":"2024-02-10T07:47:47.965930+0000","last_unstale":"2024-02-10T07:54:10.279102+0000","last_undegraded":"2024-02-10T07:54:10.279102+0000","last_fullsized":"2024-02-10T07:54:10.279102+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:49.553517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.553517+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876573+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":775,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.686328+0000","last_change":"2024-02-10T07:49:56.718482+0000","last_active":"2024-02-10T07:54:10.686328+0000","last_peered":"2024-02-10T07:54:10.686328+0000","last_clean":"2024-02-10T07:54:10.686328+0000","last_became_active":"2024-02-10T07:49:06.266689+0000","last_became_peered":"2024-02-10T07:49:06.266689+0000","last_unstale":"2024-02-10T07:54:10.686328+0000","last_undegraded":"2024-02-10T07:54:10.686328+0000","last_fullsized":"2024-02-10T07:54:10.686328+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.718382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.718382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651921+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":866,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278691+0000","last_change":"2024-02-10T07:48:18.069773+0000","last_active":"2024-02-10T07:54:10.278691+0000","last_peered":"2024-02-10T07:54:10.278691+0000","last_clean":"2024-02-10T07:54:10.278691+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:54:10.278691+0000","last_undegraded":"2024-02-10T07:54:10.278691+0000","last_fullsized":"2024-02-10T07:54:10.278691+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:18.069641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:18.069641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":392,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279205+0000","last_change":"2024-02-10T07:49:06.115547+0000","last_active":"2024-02-10T07:54:10.279205+0000","last_peered":"2024-02-10T07:54:10.279205+0000","last_clean":"2024-02-10T07:54:10.279205+0000","last_became_active":"2024-02-10T07:48:02.329859+0000","last_became_peered":"2024-02-10T07:48:02.329859+0000","last_unstale":"2024-02-10T07:54:10.279205+0000","last_undegraded":"2024-02-10T07:54:10.279205+0000","last_fullsized":"2024-02-10T07:54:10.279205+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.115466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.473686+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.115466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"430'9","reported_seq":879,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279558+0000","last_change":"2024-02-10T07:48:11.611397+0000","last_active":"2024-02-10T07:54:10.279558+0000","last_peered":"2024-02-10T07:54:10.279558+0000","last_clean":"2024-02-10T07:54:10.279558+0000","last_became_active":"2024-02-10T07:47:47.966093+0000","last_became_peered":"2024-02-10T07:47:47.966093+0000","last_unstale":"2024-02-10T07:54:10.279558+0000","last_undegraded":"2024-02-10T07:54:10.279558+0000","last_fullsized":"2024-02-10T07:54:10.279558+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7","last_scrub_stamp":"2024-02-10T07:48:11.608499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.608499+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876630+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":866,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.280260+0000","last_change":"2024-02-10T07:48:13.621135+0000","last_active":"2024-02-10T07:54:10.280260+0000","last_peered":"2024-02-10T07:54:10.280260+0000","last_clean":"2024-02-10T07:54:10.280260+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:54:10.280260+0000","last_undegraded":"2024-02-10T07:54:10.280260+0000","last_fullsized":"2024-02-10T07:54:10.280260+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:13.620938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.620938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876591+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":872,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.687367+0000","last_change":"2024-02-10T07:49:51.810158+0000","last_active":"2024-02-10T07:54:10.687367+0000","last_peered":"2024-02-10T07:54:10.687367+0000","last_clean":"2024-02-10T07:54:10.687367+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:54:10.687367+0000","last_undegraded":"2024-02-10T07:54:10.687367+0000","last_fullsized":"2024-02-10T07:54:10.687367+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:51.810062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:51.810062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":874,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.473406+0000","last_change":"2024-02-10T07:49:16.313655+0000","last_active":"2024-02-10T07:54:10.473406+0000","last_peered":"2024-02-10T07:54:10.473406+0000","last_clean":"2024-02-10T07:54:10.473406+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:54:10.473406+0000","last_undegraded":"2024-02-10T07:54:10.473406+0000","last_fullsized":"2024-02-10T07:54:10.473406+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:16.313573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.313573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.477692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"430'46","reported_seq":162,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.969777+0000","last_change":"2024-02-10T07:50:43.322322+0000","last_active":"2024-02-10T07:54:10.969777+0000","last_peered":"2024-02-10T07:54:10.969777+0000","last_clean":"2024-02-10T07:54:10.969777+0000","last_became_active":"2024-02-10T07:50:43.300153+0000","last_became_peered":"2024-02-10T07:50:43.300153+0000","last_unstale":"2024-02-10T07:54:10.969777+0000","last_undegraded":"2024-02-10T07:54:10.969777+0000","last_fullsized":"2024-02-10T07:54:10.969777+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:49:00.551605+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.551605+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.506591+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":569,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.972226+0000","last_change":"2024-02-10T07:52:11.043473+0000","last_active":"2024-02-10T07:54:10.972226+0000","last_peered":"2024-02-10T07:54:10.972226+0000","last_clean":"2024-02-10T07:54:10.972226+0000","last_became_active":"2024-02-10T07:49:06.263790+0000","last_became_peered":"2024-02-10T07:49:06.263790+0000","last_unstale":"2024-02-10T07:54:10.972226+0000","last_undegraded":"2024-02-10T07:54:10.972226+0000","last_fullsized":"2024-02-10T07:54:10.972226+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:11.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:11.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":883,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279870+0000","last_change":"2024-02-10T07:50:43.286623+0000","last_active":"2024-02-10T07:54:10.279870+0000","last_peered":"2024-02-10T07:54:10.279870+0000","last_clean":"2024-02-10T07:54:10.279870+0000","last_became_active":"2024-02-10T07:50:43.285201+0000","last_became_peered":"2024-02-10T07:50:43.285201+0000","last_unstale":"2024-02-10T07:54:10.279870+0000","last_undegraded":"2024-02-10T07:54:10.279870+0000","last_fullsized":"2024-02-10T07:54:10.279870+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.680331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.680331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876647+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":886,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279131+0000","last_change":"2024-02-10T07:50:43.282584+0000","last_active":"2024-02-10T07:54:10.279131+0000","last_peered":"2024-02-10T07:54:10.279131+0000","last_clean":"2024-02-10T07:54:10.279131+0000","last_became_active":"2024-02-10T07:50:43.281382+0000","last_became_peered":"2024-02-10T07:50:43.281382+0000","last_unstale":"2024-02-10T07:54:10.279131+0000","last_undegraded":"2024-02-10T07:54:10.279131+0000","last_fullsized":"2024-02-10T07:54:10.279131+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:04.396771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.396771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876865+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"430'24","reported_seq":842,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.686442+0000","last_change":"2024-02-10T07:49:58.618564+0000","last_active":"2024-02-10T07:54:10.686442+0000","last_peered":"2024-02-10T07:54:10.686442+0000","last_clean":"2024-02-10T07:54:10.686442+0000","last_became_active":"2024-02-10T07:49:06.263261+0000","last_became_peered":"2024-02-10T07:49:06.263261+0000","last_unstale":"2024-02-10T07:54:10.686442+0000","last_undegraded":"2024-02-10T07:54:10.686442+0000","last_fullsized":"2024-02-10T07:54:10.686442+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'18","last_scrub_stamp":"2024-02-10T07:49:58.618464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:58.618464+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":898,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279817+0000","last_change":"2024-02-10T07:50:43.307775+0000","last_active":"2024-02-10T07:54:10.279817+0000","last_peered":"2024-02-10T07:54:10.279817+0000","last_clean":"2024-02-10T07:54:10.279817+0000","last_became_active":"2024-02-10T07:50:43.285254+0000","last_became_peered":"2024-02-10T07:50:43.285254+0000","last_unstale":"2024-02-10T07:54:10.279817+0000","last_undegraded":"2024-02-10T07:54:10.279817+0000","last_fullsized":"2024-02-10T07:54:10.279817+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:47:34.074746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:47:34.074746+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"414'9","reported_seq":1180,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.972051+0000","last_change":"2024-02-10T07:51:13.307061+0000","last_active":"2024-02-10T07:54:10.972051+0000","last_peered":"2024-02-10T07:54:10.972051+0000","last_clean":"2024-02-10T07:54:10.972051+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:54:10.972051+0000","last_undegraded":"2024-02-10T07:54:10.972051+0000","last_fullsized":"2024-02-10T07:54:10.972051+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'9","last_scrub_stamp":"2024-02-10T07:51:13.306986+0000","last_deep_scrub":"414'9","last_deep_scrub_stamp":"2024-02-10T07:51:13.306986+0000","last_clean_scrub_stamp":"2024-02-10T07:51:13.306986+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:16.199214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":883,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.686939+0000","last_change":"2024-02-10T07:48:30.384635+0000","last_active":"2024-02-10T07:54:10.686939+0000","last_peered":"2024-02-10T07:54:10.686939+0000","last_clean":"2024-02-10T07:54:10.686939+0000","last_became_active":"2024-02-10T07:47:47.966127+0000","last_became_peered":"2024-02-10T07:47:47.966127+0000","last_unstale":"2024-02-10T07:54:10.686939+0000","last_undegraded":"2024-02-10T07:54:10.686939+0000","last_fullsized":"2024-02-10T07:54:10.686939+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:30.384546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:30.384546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651660+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"430'35","reported_seq":962,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.473000+0000","last_change":"2024-02-10T07:49:06.308245+0000","last_active":"2024-02-10T07:54:10.473000+0000","last_peered":"2024-02-10T07:54:10.473000+0000","last_clean":"2024-02-10T07:54:10.473000+0000","last_became_active":"2024-02-10T07:49:06.261487+0000","last_became_peered":"2024-02-10T07:49:06.261487+0000","last_unstale":"2024-02-10T07:54:10.473000+0000","last_undegraded":"2024-02-10T07:54:10.473000+0000","last_fullsized":"2024-02-10T07:54:10.473000+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:48:12.278982+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-02-10T07:42:17.820959+0000","last_clean_scrub_stamp":"2024-02-10T07:48:12.278982+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.477565+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":869,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.687427+0000","last_change":"2024-02-10T07:49:11.231819+0000","last_active":"2024-02-10T07:54:10.687427+0000","last_peered":"2024-02-10T07:54:10.687427+0000","last_clean":"2024-02-10T07:54:10.687427+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:54:10.687427+0000","last_undegraded":"2024-02-10T07:54:10.687427+0000","last_fullsized":"2024-02-10T07:54:10.687427+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.231704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.231704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651793+0000","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":873,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.473570+0000","last_change":"2024-02-10T07:50:43.289955+0000","last_active":"2024-02-10T07:54:10.473570+0000","last_peered":"2024-02-10T07:54:10.473570+0000","last_clean":"2024-02-10T07:54:10.473570+0000","last_became_active":"2024-02-10T07:50:43.289011+0000","last_became_peered":"2024-02-10T07:50:43.289011+0000","last_unstale":"2024-02-10T07:54:10.473570+0000","last_undegraded":"2024-02-10T07:54:10.473570+0000","last_fullsized":"2024-02-10T07:54:10.473570+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.208451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:06.200009+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.208451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.477661+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":378,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.472734+0000","last_change":"2024-02-10T07:53:21.437757+0000","last_active":"2024-02-10T07:54:10.472734+0000","last_peered":"2024-02-10T07:54:10.472734+0000","last_clean":"2024-02-10T07:54:10.472734+0000","last_became_active":"2024-02-10T07:49:06.258789+0000","last_became_peered":"2024-02-10T07:49:06.258789+0000","last_unstale":"2024-02-10T07:54:10.472734+0000","last_undegraded":"2024-02-10T07:54:10.472734+0000","last_fullsized":"2024-02-10T07:54:10.472734+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:21.437653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:53:21.437653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.318460+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"430'27","reported_seq":939,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.972107+0000","last_change":"2024-02-10T07:52:11.834589+0000","last_active":"2024-02-10T07:54:10.972107+0000","last_peered":"2024-02-10T07:54:10.972107+0000","last_clean":"2024-02-10T07:54:10.972107+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:54:10.972107+0000","last_undegraded":"2024-02-10T07:54:10.972107+0000","last_fullsized":"2024-02-10T07:54:10.972107+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'27","last_scrub_stamp":"2024-02-10T07:52:11.834490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:52:11.834490+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":885,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278605+0000","last_change":"2024-02-10T07:50:43.284049+0000","last_active":"2024-02-10T07:54:10.278605+0000","last_peered":"2024-02-10T07:54:10.278605+0000","last_clean":"2024-02-10T07:54:10.278605+0000","last_became_active":"2024-02-10T07:50:43.283688+0000","last_became_peered":"2024-02-10T07:50:43.283688+0000","last_unstale":"2024-02-10T07:54:10.278605+0000","last_undegraded":"2024-02-10T07:54:10.278605+0000","last_fullsized":"2024-02-10T07:54:10.278605+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.087614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.087614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":890,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.971329+0000","last_change":"2024-02-10T07:52:22.845994+0000","last_active":"2024-02-10T07:54:10.971329+0000","last_peered":"2024-02-10T07:54:10.971329+0000","last_clean":"2024-02-10T07:54:10.971329+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:54:10.971329+0000","last_undegraded":"2024-02-10T07:54:10.971329+0000","last_fullsized":"2024-02-10T07:54:10.971329+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:22.845773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:52:22.845773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":899,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.472873+0000","last_change":"2024-02-10T07:49:06.259100+0000","last_active":"2024-02-10T07:54:10.472873+0000","last_peered":"2024-02-10T07:54:10.472873+0000","last_clean":"2024-02-10T07:54:10.472873+0000","last_became_active":"2024-02-10T07:49:06.258763+0000","last_became_peered":"2024-02-10T07:49:06.258763+0000","last_unstale":"2024-02-10T07:54:10.472873+0000","last_undegraded":"2024-02-10T07:54:10.472873+0000","last_fullsized":"2024-02-10T07:54:10.472873+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:29.188832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:29.188832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.477597+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"430'14","reported_seq":215,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278012+0000","last_change":"2024-02-10T07:53:28.121930+0000","last_active":"2024-02-10T07:54:10.278012+0000","last_peered":"2024-02-10T07:54:10.278012+0000","last_clean":"2024-02-10T07:54:10.278012+0000","last_became_active":"2024-02-10T07:49:06.250577+0000","last_became_peered":"2024-02-10T07:49:06.250577+0000","last_unstale":"2024-02-10T07:54:10.278012+0000","last_undegraded":"2024-02-10T07:54:10.278012+0000","last_fullsized":"2024-02-10T07:54:10.278012+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'14","last_scrub_stamp":"2024-02-10T07:53:28.117356+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:53:28.117356+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:28.261781+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":372,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.472982+0000","last_change":"2024-02-10T07:53:33.873865+0000","last_active":"2024-02-10T07:54:10.472982+0000","last_peered":"2024-02-10T07:54:10.472982+0000","last_clean":"2024-02-10T07:54:10.472982+0000","last_became_active":"2024-02-10T07:49:06.259341+0000","last_became_peered":"2024-02-10T07:49:06.259341+0000","last_unstale":"2024-02-10T07:54:10.472982+0000","last_undegraded":"2024-02-10T07:54:10.472982+0000","last_fullsized":"2024-02-10T07:54:10.472982+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:33.873724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:53:33.873724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:51.425277+0000","scrub_duration":324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":867,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278631+0000","last_change":"2024-02-10T07:48:24.171978+0000","last_active":"2024-02-10T07:54:10.278631+0000","last_peered":"2024-02-10T07:54:10.278631+0000","last_clean":"2024-02-10T07:54:10.278631+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:54:10.278631+0000","last_undegraded":"2024-02-10T07:54:10.278631+0000","last_fullsized":"2024-02-10T07:54:10.278631+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:24.171859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:24.171859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":824,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.972190+0000","last_change":"2024-02-10T07:52:27.104889+0000","last_active":"2024-02-10T07:54:10.972190+0000","last_peered":"2024-02-10T07:54:10.972190+0000","last_clean":"2024-02-10T07:54:10.972190+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:54:10.972190+0000","last_undegraded":"2024-02-10T07:54:10.972190+0000","last_fullsized":"2024-02-10T07:54:10.972190+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:27.104803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:52:27.104803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"430'27","reported_seq":845,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279186+0000","last_change":"2024-02-10T07:49:53.245490+0000","last_active":"2024-02-10T07:54:10.279186+0000","last_peered":"2024-02-10T07:54:10.279186+0000","last_clean":"2024-02-10T07:54:10.279186+0000","last_became_active":"2024-02-10T07:49:06.239701+0000","last_became_peered":"2024-02-10T07:49:06.239701+0000","last_unstale":"2024-02-10T07:54:10.279186+0000","last_undegraded":"2024-02-10T07:54:10.279186+0000","last_fullsized":"2024-02-10T07:54:10.279186+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:53.245410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:53.245410+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":864,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.472892+0000","last_change":"2024-02-10T07:49:52.530010+0000","last_active":"2024-02-10T07:54:10.472892+0000","last_peered":"2024-02-10T07:54:10.472892+0000","last_clean":"2024-02-10T07:54:10.472892+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:54:10.472892+0000","last_undegraded":"2024-02-10T07:54:10.472892+0000","last_fullsized":"2024-02-10T07:54:10.472892+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.529908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:59.179342+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.529908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.477864+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":372,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.472939+0000","last_change":"2024-02-10T07:53:19.493460+0000","last_active":"2024-02-10T07:54:10.472939+0000","last_peered":"2024-02-10T07:54:10.472939+0000","last_clean":"2024-02-10T07:54:10.472939+0000","last_became_active":"2024-02-10T07:49:06.265267+0000","last_became_peered":"2024-02-10T07:49:06.265267+0000","last_unstale":"2024-02-10T07:54:10.472939+0000","last_undegraded":"2024-02-10T07:54:10.472939+0000","last_fullsized":"2024-02-10T07:54:10.472939+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:19.492989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:19.492989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:32.822823+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":884,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278857+0000","last_change":"2024-02-10T07:50:00.265308+0000","last_active":"2024-02-10T07:54:10.278857+0000","last_peered":"2024-02-10T07:54:10.278857+0000","last_clean":"2024-02-10T07:54:10.278857+0000","last_became_active":"2024-02-10T07:47:47.966472+0000","last_became_peered":"2024-02-10T07:47:47.966472+0000","last_unstale":"2024-02-10T07:54:10.278857+0000","last_undegraded":"2024-02-10T07:54:10.278857+0000","last_fullsized":"2024-02-10T07:54:10.278857+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.265208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.265208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"430'25","reported_seq":717,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.687006+0000","last_change":"2024-02-10T07:50:43.310533+0000","last_active":"2024-02-10T07:54:10.687006+0000","last_peered":"2024-02-10T07:54:10.687006+0000","last_clean":"2024-02-10T07:54:10.687006+0000","last_became_active":"2024-02-10T07:50:43.296447+0000","last_became_peered":"2024-02-10T07:50:43.296447+0000","last_unstale":"2024-02-10T07:54:10.687006+0000","last_undegraded":"2024-02-10T07:54:10.687006+0000","last_fullsized":"2024-02-10T07:54:10.687006+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:09.116615+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-02-10T07:39:32.934269+0000","last_clean_scrub_stamp":"2024-02-10T07:49:09.116615+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651728+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":560,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.472776+0000","last_change":"2024-02-10T07:53:35.716416+0000","last_active":"2024-02-10T07:54:10.472776+0000","last_peered":"2024-02-10T07:54:10.472776+0000","last_clean":"2024-02-10T07:54:10.472776+0000","last_became_active":"2024-02-10T07:49:06.260569+0000","last_became_peered":"2024-02-10T07:49:06.260569+0000","last_unstale":"2024-02-10T07:54:10.472776+0000","last_undegraded":"2024-02-10T07:54:10.472776+0000","last_fullsized":"2024-02-10T07:54:10.472776+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:35.715987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:53:35.715987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:50.372195+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":897,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.686365+0000","last_change":"2024-02-10T07:50:02.685941+0000","last_active":"2024-02-10T07:54:10.686365+0000","last_peered":"2024-02-10T07:54:10.686365+0000","last_clean":"2024-02-10T07:54:10.686365+0000","last_became_active":"2024-02-10T07:49:06.252380+0000","last_became_peered":"2024-02-10T07:49:06.252380+0000","last_unstale":"2024-02-10T07:54:10.686365+0000","last_undegraded":"2024-02-10T07:54:10.686365+0000","last_fullsized":"2024-02-10T07:54:10.686365+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_clean_scrub_stamp":"2024-02-10T07:50:02.685831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":875,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279288+0000","last_change":"2024-02-10T07:50:00.345087+0000","last_active":"2024-02-10T07:54:10.279288+0000","last_peered":"2024-02-10T07:54:10.279288+0000","last_clean":"2024-02-10T07:54:10.279288+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:54:10.279288+0000","last_undegraded":"2024-02-10T07:54:10.279288+0000","last_fullsized":"2024-02-10T07:54:10.279288+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.344993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.344993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876957+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"430'32","reported_seq":947,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279387+0000","last_change":"2024-02-10T07:47:43.494877+0000","last_active":"2024-02-10T07:54:10.279387+0000","last_peered":"2024-02-10T07:54:10.279387+0000","last_clean":"2024-02-10T07:54:10.279387+0000","last_became_active":"2024-02-10T07:46:10.721336+0000","last_became_peered":"2024-02-10T07:46:10.721336+0000","last_unstale":"2024-02-10T07:54:10.279387+0000","last_undegraded":"2024-02-10T07:54:10.279387+0000","last_fullsized":"2024-02-10T07:54:10.279387+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:47:43.494350+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.494350+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876554+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":878,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.473185+0000","last_change":"2024-02-10T07:53:42.705767+0000","last_active":"2024-02-10T07:54:10.473185+0000","last_peered":"2024-02-10T07:54:10.473185+0000","last_clean":"2024-02-10T07:54:10.473185+0000","last_became_active":"2024-02-10T07:49:06.261570+0000","last_became_peered":"2024-02-10T07:49:06.261570+0000","last_unstale":"2024-02-10T07:54:10.473185+0000","last_undegraded":"2024-02-10T07:54:10.473185+0000","last_fullsized":"2024-02-10T07:54:10.473185+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:42.705250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:53:42.705250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:05.715285+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"356'11","reported_seq":380,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.686461+0000","last_change":"2024-02-10T07:48:25.339974+0000","last_active":"2024-02-10T07:54:10.686461+0000","last_peered":"2024-02-10T07:54:10.686461+0000","last_clean":"2024-02-10T07:54:10.686461+0000","last_became_active":"2024-02-10T07:48:02.371888+0000","last_became_peered":"2024-02-10T07:48:02.371888+0000","last_unstale":"2024-02-10T07:54:10.686461+0000","last_undegraded":"2024-02-10T07:54:10.686461+0000","last_fullsized":"2024-02-10T07:54:10.686461+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'11","last_scrub_stamp":"2024-02-10T07:48:25.339855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:25.339855+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651640+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":895,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279367+0000","last_change":"2024-02-10T07:50:43.283263+0000","last_active":"2024-02-10T07:54:10.279367+0000","last_peered":"2024-02-10T07:54:10.279367+0000","last_clean":"2024-02-10T07:54:10.279367+0000","last_became_active":"2024-02-10T07:50:43.282925+0000","last_became_peered":"2024-02-10T07:50:43.282925+0000","last_unstale":"2024-02-10T07:54:10.279367+0000","last_undegraded":"2024-02-10T07:54:10.279367+0000","last_fullsized":"2024-02-10T07:54:10.279367+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.335455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.335455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"430'35","reported_seq":511,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.686481+0000","last_change":"2024-02-10T07:49:12.150442+0000","last_active":"2024-02-10T07:54:10.686481+0000","last_peered":"2024-02-10T07:54:10.686481+0000","last_clean":"2024-02-10T07:54:10.686481+0000","last_became_active":"2024-02-10T07:48:02.329187+0000","last_became_peered":"2024-02-10T07:48:02.329187+0000","last_unstale":"2024-02-10T07:54:10.686481+0000","last_undegraded":"2024-02-10T07:54:10.686481+0000","last_fullsized":"2024-02-10T07:54:10.686481+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:49:12.150298+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:49:12.150298+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651745+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":876,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279417+0000","last_change":"2024-02-10T07:50:43.285390+0000","last_active":"2024-02-10T07:54:10.279417+0000","last_peered":"2024-02-10T07:54:10.279417+0000","last_clean":"2024-02-10T07:54:10.279417+0000","last_became_active":"2024-02-10T07:50:43.284846+0000","last_became_peered":"2024-02-10T07:50:43.284846+0000","last_unstale":"2024-02-10T07:54:10.279417+0000","last_undegraded":"2024-02-10T07:54:10.279417+0000","last_fullsized":"2024-02-10T07:54:10.279417+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:01.513972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:01.513972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876849+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"405'15","reported_seq":536,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279802+0000","last_change":"2024-02-10T07:50:43.310228+0000","last_active":"2024-02-10T07:54:10.279802+0000","last_peered":"2024-02-10T07:54:10.279802+0000","last_clean":"2024-02-10T07:54:10.279802+0000","last_became_active":"2024-02-10T07:50:43.285213+0000","last_became_peered":"2024-02-10T07:50:43.285213+0000","last_unstale":"2024-02-10T07:54:10.279802+0000","last_undegraded":"2024-02-10T07:54:10.279802+0000","last_fullsized":"2024-02-10T07:54:10.279802+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'13","last_scrub_stamp":"2024-02-10T07:49:00.366610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.366610+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":890,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.472764+0000","last_change":"2024-02-10T07:50:07.501014+0000","last_active":"2024-02-10T07:54:10.472764+0000","last_peered":"2024-02-10T07:54:10.472764+0000","last_clean":"2024-02-10T07:54:10.472764+0000","last_became_active":"2024-02-10T07:49:06.257043+0000","last_became_peered":"2024-02-10T07:49:06.257043+0000","last_unstale":"2024-02-10T07:54:10.472764+0000","last_undegraded":"2024-02-10T07:54:10.472764+0000","last_fullsized":"2024-02-10T07:54:10.472764+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.500890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.478008+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"430'21","reported_seq":637,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279929+0000","last_change":"2024-02-10T07:49:06.277154+0000","last_active":"2024-02-10T07:54:10.279929+0000","last_peered":"2024-02-10T07:54:10.279929+0000","last_clean":"2024-02-10T07:54:10.279929+0000","last_became_active":"2024-02-10T07:49:06.255239+0000","last_became_peered":"2024-02-10T07:49:06.255239+0000","last_unstale":"2024-02-10T07:54:10.279929+0000","last_undegraded":"2024-02-10T07:54:10.279929+0000","last_fullsized":"2024-02-10T07:54:10.279929+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:39.610017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:39.610017+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":128,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.969949+0000","last_change":"2024-02-10T07:50:43.282730+0000","last_active":"2024-02-10T07:54:10.969949+0000","last_peered":"2024-02-10T07:54:10.969949+0000","last_clean":"2024-02-10T07:54:10.969949+0000","last_became_active":"2024-02-10T07:50:43.281549+0000","last_became_peered":"2024-02-10T07:50:43.281549+0000","last_unstale":"2024-02-10T07:54:10.969949+0000","last_undegraded":"2024-02-10T07:54:10.969949+0000","last_fullsized":"2024-02-10T07:54:10.969949+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:44.324046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:49:44.324046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.506674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":231,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.971895+0000","last_change":"2024-02-10T07:52:34.860375+0000","last_active":"2024-02-10T07:54:10.971895+0000","last_peered":"2024-02-10T07:54:10.971895+0000","last_clean":"2024-02-10T07:54:10.971895+0000","last_became_active":"2024-02-10T07:49:06.262933+0000","last_became_peered":"2024-02-10T07:49:06.262933+0000","last_unstale":"2024-02-10T07:54:10.971895+0000","last_undegraded":"2024-02-10T07:54:10.971895+0000","last_fullsized":"2024-02-10T07:54:10.971895+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:34.860273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:52:34.860273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":883,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279038+0000","last_change":"2024-02-10T07:49:59.278666+0000","last_active":"2024-02-10T07:54:10.279038+0000","last_peered":"2024-02-10T07:54:10.279038+0000","last_clean":"2024-02-10T07:54:10.279038+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:54:10.279038+0000","last_undegraded":"2024-02-10T07:54:10.279038+0000","last_fullsized":"2024-02-10T07:54:10.279038+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.278574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.278574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"430'25","reported_seq":952,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.472857+0000","last_change":"2024-02-10T07:53:44.569050+0000","last_active":"2024-02-10T07:54:10.472857+0000","last_peered":"2024-02-10T07:54:10.472857+0000","last_clean":"2024-02-10T07:54:10.472857+0000","last_became_active":"2024-02-10T07:49:06.260962+0000","last_became_peered":"2024-02-10T07:49:06.260962+0000","last_unstale":"2024-02-10T07:54:10.472857+0000","last_undegraded":"2024-02-10T07:54:10.472857+0000","last_fullsized":"2024-02-10T07:54:10.472857+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'25","last_scrub_stamp":"2024-02-10T07:53:44.568910+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:53:44.568910+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:59.428046+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":863,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278759+0000","last_change":"2024-02-10T07:48:58.511630+0000","last_active":"2024-02-10T07:54:10.278759+0000","last_peered":"2024-02-10T07:54:10.278759+0000","last_clean":"2024-02-10T07:54:10.278759+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:54:10.278759+0000","last_undegraded":"2024-02-10T07:54:10.278759+0000","last_fullsized":"2024-02-10T07:54:10.278759+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:58.511251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.511251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":203,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278701+0000","last_change":"2024-02-10T07:53:42.214117+0000","last_active":"2024-02-10T07:54:10.278701+0000","last_peered":"2024-02-10T07:54:10.278701+0000","last_clean":"2024-02-10T07:54:10.278701+0000","last_became_active":"2024-02-10T07:49:06.250832+0000","last_became_peered":"2024-02-10T07:49:06.250832+0000","last_unstale":"2024-02-10T07:54:10.278701+0000","last_undegraded":"2024-02-10T07:54:10.278701+0000","last_fullsized":"2024-02-10T07:54:10.278701+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:42.213981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:42.213981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:57.611769+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":915,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.971058+0000","last_change":"2024-02-10T07:52:16.809094+0000","last_active":"2024-02-10T07:54:10.971058+0000","last_peered":"2024-02-10T07:54:10.971058+0000","last_clean":"2024-02-10T07:54:10.971058+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:54:10.971058+0000","last_undegraded":"2024-02-10T07:54:10.971058+0000","last_fullsized":"2024-02-10T07:54:10.971058+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:16.808978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:52:16.808978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"430'18","reported_seq":220,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278636+0000","last_change":"2024-02-10T07:53:29.079571+0000","last_active":"2024-02-10T07:54:10.278636+0000","last_peered":"2024-02-10T07:54:10.278636+0000","last_clean":"2024-02-10T07:54:10.278636+0000","last_became_active":"2024-02-10T07:49:06.258281+0000","last_became_peered":"2024-02-10T07:49:06.258281+0000","last_unstale":"2024-02-10T07:54:10.278636+0000","last_undegraded":"2024-02-10T07:54:10.278636+0000","last_fullsized":"2024-02-10T07:54:10.278636+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'18","last_scrub_stamp":"2024-02-10T07:53:29.076952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:53:29.076952+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:30.380886+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":404,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.971343+0000","last_change":"2024-02-10T07:52:21.875347+0000","last_active":"2024-02-10T07:54:10.971343+0000","last_peered":"2024-02-10T07:54:10.971343+0000","last_clean":"2024-02-10T07:54:10.971343+0000","last_became_active":"2024-02-10T07:48:02.366002+0000","last_became_peered":"2024-02-10T07:48:02.366002+0000","last_unstale":"2024-02-10T07:54:10.971343+0000","last_undegraded":"2024-02-10T07:54:10.971343+0000","last_fullsized":"2024-02-10T07:54:10.971343+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:21.875258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:21.875258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"245'4","reported_seq":897,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279750+0000","last_change":"2024-02-10T07:49:08.311056+0000","last_active":"2024-02-10T07:54:10.279750+0000","last_peered":"2024-02-10T07:54:10.279750+0000","last_clean":"2024-02-10T07:54:10.279750+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:54:10.279750+0000","last_undegraded":"2024-02-10T07:54:10.279750+0000","last_fullsized":"2024-02-10T07:54:10.279750+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'4","last_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_deep_scrub":"245'4","last_deep_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.310917+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"430'46","reported_seq":641,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278540+0000","last_change":"2024-02-10T07:48:11.967227+0000","last_active":"2024-02-10T07:54:10.278540+0000","last_peered":"2024-02-10T07:54:10.278540+0000","last_clean":"2024-02-10T07:54:10.278540+0000","last_became_active":"2024-02-10T07:48:02.330130+0000","last_became_peered":"2024-02-10T07:48:02.330130+0000","last_unstale":"2024-02-10T07:54:10.278540+0000","last_undegraded":"2024-02-10T07:54:10.278540+0000","last_fullsized":"2024-02-10T07:54:10.278540+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:48:11.966957+0000","last_deep_scrub":"246'22","last_deep_scrub_stamp":"2024-02-10T07:41:26.771112+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.966957+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":1153,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279233+0000","last_change":"2024-02-10T07:48:53.423840+0000","last_active":"2024-02-10T07:54:10.279233+0000","last_peered":"2024-02-10T07:54:10.279233+0000","last_clean":"2024-02-10T07:54:10.279233+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:54:10.279233+0000","last_undegraded":"2024-02-10T07:54:10.279233+0000","last_fullsized":"2024-02-10T07:54:10.279233+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:48:53.423613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:53.423613+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":709,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278384+0000","last_change":"2024-02-10T07:53:32.243227+0000","last_active":"2024-02-10T07:54:10.278384+0000","last_peered":"2024-02-10T07:54:10.278384+0000","last_clean":"2024-02-10T07:54:10.278384+0000","last_became_active":"2024-02-10T07:47:47.966183+0000","last_became_peered":"2024-02-10T07:47:47.966183+0000","last_unstale":"2024-02-10T07:54:10.278384+0000","last_undegraded":"2024-02-10T07:54:10.278384+0000","last_fullsized":"2024-02-10T07:54:10.278384+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:32.242307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:53:32.242307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:47.454633+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":875,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.281127+0000","last_change":"2024-02-10T07:48:56.416159+0000","last_active":"2024-02-10T07:54:10.281127+0000","last_peered":"2024-02-10T07:54:10.281127+0000","last_clean":"2024-02-10T07:54:10.281127+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:54:10.281127+0000","last_undegraded":"2024-02-10T07:54:10.281127+0000","last_fullsized":"2024-02-10T07:54:10.281127+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.416037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.416037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"430'29","reported_seq":991,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279005+0000","last_change":"2024-02-10T07:49:43.339027+0000","last_active":"2024-02-10T07:54:10.279005+0000","last_peered":"2024-02-10T07:54:10.279005+0000","last_clean":"2024-02-10T07:54:10.279005+0000","last_became_active":"2024-02-10T07:49:06.243569+0000","last_became_peered":"2024-02-10T07:49:06.243569+0000","last_unstale":"2024-02-10T07:54:10.279005+0000","last_undegraded":"2024-02-10T07:54:10.279005+0000","last_fullsized":"2024-02-10T07:54:10.279005+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-02-10T07:49:43.338928+0000","last_deep_scrub":"245'17","last_deep_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_clean_scrub_stamp":"2024-02-10T07:49:43.338928+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":128,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.969462+0000","last_change":"2024-02-10T07:50:43.285105+0000","last_active":"2024-02-10T07:54:10.969462+0000","last_peered":"2024-02-10T07:54:10.969462+0000","last_clean":"2024-02-10T07:54:10.969462+0000","last_became_active":"2024-02-10T07:50:43.284749+0000","last_became_peered":"2024-02-10T07:50:43.284749+0000","last_unstale":"2024-02-10T07:54:10.969462+0000","last_undegraded":"2024-02-10T07:54:10.969462+0000","last_fullsized":"2024-02-10T07:54:10.969462+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:04.308807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.308807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.506574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":897,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278330+0000","last_change":"2024-02-10T07:50:43.281637+0000","last_active":"2024-02-10T07:54:10.278330+0000","last_peered":"2024-02-10T07:54:10.278330+0000","last_clean":"2024-02-10T07:54:10.278330+0000","last_became_active":"2024-02-10T07:50:43.281262+0000","last_became_peered":"2024-02-10T07:50:43.281262+0000","last_unstale":"2024-02-10T07:54:10.278330+0000","last_undegraded":"2024-02-10T07:54:10.278330+0000","last_fullsized":"2024-02-10T07:54:10.278330+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.361139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":128,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.969611+0000","last_change":"2024-02-10T07:50:43.281022+0000","last_active":"2024-02-10T07:54:10.969611+0000","last_peered":"2024-02-10T07:54:10.969611+0000","last_clean":"2024-02-10T07:54:10.969611+0000","last_became_active":"2024-02-10T07:50:43.280617+0000","last_became_peered":"2024-02-10T07:50:43.280617+0000","last_unstale":"2024-02-10T07:54:10.969611+0000","last_undegraded":"2024-02-10T07:54:10.969611+0000","last_fullsized":"2024-02-10T07:54:10.969611+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:21.172601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:21.172601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.506543+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":892,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279844+0000","last_change":"2024-02-10T07:50:07.252577+0000","last_active":"2024-02-10T07:54:10.279844+0000","last_peered":"2024-02-10T07:54:10.279844+0000","last_clean":"2024-02-10T07:54:10.279844+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:54:10.279844+0000","last_undegraded":"2024-02-10T07:54:10.279844+0000","last_fullsized":"2024-02-10T07:54:10.279844+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.252458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.252458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"430'51","reported_seq":974,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.971770+0000","last_change":"2024-02-10T07:51:10.348915+0000","last_active":"2024-02-10T07:54:10.971770+0000","last_peered":"2024-02-10T07:54:10.971770+0000","last_clean":"2024-02-10T07:54:10.971770+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:54:10.971770+0000","last_undegraded":"2024-02-10T07:54:10.971770+0000","last_fullsized":"2024-02-10T07:54:10.971770+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'35","last_scrub_stamp":"2024-02-10T07:51:10.348752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:51:10.348752+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":886,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.473141+0000","last_change":"2024-02-10T07:50:43.284817+0000","last_active":"2024-02-10T07:54:10.473141+0000","last_peered":"2024-02-10T07:54:10.473141+0000","last_clean":"2024-02-10T07:54:10.473141+0000","last_became_active":"2024-02-10T07:50:43.284289+0000","last_became_peered":"2024-02-10T07:50:43.284289+0000","last_unstale":"2024-02-10T07:54:10.473141+0000","last_undegraded":"2024-02-10T07:54:10.473141+0000","last_fullsized":"2024-02-10T07:54:10.473141+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:00.336175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.336175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.477630+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":128,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.969633+0000","last_change":"2024-02-10T07:50:43.295279+0000","last_active":"2024-02-10T07:54:10.969633+0000","last_peered":"2024-02-10T07:54:10.969633+0000","last_clean":"2024-02-10T07:54:10.969633+0000","last_became_active":"2024-02-10T07:50:43.294972+0000","last_became_peered":"2024-02-10T07:50:43.294972+0000","last_unstale":"2024-02-10T07:54:10.969633+0000","last_undegraded":"2024-02-10T07:54:10.969633+0000","last_fullsized":"2024-02-10T07:54:10.969633+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:28.822700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:28.822700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.506490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":203,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278539+0000","last_change":"2024-02-10T07:53:35.090294+0000","last_active":"2024-02-10T07:54:10.278539+0000","last_peered":"2024-02-10T07:54:10.278539+0000","last_clean":"2024-02-10T07:54:10.278539+0000","last_became_active":"2024-02-10T07:49:06.249431+0000","last_became_peered":"2024-02-10T07:49:06.249431+0000","last_unstale":"2024-02-10T07:54:10.278539+0000","last_undegraded":"2024-02-10T07:54:10.278539+0000","last_fullsized":"2024-02-10T07:54:10.278539+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:35.090175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:53:35.090175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:50.135166+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"430'21","reported_seq":905,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.473457+0000","last_change":"2024-02-10T07:48:59.304277+0000","last_active":"2024-02-10T07:54:10.473457+0000","last_peered":"2024-02-10T07:54:10.473457+0000","last_clean":"2024-02-10T07:54:10.473457+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:54:10.473457+0000","last_undegraded":"2024-02-10T07:54:10.473457+0000","last_fullsized":"2024-02-10T07:54:10.473457+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:48:59.304199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.304199+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.477729+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":203,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278710+0000","last_change":"2024-02-10T07:53:41.173096+0000","last_active":"2024-02-10T07:54:10.278710+0000","last_peered":"2024-02-10T07:54:10.278710+0000","last_clean":"2024-02-10T07:54:10.278710+0000","last_became_active":"2024-02-10T07:49:06.238367+0000","last_became_peered":"2024-02-10T07:49:06.238367+0000","last_unstale":"2024-02-10T07:54:10.278710+0000","last_undegraded":"2024-02-10T07:54:10.278710+0000","last_fullsized":"2024-02-10T07:54:10.278710+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:41.172967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:53:41.172967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:48.222331+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":295,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.687095+0000","last_change":"2024-02-10T07:49:10.087939+0000","last_active":"2024-02-10T07:54:10.687095+0000","last_peered":"2024-02-10T07:54:10.687095+0000","last_clean":"2024-02-10T07:54:10.687095+0000","last_became_active":"2024-02-10T07:48:02.328316+0000","last_became_peered":"2024-02-10T07:48:02.328316+0000","last_unstale":"2024-02-10T07:54:10.687095+0000","last_undegraded":"2024-02-10T07:54:10.687095+0000","last_fullsized":"2024-02-10T07:54:10.687095+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:49:10.087795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:49:10.087795+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651709+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":868,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279245+0000","last_change":"2024-02-10T07:49:19.198419+0000","last_active":"2024-02-10T07:54:10.279245+0000","last_peered":"2024-02-10T07:54:10.279245+0000","last_clean":"2024-02-10T07:54:10.279245+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:54:10.279245+0000","last_undegraded":"2024-02-10T07:54:10.279245+0000","last_fullsized":"2024-02-10T07:54:10.279245+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:19.198312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:04.220328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.198312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876728+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"430'59","reported_seq":1341,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.687269+0000","last_change":"2024-02-10T07:50:43.327112+0000","last_active":"2024-02-10T07:54:10.687269+0000","last_peered":"2024-02-10T07:54:10.687269+0000","last_clean":"2024-02-10T07:54:10.687269+0000","last_became_active":"2024-02-10T07:50:43.297262+0000","last_became_peered":"2024-02-10T07:50:43.297262+0000","last_unstale":"2024-02-10T07:54:10.687269+0000","last_undegraded":"2024-02-10T07:54:10.687269+0000","last_fullsized":"2024-02-10T07:54:10.687269+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'46","last_scrub_stamp":"2024-02-10T07:49:55.714323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:55.714323+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":876,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.473311+0000","last_change":"2024-02-10T07:49:17.357664+0000","last_active":"2024-02-10T07:54:10.473311+0000","last_peered":"2024-02-10T07:54:10.473311+0000","last_clean":"2024-02-10T07:54:10.473311+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:54:10.473311+0000","last_undegraded":"2024-02-10T07:54:10.473311+0000","last_fullsized":"2024-02-10T07:54:10.473311+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.357533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:22.786949+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.357533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.477760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":865,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.280017+0000","last_change":"2024-02-10T07:47:50.081222+0000","last_active":"2024-02-10T07:54:10.280017+0000","last_peered":"2024-02-10T07:54:10.280017+0000","last_clean":"2024-02-10T07:54:10.280017+0000","last_became_active":"2024-02-10T07:47:48.149256+0000","last_became_peered":"2024-02-10T07:47:48.149256+0000","last_unstale":"2024-02-10T07:54:10.280017+0000","last_undegraded":"2024-02-10T07:54:10.280017+0000","last_fullsized":"2024-02-10T07:54:10.280017+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.080714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:42.730875+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.080714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":895,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.686979+0000","last_change":"2024-02-10T07:50:43.288780+0000","last_active":"2024-02-10T07:54:10.686979+0000","last_peered":"2024-02-10T07:54:10.686979+0000","last_clean":"2024-02-10T07:54:10.686979+0000","last_became_active":"2024-02-10T07:50:43.288388+0000","last_became_peered":"2024-02-10T07:50:43.288388+0000","last_unstale":"2024-02-10T07:54:10.686979+0000","last_undegraded":"2024-02-10T07:54:10.686979+0000","last_fullsized":"2024-02-10T07:54:10.686979+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:49.886354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:49.886354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":128,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.969468+0000","last_change":"2024-02-10T07:50:43.287356+0000","last_active":"2024-02-10T07:54:10.969468+0000","last_peered":"2024-02-10T07:54:10.969468+0000","last_clean":"2024-02-10T07:54:10.969468+0000","last_became_active":"2024-02-10T07:50:43.287043+0000","last_became_peered":"2024-02-10T07:50:43.287043+0000","last_unstale":"2024-02-10T07:54:10.969468+0000","last_undegraded":"2024-02-10T07:54:10.969468+0000","last_fullsized":"2024-02-10T07:54:10.969468+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:15.234112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.234112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.506527+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"430'21","reported_seq":235,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279504+0000","last_change":"2024-02-10T07:49:04.340445+0000","last_active":"2024-02-10T07:54:10.279504+0000","last_peered":"2024-02-10T07:54:10.279504+0000","last_clean":"2024-02-10T07:54:10.279504+0000","last_became_active":"2024-02-10T07:48:35.057662+0000","last_became_peered":"2024-02-10T07:48:35.057662+0000","last_unstale":"2024-02-10T07:54:10.279504+0000","last_undegraded":"2024-02-10T07:54:10.279504+0000","last_fullsized":"2024-02-10T07:54:10.279504+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'17","last_scrub_stamp":"2024-02-10T07:49:04.340366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.340366+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":128,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.969902+0000","last_change":"2024-02-10T07:50:43.280896+0000","last_active":"2024-02-10T07:54:10.969902+0000","last_peered":"2024-02-10T07:54:10.969902+0000","last_clean":"2024-02-10T07:54:10.969902+0000","last_became_active":"2024-02-10T07:50:43.279125+0000","last_became_peered":"2024-02-10T07:50:43.279125+0000","last_unstale":"2024-02-10T07:54:10.969902+0000","last_undegraded":"2024-02-10T07:54:10.969902+0000","last_fullsized":"2024-02-10T07:54:10.969902+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:50.232544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.232544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.506624+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":284,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278779+0000","last_change":"2024-02-10T07:48:02.330253+0000","last_active":"2024-02-10T07:54:10.278779+0000","last_peered":"2024-02-10T07:54:10.278779+0000","last_clean":"2024-02-10T07:54:10.278779+0000","last_became_active":"2024-02-10T07:48:02.329687+0000","last_became_peered":"2024-02-10T07:48:02.329687+0000","last_unstale":"2024-02-10T07:54:10.278779+0000","last_undegraded":"2024-02-10T07:54:10.278779+0000","last_fullsized":"2024-02-10T07:54:10.278779+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:47:49.750508+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.750508+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1122,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278454+0000","last_change":"2024-02-10T07:48:15.033077+0000","last_active":"2024-02-10T07:54:10.278454+0000","last_peered":"2024-02-10T07:54:10.278454+0000","last_clean":"2024-02-10T07:54:10.278454+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:54:10.278454+0000","last_undegraded":"2024-02-10T07:54:10.278454+0000","last_fullsized":"2024-02-10T07:54:10.278454+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:48:15.032944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.032944+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":513,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.971989+0000","last_change":"2024-02-10T07:50:58.025300+0000","last_active":"2024-02-10T07:54:10.971989+0000","last_peered":"2024-02-10T07:54:10.971989+0000","last_clean":"2024-02-10T07:54:10.971989+0000","last_became_active":"2024-02-10T07:49:06.256223+0000","last_became_peered":"2024-02-10T07:49:06.256223+0000","last_unstale":"2024-02-10T07:54:10.971989+0000","last_undegraded":"2024-02-10T07:54:10.971989+0000","last_fullsized":"2024-02-10T07:54:10.971989+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:58.025193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:58.025193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"430'48","reported_seq":1230,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.971694+0000","last_change":"2024-02-10T07:52:03.839258+0000","last_active":"2024-02-10T07:54:10.971694+0000","last_peered":"2024-02-10T07:54:10.971694+0000","last_clean":"2024-02-10T07:54:10.971694+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:54:10.971694+0000","last_undegraded":"2024-02-10T07:54:10.971694+0000","last_fullsized":"2024-02-10T07:54:10.971694+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'48","last_scrub_stamp":"2024-02-10T07:52:03.839174+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:52:03.839174+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:16.199329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":897,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279865+0000","last_change":"2024-02-10T07:50:43.285613+0000","last_active":"2024-02-10T07:54:10.279865+0000","last_peered":"2024-02-10T07:54:10.279865+0000","last_clean":"2024-02-10T07:54:10.279865+0000","last_became_active":"2024-02-10T07:50:43.284708+0000","last_became_peered":"2024-02-10T07:50:43.284708+0000","last_unstale":"2024-02-10T07:54:10.279865+0000","last_undegraded":"2024-02-10T07:54:10.279865+0000","last_fullsized":"2024-02-10T07:54:10.279865+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:13.369607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.736901+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.369607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":669,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.686548+0000","last_change":"2024-02-10T07:50:07.589020+0000","last_active":"2024-02-10T07:54:10.686548+0000","last_peered":"2024-02-10T07:54:10.686548+0000","last_clean":"2024-02-10T07:54:10.686548+0000","last_became_active":"2024-02-10T07:49:06.270457+0000","last_became_peered":"2024-02-10T07:49:06.270457+0000","last_unstale":"2024-02-10T07:54:10.686548+0000","last_undegraded":"2024-02-10T07:54:10.686548+0000","last_fullsized":"2024-02-10T07:54:10.686548+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.588888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.588888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651936+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":891,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.472786+0000","last_change":"2024-02-10T07:53:41.577827+0000","last_active":"2024-02-10T07:54:10.472786+0000","last_peered":"2024-02-10T07:54:10.472786+0000","last_clean":"2024-02-10T07:54:10.472786+0000","last_became_active":"2024-02-10T07:49:06.266625+0000","last_became_peered":"2024-02-10T07:49:06.266625+0000","last_unstale":"2024-02-10T07:54:10.472786+0000","last_undegraded":"2024-02-10T07:54:10.472786+0000","last_fullsized":"2024-02-10T07:54:10.472786+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:41.577586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_clean_scrub_stamp":"2024-02-10T07:53:41.577586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:59.035911+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2232,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:12.326356+0000","last_change":"2024-02-10T07:50:04.159154+0000","last_active":"2024-02-10T07:54:12.326356+0000","last_peered":"2024-02-10T07:54:12.326356+0000","last_clean":"2024-02-10T07:54:12.326356+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:54:12.326356+0000","last_undegraded":"2024-02-10T07:54:12.326356+0000","last_fullsized":"2024-02-10T07:54:12.326356+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:50:04.159046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.159046+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1335,"num_read_kb":1330,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"430'42","reported_seq":972,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.473380+0000","last_change":"2024-02-10T07:50:43.324808+0000","last_active":"2024-02-10T07:54:10.473380+0000","last_peered":"2024-02-10T07:54:10.473380+0000","last_clean":"2024-02-10T07:54:10.473380+0000","last_became_active":"2024-02-10T07:50:43.291771+0000","last_became_peered":"2024-02-10T07:50:43.291771+0000","last_unstale":"2024-02-10T07:54:10.473380+0000","last_undegraded":"2024-02-10T07:54:10.473380+0000","last_fullsized":"2024-02-10T07:54:10.473380+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:47.417096+0000","last_deep_scrub":"237'18","last_deep_scrub_stamp":"2024-02-10T07:40:31.005376+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.417096+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.477900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":847,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.972286+0000","last_change":"2024-02-10T07:53:34.104503+0000","last_active":"2024-02-10T07:54:10.972286+0000","last_peered":"2024-02-10T07:54:10.972286+0000","last_clean":"2024-02-10T07:54:10.972286+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:54:10.972286+0000","last_undegraded":"2024-02-10T07:54:10.972286+0000","last_fullsized":"2024-02-10T07:54:10.972286+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:34.104386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:34.104386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:46.902238+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":890,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.687046+0000","last_change":"2024-02-10T07:50:43.296823+0000","last_active":"2024-02-10T07:54:10.687046+0000","last_peered":"2024-02-10T07:54:10.687046+0000","last_clean":"2024-02-10T07:54:10.687046+0000","last_became_active":"2024-02-10T07:50:43.296488+0000","last_became_peered":"2024-02-10T07:50:43.296488+0000","last_unstale":"2024-02-10T07:54:10.687046+0000","last_undegraded":"2024-02-10T07:54:10.687046+0000","last_fullsized":"2024-02-10T07:54:10.687046+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.630083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.630083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651968+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":895,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.687210+0000","last_change":"2024-02-10T07:50:43.298468+0000","last_active":"2024-02-10T07:54:10.687210+0000","last_peered":"2024-02-10T07:54:10.687210+0000","last_clean":"2024-02-10T07:54:10.687210+0000","last_became_active":"2024-02-10T07:50:43.297861+0000","last_became_peered":"2024-02-10T07:50:43.297861+0000","last_unstale":"2024-02-10T07:54:10.687210+0000","last_undegraded":"2024-02-10T07:54:10.687210+0000","last_fullsized":"2024-02-10T07:54:10.687210+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:05.193158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:49:05.193158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"430'47","reported_seq":955,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.473359+0000","last_change":"2024-02-10T07:49:57.522503+0000","last_active":"2024-02-10T07:54:10.473359+0000","last_peered":"2024-02-10T07:54:10.473359+0000","last_clean":"2024-02-10T07:54:10.473359+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:54:10.473359+0000","last_undegraded":"2024-02-10T07:54:10.473359+0000","last_fullsized":"2024-02-10T07:54:10.473359+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'37","last_scrub_stamp":"2024-02-10T07:49:57.522391+0000","last_deep_scrub":"271'37","last_deep_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.522391+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.477984+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":906,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.971794+0000","last_change":"2024-02-10T07:51:02.173651+0000","last_active":"2024-02-10T07:54:10.971794+0000","last_peered":"2024-02-10T07:54:10.971794+0000","last_clean":"2024-02-10T07:54:10.971794+0000","last_became_active":"2024-02-10T07:48:35.062118+0000","last_became_peered":"2024-02-10T07:48:35.062118+0000","last_unstale":"2024-02-10T07:54:10.971794+0000","last_undegraded":"2024-02-10T07:54:10.971794+0000","last_fullsized":"2024-02-10T07:54:10.971794+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:51:02.173535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:00.292320+0000","last_clean_scrub_stamp":"2024-02-10T07:51:02.173535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":767,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.970879+0000","last_change":"2024-02-10T07:52:23.855559+0000","last_active":"2024-02-10T07:54:10.970879+0000","last_peered":"2024-02-10T07:54:10.970879+0000","last_clean":"2024-02-10T07:54:10.970879+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:54:10.970879+0000","last_undegraded":"2024-02-10T07:54:10.970879+0000","last_fullsized":"2024-02-10T07:54:10.970879+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:23.855477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:23.855477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":128,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.969741+0000","last_change":"2024-02-10T07:50:43.289736+0000","last_active":"2024-02-10T07:54:10.969741+0000","last_peered":"2024-02-10T07:54:10.969741+0000","last_clean":"2024-02-10T07:54:10.969741+0000","last_became_active":"2024-02-10T07:50:43.289458+0000","last_became_peered":"2024-02-10T07:50:43.289458+0000","last_unstale":"2024-02-10T07:54:10.969741+0000","last_undegraded":"2024-02-10T07:54:10.969741+0000","last_fullsized":"2024-02-10T07:54:10.969741+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:48.809513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:48.809513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.506639+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"430'35","reported_seq":701,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.686376+0000","last_change":"2024-02-10T07:49:19.084010+0000","last_active":"2024-02-10T07:54:10.686376+0000","last_peered":"2024-02-10T07:54:10.686376+0000","last_clean":"2024-02-10T07:54:10.686376+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:54:10.686376+0000","last_undegraded":"2024-02-10T07:54:10.686376+0000","last_fullsized":"2024-02-10T07:54:10.686376+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'25","last_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_deep_scrub":"382'25","last_deep_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.083856+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651809+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"158'10","reported_seq":1155,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.473598+0000","last_change":"2024-02-10T07:49:50.396822+0000","last_active":"2024-02-10T07:54:10.473598+0000","last_peered":"2024-02-10T07:54:10.473598+0000","last_clean":"2024-02-10T07:54:10.473598+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:54:10.473598+0000","last_undegraded":"2024-02-10T07:54:10.473598+0000","last_fullsized":"2024-02-10T07:54:10.473598+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:49:50.396730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.396730+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.477796+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":876,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279508+0000","last_change":"2024-02-10T07:50:43.283505+0000","last_active":"2024-02-10T07:54:10.279508+0000","last_peered":"2024-02-10T07:54:10.279508+0000","last_clean":"2024-02-10T07:54:10.279508+0000","last_became_active":"2024-02-10T07:50:43.283161+0000","last_became_peered":"2024-02-10T07:50:43.283161+0000","last_unstale":"2024-02-10T07:54:10.279508+0000","last_undegraded":"2024-02-10T07:54:10.279508+0000","last_fullsized":"2024-02-10T07:54:10.279508+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_clean_scrub_stamp":"2024-02-10T07:48:23.271454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"296'2","reported_seq":886,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.473481+0000","last_change":"2024-02-10T07:50:03.643473+0000","last_active":"2024-02-10T07:54:10.473481+0000","last_peered":"2024-02-10T07:54:10.473481+0000","last_clean":"2024-02-10T07:54:10.473481+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:54:10.473481+0000","last_undegraded":"2024-02-10T07:54:10.473481+0000","last_fullsized":"2024-02-10T07:54:10.473481+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'2","last_scrub_stamp":"2024-02-10T07:50:03.643367+0000","last_deep_scrub":"296'2","last_deep_scrub_stamp":"2024-02-10T07:45:03.233375+0000","last_clean_scrub_stamp":"2024-02-10T07:50:03.643367+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:12.477965+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"430'42","reported_seq":936,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278785+0000","last_change":"2024-02-10T07:49:20.216865+0000","last_active":"2024-02-10T07:54:10.278785+0000","last_peered":"2024-02-10T07:54:10.278785+0000","last_clean":"2024-02-10T07:54:10.278785+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:54:10.278785+0000","last_undegraded":"2024-02-10T07:54:10.278785+0000","last_fullsized":"2024-02-10T07:54:10.278785+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-02-10T07:49:20.216750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:20.216750+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"376'12","reported_seq":942,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.686609+0000","last_change":"2024-02-10T07:49:06.282456+0000","last_active":"2024-02-10T07:54:10.686609+0000","last_peered":"2024-02-10T07:54:10.686609+0000","last_clean":"2024-02-10T07:54:10.686609+0000","last_became_active":"2024-02-10T07:49:06.255324+0000","last_became_peered":"2024-02-10T07:49:06.255324+0000","last_unstale":"2024-02-10T07:54:10.686609+0000","last_undegraded":"2024-02-10T07:54:10.686609+0000","last_fullsized":"2024-02-10T07:54:10.686609+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'11","last_scrub_stamp":"2024-02-10T07:48:58.368663+0000","last_deep_scrub":"227'9","last_deep_scrub_stamp":"2024-02-10T07:39:53.440221+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.368663+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651762+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":873,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278912+0000","last_change":"2024-02-10T07:50:43.282245+0000","last_active":"2024-02-10T07:54:10.278912+0000","last_peered":"2024-02-10T07:54:10.278912+0000","last_clean":"2024-02-10T07:54:10.278912+0000","last_became_active":"2024-02-10T07:50:43.281465+0000","last_became_peered":"2024-02-10T07:50:43.281465+0000","last_unstale":"2024-02-10T07:54:10.278912+0000","last_undegraded":"2024-02-10T07:54:10.278912+0000","last_fullsized":"2024-02-10T07:54:10.278912+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:27.819612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:46:27.819612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876459+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":358,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279595+0000","last_change":"2024-02-10T07:47:50.948710+0000","last_active":"2024-02-10T07:54:10.279595+0000","last_peered":"2024-02-10T07:54:10.279595+0000","last_clean":"2024-02-10T07:54:10.279595+0000","last_became_active":"2024-02-10T07:47:47.966443+0000","last_became_peered":"2024-02-10T07:47:47.966443+0000","last_unstale":"2024-02-10T07:54:10.279595+0000","last_undegraded":"2024-02-10T07:54:10.279595+0000","last_fullsized":"2024-02-10T07:54:10.279595+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.948624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.948624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"430'42","reported_seq":875,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278572+0000","last_change":"2024-02-10T07:49:57.303310+0000","last_active":"2024-02-10T07:54:10.278572+0000","last_peered":"2024-02-10T07:54:10.278572+0000","last_clean":"2024-02-10T07:54:10.278572+0000","last_became_active":"2024-02-10T07:49:06.252935+0000","last_became_peered":"2024-02-10T07:49:06.252935+0000","last_unstale":"2024-02-10T07:54:10.278572+0000","last_undegraded":"2024-02-10T07:54:10.278572+0000","last_fullsized":"2024-02-10T07:54:10.278572+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:57.303200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.303200+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":875,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278519+0000","last_change":"2024-02-10T07:49:06.239655+0000","last_active":"2024-02-10T07:54:10.278519+0000","last_peered":"2024-02-10T07:54:10.278519+0000","last_clean":"2024-02-10T07:54:10.278519+0000","last_became_active":"2024-02-10T07:49:06.239190+0000","last_became_peered":"2024-02-10T07:49:06.239190+0000","last_unstale":"2024-02-10T07:54:10.278519+0000","last_undegraded":"2024-02-10T07:54:10.278519+0000","last_fullsized":"2024-02-10T07:54:10.278519+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:42.549018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.549018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876515+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":382,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.686566+0000","last_change":"2024-02-10T07:49:06.274374+0000","last_active":"2024-02-10T07:54:10.686566+0000","last_peered":"2024-02-10T07:54:10.686566+0000","last_clean":"2024-02-10T07:54:10.686566+0000","last_became_active":"2024-02-10T07:48:02.371926+0000","last_became_peered":"2024-02-10T07:48:02.371926+0000","last_unstale":"2024-02-10T07:54:10.686566+0000","last_undegraded":"2024-02-10T07:54:10.686566+0000","last_fullsized":"2024-02-10T07:54:10.686566+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.274263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:49.704024+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.274263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651677+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"395'12","reported_seq":298,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.971854+0000","last_change":"2024-02-10T07:52:24.850402+0000","last_active":"2024-02-10T07:54:10.971854+0000","last_peered":"2024-02-10T07:54:10.971854+0000","last_clean":"2024-02-10T07:54:10.971854+0000","last_became_active":"2024-02-10T07:48:35.071916+0000","last_became_peered":"2024-02-10T07:48:35.071916+0000","last_unstale":"2024-02-10T07:54:10.971854+0000","last_undegraded":"2024-02-10T07:54:10.971854+0000","last_fullsized":"2024-02-10T07:54:10.971854+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'12","last_scrub_stamp":"2024-02-10T07:52:24.850269+0000","last_deep_scrub":"395'12","last_deep_scrub_stamp":"2024-02-10T07:52:24.850269+0000","last_clean_scrub_stamp":"2024-02-10T07:52:24.850269+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:16.199458+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":203,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278016+0000","last_change":"2024-02-10T07:53:37.144276+0000","last_active":"2024-02-10T07:54:10.278016+0000","last_peered":"2024-02-10T07:54:10.278016+0000","last_clean":"2024-02-10T07:54:10.278016+0000","last_became_active":"2024-02-10T07:49:06.251038+0000","last_became_peered":"2024-02-10T07:49:06.251038+0000","last_unstale":"2024-02-10T07:54:10.278016+0000","last_undegraded":"2024-02-10T07:54:10.278016+0000","last_fullsized":"2024-02-10T07:54:10.278016+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:37.144142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:53:37.144142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:01.122427+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"430'39","reported_seq":947,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.687376+0000","last_change":"2024-02-10T07:50:00.655466+0000","last_active":"2024-02-10T07:54:10.687376+0000","last_peered":"2024-02-10T07:54:10.687376+0000","last_clean":"2024-02-10T07:54:10.687376+0000","last_became_active":"2024-02-10T07:47:47.967427+0000","last_became_peered":"2024-02-10T07:47:47.967427+0000","last_unstale":"2024-02-10T07:54:10.687376+0000","last_undegraded":"2024-02-10T07:54:10.687376+0000","last_fullsized":"2024-02-10T07:54:10.687376+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:50:00.655377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.655377+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":577,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.970035+0000","last_change":"2024-02-10T07:51:59.948039+0000","last_active":"2024-02-10T07:54:10.970035+0000","last_peered":"2024-02-10T07:54:10.970035+0000","last_clean":"2024-02-10T07:54:10.970035+0000","last_became_active":"2024-02-10T07:49:06.268874+0000","last_became_peered":"2024-02-10T07:49:06.268874+0000","last_unstale":"2024-02-10T07:54:10.970035+0000","last_undegraded":"2024-02-10T07:54:10.970035+0000","last_fullsized":"2024-02-10T07:54:10.970035+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:51:59.947908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_clean_scrub_stamp":"2024-02-10T07:51:59.947908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":356,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279106+0000","last_change":"2024-02-10T07:49:57.310447+0000","last_active":"2024-02-10T07:54:10.279106+0000","last_peered":"2024-02-10T07:54:10.279106+0000","last_clean":"2024-02-10T07:54:10.279106+0000","last_became_active":"2024-02-10T07:48:02.333689+0000","last_became_peered":"2024-02-10T07:48:02.333689+0000","last_unstale":"2024-02-10T07:54:10.279106+0000","last_undegraded":"2024-02-10T07:54:10.279106+0000","last_fullsized":"2024-02-10T07:54:10.279106+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:57.310333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.310333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":880,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.278612+0000","last_change":"2024-02-10T07:50:43.283476+0000","last_active":"2024-02-10T07:54:10.278612+0000","last_peered":"2024-02-10T07:54:10.278612+0000","last_clean":"2024-02-10T07:54:10.278612+0000","last_became_active":"2024-02-10T07:50:43.283104+0000","last_became_peered":"2024-02-10T07:50:43.283104+0000","last_unstale":"2024-02-10T07:54:10.278612+0000","last_undegraded":"2024-02-10T07:54:10.278612+0000","last_fullsized":"2024-02-10T07:54:10.278612+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:37.995362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.995362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876479+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"430'32","reported_seq":965,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.280240+0000","last_change":"2024-02-10T07:49:16.249257+0000","last_active":"2024-02-10T07:54:10.280240+0000","last_peered":"2024-02-10T07:54:10.280240+0000","last_clean":"2024-02-10T07:54:10.280240+0000","last_became_active":"2024-02-10T07:48:35.062573+0000","last_became_peered":"2024-02-10T07:48:35.062573+0000","last_unstale":"2024-02-10T07:54:10.280240+0000","last_undegraded":"2024-02-10T07:54:10.280240+0000","last_fullsized":"2024-02-10T07:54:10.280240+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:49:16.248936+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.248936+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876711+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":128,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.969934+0000","last_change":"2024-02-10T07:50:43.295417+0000","last_active":"2024-02-10T07:54:10.969934+0000","last_peered":"2024-02-10T07:54:10.969934+0000","last_clean":"2024-02-10T07:54:10.969934+0000","last_became_active":"2024-02-10T07:50:43.290295+0000","last_became_peered":"2024-02-10T07:50:43.290295+0000","last_unstale":"2024-02-10T07:54:10.969934+0000","last_undegraded":"2024-02-10T07:54:10.969934+0000","last_fullsized":"2024-02-10T07:54:10.969934+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.153368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.153368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.506559+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":865,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.687299+0000","last_change":"2024-02-10T07:49:07.170105+0000","last_active":"2024-02-10T07:54:10.687299+0000","last_peered":"2024-02-10T07:54:10.687299+0000","last_clean":"2024-02-10T07:54:10.687299+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:54:10.687299+0000","last_undegraded":"2024-02-10T07:54:10.687299+0000","last_fullsized":"2024-02-10T07:54:10.687299+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:07.169962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:07.169962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:14.651778+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":907,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.971896+0000","last_change":"2024-02-10T07:52:26.772420+0000","last_active":"2024-02-10T07:54:10.971896+0000","last_peered":"2024-02-10T07:54:10.971896+0000","last_clean":"2024-02-10T07:54:10.971896+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:54:10.971896+0000","last_undegraded":"2024-02-10T07:54:10.971896+0000","last_fullsized":"2024-02-10T07:54:10.971896+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:26.772322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:03.803361+0000","last_clean_scrub_stamp":"2024-02-10T07:52:26.772322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"430'53","reported_seq":1013,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279342+0000","last_change":"2024-02-10T07:50:43.316983+0000","last_active":"2024-02-10T07:54:10.279342+0000","last_peered":"2024-02-10T07:54:10.279342+0000","last_clean":"2024-02-10T07:54:10.279342+0000","last_became_active":"2024-02-10T07:50:43.281879+0000","last_became_peered":"2024-02-10T07:50:43.281879+0000","last_unstale":"2024-02-10T07:54:10.279342+0000","last_undegraded":"2024-02-10T07:54:10.279342+0000","last_fullsized":"2024-02-10T07:54:10.279342+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T07:48:59.066256+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.066256+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876666+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":867,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.280355+0000","last_change":"2024-02-10T07:50:43.290883+0000","last_active":"2024-02-10T07:54:10.280355+0000","last_peered":"2024-02-10T07:54:10.280355+0000","last_clean":"2024-02-10T07:54:10.280355+0000","last_became_active":"2024-02-10T07:50:43.290334+0000","last_became_peered":"2024-02-10T07:50:43.290334+0000","last_unstale":"2024-02-10T07:54:10.280355+0000","last_undegraded":"2024-02-10T07:54:10.280355+0000","last_fullsized":"2024-02-10T07:54:10.280355+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:47.275882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.275882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876812+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"405'12","reported_seq":175,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.969882+0000","last_change":"2024-02-10T07:50:43.294666+0000","last_active":"2024-02-10T07:54:10.969882+0000","last_peered":"2024-02-10T07:54:10.969882+0000","last_clean":"2024-02-10T07:54:10.969882+0000","last_became_active":"2024-02-10T07:50:43.280142+0000","last_became_peered":"2024-02-10T07:50:43.280142+0000","last_unstale":"2024-02-10T07:54:10.969882+0000","last_undegraded":"2024-02-10T07:54:10.969882+0000","last_fullsized":"2024-02-10T07:54:10.969882+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'11","last_scrub_stamp":"2024-02-10T07:49:14.335064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:14.335064+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.506608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":373,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.472787+0000","last_change":"2024-02-10T07:53:20.474601+0000","last_active":"2024-02-10T07:54:10.472787+0000","last_peered":"2024-02-10T07:54:10.472787+0000","last_clean":"2024-02-10T07:54:10.472787+0000","last_became_active":"2024-02-10T07:49:06.258926+0000","last_became_peered":"2024-02-10T07:49:06.258926+0000","last_unstale":"2024-02-10T07:54:10.472787+0000","last_undegraded":"2024-02-10T07:54:10.472787+0000","last_fullsized":"2024-02-10T07:54:10.472787+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:20.474497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_clean_scrub_stamp":"2024-02-10T07:53:20.474497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:48.442368+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"430'30","reported_seq":947,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-02-10T07:54:10.279538+0000","last_change":"2024-02-10T07:49:06.289632+0000","last_active":"2024-02-10T07:54:10.279538+0000","last_peered":"2024-02-10T07:54:10.279538+0000","last_clean":"2024-02-10T07:54:10.279538+0000","last_became_active":"2024-02-10T07:49:06.238858+0000","last_became_peered":"2024-02-10T07:49:06.238858+0000","last_unstale":"2024-02-10T07:54:10.279538+0000","last_undegraded":"2024-02-10T07:54:10.279538+0000","last_fullsized":"2024-02-10T07:54:10.279538+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:48:15.588199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.588199+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:13.876612+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1335,"num_read_kb":1330,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1540,"num_read_kb":1297,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":6382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1060,"ondisk_log_size":1060,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038033,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1251400,"kb_used_data":435836,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92493752,"statfs":{"total":95995035648,"available":94713602048,"internally_reserved":0,"allocated":446296064,"data_stored":445750721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":835123658},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":3.0030000000000001,"5min":1.8480000000000001,"15min":1.6899999999999999},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":18.202999999999999,"5min":39.494,"15min":39.494},"last":0.55200000000000005},{"interface":"front","average":{"1min":2.968,"5min":1.8959999999999999,"15min":1.7030000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.23000000000000001},"max":{"1min":19.745000000000001,"5min":38.524000000000001,"15min":38.524000000000001},"last":0.69299999999999995}]},{"osd":1,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":6.6909999999999998,"5min":2.4009999999999998,"15min":2.069},"min":{"1min":0.33400000000000002,"5min":0.251,"15min":0.24399999999999999},"max":{"1min":50.149000000000001,"5min":50.149000000000001,"15min":50.149000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":6.5720000000000001,"5min":2.431,"15min":2.0190000000000001},"min":{"1min":0.33100000000000002,"5min":0.26300000000000001,"15min":0.24399999999999999},"max":{"1min":54.048000000000002,"5min":54.048000000000002,"15min":54.048000000000002},"last":0.53900000000000003}]},{"osd":2,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":1.25,"15min":1.141},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.255},"max":{"1min":7.0019999999999998,"5min":7.0019999999999998,"15min":18.57},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.4510000000000001,"15min":1.2989999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":7.8410000000000002,"5min":8.3949999999999996,"15min":18.521000000000001},"last":0.48799999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:54:05 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.1950000000000001,"15min":1.099},"min":{"1min":0.35499999999999998,"5min":0.32400000000000001,"15min":0.307},"max":{"1min":1.6930000000000001,"5min":4.7590000000000003,"15min":4.7590000000000003},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.222,"15min":1.022},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.313},"max":{"1min":3.1859999999999999,"5min":6.2839999999999998,"15min":6.2839999999999998},"last":0.61899999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.347,"15min":1.1719999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.22},"max":{"1min":1.5960000000000001,"5min":26.001000000000001,"15min":26.001000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.45700000000000002,"5min":1.3089999999999999,"15min":1.1539999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.94199999999999995,"5min":27.09,"15min":27.09},"last":0.45600000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.75900000000000001,"15min":1.0449999999999999},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.245},"max":{"1min":1.6479999999999999,"5min":3.5489999999999999,"15min":18.126999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.98099999999999998,"15min":1.1699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.6639999999999999,"5min":5.8490000000000002,"15min":19.548999999999999},"last":0.63300000000000001}]},{"osd":6,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":1.1599999999999999,"15min":1.159},"min":{"1min":0.30299999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.0409999999999999,"5min":7.4329999999999998,"15min":17.832000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":1.077,"15min":1.179},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.29799999999999999},"max":{"1min":1.1719999999999999,"5min":6.101,"15min":20.449999999999999},"last":0.57099999999999995}]}]},{"osd":6,"up_from":50,"seq":214748365078,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":646512,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93098640,"statfs":{"total":95995035648,"available":95333007360,"internally_reserved":0,"allocated":1736704,"data_stored":1219050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":660273606},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":1.9199999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.1580000000000004,"5min":19.484999999999999,"15min":24.727},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.4419999999999999,"15min":1.407},"min":{"1min":0.25700000000000001,"5min":0.25600000000000001,"15min":0.23100000000000001},"max":{"1min":6.0469999999999997,"5min":14.808,"15min":29.084},"last":0.61799999999999999}]},{"osd":1,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.7310000000000001,"15min":1.907},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.23799999999999999},"max":{"1min":9.9710000000000001,"5min":15.007999999999999,"15min":37.659999999999997},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.651,"5min":1.6759999999999999,"15min":1.948},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.253},"max":{"1min":10.029999999999999,"5min":19.524999999999999,"15min":37.75},"last":0.44700000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":1.7929999999999999,"15min":1.409},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":11.839,"5min":15.768000000000001,"15min":15.768000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":1.853,"5min":1.5980000000000001,"15min":1.2430000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.013999999999999,"5min":15.728999999999999,"15min":15.728999999999999},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.284,"15min":1.5980000000000001},"min":{"1min":0.371,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.7779999999999996,"5min":5.6500000000000004,"15min":13.279},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.2470000000000001,"15min":1.2450000000000001},"min":{"1min":0.376,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.8929999999999998,"5min":7.7949999999999999,"15min":7.7949999999999999},"last":0.46600000000000003}]},{"osd":4,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.60699999999999998,"15min":0.85499999999999998},"min":{"1min":0.26800000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":3.6099999999999999,"5min":3.6099999999999999,"15min":7.6509999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.70099999999999996,"15min":0.88600000000000001},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.26000000000000001},"max":{"1min":2.3220000000000001,"5min":4.8600000000000003,"15min":7.0300000000000002},"last":0.57799999999999996}]},{"osd":5,"last update":"Sat Feb 10 07:54:05 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.58899999999999997,"15min":1.0780000000000001},"min":{"1min":0.308,"5min":0.28799999999999998,"15min":0.24099999999999999},"max":{"1min":4.5720000000000001,"5min":4.5720000000000001,"15min":44.689},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.748,"15min":1.21},"min":{"1min":0.32700000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.4460000000000002,"5min":4.8940000000000001,"15min":44.465000000000003},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:53:15 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.121,"15min":1.3089999999999999},"min":{"1min":0.314,"5min":0.29999999999999999,"15min":0.28100000000000003},"max":{"1min":2.9449999999999998,"5min":4.9379999999999997,"15min":42.631999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":1.0589999999999999,"15min":1.224},"min":{"1min":0.376,"5min":0.32500000000000001,"15min":0.315},"max":{"1min":2.5990000000000002,"5min":6.5419999999999998,"15min":44.792999999999999},"last":0.63900000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575152,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852812,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892340,"statfs":{"total":95995035648,"available":95121756160,"internally_reserved":0,"allocated":1179648,"data_stored":629409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":872085958},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:49 2024","interfaces":[{"interface":"back","average":{"1min":2.3170000000000002,"5min":2.117,"15min":1.589},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":9.0530000000000008,"5min":14.382999999999999,"15min":14.382999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":3.2719999999999998,"5min":2.109,"15min":1.5920000000000001},"min":{"1min":0.47699999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":7.6779999999999999,"5min":14.16,"15min":14.16},"last":0.315}]},{"osd":2,"last update":"Sat Feb 10 07:53:19 2024","interfaces":[{"interface":"back","average":{"1min":2.8029999999999999,"5min":2.1200000000000001,"15min":1.4570000000000001},"min":{"1min":0.246,"5min":0.23400000000000001,"15min":0.20399999999999999},"max":{"1min":6.2510000000000003,"5min":14.343,"15min":14.343},"last":0.29199999999999998},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.8129999999999999,"15min":1.45},"min":{"1min":0.58999999999999997,"5min":0.24099999999999999,"15min":0.224},"max":{"1min":7.5279999999999996,"5min":14.204000000000001,"15min":14.204000000000001},"last":0.44400000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:54:02 2024","interfaces":[{"interface":"back","average":{"1min":2.714,"5min":2.1890000000000001,"15min":1.0549999999999999},"min":{"1min":0.314,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":9.0830000000000002,"5min":22.367999999999999,"15min":22.367999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.2269999999999999,"15min":1},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":7.6020000000000003,"5min":22.199999999999999,"15min":22.199999999999999},"last":0.40200000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:54:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9319999999999999,"5min":1.5900000000000001,"15min":2.2370000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28399999999999997},"max":{"1min":9.2010000000000005,"5min":9.2010000000000005,"15min":207.45699999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2130000000000001,"15min":1.3160000000000001},"min":{"1min":0.35299999999999998,"5min":0.29599999999999999,"15min":0.28899999999999998},"max":{"1min":7.9749999999999996,"5min":10.327999999999999,"15min":32.201000000000001},"last":0.60299999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:54:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9970000000000001,"5min":1.9119999999999999,"15min":2.1269999999999998},"min":{"1min":0.35899999999999999,"5min":0.34200000000000003,"15min":0.308},"max":{"1min":9.1419999999999995,"5min":23.978999999999999,"15min":98.703000000000003},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.893,"5min":1.6850000000000001,"15min":2.0470000000000002},"min":{"1min":0.40999999999999998,"5min":0.32600000000000001,"15min":0.30199999999999999},"max":{"1min":7.4320000000000004,"5min":24.024999999999999,"15min":98.664000000000001},"last":0.78800000000000003}]},{"osd":6,"last update":"Sat Feb 10 07:53:49 2024","interfaces":[{"interface":"back","average":{"1min":1.7270000000000001,"5min":1.514,"15min":1.47},"min":{"1min":0.40400000000000003,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":8.0600000000000005,"5min":10.291,"15min":36.808},"last":0.64600000000000002},{"interface":"front","average":{"1min":2.2629999999999999,"5min":1.837,"15min":1.7749999999999999},"min":{"1min":0.47399999999999998,"5min":0.40100000000000002,"15min":0.33300000000000002},"max":{"1min":9.1750000000000007,"5min":9.1750000000000007,"15min":33.774999999999999},"last":0.84199999999999997}]},{"osd":7,"last update":"Sat Feb 10 07:53:19 2024","interfaces":[{"interface":"back","average":{"1min":2.048,"5min":1.7709999999999999,"15min":1.5489999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":7.649,"5min":24.099,"15min":24.099},"last":0.755},{"interface":"front","average":{"1min":1.639,"5min":1.331,"15min":1.288},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.34300000000000003},"max":{"1min":4.0439999999999996,"5min":10.356999999999999,"15min":10.356999999999999},"last":0.627}]}]},{"osd":0,"up_from":8,"seq":34359738677,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799240,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92945912,"statfs":{"total":95995035648,"available":95176613888,"internally_reserved":0,"allocated":1765376,"data_stored":1220430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":816642501},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:53:44 2024","interfaces":[{"interface":"back","average":{"1min":4.194,"5min":2.5979999999999999,"15min":2.625},"min":{"1min":0.23899999999999999,"5min":0.20499999999999999,"15min":0.182},"max":{"1min":11.933,"5min":16.942,"15min":156.16200000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":3.29,"5min":1.9790000000000001,"15min":2.4220000000000002},"min":{"1min":0.42099999999999999,"5min":0.22,"15min":0.183},"max":{"1min":8.891,"5min":8.891,"15min":156.941},"last":0.443}]},{"osd":2,"last update":"Sat Feb 10 07:53:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5739999999999998,"5min":2.0409999999999999,"15min":1.6499999999999999},"min":{"1min":0.65700000000000003,"5min":0.221,"15min":0.20000000000000001},"max":{"1min":19.733000000000001,"5min":19.733000000000001,"15min":34.603000000000002},"last":0.38500000000000001},{"interface":"front","average":{"1min":2.9100000000000001,"5min":1.9370000000000001,"15min":1.444},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.22600000000000001},"max":{"1min":8.4060000000000006,"5min":10.116,"15min":34.718000000000004},"last":0.40300000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":4.2039999999999997,"5min":2.21,"15min":1.1579999999999999},"min":{"1min":0.45900000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":29.792999999999999,"5min":29.792999999999999,"15min":29.792999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":4.1929999999999996,"5min":2.444,"15min":1.23},"min":{"1min":0.38800000000000001,"5min":0.214,"15min":0.214},"max":{"1min":29.506,"5min":29.506,"15min":29.506},"last":0.42799999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:54:03 2024","interfaces":[{"interface":"back","average":{"1min":1.7170000000000001,"5min":1.825,"15min":1.4890000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27600000000000002},"max":{"1min":7.181,"5min":16.991,"15min":37.286000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.46,"5min":1.5980000000000001,"15min":1.4350000000000001},"min":{"1min":0.32400000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.0709999999999997,"5min":8.5939999999999994,"15min":34.683},"last":0.64100000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:54:03 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.925,"15min":2.0379999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":7.1120000000000001,"5min":17.062999999999999,"15min":41.429000000000002},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.625,"15min":1.899},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.308},"max":{"1min":6,"5min":8.6460000000000008,"15min":41.456000000000003},"last":0.67000000000000004}]},{"osd":6,"last update":"Sat Feb 10 07:53:44 2024","interfaces":[{"interface":"back","average":{"1min":2.1320000000000001,"5min":1.6100000000000001,"15min":1.6020000000000001},"min":{"1min":0.45400000000000001,"5min":0.34599999999999997,"15min":0.33600000000000002},"max":{"1min":8.2349999999999994,"5min":8.5630000000000006,"15min":41.587000000000003},"last":0.627},{"interface":"front","average":{"1min":2.694,"5min":2.0310000000000001,"15min":1.879},"min":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":7.1520000000000001,"5min":17.027999999999999,"15min":41.256},"last":0.67900000000000005}]},{"osd":7,"last update":"Sat Feb 10 07:53:22 2024","interfaces":[{"interface":"back","average":{"1min":2.6680000000000001,"5min":1.7669999999999999,"15min":1.6160000000000001},"min":{"1min":0.41199999999999998,"5min":0.39600000000000002,"15min":0.33300000000000002},"max":{"1min":8.6809999999999992,"5min":8.6809999999999992,"15min":34.710999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.831,"5min":1.589,"15min":1.4039999999999999},"min":{"1min":0.45400000000000001,"5min":0.39200000000000002,"15min":0.379},"max":{"1min":8.3230000000000004,"5min":8.5999999999999996,"15min":34.997999999999998},"last":0.57399999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411626,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":694504,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":93050648,"statfs":{"total":95995035648,"available":95283863552,"internally_reserved":0,"allocated":1138688,"data_stored":629272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":710015429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:23 2024","interfaces":[{"interface":"back","average":{"1min":3.5510000000000002,"5min":2.46,"15min":1.8029999999999999},"min":{"1min":0.47499999999999998,"5min":0.20699999999999999,"15min":0.191},"max":{"1min":32.371000000000002,"5min":32.371000000000002,"15min":65.430999999999997},"last":0.27500000000000002},{"interface":"front","average":{"1min":4.0869999999999997,"5min":2.6749999999999998,"15min":1.6990000000000001},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.20100000000000001},"max":{"1min":33.280000000000001,"5min":38.173000000000002,"15min":38.173000000000002},"last":0.35799999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:53:23 2024","interfaces":[{"interface":"back","average":{"1min":3.2490000000000001,"5min":1.9690000000000001,"15min":1.736},"min":{"1min":0.46700000000000003,"5min":0.21299999999999999,"15min":0.20599999999999999},"max":{"1min":13.071999999999999,"5min":27.606000000000002,"15min":65.795000000000002},"last":0.438},{"interface":"front","average":{"1min":4.0030000000000001,"5min":2.456,"15min":1.681},"min":{"1min":0.38500000000000001,"5min":0.219,"15min":0.20499999999999999},"max":{"1min":12.17,"5min":29.709,"15min":29.709},"last":0.39500000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:54:01 2024","interfaces":[{"interface":"back","average":{"1min":1.829,"5min":1.4139999999999999,"15min":0.85199999999999998},"min":{"1min":0.28199999999999997,"5min":0.217,"15min":0.217},"max":{"1min":6.5259999999999998,"5min":7.0819999999999999,"15min":7.0819999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":2.415,"5min":1.776,"15min":0.91700000000000004},"min":{"1min":0.29299999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":5.633,"5min":7.7530000000000001,"15min":7.7530000000000001},"last":0.33800000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:54:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1890000000000001,"15min":1.401},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29599999999999999},"max":{"1min":3.4820000000000002,"5min":11.395,"15min":65.494},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.881,"5min":1.6779999999999999,"15min":1.3169999999999999},"min":{"1min":0.40100000000000002,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":5.5890000000000004,"5min":7.7149999999999999,"15min":7.7149999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Sat Feb 10 07:54:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.4339999999999999,"15min":1.4910000000000001},"min":{"1min":0.36099999999999999,"5min":0.33400000000000002,"15min":0.32600000000000001},"max":{"1min":4.7480000000000002,"5min":10.807,"15min":25.533000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.105,"15min":1.536},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.5310000000000001,"5min":4.9749999999999996,"15min":65.834000000000003},"last":0.66700000000000004}]},{"osd":6,"last update":"Sat Feb 10 07:53:41 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.2110000000000001,"15min":1.6890000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.35199999999999998},"max":{"1min":4.7400000000000002,"5min":5.069,"15min":65.748000000000005},"last":0.65000000000000002},{"interface":"front","average":{"1min":2.4780000000000002,"5min":1.5,"15min":1.4550000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":10.893000000000001,"5min":10.893000000000001,"15min":25.901},"last":0.61499999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:53:23 2024","interfaces":[{"interface":"back","average":{"1min":2.5030000000000001,"5min":1.702,"15min":1.3979999999999999},"min":{"1min":0.42299999999999999,"5min":0.36899999999999999,"15min":0.36699999999999999},"max":{"1min":10.859999999999999,"5min":10.859999999999999,"15min":10.859999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.2210000000000001,"15min":1.444},"min":{"1min":0.42799999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.609,"5min":5.0979999999999999,"15min":65.539000000000001},"last":0.58599999999999997}]}]},{"osd":3,"up_from":347,"seq":1490353651790,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":269004,"kb_used_data":1152,"kb_used_omap":5,"kb_used_meta":267834,"kb_avail":93476148,"statfs":{"total":95995035648,"available":95719575552,"internally_reserved":0,"allocated":1179648,"data_stored":629188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5625,"internal_metadata":274262535},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.8700000000000001,"5min":2.0329999999999999,"15min":1.028},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23699999999999999},"max":{"1min":6.694,"5min":28.247,"15min":28.247},"last":0.317},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0939999999999999,"15min":1.115},"min":{"1min":0.24399999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.9409999999999998,"5min":27.361999999999998,"15min":27.361999999999998},"last":0.45100000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":3.0760000000000001,"5min":2.8860000000000001,"15min":1.226},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":15.547000000000001,"5min":30.718,"15min":30.718},"last":0.42899999999999999},{"interface":"front","average":{"1min":2.4940000000000002,"5min":2.5939999999999999,"15min":1.1499999999999999},"min":{"1min":0.32800000000000001,"5min":0.22,"15min":0.22},"max":{"1min":12.583,"5min":32.927999999999997,"15min":32.927999999999997},"last":0.501}]},{"osd":2,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":2.0350000000000001,"5min":1.7130000000000001,"15min":0.879},"min":{"1min":0.30599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.7430000000000003,"5min":10.686,"15min":10.686},"last":0.46800000000000003},{"interface":"front","average":{"1min":2.3010000000000002,"5min":2.0390000000000001,"15min":0.996},"min":{"1min":0.39100000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.4209999999999994,"5min":11.25,"15min":11.25},"last":0.36199999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.637,"15min":2.383},"min":{"1min":0.34200000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":7.782,"5min":15.505000000000001,"15min":28.707999999999998},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.7849999999999999,"5min":1.514,"15min":2.5529999999999999},"min":{"1min":0.39100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":6.7759999999999998,"5min":15.551,"15min":28.760999999999999},"last":0.63800000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.655,"15min":1.1160000000000001},"min":{"1min":0.371,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":6.343,"5min":15.525,"15min":15.525},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.4810000000000001,"15min":0.99199999999999999},"min":{"1min":0.36199999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":7.8280000000000003,"5min":10.298,"15min":10.298},"last":0.70299999999999996}]},{"osd":6,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":2.1850000000000001,"15min":2.444},"min":{"1min":0.40799999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":6.3920000000000003,"5min":55.750999999999998,"15min":55.750999999999998},"last":0.621},{"interface":"front","average":{"1min":2.0169999999999999,"5min":2.4100000000000001,"15min":2.956},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":8.5559999999999992,"5min":55.807000000000002,"15min":55.807000000000002},"last":0.57699999999999996}]},{"osd":7,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":1.8720000000000001,"15min":1.367},"min":{"1min":0.433,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.8760000000000003,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":2.0819999999999999,"5min":1.5920000000000001,"15min":1.3759999999999999},"min":{"1min":0.499,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.8049999999999997,"5min":7.8650000000000002,"15min":7.8650000000000002},"last":0.54300000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953760,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758408,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92986744,"statfs":{"total":95995035648,"available":95218425856,"internally_reserved":0,"allocated":1175552,"data_stored":629300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:54:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":1.341,"15min":1.7030000000000001},"min":{"1min":0.26300000000000001,"5min":0.251,"15min":0.20499999999999999},"max":{"1min":4.0339999999999998,"5min":20.004999999999999,"15min":74.533000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.7050000000000001,"15min":1.839},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.22700000000000001},"max":{"1min":4.8710000000000004,"5min":20.050000000000001,"15min":74.340999999999994},"last":0.53100000000000003}]},{"osd":1,"last update":"Sat Feb 10 07:54:04 2024","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.1030000000000002,"15min":2.5489999999999999},"min":{"1min":0.27400000000000002,"5min":0.245,"15min":0.23200000000000001},"max":{"1min":23.460000000000001,"5min":35.673999999999999,"15min":69.852000000000004},"last":0.63500000000000001},{"interface":"front","average":{"1min":2.117,"5min":2.3799999999999999,"15min":2.5},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":18.305,"5min":35.375,"15min":70.692999999999998},"last":0.54400000000000004}]},{"osd":2,"last update":"Sat Feb 10 07:54:04 2024","interfaces":[{"interface":"back","average":{"1min":2.0710000000000002,"5min":1.4890000000000001,"15min":1.262},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.26600000000000001},"max":{"1min":25.32,"5min":25.32,"15min":25.32},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.841,"5min":1.2989999999999999,"15min":1.1970000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":20.535,"5min":20.535,"15min":20.535},"last":0.623}]},{"osd":3,"last update":"Sat Feb 10 07:53:58 2024","interfaces":[{"interface":"back","average":{"1min":1.7609999999999999,"5min":1.3959999999999999,"15min":3.6280000000000001},"min":{"1min":0.30499999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":20.318000000000001,"5min":20.318000000000001,"15min":66.379999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.8009999999999999,"5min":1.3400000000000001,"15min":1.629},"min":{"1min":0.314,"5min":0.314,"15min":0.309},"max":{"1min":22.917000000000002,"5min":22.917000000000002,"15min":22.917000000000002},"last":0.64600000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:54:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.70099999999999996,"15min":0.98799999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.3109999999999999,"5min":5.9720000000000004,"15min":10.502000000000001},"last":0.313},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60299999999999998,"15min":0.98499999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.247},"max":{"1min":2.0289999999999999,"5min":2.9820000000000002,"15min":9.2460000000000004},"last":0.37}]},{"osd":6,"last update":"Sat Feb 10 07:53:49 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.77100000000000002,"15min":1.2070000000000001},"min":{"1min":0.27800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.2090000000000001,"5min":6.0039999999999996,"15min":67.433999999999997},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.79800000000000004,"15min":1.2849999999999999},"min":{"1min":0.29499999999999998,"5min":0.29199999999999998,"15min":0.26600000000000001},"max":{"1min":1.798,"5min":5.0090000000000003,"15min":66.876000000000005},"last":0.39600000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:53:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":1.0189999999999999,"15min":0.97899999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.307},"max":{"1min":1.879,"5min":7.4930000000000003,"15min":7.4930000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.86099999999999999,"15min":1.006},"min":{"1min":0.36699999999999999,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":1.8420000000000001,"5min":3.8479999999999999,"15min":10.919},"last":0.41399999999999998}]}]},{"osd":5,"up_from":156,"seq":670014898389,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":697728,"kb_used_data":1136,"kb_used_omap":4,"kb_used_meta":696571,"kb_avail":93047424,"statfs":{"total":95995035648,"available":95280562176,"internally_reserved":0,"allocated":1163264,"data_stored":630114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":713289378},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":1.6719999999999999,"15min":1.772},"min":{"1min":0.24299999999999999,"5min":0.22600000000000001,"15min":0.20899999999999999},"max":{"1min":3.0739999999999998,"5min":24.292999999999999,"15min":52.938000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.4970000000000001,"15min":1.974},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.22900000000000001},"max":{"1min":3.2589999999999999,"5min":23.908000000000001,"15min":52.805},"last":0.48899999999999999}]},{"osd":1,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0950000000000002,"5min":2.2309999999999999,"15min":2.1360000000000001},"min":{"1min":0.26700000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":12.362,"5min":28.780999999999999,"15min":54.298000000000002},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.843,"5min":2.2850000000000001,"15min":2.0569999999999999},"min":{"1min":0.27900000000000003,"5min":0.248,"15min":0.246},"max":{"1min":6.1630000000000003,"5min":28.356000000000002,"15min":52.969999999999999},"last":0.52600000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.329,"15min":1.244},"min":{"1min":0.29899999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.2709999999999999,"5min":10.039999999999999,"15min":10.039999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.587,"5min":1.347,"15min":1.2789999999999999},"min":{"1min":0.30199999999999999,"5min":0.28799999999999998,"15min":0.28100000000000003},"max":{"1min":5.2270000000000003,"5min":9.6069999999999993,"15min":13.662000000000001},"last":0.61899999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":2.1280000000000001,"15min":1.8620000000000001},"min":{"1min":0.33400000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.4540000000000002,"5min":97.119,"15min":97.119},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.0449999999999999,"5min":2.4020000000000001,"15min":1.784},"min":{"1min":0.32700000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.1819999999999999,"5min":97.141999999999996,"15min":97.141999999999996},"last":0.51100000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.81999999999999995,"15min":1.1439999999999999},"min":{"1min":0.27300000000000002,"5min":0.214,"15min":0.214},"max":{"1min":2.4830000000000001,"5min":6.7759999999999998,"15min":8.952},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.82999999999999996,"15min":1.0620000000000001},"min":{"1min":0.27500000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.411,"5min":6.202,"15min":10.180999999999999},"last":0.55200000000000005}]},{"osd":6,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":1.1200000000000001,"15min":1.1539999999999999},"min":{"1min":0.26900000000000002,"5min":0.26500000000000001,"15min":0.23999999999999999},"max":{"1min":2.508,"5min":21.748000000000001,"15min":21.748000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":1.155,"15min":1.0960000000000001},"min":{"1min":0.32300000000000001,"5min":0.28599999999999998,"15min":0.26300000000000001},"max":{"1min":2.4420000000000002,"5min":22.187999999999999,"15min":22.187999999999999},"last":0.58999999999999997}]},{"osd":7,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":1.0009999999999999,"15min":1.25},"min":{"1min":0.313,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.98099999999999998,"5min":22.302,"15min":22.302},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.133,"15min":1.1479999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.26300000000000001},"max":{"1min":2.4580000000000002,"5min":21.779,"15min":21.779},"last":0.57399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:54:19.078 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:54:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:54:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:54:19.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:19 smithi008 ceph-mon[29816]: from='client.16149 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:54:19.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:19 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:54:19.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:19 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:54:19.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:19 smithi008 ceph-mon[29816]: pgmap v1037: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-02-10T07:54:19.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:19 smithi008 ceph-mon[29816]: osdmap e464: 8 total, 8 up, 8 in 2024-02-10T07:54:19.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:19 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:54:19.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:19 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:54:19.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:19 smithi008 ceph-mon[40743]: from='client.16149 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:54:19.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:19 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:54:19.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:19 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:54:19.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:19 smithi008 ceph-mon[40743]: pgmap v1037: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-02-10T07:54:19.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:19 smithi008 ceph-mon[40743]: osdmap e464: 8 total, 8 up, 8 in 2024-02-10T07:54:19.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:19 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:54:19.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:19 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:54:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:19 smithi170 ceph-mon[33394]: from='client.16149 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:54:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:19 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:54:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:19 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:54:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:19 smithi170 ceph-mon[33394]: pgmap v1037: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-02-10T07:54:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:19 smithi170 ceph-mon[33394]: osdmap e464: 8 total, 8 up, 8 in 2024-02-10T07:54:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:19 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:54:19.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:19 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:54:20.829 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:54:20.830 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:54:21.282 INFO:teuthology.orchestra.run.smithi008.stderr:noscrub is unset 2024-02-10T07:54:21.540 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:54:21.568 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:21 smithi008 ceph-mon[29816]: pgmap v1039: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:54:21.568 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:21 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/410957773' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:54:21.568 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:21 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:54:21.568 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:21 smithi008 ceph-mon[40743]: pgmap v1039: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:54:21.568 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:21 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/410957773' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:54:21.569 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:21 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:54:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:21 smithi170 ceph-mon[33394]: pgmap v1039: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:54:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:21 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/410957773' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:54:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:21 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-10T07:54:21.734 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-10T07:54:22.034 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:22.038 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:22.475 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:22 smithi008 ceph-mon[29816]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:54:22.475 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:22 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:54:22.475 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:22 smithi008 ceph-mon[29816]: osdmap e465: 8 total, 8 up, 8 in 2024-02-10T07:54:22.475 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:22 smithi008 ceph-mon[29816]: 6.2 scrub starts 2024-02-10T07:54:22.475 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:22 smithi008 ceph-mon[29816]: 6.2 scrub ok 2024-02-10T07:54:22.476 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:22 smithi008 ceph-mon[40743]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:54:22.476 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:22 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:54:22.476 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:22 smithi008 ceph-mon[40743]: osdmap e465: 8 total, 8 up, 8 in 2024-02-10T07:54:22.476 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:22 smithi008 ceph-mon[40743]: 6.2 scrub starts 2024-02-10T07:54:22.476 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:22 smithi008 ceph-mon[40743]: 6.2 scrub ok 2024-02-10T07:54:22.507 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-10T07:54:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:22 smithi170 ceph-mon[33394]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-10T07:54:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:22 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-10T07:54:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:22 smithi170 ceph-mon[33394]: osdmap e465: 8 total, 8 up, 8 in 2024-02-10T07:54:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:22 smithi170 ceph-mon[33394]: 6.2 scrub starts 2024-02-10T07:54:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:22 smithi170 ceph-mon[33394]: 6.2 scrub ok 2024-02-10T07:54:23.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:23 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:54:23.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:23 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:54:23.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:23 smithi008 ceph-mon[29816]: pgmap v1041: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:54:23.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:23 smithi008 ceph-mon[29816]: osdmap e466: 8 total, 8 up, 8 in 2024-02-10T07:54:23.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:23 smithi008 ceph-mon[29816]: 3.9 scrub starts 2024-02-10T07:54:23.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:23 smithi008 ceph-mon[29816]: 3.9 scrub ok 2024-02-10T07:54:23.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:23 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:54:23.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:23 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:54:23.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:23 smithi008 ceph-mon[40743]: pgmap v1041: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:54:23.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:23 smithi008 ceph-mon[40743]: osdmap e466: 8 total, 8 up, 8 in 2024-02-10T07:54:23.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:23 smithi008 ceph-mon[40743]: 3.9 scrub starts 2024-02-10T07:54:23.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:23 smithi008 ceph-mon[40743]: 3.9 scrub ok 2024-02-10T07:54:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:23 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:54:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:23 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:54:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:23 smithi170 ceph-mon[33394]: pgmap v1041: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:54:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:23 smithi170 ceph-mon[33394]: osdmap e466: 8 total, 8 up, 8 in 2024-02-10T07:54:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:23 smithi170 ceph-mon[33394]: 3.9 scrub starts 2024-02-10T07:54:23.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:23 smithi170 ceph-mon[33394]: 3.9 scrub ok 2024-02-10T07:54:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[29816]: 6.1c scrub starts 2024-02-10T07:54:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[29816]: 6.1c scrub ok 2024-02-10T07:54:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[29816]: 3.16 scrub starts 2024-02-10T07:54:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[29816]: 3.16 scrub ok 2024-02-10T07:54:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[29816]: 5.2 scrub starts 2024-02-10T07:54:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[29816]: 5.2 scrub ok 2024-02-10T07:54:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[29816]: 4.18 scrub starts 2024-02-10T07:54:24.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[29816]: 4.18 scrub ok 2024-02-10T07:54:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[40743]: 6.1c scrub starts 2024-02-10T07:54:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[40743]: 6.1c scrub ok 2024-02-10T07:54:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[40743]: 3.16 scrub starts 2024-02-10T07:54:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[40743]: 3.16 scrub ok 2024-02-10T07:54:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[40743]: 5.2 scrub starts 2024-02-10T07:54:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[40743]: 5.2 scrub ok 2024-02-10T07:54:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[40743]: 4.18 scrub starts 2024-02-10T07:54:24.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:24 smithi008 ceph-mon[40743]: 4.18 scrub ok 2024-02-10T07:54:24.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:54:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:54:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:24 smithi170 ceph-mon[33394]: 6.1c scrub starts 2024-02-10T07:54:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:24 smithi170 ceph-mon[33394]: 6.1c scrub ok 2024-02-10T07:54:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:24 smithi170 ceph-mon[33394]: 3.16 scrub starts 2024-02-10T07:54:24.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:24 smithi170 ceph-mon[33394]: 3.16 scrub ok 2024-02-10T07:54:24.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:24 smithi170 ceph-mon[33394]: 5.2 scrub starts 2024-02-10T07:54:24.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:24 smithi170 ceph-mon[33394]: 5.2 scrub ok 2024-02-10T07:54:24.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:24 smithi170 ceph-mon[33394]: 4.18 scrub starts 2024-02-10T07:54:24.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:24 smithi170 ceph-mon[33394]: 4.18 scrub ok 2024-02-10T07:54:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:25 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:25 smithi008 ceph-mon[29816]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:54:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:25 smithi008 ceph-mon[29816]: 5.1 scrub starts 2024-02-10T07:54:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:25 smithi008 ceph-mon[29816]: 5.1 scrub ok 2024-02-10T07:54:25.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:25 smithi008 ceph-mon[29816]: 6.5 scrub starts 2024-02-10T07:54:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:25 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:25 smithi008 ceph-mon[40743]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:54:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:25 smithi008 ceph-mon[40743]: 5.1 scrub starts 2024-02-10T07:54:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:25 smithi008 ceph-mon[40743]: 5.1 scrub ok 2024-02-10T07:54:25.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:25 smithi008 ceph-mon[40743]: 6.5 scrub starts 2024-02-10T07:54:25.590 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:54:25.590 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:54:25.603 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:25 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:25 smithi170 ceph-mon[33394]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:54:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:25 smithi170 ceph-mon[33394]: 5.1 scrub starts 2024-02-10T07:54:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:25 smithi170 ceph-mon[33394]: 5.1 scrub ok 2024-02-10T07:54:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:25 smithi170 ceph-mon[33394]: 6.5 scrub starts 2024-02-10T07:54:26.245 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:54:26.247 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:54:26.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:26 smithi008 ceph-mon[29816]: 6.5 scrub ok 2024-02-10T07:54:26.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:26 smithi008 ceph-mon[29816]: 3.2 scrub starts 2024-02-10T07:54:26.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:26 smithi008 ceph-mon[29816]: 3.2 scrub ok 2024-02-10T07:54:26.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:26 smithi008 ceph-mon[40743]: 6.5 scrub ok 2024-02-10T07:54:26.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:26 smithi008 ceph-mon[40743]: 3.2 scrub starts 2024-02-10T07:54:26.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:26 smithi008 ceph-mon[40743]: 3.2 scrub ok 2024-02-10T07:54:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:26 smithi170 ceph-mon[33394]: 6.5 scrub ok 2024-02-10T07:54:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:26 smithi170 ceph-mon[33394]: 3.2 scrub starts 2024-02-10T07:54:26.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:26 smithi170 ceph-mon[33394]: 3.2 scrub ok 2024-02-10T07:54:26.641 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-10T07:54:26.898 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:54:27.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:27 smithi008 ceph-mon[29816]: 3.1f scrub starts 2024-02-10T07:54:27.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:27 smithi008 ceph-mon[29816]: 3.1f scrub ok 2024-02-10T07:54:27.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:27 smithi008 ceph-mon[29816]: pgmap v1044: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:54:27.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:27 smithi008 ceph-mon[29816]: from='client.16161 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:54:27.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:27 smithi008 ceph-mon[29816]: 6.7 scrub starts 2024-02-10T07:54:27.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:27 smithi008 ceph-mon[29816]: osdmap e467: 8 total, 8 up, 8 in 2024-02-10T07:54:27.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:27 smithi008 ceph-mon[40743]: 3.1f scrub starts 2024-02-10T07:54:27.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:27 smithi008 ceph-mon[40743]: 3.1f scrub ok 2024-02-10T07:54:27.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:27 smithi008 ceph-mon[40743]: pgmap v1044: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:54:27.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:27 smithi008 ceph-mon[40743]: from='client.16161 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:54:27.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:27 smithi008 ceph-mon[40743]: 6.7 scrub starts 2024-02-10T07:54:27.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:27 smithi008 ceph-mon[40743]: osdmap e467: 8 total, 8 up, 8 in 2024-02-10T07:54:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:27 smithi170 ceph-mon[33394]: 3.1f scrub starts 2024-02-10T07:54:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:27 smithi170 ceph-mon[33394]: 3.1f scrub ok 2024-02-10T07:54:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:27 smithi170 ceph-mon[33394]: pgmap v1044: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:54:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:27 smithi170 ceph-mon[33394]: from='client.16161 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:54:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:27 smithi170 ceph-mon[33394]: 6.7 scrub starts 2024-02-10T07:54:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:27 smithi170 ceph-mon[33394]: osdmap e467: 8 total, 8 up, 8 in 2024-02-10T07:54:28.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:28 smithi008 ceph-mon[29816]: 6.7 scrub ok 2024-02-10T07:54:28.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:28 smithi008 ceph-mon[29816]: 5.15 scrub starts 2024-02-10T07:54:28.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:28 smithi008 ceph-mon[29816]: 5.15 scrub ok 2024-02-10T07:54:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:28 smithi008 ceph-mon[40743]: 6.7 scrub ok 2024-02-10T07:54:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:28 smithi008 ceph-mon[40743]: 5.15 scrub starts 2024-02-10T07:54:28.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:28 smithi008 ceph-mon[40743]: 5.15 scrub ok 2024-02-10T07:54:28.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:28 smithi170 ceph-mon[33394]: 6.7 scrub ok 2024-02-10T07:54:28.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:28 smithi170 ceph-mon[33394]: 5.15 scrub starts 2024-02-10T07:54:28.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:28 smithi170 ceph-mon[33394]: 5.15 scrub ok 2024-02-10T07:54:28.833 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":1044,"stamp":"2024-02-10T07:54:26.216374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3121,"num_read_kb":2967,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":1780341,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1297,"ondisk_log_size":1297,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":398,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5931472,"kb_used_data":406820,"kb_used_omap":19,"kb_used_meta":5524524,"kb_avail":744029744,"statfs":{"total":767960285184,"available":761886457856,"internally_reserved":0,"allocated":416583680,"data_stored":412190780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19634,"internal_metadata":5657113422},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.944731"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":869,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662074+0000","last_change":"2024-02-10T07:49:56.315126+0000","last_active":"2024-02-10T07:54:22.662074+0000","last_peered":"2024-02-10T07:54:22.662074+0000","last_clean":"2024-02-10T07:54:22.662074+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:54:22.662074+0000","last_undegraded":"2024-02-10T07:54:22.662074+0000","last_fullsized":"2024-02-10T07:54:22.662074+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.315029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.315029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":880,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753328+0000","last_change":"2024-02-10T07:49:08.138553+0000","last_active":"2024-02-10T07:54:22.753328+0000","last_peered":"2024-02-10T07:54:22.753328+0000","last_clean":"2024-02-10T07:54:22.753328+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:54:22.753328+0000","last_undegraded":"2024-02-10T07:54:22.753328+0000","last_fullsized":"2024-02-10T07:54:22.753328+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:08.138442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.138442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":917,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:25.476870+0000","last_change":"2024-02-10T07:54:25.476912+0000","last_active":"2024-02-10T07:54:25.476870+0000","last_peered":"2024-02-10T07:54:25.476870+0000","last_clean":"2024-02-10T07:54:25.476870+0000","last_became_active":"2024-02-10T07:50:43.281905+0000","last_became_peered":"2024-02-10T07:50:43.281905+0000","last_unstale":"2024-02-10T07:54:25.476870+0000","last_undegraded":"2024-02-10T07:54:25.476870+0000","last_fullsized":"2024-02-10T07:54:25.476870+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:54:25.476772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:54:25.476772+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:30.916677+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"430'25","reported_seq":939,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.661858+0000","last_change":"2024-02-10T07:46:37.008978+0000","last_active":"2024-02-10T07:54:22.661858+0000","last_peered":"2024-02-10T07:54:22.661858+0000","last_clean":"2024-02-10T07:54:22.661858+0000","last_became_active":"2024-02-10T07:46:10.717475+0000","last_became_peered":"2024-02-10T07:46:10.717475+0000","last_unstale":"2024-02-10T07:54:22.661858+0000","last_undegraded":"2024-02-10T07:54:22.661858+0000","last_fullsized":"2024-02-10T07:54:22.661858+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'19","last_scrub_stamp":"2024-02-10T07:46:37.008841+0000","last_deep_scrub":"287'19","last_deep_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_clean_scrub_stamp":"2024-02-10T07:46:37.008841+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":207,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-10T07:54:21.277420+0000","last_change":"2024-02-10T07:53:26.218018+0000","last_active":"2024-02-10T07:54:21.277420+0000","last_peered":"2024-02-10T07:54:21.277420+0000","last_clean":"2024-02-10T07:54:21.277420+0000","last_became_active":"2024-02-10T07:49:06.236235+0000","last_became_peered":"2024-02-10T07:49:06.236235+0000","last_unstale":"2024-02-10T07:54:21.277420+0000","last_undegraded":"2024-02-10T07:54:21.277420+0000","last_fullsized":"2024-02-10T07:54:21.277420+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:26.217749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:15.079428+0000","last_clean_scrub_stamp":"2024-02-10T07:53:26.217749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:52.906075+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":873,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.661854+0000","last_change":"2024-02-10T07:50:06.335390+0000","last_active":"2024-02-10T07:54:22.661854+0000","last_peered":"2024-02-10T07:54:22.661854+0000","last_clean":"2024-02-10T07:54:22.661854+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:54:22.661854+0000","last_undegraded":"2024-02-10T07:54:22.661854+0000","last_fullsized":"2024-02-10T07:54:22.661854+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:06.335262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:50:06.335262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":396,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.287825+0000","last_change":"2024-02-10T07:51:05.208836+0000","last_active":"2024-02-10T07:54:22.287825+0000","last_peered":"2024-02-10T07:54:22.287825+0000","last_clean":"2024-02-10T07:54:22.287825+0000","last_became_active":"2024-02-10T07:50:43.282449+0000","last_became_peered":"2024-02-10T07:50:43.282449+0000","last_unstale":"2024-02-10T07:54:22.287825+0000","last_undegraded":"2024-02-10T07:54:22.287825+0000","last_fullsized":"2024-02-10T07:54:22.287825+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:51:05.208755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:51:05.208755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.206989+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"430'39","reported_seq":168,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.713872+0000","last_change":"2024-02-10T07:50:43.318008+0000","last_active":"2024-02-10T07:54:22.713872+0000","last_peered":"2024-02-10T07:54:22.713872+0000","last_clean":"2024-02-10T07:54:22.713872+0000","last_became_active":"2024-02-10T07:50:43.290681+0000","last_became_peered":"2024-02-10T07:50:43.290681+0000","last_unstale":"2024-02-10T07:54:22.713872+0000","last_undegraded":"2024-02-10T07:54:22.713872+0000","last_fullsized":"2024-02-10T07:54:22.713872+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:48:13.230908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.230908+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.254457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":869,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.393721+0000","last_change":"2024-02-10T07:50:05.620136+0000","last_active":"2024-02-10T07:54:22.393721+0000","last_peered":"2024-02-10T07:54:22.393721+0000","last_clean":"2024-02-10T07:54:22.393721+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:54:22.393721+0000","last_undegraded":"2024-02-10T07:54:22.393721+0000","last_fullsized":"2024-02-10T07:54:22.393721+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:05.619958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:05.619958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":858,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.442497+0000","last_change":"2024-02-10T07:53:23.117713+0000","last_active":"2024-02-10T07:54:22.442497+0000","last_peered":"2024-02-10T07:54:22.442497+0000","last_clean":"2024-02-10T07:54:22.442497+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:54:22.442497+0000","last_undegraded":"2024-02-10T07:54:22.442497+0000","last_fullsized":"2024-02-10T07:54:22.442497+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:23.117597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:23.117597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:27.492528+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":410,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662802+0000","last_change":"2024-02-10T07:49:13.218634+0000","last_active":"2024-02-10T07:54:22.662802+0000","last_peered":"2024-02-10T07:54:22.662802+0000","last_clean":"2024-02-10T07:54:22.662802+0000","last_became_active":"2024-02-10T07:48:02.332960+0000","last_became_peered":"2024-02-10T07:48:02.332960+0000","last_unstale":"2024-02-10T07:54:22.662802+0000","last_undegraded":"2024-02-10T07:54:22.662802+0000","last_fullsized":"2024-02-10T07:54:22.662802+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.218549+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"430'21","reported_seq":903,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662260+0000","last_change":"2024-02-10T07:47:49.553645+0000","last_active":"2024-02-10T07:54:22.662260+0000","last_peered":"2024-02-10T07:54:22.662260+0000","last_clean":"2024-02-10T07:54:22.662260+0000","last_became_active":"2024-02-10T07:47:47.965930+0000","last_became_peered":"2024-02-10T07:47:47.965930+0000","last_unstale":"2024-02-10T07:54:22.662260+0000","last_undegraded":"2024-02-10T07:54:22.662260+0000","last_fullsized":"2024-02-10T07:54:22.662260+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:49.553517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.553517+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":781,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753426+0000","last_change":"2024-02-10T07:49:56.718482+0000","last_active":"2024-02-10T07:54:22.753426+0000","last_peered":"2024-02-10T07:54:22.753426+0000","last_clean":"2024-02-10T07:54:22.753426+0000","last_became_active":"2024-02-10T07:49:06.266689+0000","last_became_peered":"2024-02-10T07:49:06.266689+0000","last_unstale":"2024-02-10T07:54:22.753426+0000","last_undegraded":"2024-02-10T07:54:22.753426+0000","last_fullsized":"2024-02-10T07:54:22.753426+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.718382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.718382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":872,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288942+0000","last_change":"2024-02-10T07:48:18.069773+0000","last_active":"2024-02-10T07:54:22.288942+0000","last_peered":"2024-02-10T07:54:22.288942+0000","last_clean":"2024-02-10T07:54:22.288942+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:54:22.288942+0000","last_undegraded":"2024-02-10T07:54:22.288942+0000","last_fullsized":"2024-02-10T07:54:22.288942+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:18.069641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:18.069641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":398,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662281+0000","last_change":"2024-02-10T07:49:06.115547+0000","last_active":"2024-02-10T07:54:22.662281+0000","last_peered":"2024-02-10T07:54:22.662281+0000","last_clean":"2024-02-10T07:54:22.662281+0000","last_became_active":"2024-02-10T07:48:02.329859+0000","last_became_peered":"2024-02-10T07:48:02.329859+0000","last_unstale":"2024-02-10T07:54:22.662281+0000","last_undegraded":"2024-02-10T07:54:22.662281+0000","last_fullsized":"2024-02-10T07:54:22.662281+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.115466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.473686+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.115466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"430'9","reported_seq":885,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662617+0000","last_change":"2024-02-10T07:48:11.611397+0000","last_active":"2024-02-10T07:54:22.662617+0000","last_peered":"2024-02-10T07:54:22.662617+0000","last_clean":"2024-02-10T07:54:22.662617+0000","last_became_active":"2024-02-10T07:47:47.966093+0000","last_became_peered":"2024-02-10T07:47:47.966093+0000","last_unstale":"2024-02-10T07:54:22.662617+0000","last_undegraded":"2024-02-10T07:54:22.662617+0000","last_fullsized":"2024-02-10T07:54:22.662617+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7","last_scrub_stamp":"2024-02-10T07:48:11.608499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.608499+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":872,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662925+0000","last_change":"2024-02-10T07:48:13.621135+0000","last_active":"2024-02-10T07:54:22.662925+0000","last_peered":"2024-02-10T07:54:22.662925+0000","last_clean":"2024-02-10T07:54:22.662925+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:54:22.662925+0000","last_undegraded":"2024-02-10T07:54:22.662925+0000","last_fullsized":"2024-02-10T07:54:22.662925+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:13.620938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.620938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":878,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753904+0000","last_change":"2024-02-10T07:49:51.810158+0000","last_active":"2024-02-10T07:54:22.753904+0000","last_peered":"2024-02-10T07:54:22.753904+0000","last_clean":"2024-02-10T07:54:22.753904+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:54:22.753904+0000","last_undegraded":"2024-02-10T07:54:22.753904+0000","last_fullsized":"2024-02-10T07:54:22.753904+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:51.810062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:51.810062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":881,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.394105+0000","last_change":"2024-02-10T07:49:16.313655+0000","last_active":"2024-02-10T07:54:22.394105+0000","last_peered":"2024-02-10T07:54:22.394105+0000","last_clean":"2024-02-10T07:54:22.394105+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:54:22.394105+0000","last_undegraded":"2024-02-10T07:54:22.394105+0000","last_fullsized":"2024-02-10T07:54:22.394105+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:16.313573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.313573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"430'46","reported_seq":168,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.714094+0000","last_change":"2024-02-10T07:50:43.322322+0000","last_active":"2024-02-10T07:54:22.714094+0000","last_peered":"2024-02-10T07:54:22.714094+0000","last_clean":"2024-02-10T07:54:22.714094+0000","last_became_active":"2024-02-10T07:50:43.300153+0000","last_became_peered":"2024-02-10T07:50:43.300153+0000","last_unstale":"2024-02-10T07:54:22.714094+0000","last_undegraded":"2024-02-10T07:54:22.714094+0000","last_fullsized":"2024-02-10T07:54:22.714094+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:49:00.551605+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.551605+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.254539+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":575,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288145+0000","last_change":"2024-02-10T07:52:11.043473+0000","last_active":"2024-02-10T07:54:22.288145+0000","last_peered":"2024-02-10T07:54:22.288145+0000","last_clean":"2024-02-10T07:54:22.288145+0000","last_became_active":"2024-02-10T07:49:06.263790+0000","last_became_peered":"2024-02-10T07:49:06.263790+0000","last_unstale":"2024-02-10T07:54:22.288145+0000","last_undegraded":"2024-02-10T07:54:22.288145+0000","last_fullsized":"2024-02-10T07:54:22.288145+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:11.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:11.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.207029+0000","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":889,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662538+0000","last_change":"2024-02-10T07:50:43.286623+0000","last_active":"2024-02-10T07:54:22.662538+0000","last_peered":"2024-02-10T07:54:22.662538+0000","last_clean":"2024-02-10T07:54:22.662538+0000","last_became_active":"2024-02-10T07:50:43.285201+0000","last_became_peered":"2024-02-10T07:50:43.285201+0000","last_unstale":"2024-02-10T07:54:22.662538+0000","last_undegraded":"2024-02-10T07:54:22.662538+0000","last_fullsized":"2024-02-10T07:54:22.662538+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.680331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.680331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":893,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662374+0000","last_change":"2024-02-10T07:50:43.282584+0000","last_active":"2024-02-10T07:54:22.662374+0000","last_peered":"2024-02-10T07:54:22.662374+0000","last_clean":"2024-02-10T07:54:22.662374+0000","last_became_active":"2024-02-10T07:50:43.281382+0000","last_became_peered":"2024-02-10T07:50:43.281382+0000","last_unstale":"2024-02-10T07:54:22.662374+0000","last_undegraded":"2024-02-10T07:54:22.662374+0000","last_fullsized":"2024-02-10T07:54:22.662374+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:04.396771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.396771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"430'24","reported_seq":848,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753787+0000","last_change":"2024-02-10T07:49:58.618564+0000","last_active":"2024-02-10T07:54:22.753787+0000","last_peered":"2024-02-10T07:54:22.753787+0000","last_clean":"2024-02-10T07:54:22.753787+0000","last_became_active":"2024-02-10T07:49:06.263261+0000","last_became_peered":"2024-02-10T07:49:06.263261+0000","last_unstale":"2024-02-10T07:54:22.753787+0000","last_undegraded":"2024-02-10T07:54:22.753787+0000","last_fullsized":"2024-02-10T07:54:22.753787+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'18","last_scrub_stamp":"2024-02-10T07:49:58.618464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:58.618464+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":912,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.819304+0000","last_change":"2024-02-10T07:54:22.819331+0000","last_active":"2024-02-10T07:54:22.819304+0000","last_peered":"2024-02-10T07:54:22.819304+0000","last_clean":"2024-02-10T07:54:22.819304+0000","last_became_active":"2024-02-10T07:50:43.285254+0000","last_became_peered":"2024-02-10T07:50:43.285254+0000","last_unstale":"2024-02-10T07:54:22.819304+0000","last_undegraded":"2024-02-10T07:54:22.819304+0000","last_fullsized":"2024-02-10T07:54:22.819304+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:54:22.819236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:54:22.819236+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:42.843356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"414'9","reported_seq":1188,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288021+0000","last_change":"2024-02-10T07:51:13.307061+0000","last_active":"2024-02-10T07:54:22.288021+0000","last_peered":"2024-02-10T07:54:22.288021+0000","last_clean":"2024-02-10T07:54:22.288021+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:54:22.288021+0000","last_undegraded":"2024-02-10T07:54:22.288021+0000","last_fullsized":"2024-02-10T07:54:22.288021+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'9","last_scrub_stamp":"2024-02-10T07:51:13.306986+0000","last_deep_scrub":"414'9","last_deep_scrub_stamp":"2024-02-10T07:51:13.306986+0000","last_clean_scrub_stamp":"2024-02-10T07:51:13.306986+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.207008+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":889,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753293+0000","last_change":"2024-02-10T07:48:30.384635+0000","last_active":"2024-02-10T07:54:22.753293+0000","last_peered":"2024-02-10T07:54:22.753293+0000","last_clean":"2024-02-10T07:54:22.753293+0000","last_became_active":"2024-02-10T07:47:47.966127+0000","last_became_peered":"2024-02-10T07:47:47.966127+0000","last_unstale":"2024-02-10T07:54:22.753293+0000","last_undegraded":"2024-02-10T07:54:22.753293+0000","last_fullsized":"2024-02-10T07:54:22.753293+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:30.384546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:30.384546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"430'35","reported_seq":968,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.394276+0000","last_change":"2024-02-10T07:49:06.308245+0000","last_active":"2024-02-10T07:54:22.394276+0000","last_peered":"2024-02-10T07:54:22.394276+0000","last_clean":"2024-02-10T07:54:22.394276+0000","last_became_active":"2024-02-10T07:49:06.261487+0000","last_became_peered":"2024-02-10T07:49:06.261487+0000","last_unstale":"2024-02-10T07:54:22.394276+0000","last_undegraded":"2024-02-10T07:54:22.394276+0000","last_fullsized":"2024-02-10T07:54:22.394276+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:48:12.278982+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-02-10T07:42:17.820959+0000","last_clean_scrub_stamp":"2024-02-10T07:48:12.278982+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":875,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753981+0000","last_change":"2024-02-10T07:49:11.231819+0000","last_active":"2024-02-10T07:54:22.753981+0000","last_peered":"2024-02-10T07:54:22.753981+0000","last_clean":"2024-02-10T07:54:22.753981+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:54:22.753981+0000","last_undegraded":"2024-02-10T07:54:22.753981+0000","last_fullsized":"2024-02-10T07:54:22.753981+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.231704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.231704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":879,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.394464+0000","last_change":"2024-02-10T07:50:43.289955+0000","last_active":"2024-02-10T07:54:22.394464+0000","last_peered":"2024-02-10T07:54:22.394464+0000","last_clean":"2024-02-10T07:54:22.394464+0000","last_became_active":"2024-02-10T07:50:43.289011+0000","last_became_peered":"2024-02-10T07:50:43.289011+0000","last_unstale":"2024-02-10T07:54:22.394464+0000","last_undegraded":"2024-02-10T07:54:22.394464+0000","last_fullsized":"2024-02-10T07:54:22.394464+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.208451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:06.200009+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.208451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":384,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.393619+0000","last_change":"2024-02-10T07:53:21.437757+0000","last_active":"2024-02-10T07:54:22.393619+0000","last_peered":"2024-02-10T07:54:22.393619+0000","last_clean":"2024-02-10T07:54:22.393619+0000","last_became_active":"2024-02-10T07:49:06.258789+0000","last_became_peered":"2024-02-10T07:49:06.258789+0000","last_unstale":"2024-02-10T07:54:22.393619+0000","last_undegraded":"2024-02-10T07:54:22.393619+0000","last_fullsized":"2024-02-10T07:54:22.393619+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:21.437653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:53:21.437653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.318460+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"430'27","reported_seq":945,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288029+0000","last_change":"2024-02-10T07:52:11.834589+0000","last_active":"2024-02-10T07:54:22.288029+0000","last_peered":"2024-02-10T07:54:22.288029+0000","last_clean":"2024-02-10T07:54:22.288029+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:54:22.288029+0000","last_undegraded":"2024-02-10T07:54:22.288029+0000","last_fullsized":"2024-02-10T07:54:22.288029+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'27","last_scrub_stamp":"2024-02-10T07:52:11.834490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:52:11.834490+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.207109+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":891,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288797+0000","last_change":"2024-02-10T07:50:43.284049+0000","last_active":"2024-02-10T07:54:22.288797+0000","last_peered":"2024-02-10T07:54:22.288797+0000","last_clean":"2024-02-10T07:54:22.288797+0000","last_became_active":"2024-02-10T07:50:43.283688+0000","last_became_peered":"2024-02-10T07:50:43.283688+0000","last_unstale":"2024-02-10T07:54:22.288797+0000","last_undegraded":"2024-02-10T07:54:22.288797+0000","last_fullsized":"2024-02-10T07:54:22.288797+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.087614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.087614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":896,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.287658+0000","last_change":"2024-02-10T07:52:22.845994+0000","last_active":"2024-02-10T07:54:22.287658+0000","last_peered":"2024-02-10T07:54:22.287658+0000","last_clean":"2024-02-10T07:54:22.287658+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:54:22.287658+0000","last_undegraded":"2024-02-10T07:54:22.287658+0000","last_fullsized":"2024-02-10T07:54:22.287658+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:22.845773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:52:22.845773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.207066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":905,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.394271+0000","last_change":"2024-02-10T07:49:06.259100+0000","last_active":"2024-02-10T07:54:22.394271+0000","last_peered":"2024-02-10T07:54:22.394271+0000","last_clean":"2024-02-10T07:54:22.394271+0000","last_became_active":"2024-02-10T07:49:06.258763+0000","last_became_peered":"2024-02-10T07:49:06.258763+0000","last_unstale":"2024-02-10T07:54:22.394271+0000","last_undegraded":"2024-02-10T07:54:22.394271+0000","last_fullsized":"2024-02-10T07:54:22.394271+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:29.188832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:29.188832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"430'14","reported_seq":219,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-10T07:54:21.277089+0000","last_change":"2024-02-10T07:53:28.121930+0000","last_active":"2024-02-10T07:54:21.277089+0000","last_peered":"2024-02-10T07:54:21.277089+0000","last_clean":"2024-02-10T07:54:21.277089+0000","last_became_active":"2024-02-10T07:49:06.250577+0000","last_became_peered":"2024-02-10T07:49:06.250577+0000","last_unstale":"2024-02-10T07:54:21.277089+0000","last_undegraded":"2024-02-10T07:54:21.277089+0000","last_fullsized":"2024-02-10T07:54:21.277089+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'14","last_scrub_stamp":"2024-02-10T07:53:28.117356+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:53:28.117356+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:28.261781+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":379,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.393790+0000","last_change":"2024-02-10T07:53:33.873865+0000","last_active":"2024-02-10T07:54:22.393790+0000","last_peered":"2024-02-10T07:54:22.393790+0000","last_clean":"2024-02-10T07:54:22.393790+0000","last_became_active":"2024-02-10T07:49:06.259341+0000","last_became_peered":"2024-02-10T07:49:06.259341+0000","last_unstale":"2024-02-10T07:54:22.393790+0000","last_undegraded":"2024-02-10T07:54:22.393790+0000","last_fullsized":"2024-02-10T07:54:22.393790+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:33.873724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:53:33.873724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:51.425277+0000","scrub_duration":324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":873,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288312+0000","last_change":"2024-02-10T07:48:24.171978+0000","last_active":"2024-02-10T07:54:22.288312+0000","last_peered":"2024-02-10T07:54:22.288312+0000","last_clean":"2024-02-10T07:54:22.288312+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:54:22.288312+0000","last_undegraded":"2024-02-10T07:54:22.288312+0000","last_fullsized":"2024-02-10T07:54:22.288312+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:24.171859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:24.171859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":830,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.442931+0000","last_change":"2024-02-10T07:52:27.104889+0000","last_active":"2024-02-10T07:54:22.442931+0000","last_peered":"2024-02-10T07:54:22.442931+0000","last_clean":"2024-02-10T07:54:22.442931+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:54:22.442931+0000","last_undegraded":"2024-02-10T07:54:22.442931+0000","last_fullsized":"2024-02-10T07:54:22.442931+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:27.104803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:52:27.104803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"430'27","reported_seq":852,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288562+0000","last_change":"2024-02-10T07:49:53.245490+0000","last_active":"2024-02-10T07:54:22.288562+0000","last_peered":"2024-02-10T07:54:22.288562+0000","last_clean":"2024-02-10T07:54:22.288562+0000","last_became_active":"2024-02-10T07:49:06.239701+0000","last_became_peered":"2024-02-10T07:49:06.239701+0000","last_unstale":"2024-02-10T07:54:22.288562+0000","last_undegraded":"2024-02-10T07:54:22.288562+0000","last_fullsized":"2024-02-10T07:54:22.288562+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:53.245410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:53.245410+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":870,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.393516+0000","last_change":"2024-02-10T07:49:52.530010+0000","last_active":"2024-02-10T07:54:22.393516+0000","last_peered":"2024-02-10T07:54:22.393516+0000","last_clean":"2024-02-10T07:54:22.393516+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:54:22.393516+0000","last_undegraded":"2024-02-10T07:54:22.393516+0000","last_fullsized":"2024-02-10T07:54:22.393516+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.529908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:59.179342+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.529908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":379,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.393977+0000","last_change":"2024-02-10T07:53:19.493460+0000","last_active":"2024-02-10T07:54:22.393977+0000","last_peered":"2024-02-10T07:54:22.393977+0000","last_clean":"2024-02-10T07:54:22.393977+0000","last_became_active":"2024-02-10T07:49:06.265267+0000","last_became_peered":"2024-02-10T07:49:06.265267+0000","last_unstale":"2024-02-10T07:54:22.393977+0000","last_undegraded":"2024-02-10T07:54:22.393977+0000","last_fullsized":"2024-02-10T07:54:22.393977+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:19.492989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:19.492989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:32.822823+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":890,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288376+0000","last_change":"2024-02-10T07:50:00.265308+0000","last_active":"2024-02-10T07:54:22.288376+0000","last_peered":"2024-02-10T07:54:22.288376+0000","last_clean":"2024-02-10T07:54:22.288376+0000","last_became_active":"2024-02-10T07:47:47.966472+0000","last_became_peered":"2024-02-10T07:47:47.966472+0000","last_unstale":"2024-02-10T07:54:22.288376+0000","last_undegraded":"2024-02-10T07:54:22.288376+0000","last_fullsized":"2024-02-10T07:54:22.288376+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.265208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.265208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"430'25","reported_seq":723,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753345+0000","last_change":"2024-02-10T07:50:43.310533+0000","last_active":"2024-02-10T07:54:22.753345+0000","last_peered":"2024-02-10T07:54:22.753345+0000","last_clean":"2024-02-10T07:54:22.753345+0000","last_became_active":"2024-02-10T07:50:43.296447+0000","last_became_peered":"2024-02-10T07:50:43.296447+0000","last_unstale":"2024-02-10T07:54:22.753345+0000","last_undegraded":"2024-02-10T07:54:22.753345+0000","last_fullsized":"2024-02-10T07:54:22.753345+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:09.116615+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-02-10T07:39:32.934269+0000","last_clean_scrub_stamp":"2024-02-10T07:49:09.116615+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":567,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.393716+0000","last_change":"2024-02-10T07:53:35.716416+0000","last_active":"2024-02-10T07:54:22.393716+0000","last_peered":"2024-02-10T07:54:22.393716+0000","last_clean":"2024-02-10T07:54:22.393716+0000","last_became_active":"2024-02-10T07:49:06.260569+0000","last_became_peered":"2024-02-10T07:49:06.260569+0000","last_unstale":"2024-02-10T07:54:22.393716+0000","last_undegraded":"2024-02-10T07:54:22.393716+0000","last_fullsized":"2024-02-10T07:54:22.393716+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:35.715987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:53:35.715987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:50.372195+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":903,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753737+0000","last_change":"2024-02-10T07:50:02.685941+0000","last_active":"2024-02-10T07:54:22.753737+0000","last_peered":"2024-02-10T07:54:22.753737+0000","last_clean":"2024-02-10T07:54:22.753737+0000","last_became_active":"2024-02-10T07:49:06.252380+0000","last_became_peered":"2024-02-10T07:49:06.252380+0000","last_unstale":"2024-02-10T07:54:22.753737+0000","last_undegraded":"2024-02-10T07:54:22.753737+0000","last_fullsized":"2024-02-10T07:54:22.753737+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_clean_scrub_stamp":"2024-02-10T07:50:02.685831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":881,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662349+0000","last_change":"2024-02-10T07:50:00.345087+0000","last_active":"2024-02-10T07:54:22.662349+0000","last_peered":"2024-02-10T07:54:22.662349+0000","last_clean":"2024-02-10T07:54:22.662349+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:54:22.662349+0000","last_undegraded":"2024-02-10T07:54:22.662349+0000","last_fullsized":"2024-02-10T07:54:22.662349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.344993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.344993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"430'32","reported_seq":953,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662479+0000","last_change":"2024-02-10T07:47:43.494877+0000","last_active":"2024-02-10T07:54:22.662479+0000","last_peered":"2024-02-10T07:54:22.662479+0000","last_clean":"2024-02-10T07:54:22.662479+0000","last_became_active":"2024-02-10T07:46:10.721336+0000","last_became_peered":"2024-02-10T07:46:10.721336+0000","last_unstale":"2024-02-10T07:54:22.662479+0000","last_undegraded":"2024-02-10T07:54:22.662479+0000","last_fullsized":"2024-02-10T07:54:22.662479+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:47:43.494350+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.494350+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":884,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.394019+0000","last_change":"2024-02-10T07:53:42.705767+0000","last_active":"2024-02-10T07:54:22.394019+0000","last_peered":"2024-02-10T07:54:22.394019+0000","last_clean":"2024-02-10T07:54:22.394019+0000","last_became_active":"2024-02-10T07:49:06.261570+0000","last_became_peered":"2024-02-10T07:49:06.261570+0000","last_unstale":"2024-02-10T07:54:22.394019+0000","last_undegraded":"2024-02-10T07:54:22.394019+0000","last_fullsized":"2024-02-10T07:54:22.394019+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:42.705250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:53:42.705250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:05.715285+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"356'11","reported_seq":388,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753823+0000","last_change":"2024-02-10T07:48:25.339974+0000","last_active":"2024-02-10T07:54:22.753823+0000","last_peered":"2024-02-10T07:54:22.753823+0000","last_clean":"2024-02-10T07:54:22.753823+0000","last_became_active":"2024-02-10T07:48:02.371888+0000","last_became_peered":"2024-02-10T07:48:02.371888+0000","last_unstale":"2024-02-10T07:54:22.753823+0000","last_undegraded":"2024-02-10T07:54:22.753823+0000","last_fullsized":"2024-02-10T07:54:22.753823+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'11","last_scrub_stamp":"2024-02-10T07:48:25.339855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:25.339855+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":901,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288281+0000","last_change":"2024-02-10T07:50:43.283263+0000","last_active":"2024-02-10T07:54:22.288281+0000","last_peered":"2024-02-10T07:54:22.288281+0000","last_clean":"2024-02-10T07:54:22.288281+0000","last_became_active":"2024-02-10T07:50:43.282925+0000","last_became_peered":"2024-02-10T07:50:43.282925+0000","last_unstale":"2024-02-10T07:54:22.288281+0000","last_undegraded":"2024-02-10T07:54:22.288281+0000","last_fullsized":"2024-02-10T07:54:22.288281+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.335455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.335455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"430'35","reported_seq":517,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753589+0000","last_change":"2024-02-10T07:49:12.150442+0000","last_active":"2024-02-10T07:54:22.753589+0000","last_peered":"2024-02-10T07:54:22.753589+0000","last_clean":"2024-02-10T07:54:22.753589+0000","last_became_active":"2024-02-10T07:48:02.329187+0000","last_became_peered":"2024-02-10T07:48:02.329187+0000","last_unstale":"2024-02-10T07:54:22.753589+0000","last_undegraded":"2024-02-10T07:54:22.753589+0000","last_fullsized":"2024-02-10T07:54:22.753589+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:49:12.150298+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:49:12.150298+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":882,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.663120+0000","last_change":"2024-02-10T07:50:43.285390+0000","last_active":"2024-02-10T07:54:22.663120+0000","last_peered":"2024-02-10T07:54:22.663120+0000","last_clean":"2024-02-10T07:54:22.663120+0000","last_became_active":"2024-02-10T07:50:43.284846+0000","last_became_peered":"2024-02-10T07:50:43.284846+0000","last_unstale":"2024-02-10T07:54:22.663120+0000","last_undegraded":"2024-02-10T07:54:22.663120+0000","last_fullsized":"2024-02-10T07:54:22.663120+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:01.513972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:01.513972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"405'15","reported_seq":545,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288932+0000","last_change":"2024-02-10T07:50:43.310228+0000","last_active":"2024-02-10T07:54:22.288932+0000","last_peered":"2024-02-10T07:54:22.288932+0000","last_clean":"2024-02-10T07:54:22.288932+0000","last_became_active":"2024-02-10T07:50:43.285213+0000","last_became_peered":"2024-02-10T07:50:43.285213+0000","last_unstale":"2024-02-10T07:54:22.288932+0000","last_undegraded":"2024-02-10T07:54:22.288932+0000","last_fullsized":"2024-02-10T07:54:22.288932+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'13","last_scrub_stamp":"2024-02-10T07:49:00.366610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.366610+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":896,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.393708+0000","last_change":"2024-02-10T07:50:07.501014+0000","last_active":"2024-02-10T07:54:22.393708+0000","last_peered":"2024-02-10T07:54:22.393708+0000","last_clean":"2024-02-10T07:54:22.393708+0000","last_became_active":"2024-02-10T07:49:06.257043+0000","last_became_peered":"2024-02-10T07:49:06.257043+0000","last_unstale":"2024-02-10T07:54:22.393708+0000","last_undegraded":"2024-02-10T07:54:22.393708+0000","last_fullsized":"2024-02-10T07:54:22.393708+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.500890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"430'21","reported_seq":643,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288712+0000","last_change":"2024-02-10T07:49:06.277154+0000","last_active":"2024-02-10T07:54:22.288712+0000","last_peered":"2024-02-10T07:54:22.288712+0000","last_clean":"2024-02-10T07:54:22.288712+0000","last_became_active":"2024-02-10T07:49:06.255239+0000","last_became_peered":"2024-02-10T07:49:06.255239+0000","last_unstale":"2024-02-10T07:54:22.288712+0000","last_undegraded":"2024-02-10T07:54:22.288712+0000","last_fullsized":"2024-02-10T07:54:22.288712+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:39.610017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:39.610017+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":134,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.714192+0000","last_change":"2024-02-10T07:50:43.282730+0000","last_active":"2024-02-10T07:54:22.714192+0000","last_peered":"2024-02-10T07:54:22.714192+0000","last_clean":"2024-02-10T07:54:22.714192+0000","last_became_active":"2024-02-10T07:50:43.281549+0000","last_became_peered":"2024-02-10T07:50:43.281549+0000","last_unstale":"2024-02-10T07:54:22.714192+0000","last_undegraded":"2024-02-10T07:54:22.714192+0000","last_fullsized":"2024-02-10T07:54:22.714192+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:44.324046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:49:44.324046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.254603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":237,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.287607+0000","last_change":"2024-02-10T07:52:34.860375+0000","last_active":"2024-02-10T07:54:22.287607+0000","last_peered":"2024-02-10T07:54:22.287607+0000","last_clean":"2024-02-10T07:54:22.287607+0000","last_became_active":"2024-02-10T07:49:06.262933+0000","last_became_peered":"2024-02-10T07:49:06.262933+0000","last_unstale":"2024-02-10T07:54:22.287607+0000","last_undegraded":"2024-02-10T07:54:22.287607+0000","last_fullsized":"2024-02-10T07:54:22.287607+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:34.860273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:52:34.860273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.207182+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":889,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288262+0000","last_change":"2024-02-10T07:49:59.278666+0000","last_active":"2024-02-10T07:54:22.288262+0000","last_peered":"2024-02-10T07:54:22.288262+0000","last_clean":"2024-02-10T07:54:22.288262+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:54:22.288262+0000","last_undegraded":"2024-02-10T07:54:22.288262+0000","last_fullsized":"2024-02-10T07:54:22.288262+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.278574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.278574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"430'25","reported_seq":958,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.394371+0000","last_change":"2024-02-10T07:53:44.569050+0000","last_active":"2024-02-10T07:54:22.394371+0000","last_peered":"2024-02-10T07:54:22.394371+0000","last_clean":"2024-02-10T07:54:22.394371+0000","last_became_active":"2024-02-10T07:49:06.260962+0000","last_became_peered":"2024-02-10T07:49:06.260962+0000","last_unstale":"2024-02-10T07:54:22.394371+0000","last_undegraded":"2024-02-10T07:54:22.394371+0000","last_fullsized":"2024-02-10T07:54:22.394371+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'25","last_scrub_stamp":"2024-02-10T07:53:44.568910+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:53:44.568910+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:59.428046+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":869,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288386+0000","last_change":"2024-02-10T07:48:58.511630+0000","last_active":"2024-02-10T07:54:22.288386+0000","last_peered":"2024-02-10T07:54:22.288386+0000","last_clean":"2024-02-10T07:54:22.288386+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:54:22.288386+0000","last_undegraded":"2024-02-10T07:54:22.288386+0000","last_fullsized":"2024-02-10T07:54:22.288386+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:58.511251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.511251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":207,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-10T07:54:21.277647+0000","last_change":"2024-02-10T07:53:42.214117+0000","last_active":"2024-02-10T07:54:21.277647+0000","last_peered":"2024-02-10T07:54:21.277647+0000","last_clean":"2024-02-10T07:54:21.277647+0000","last_became_active":"2024-02-10T07:49:06.250832+0000","last_became_peered":"2024-02-10T07:49:06.250832+0000","last_unstale":"2024-02-10T07:54:21.277647+0000","last_undegraded":"2024-02-10T07:54:21.277647+0000","last_fullsized":"2024-02-10T07:54:21.277647+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:42.213981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:42.213981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:57.611769+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":921,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.287070+0000","last_change":"2024-02-10T07:52:16.809094+0000","last_active":"2024-02-10T07:54:22.287070+0000","last_peered":"2024-02-10T07:54:22.287070+0000","last_clean":"2024-02-10T07:54:22.287070+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:54:22.287070+0000","last_undegraded":"2024-02-10T07:54:22.287070+0000","last_fullsized":"2024-02-10T07:54:22.287070+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:16.808978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:52:16.808978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.207145+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"430'18","reported_seq":224,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-10T07:54:21.277523+0000","last_change":"2024-02-10T07:53:29.079571+0000","last_active":"2024-02-10T07:54:21.277523+0000","last_peered":"2024-02-10T07:54:21.277523+0000","last_clean":"2024-02-10T07:54:21.277523+0000","last_became_active":"2024-02-10T07:49:06.258281+0000","last_became_peered":"2024-02-10T07:49:06.258281+0000","last_unstale":"2024-02-10T07:54:21.277523+0000","last_undegraded":"2024-02-10T07:54:21.277523+0000","last_fullsized":"2024-02-10T07:54:21.277523+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'18","last_scrub_stamp":"2024-02-10T07:53:29.076952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:53:29.076952+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:30.380886+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":410,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.287496+0000","last_change":"2024-02-10T07:52:21.875347+0000","last_active":"2024-02-10T07:54:22.287496+0000","last_peered":"2024-02-10T07:54:22.287496+0000","last_clean":"2024-02-10T07:54:22.287496+0000","last_became_active":"2024-02-10T07:48:02.366002+0000","last_became_peered":"2024-02-10T07:48:02.366002+0000","last_unstale":"2024-02-10T07:54:22.287496+0000","last_undegraded":"2024-02-10T07:54:22.287496+0000","last_fullsized":"2024-02-10T07:54:22.287496+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:21.875258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:21.875258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.207200+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"245'4","reported_seq":904,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288933+0000","last_change":"2024-02-10T07:49:08.311056+0000","last_active":"2024-02-10T07:54:22.288933+0000","last_peered":"2024-02-10T07:54:22.288933+0000","last_clean":"2024-02-10T07:54:22.288933+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:54:22.288933+0000","last_undegraded":"2024-02-10T07:54:22.288933+0000","last_fullsized":"2024-02-10T07:54:22.288933+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'4","last_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_deep_scrub":"245'4","last_deep_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.310917+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"430'46","reported_seq":647,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288207+0000","last_change":"2024-02-10T07:48:11.967227+0000","last_active":"2024-02-10T07:54:22.288207+0000","last_peered":"2024-02-10T07:54:22.288207+0000","last_clean":"2024-02-10T07:54:22.288207+0000","last_became_active":"2024-02-10T07:48:02.330130+0000","last_became_peered":"2024-02-10T07:48:02.330130+0000","last_unstale":"2024-02-10T07:54:22.288207+0000","last_undegraded":"2024-02-10T07:54:22.288207+0000","last_fullsized":"2024-02-10T07:54:22.288207+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:48:11.966957+0000","last_deep_scrub":"246'22","last_deep_scrub_stamp":"2024-02-10T07:41:26.771112+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.966957+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":1161,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288561+0000","last_change":"2024-02-10T07:48:53.423840+0000","last_active":"2024-02-10T07:54:22.288561+0000","last_peered":"2024-02-10T07:54:22.288561+0000","last_clean":"2024-02-10T07:54:22.288561+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:54:22.288561+0000","last_undegraded":"2024-02-10T07:54:22.288561+0000","last_fullsized":"2024-02-10T07:54:22.288561+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:48:53.423613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:53.423613+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":713,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-10T07:54:21.277600+0000","last_change":"2024-02-10T07:53:32.243227+0000","last_active":"2024-02-10T07:54:21.277600+0000","last_peered":"2024-02-10T07:54:21.277600+0000","last_clean":"2024-02-10T07:54:21.277600+0000","last_became_active":"2024-02-10T07:47:47.966183+0000","last_became_peered":"2024-02-10T07:47:47.966183+0000","last_unstale":"2024-02-10T07:54:21.277600+0000","last_undegraded":"2024-02-10T07:54:21.277600+0000","last_fullsized":"2024-02-10T07:54:21.277600+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:32.242307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:53:32.242307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:47.454633+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":882,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288850+0000","last_change":"2024-02-10T07:48:56.416159+0000","last_active":"2024-02-10T07:54:22.288850+0000","last_peered":"2024-02-10T07:54:22.288850+0000","last_clean":"2024-02-10T07:54:22.288850+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:54:22.288850+0000","last_undegraded":"2024-02-10T07:54:22.288850+0000","last_fullsized":"2024-02-10T07:54:22.288850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.416037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.416037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"430'29","reported_seq":997,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662374+0000","last_change":"2024-02-10T07:49:43.339027+0000","last_active":"2024-02-10T07:54:22.662374+0000","last_peered":"2024-02-10T07:54:22.662374+0000","last_clean":"2024-02-10T07:54:22.662374+0000","last_became_active":"2024-02-10T07:49:06.243569+0000","last_became_peered":"2024-02-10T07:49:06.243569+0000","last_unstale":"2024-02-10T07:54:22.662374+0000","last_undegraded":"2024-02-10T07:54:22.662374+0000","last_fullsized":"2024-02-10T07:54:22.662374+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-02-10T07:49:43.338928+0000","last_deep_scrub":"245'17","last_deep_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_clean_scrub_stamp":"2024-02-10T07:49:43.338928+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":134,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.713695+0000","last_change":"2024-02-10T07:50:43.285105+0000","last_active":"2024-02-10T07:54:22.713695+0000","last_peered":"2024-02-10T07:54:22.713695+0000","last_clean":"2024-02-10T07:54:22.713695+0000","last_became_active":"2024-02-10T07:50:43.284749+0000","last_became_peered":"2024-02-10T07:50:43.284749+0000","last_unstale":"2024-02-10T07:54:22.713695+0000","last_undegraded":"2024-02-10T07:54:22.713695+0000","last_fullsized":"2024-02-10T07:54:22.713695+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:04.308807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.308807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.254521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":903,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.287239+0000","last_change":"2024-02-10T07:50:43.281637+0000","last_active":"2024-02-10T07:54:22.287239+0000","last_peered":"2024-02-10T07:54:22.287239+0000","last_clean":"2024-02-10T07:54:22.287239+0000","last_became_active":"2024-02-10T07:50:43.281262+0000","last_became_peered":"2024-02-10T07:50:43.281262+0000","last_unstale":"2024-02-10T07:54:22.287239+0000","last_undegraded":"2024-02-10T07:54:22.287239+0000","last_fullsized":"2024-02-10T07:54:22.287239+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.361139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":134,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.713952+0000","last_change":"2024-02-10T07:50:43.281022+0000","last_active":"2024-02-10T07:54:22.713952+0000","last_peered":"2024-02-10T07:54:22.713952+0000","last_clean":"2024-02-10T07:54:22.713952+0000","last_became_active":"2024-02-10T07:50:43.280617+0000","last_became_peered":"2024-02-10T07:50:43.280617+0000","last_unstale":"2024-02-10T07:54:22.713952+0000","last_undegraded":"2024-02-10T07:54:22.713952+0000","last_fullsized":"2024-02-10T07:54:22.713952+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:21.172601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:21.172601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.254490+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":899,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288806+0000","last_change":"2024-02-10T07:50:07.252577+0000","last_active":"2024-02-10T07:54:22.288806+0000","last_peered":"2024-02-10T07:54:22.288806+0000","last_clean":"2024-02-10T07:54:22.288806+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:54:22.288806+0000","last_undegraded":"2024-02-10T07:54:22.288806+0000","last_fullsized":"2024-02-10T07:54:22.288806+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.252458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.252458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"430'51","reported_seq":980,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.287485+0000","last_change":"2024-02-10T07:51:10.348915+0000","last_active":"2024-02-10T07:54:22.287485+0000","last_peered":"2024-02-10T07:54:22.287485+0000","last_clean":"2024-02-10T07:54:22.287485+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:54:22.287485+0000","last_undegraded":"2024-02-10T07:54:22.287485+0000","last_fullsized":"2024-02-10T07:54:22.287485+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'35","last_scrub_stamp":"2024-02-10T07:51:10.348752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:51:10.348752+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.206969+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":892,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.393790+0000","last_change":"2024-02-10T07:50:43.284817+0000","last_active":"2024-02-10T07:54:22.393790+0000","last_peered":"2024-02-10T07:54:22.393790+0000","last_clean":"2024-02-10T07:54:22.393790+0000","last_became_active":"2024-02-10T07:50:43.284289+0000","last_became_peered":"2024-02-10T07:50:43.284289+0000","last_unstale":"2024-02-10T07:54:22.393790+0000","last_undegraded":"2024-02-10T07:54:22.393790+0000","last_fullsized":"2024-02-10T07:54:22.393790+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:00.336175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.336175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":134,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.713841+0000","last_change":"2024-02-10T07:50:43.295279+0000","last_active":"2024-02-10T07:54:22.713841+0000","last_peered":"2024-02-10T07:54:22.713841+0000","last_clean":"2024-02-10T07:54:22.713841+0000","last_became_active":"2024-02-10T07:50:43.294972+0000","last_became_peered":"2024-02-10T07:50:43.294972+0000","last_unstale":"2024-02-10T07:54:22.713841+0000","last_undegraded":"2024-02-10T07:54:22.713841+0000","last_fullsized":"2024-02-10T07:54:22.713841+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:28.822700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:28.822700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.254435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":207,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-10T07:54:21.277702+0000","last_change":"2024-02-10T07:53:35.090294+0000","last_active":"2024-02-10T07:54:21.277702+0000","last_peered":"2024-02-10T07:54:21.277702+0000","last_clean":"2024-02-10T07:54:21.277702+0000","last_became_active":"2024-02-10T07:49:06.249431+0000","last_became_peered":"2024-02-10T07:49:06.249431+0000","last_unstale":"2024-02-10T07:54:21.277702+0000","last_undegraded":"2024-02-10T07:54:21.277702+0000","last_fullsized":"2024-02-10T07:54:21.277702+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:35.090175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:53:35.090175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:50.135166+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"430'21","reported_seq":911,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.394200+0000","last_change":"2024-02-10T07:48:59.304277+0000","last_active":"2024-02-10T07:54:22.394200+0000","last_peered":"2024-02-10T07:54:22.394200+0000","last_clean":"2024-02-10T07:54:22.394200+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:54:22.394200+0000","last_undegraded":"2024-02-10T07:54:22.394200+0000","last_fullsized":"2024-02-10T07:54:22.394200+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:48:59.304199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.304199+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":207,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-10T07:54:21.277738+0000","last_change":"2024-02-10T07:53:41.173096+0000","last_active":"2024-02-10T07:54:21.277738+0000","last_peered":"2024-02-10T07:54:21.277738+0000","last_clean":"2024-02-10T07:54:21.277738+0000","last_became_active":"2024-02-10T07:49:06.238367+0000","last_became_peered":"2024-02-10T07:49:06.238367+0000","last_unstale":"2024-02-10T07:54:21.277738+0000","last_undegraded":"2024-02-10T07:54:21.277738+0000","last_fullsized":"2024-02-10T07:54:21.277738+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:41.172967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:53:41.172967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:48.222331+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":302,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753422+0000","last_change":"2024-02-10T07:49:10.087939+0000","last_active":"2024-02-10T07:54:22.753422+0000","last_peered":"2024-02-10T07:54:22.753422+0000","last_clean":"2024-02-10T07:54:22.753422+0000","last_became_active":"2024-02-10T07:48:02.328316+0000","last_became_peered":"2024-02-10T07:48:02.328316+0000","last_unstale":"2024-02-10T07:54:22.753422+0000","last_undegraded":"2024-02-10T07:54:22.753422+0000","last_fullsized":"2024-02-10T07:54:22.753422+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:49:10.087795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:49:10.087795+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":874,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662189+0000","last_change":"2024-02-10T07:49:19.198419+0000","last_active":"2024-02-10T07:54:22.662189+0000","last_peered":"2024-02-10T07:54:22.662189+0000","last_clean":"2024-02-10T07:54:22.662189+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:54:22.662189+0000","last_undegraded":"2024-02-10T07:54:22.662189+0000","last_fullsized":"2024-02-10T07:54:22.662189+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:19.198312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:04.220328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.198312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"430'59","reported_seq":1349,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753726+0000","last_change":"2024-02-10T07:50:43.327112+0000","last_active":"2024-02-10T07:54:22.753726+0000","last_peered":"2024-02-10T07:54:22.753726+0000","last_clean":"2024-02-10T07:54:22.753726+0000","last_became_active":"2024-02-10T07:50:43.297262+0000","last_became_peered":"2024-02-10T07:50:43.297262+0000","last_unstale":"2024-02-10T07:54:22.753726+0000","last_undegraded":"2024-02-10T07:54:22.753726+0000","last_fullsized":"2024-02-10T07:54:22.753726+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'46","last_scrub_stamp":"2024-02-10T07:49:55.714323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:55.714323+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":882,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.393969+0000","last_change":"2024-02-10T07:49:17.357664+0000","last_active":"2024-02-10T07:54:22.393969+0000","last_peered":"2024-02-10T07:54:22.393969+0000","last_clean":"2024-02-10T07:54:22.393969+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:54:22.393969+0000","last_undegraded":"2024-02-10T07:54:22.393969+0000","last_fullsized":"2024-02-10T07:54:22.393969+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.357533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:22.786949+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.357533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":871,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288861+0000","last_change":"2024-02-10T07:47:50.081222+0000","last_active":"2024-02-10T07:54:22.288861+0000","last_peered":"2024-02-10T07:54:22.288861+0000","last_clean":"2024-02-10T07:54:22.288861+0000","last_became_active":"2024-02-10T07:47:48.149256+0000","last_became_peered":"2024-02-10T07:47:48.149256+0000","last_unstale":"2024-02-10T07:54:22.288861+0000","last_undegraded":"2024-02-10T07:54:22.288861+0000","last_fullsized":"2024-02-10T07:54:22.288861+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.080714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:42.730875+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.080714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":901,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753381+0000","last_change":"2024-02-10T07:50:43.288780+0000","last_active":"2024-02-10T07:54:22.753381+0000","last_peered":"2024-02-10T07:54:22.753381+0000","last_clean":"2024-02-10T07:54:22.753381+0000","last_became_active":"2024-02-10T07:50:43.288388+0000","last_became_peered":"2024-02-10T07:50:43.288388+0000","last_unstale":"2024-02-10T07:54:22.753381+0000","last_undegraded":"2024-02-10T07:54:22.753381+0000","last_fullsized":"2024-02-10T07:54:22.753381+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:49.886354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:49.886354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":134,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.713755+0000","last_change":"2024-02-10T07:50:43.287356+0000","last_active":"2024-02-10T07:54:22.713755+0000","last_peered":"2024-02-10T07:54:22.713755+0000","last_clean":"2024-02-10T07:54:22.713755+0000","last_became_active":"2024-02-10T07:50:43.287043+0000","last_became_peered":"2024-02-10T07:50:43.287043+0000","last_unstale":"2024-02-10T07:54:22.713755+0000","last_undegraded":"2024-02-10T07:54:22.713755+0000","last_fullsized":"2024-02-10T07:54:22.713755+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:15.234112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.234112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.254474+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"430'21","reported_seq":241,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.287838+0000","last_change":"2024-02-10T07:49:04.340445+0000","last_active":"2024-02-10T07:54:22.287838+0000","last_peered":"2024-02-10T07:54:22.287838+0000","last_clean":"2024-02-10T07:54:22.287838+0000","last_became_active":"2024-02-10T07:48:35.057662+0000","last_became_peered":"2024-02-10T07:48:35.057662+0000","last_unstale":"2024-02-10T07:54:22.287838+0000","last_undegraded":"2024-02-10T07:54:22.287838+0000","last_fullsized":"2024-02-10T07:54:22.287838+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'17","last_scrub_stamp":"2024-02-10T07:49:04.340366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.340366+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":134,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.714185+0000","last_change":"2024-02-10T07:50:43.280896+0000","last_active":"2024-02-10T07:54:22.714185+0000","last_peered":"2024-02-10T07:54:22.714185+0000","last_clean":"2024-02-10T07:54:22.714185+0000","last_became_active":"2024-02-10T07:50:43.279125+0000","last_became_peered":"2024-02-10T07:50:43.279125+0000","last_unstale":"2024-02-10T07:54:22.714185+0000","last_undegraded":"2024-02-10T07:54:22.714185+0000","last_fullsized":"2024-02-10T07:54:22.714185+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:50.232544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.232544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.254573+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":290,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662835+0000","last_change":"2024-02-10T07:48:02.330253+0000","last_active":"2024-02-10T07:54:22.662835+0000","last_peered":"2024-02-10T07:54:22.662835+0000","last_clean":"2024-02-10T07:54:22.662835+0000","last_became_active":"2024-02-10T07:48:02.329687+0000","last_became_peered":"2024-02-10T07:48:02.329687+0000","last_unstale":"2024-02-10T07:54:22.662835+0000","last_undegraded":"2024-02-10T07:54:22.662835+0000","last_fullsized":"2024-02-10T07:54:22.662835+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:47:49.750508+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.750508+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1128,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288079+0000","last_change":"2024-02-10T07:48:15.033077+0000","last_active":"2024-02-10T07:54:22.288079+0000","last_peered":"2024-02-10T07:54:22.288079+0000","last_clean":"2024-02-10T07:54:22.288079+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:54:22.288079+0000","last_undegraded":"2024-02-10T07:54:22.288079+0000","last_fullsized":"2024-02-10T07:54:22.288079+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:48:15.032944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.032944+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":520,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.287847+0000","last_change":"2024-02-10T07:50:58.025300+0000","last_active":"2024-02-10T07:54:22.287847+0000","last_peered":"2024-02-10T07:54:22.287847+0000","last_clean":"2024-02-10T07:54:22.287847+0000","last_became_active":"2024-02-10T07:49:06.256223+0000","last_became_peered":"2024-02-10T07:49:06.256223+0000","last_unstale":"2024-02-10T07:54:22.287847+0000","last_undegraded":"2024-02-10T07:54:22.287847+0000","last_fullsized":"2024-02-10T07:54:22.287847+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:58.025193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:58.025193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.206949+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"430'48","reported_seq":1238,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.287907+0000","last_change":"2024-02-10T07:52:03.839258+0000","last_active":"2024-02-10T07:54:22.287907+0000","last_peered":"2024-02-10T07:54:22.287907+0000","last_clean":"2024-02-10T07:54:22.287907+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:54:22.287907+0000","last_undegraded":"2024-02-10T07:54:22.287907+0000","last_fullsized":"2024-02-10T07:54:22.287907+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'48","last_scrub_stamp":"2024-02-10T07:52:03.839174+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:52:03.839174+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.207086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":903,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.289138+0000","last_change":"2024-02-10T07:50:43.285613+0000","last_active":"2024-02-10T07:54:22.289138+0000","last_peered":"2024-02-10T07:54:22.289138+0000","last_clean":"2024-02-10T07:54:22.289138+0000","last_became_active":"2024-02-10T07:50:43.284708+0000","last_became_peered":"2024-02-10T07:50:43.284708+0000","last_unstale":"2024-02-10T07:54:22.289138+0000","last_undegraded":"2024-02-10T07:54:22.289138+0000","last_fullsized":"2024-02-10T07:54:22.289138+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:13.369607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.736901+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.369607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":675,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753474+0000","last_change":"2024-02-10T07:50:07.589020+0000","last_active":"2024-02-10T07:54:22.753474+0000","last_peered":"2024-02-10T07:54:22.753474+0000","last_clean":"2024-02-10T07:54:22.753474+0000","last_became_active":"2024-02-10T07:49:06.270457+0000","last_became_peered":"2024-02-10T07:49:06.270457+0000","last_unstale":"2024-02-10T07:54:22.753474+0000","last_undegraded":"2024-02-10T07:54:22.753474+0000","last_fullsized":"2024-02-10T07:54:22.753474+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.588888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.588888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":897,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.393605+0000","last_change":"2024-02-10T07:53:41.577827+0000","last_active":"2024-02-10T07:54:22.393605+0000","last_peered":"2024-02-10T07:54:22.393605+0000","last_clean":"2024-02-10T07:54:22.393605+0000","last_became_active":"2024-02-10T07:49:06.266625+0000","last_became_peered":"2024-02-10T07:49:06.266625+0000","last_unstale":"2024-02-10T07:54:22.393605+0000","last_undegraded":"2024-02-10T07:54:22.393605+0000","last_fullsized":"2024-02-10T07:54:22.393605+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:41.577586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_clean_scrub_stamp":"2024-02-10T07:53:41.577586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:59.035911+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2248,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.342026+0000","last_change":"2024-02-10T07:50:04.159154+0000","last_active":"2024-02-10T07:54:22.342026+0000","last_peered":"2024-02-10T07:54:22.342026+0000","last_clean":"2024-02-10T07:54:22.342026+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:54:22.342026+0000","last_undegraded":"2024-02-10T07:54:22.342026+0000","last_fullsized":"2024-02-10T07:54:22.342026+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:50:04.159046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.159046+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1345,"num_read_kb":1340,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"430'42","reported_seq":978,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.394043+0000","last_change":"2024-02-10T07:50:43.324808+0000","last_active":"2024-02-10T07:54:22.394043+0000","last_peered":"2024-02-10T07:54:22.394043+0000","last_clean":"2024-02-10T07:54:22.394043+0000","last_became_active":"2024-02-10T07:50:43.291771+0000","last_became_peered":"2024-02-10T07:50:43.291771+0000","last_unstale":"2024-02-10T07:54:22.394043+0000","last_undegraded":"2024-02-10T07:54:22.394043+0000","last_fullsized":"2024-02-10T07:54:22.394043+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:47.417096+0000","last_deep_scrub":"237'18","last_deep_scrub_stamp":"2024-02-10T07:40:31.005376+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.417096+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":853,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.443064+0000","last_change":"2024-02-10T07:53:34.104503+0000","last_active":"2024-02-10T07:54:22.443064+0000","last_peered":"2024-02-10T07:54:22.443064+0000","last_clean":"2024-02-10T07:54:22.443064+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:54:22.443064+0000","last_undegraded":"2024-02-10T07:54:22.443064+0000","last_fullsized":"2024-02-10T07:54:22.443064+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:34.104386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:34.104386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:46.902238+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":896,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753419+0000","last_change":"2024-02-10T07:50:43.296823+0000","last_active":"2024-02-10T07:54:22.753419+0000","last_peered":"2024-02-10T07:54:22.753419+0000","last_clean":"2024-02-10T07:54:22.753419+0000","last_became_active":"2024-02-10T07:50:43.296488+0000","last_became_peered":"2024-02-10T07:50:43.296488+0000","last_unstale":"2024-02-10T07:54:22.753419+0000","last_undegraded":"2024-02-10T07:54:22.753419+0000","last_fullsized":"2024-02-10T07:54:22.753419+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.630083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.630083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":901,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753681+0000","last_change":"2024-02-10T07:50:43.298468+0000","last_active":"2024-02-10T07:54:22.753681+0000","last_peered":"2024-02-10T07:54:22.753681+0000","last_clean":"2024-02-10T07:54:22.753681+0000","last_became_active":"2024-02-10T07:50:43.297861+0000","last_became_peered":"2024-02-10T07:50:43.297861+0000","last_unstale":"2024-02-10T07:54:22.753681+0000","last_undegraded":"2024-02-10T07:54:22.753681+0000","last_fullsized":"2024-02-10T07:54:22.753681+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:05.193158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:49:05.193158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"430'47","reported_seq":961,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.394007+0000","last_change":"2024-02-10T07:49:57.522503+0000","last_active":"2024-02-10T07:54:22.394007+0000","last_peered":"2024-02-10T07:54:22.394007+0000","last_clean":"2024-02-10T07:54:22.394007+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:54:22.394007+0000","last_undegraded":"2024-02-10T07:54:22.394007+0000","last_fullsized":"2024-02-10T07:54:22.394007+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'37","last_scrub_stamp":"2024-02-10T07:49:57.522391+0000","last_deep_scrub":"271'37","last_deep_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.522391+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":920,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:23.265976+0000","last_change":"2024-02-10T07:54:23.265999+0000","last_active":"2024-02-10T07:54:23.265976+0000","last_peered":"2024-02-10T07:54:23.265976+0000","last_clean":"2024-02-10T07:54:23.265976+0000","last_became_active":"2024-02-10T07:48:35.062118+0000","last_became_peered":"2024-02-10T07:48:35.062118+0000","last_unstale":"2024-02-10T07:54:23.265976+0000","last_undegraded":"2024-02-10T07:54:23.265976+0000","last_fullsized":"2024-02-10T07:54:23.265976+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:23.265906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:00.292320+0000","last_clean_scrub_stamp":"2024-02-10T07:54:23.265906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:43.409955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":773,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.286975+0000","last_change":"2024-02-10T07:52:23.855559+0000","last_active":"2024-02-10T07:54:22.286975+0000","last_peered":"2024-02-10T07:54:22.286975+0000","last_clean":"2024-02-10T07:54:22.286975+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:54:22.286975+0000","last_undegraded":"2024-02-10T07:54:22.286975+0000","last_fullsized":"2024-02-10T07:54:22.286975+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:23.855477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:23.855477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.207128+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":134,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.713950+0000","last_change":"2024-02-10T07:50:43.289736+0000","last_active":"2024-02-10T07:54:22.713950+0000","last_peered":"2024-02-10T07:54:22.713950+0000","last_clean":"2024-02-10T07:54:22.713950+0000","last_became_active":"2024-02-10T07:50:43.289458+0000","last_became_peered":"2024-02-10T07:50:43.289458+0000","last_unstale":"2024-02-10T07:54:22.713950+0000","last_undegraded":"2024-02-10T07:54:22.713950+0000","last_fullsized":"2024-02-10T07:54:22.713950+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:48.809513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:48.809513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.254588+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"430'35","reported_seq":707,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753684+0000","last_change":"2024-02-10T07:49:19.084010+0000","last_active":"2024-02-10T07:54:22.753684+0000","last_peered":"2024-02-10T07:54:22.753684+0000","last_clean":"2024-02-10T07:54:22.753684+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:54:22.753684+0000","last_undegraded":"2024-02-10T07:54:22.753684+0000","last_fullsized":"2024-02-10T07:54:22.753684+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'25","last_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_deep_scrub":"382'25","last_deep_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.083856+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"158'10","reported_seq":1163,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.394273+0000","last_change":"2024-02-10T07:49:50.396822+0000","last_active":"2024-02-10T07:54:22.394273+0000","last_peered":"2024-02-10T07:54:22.394273+0000","last_clean":"2024-02-10T07:54:22.394273+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:54:22.394273+0000","last_undegraded":"2024-02-10T07:54:22.394273+0000","last_fullsized":"2024-02-10T07:54:22.394273+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:49:50.396730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.396730+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":882,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288555+0000","last_change":"2024-02-10T07:50:43.283505+0000","last_active":"2024-02-10T07:54:22.288555+0000","last_peered":"2024-02-10T07:54:22.288555+0000","last_clean":"2024-02-10T07:54:22.288555+0000","last_became_active":"2024-02-10T07:50:43.283161+0000","last_became_peered":"2024-02-10T07:50:43.283161+0000","last_unstale":"2024-02-10T07:54:22.288555+0000","last_undegraded":"2024-02-10T07:54:22.288555+0000","last_fullsized":"2024-02-10T07:54:22.288555+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_clean_scrub_stamp":"2024-02-10T07:48:23.271454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"296'2","reported_seq":892,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.394161+0000","last_change":"2024-02-10T07:50:03.643473+0000","last_active":"2024-02-10T07:54:22.394161+0000","last_peered":"2024-02-10T07:54:22.394161+0000","last_clean":"2024-02-10T07:54:22.394161+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:54:22.394161+0000","last_undegraded":"2024-02-10T07:54:22.394161+0000","last_fullsized":"2024-02-10T07:54:22.394161+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'2","last_scrub_stamp":"2024-02-10T07:50:03.643367+0000","last_deep_scrub":"296'2","last_deep_scrub_stamp":"2024-02-10T07:45:03.233375+0000","last_clean_scrub_stamp":"2024-02-10T07:50:03.643367+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"430'42","reported_seq":942,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662160+0000","last_change":"2024-02-10T07:49:20.216865+0000","last_active":"2024-02-10T07:54:22.662160+0000","last_peered":"2024-02-10T07:54:22.662160+0000","last_clean":"2024-02-10T07:54:22.662160+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:54:22.662160+0000","last_undegraded":"2024-02-10T07:54:22.662160+0000","last_fullsized":"2024-02-10T07:54:22.662160+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-02-10T07:49:20.216750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:20.216750+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"376'12","reported_seq":950,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753263+0000","last_change":"2024-02-10T07:49:06.282456+0000","last_active":"2024-02-10T07:54:22.753263+0000","last_peered":"2024-02-10T07:54:22.753263+0000","last_clean":"2024-02-10T07:54:22.753263+0000","last_became_active":"2024-02-10T07:49:06.255324+0000","last_became_peered":"2024-02-10T07:49:06.255324+0000","last_unstale":"2024-02-10T07:54:22.753263+0000","last_undegraded":"2024-02-10T07:54:22.753263+0000","last_fullsized":"2024-02-10T07:54:22.753263+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'11","last_scrub_stamp":"2024-02-10T07:48:58.368663+0000","last_deep_scrub":"227'9","last_deep_scrub_stamp":"2024-02-10T07:39:53.440221+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.368663+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":887,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662155+0000","last_change":"2024-02-10T07:54:21.727843+0000","last_active":"2024-02-10T07:54:22.662155+0000","last_peered":"2024-02-10T07:54:22.662155+0000","last_clean":"2024-02-10T07:54:22.662155+0000","last_became_active":"2024-02-10T07:50:43.281465+0000","last_became_peered":"2024-02-10T07:50:43.281465+0000","last_unstale":"2024-02-10T07:54:22.662155+0000","last_undegraded":"2024-02-10T07:54:22.662155+0000","last_fullsized":"2024-02-10T07:54:22.662155+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:21.727684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:54:21.727684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:26.036430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":364,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288922+0000","last_change":"2024-02-10T07:47:50.948710+0000","last_active":"2024-02-10T07:54:22.288922+0000","last_peered":"2024-02-10T07:54:22.288922+0000","last_clean":"2024-02-10T07:54:22.288922+0000","last_became_active":"2024-02-10T07:47:47.966443+0000","last_became_peered":"2024-02-10T07:47:47.966443+0000","last_unstale":"2024-02-10T07:54:22.288922+0000","last_undegraded":"2024-02-10T07:54:22.288922+0000","last_fullsized":"2024-02-10T07:54:22.288922+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.948624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.948624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"430'42","reported_seq":881,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.288233+0000","last_change":"2024-02-10T07:49:57.303310+0000","last_active":"2024-02-10T07:54:22.288233+0000","last_peered":"2024-02-10T07:54:22.288233+0000","last_clean":"2024-02-10T07:54:22.288233+0000","last_became_active":"2024-02-10T07:49:06.252935+0000","last_became_peered":"2024-02-10T07:49:06.252935+0000","last_unstale":"2024-02-10T07:54:22.288233+0000","last_undegraded":"2024-02-10T07:54:22.288233+0000","last_fullsized":"2024-02-10T07:54:22.288233+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:57.303200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.303200+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":881,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662031+0000","last_change":"2024-02-10T07:49:06.239655+0000","last_active":"2024-02-10T07:54:22.662031+0000","last_peered":"2024-02-10T07:54:22.662031+0000","last_clean":"2024-02-10T07:54:22.662031+0000","last_became_active":"2024-02-10T07:49:06.239190+0000","last_became_peered":"2024-02-10T07:49:06.239190+0000","last_unstale":"2024-02-10T07:54:22.662031+0000","last_undegraded":"2024-02-10T07:54:22.662031+0000","last_fullsized":"2024-02-10T07:54:22.662031+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:42.549018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:47:42.549018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":388,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753693+0000","last_change":"2024-02-10T07:49:06.274374+0000","last_active":"2024-02-10T07:54:22.753693+0000","last_peered":"2024-02-10T07:54:22.753693+0000","last_clean":"2024-02-10T07:54:22.753693+0000","last_became_active":"2024-02-10T07:48:02.371926+0000","last_became_peered":"2024-02-10T07:48:02.371926+0000","last_unstale":"2024-02-10T07:54:22.753693+0000","last_undegraded":"2024-02-10T07:54:22.753693+0000","last_fullsized":"2024-02-10T07:54:22.753693+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.274263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:49.704024+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.274263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"395'12","reported_seq":306,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.287536+0000","last_change":"2024-02-10T07:52:24.850402+0000","last_active":"2024-02-10T07:54:22.287536+0000","last_peered":"2024-02-10T07:54:22.287536+0000","last_clean":"2024-02-10T07:54:22.287536+0000","last_became_active":"2024-02-10T07:48:35.071916+0000","last_became_peered":"2024-02-10T07:48:35.071916+0000","last_unstale":"2024-02-10T07:54:22.287536+0000","last_undegraded":"2024-02-10T07:54:22.287536+0000","last_fullsized":"2024-02-10T07:54:22.287536+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'12","last_scrub_stamp":"2024-02-10T07:52:24.850269+0000","last_deep_scrub":"395'12","last_deep_scrub_stamp":"2024-02-10T07:52:24.850269+0000","last_clean_scrub_stamp":"2024-02-10T07:52:24.850269+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.207164+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":207,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-10T07:54:21.277097+0000","last_change":"2024-02-10T07:53:37.144276+0000","last_active":"2024-02-10T07:54:21.277097+0000","last_peered":"2024-02-10T07:54:21.277097+0000","last_clean":"2024-02-10T07:54:21.277097+0000","last_became_active":"2024-02-10T07:49:06.251038+0000","last_became_peered":"2024-02-10T07:49:06.251038+0000","last_unstale":"2024-02-10T07:54:21.277097+0000","last_undegraded":"2024-02-10T07:54:21.277097+0000","last_fullsized":"2024-02-10T07:54:21.277097+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:37.144142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:53:37.144142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:01.122427+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"430'39","reported_seq":953,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753791+0000","last_change":"2024-02-10T07:50:00.655466+0000","last_active":"2024-02-10T07:54:22.753791+0000","last_peered":"2024-02-10T07:54:22.753791+0000","last_clean":"2024-02-10T07:54:22.753791+0000","last_became_active":"2024-02-10T07:47:47.967427+0000","last_became_peered":"2024-02-10T07:47:47.967427+0000","last_unstale":"2024-02-10T07:54:22.753791+0000","last_undegraded":"2024-02-10T07:54:22.753791+0000","last_fullsized":"2024-02-10T07:54:22.753791+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:50:00.655377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.655377+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":583,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.287495+0000","last_change":"2024-02-10T07:51:59.948039+0000","last_active":"2024-02-10T07:54:22.287495+0000","last_peered":"2024-02-10T07:54:22.287495+0000","last_clean":"2024-02-10T07:54:22.287495+0000","last_became_active":"2024-02-10T07:49:06.268874+0000","last_became_peered":"2024-02-10T07:49:06.268874+0000","last_unstale":"2024-02-10T07:54:22.287495+0000","last_undegraded":"2024-02-10T07:54:22.287495+0000","last_fullsized":"2024-02-10T07:54:22.287495+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:51:59.947908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_clean_scrub_stamp":"2024-02-10T07:51:59.947908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.207049+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":362,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.663063+0000","last_change":"2024-02-10T07:49:57.310447+0000","last_active":"2024-02-10T07:54:22.663063+0000","last_peered":"2024-02-10T07:54:22.663063+0000","last_clean":"2024-02-10T07:54:22.663063+0000","last_became_active":"2024-02-10T07:48:02.333689+0000","last_became_peered":"2024-02-10T07:48:02.333689+0000","last_unstale":"2024-02-10T07:54:22.663063+0000","last_undegraded":"2024-02-10T07:54:22.663063+0000","last_fullsized":"2024-02-10T07:54:22.663063+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:57.310333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.310333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":894,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.715889+0000","last_change":"2024-02-10T07:54:22.715912+0000","last_active":"2024-02-10T07:54:22.715889+0000","last_peered":"2024-02-10T07:54:22.715889+0000","last_clean":"2024-02-10T07:54:22.715889+0000","last_became_active":"2024-02-10T07:50:43.283104+0000","last_became_peered":"2024-02-10T07:50:43.283104+0000","last_unstale":"2024-02-10T07:54:22.715889+0000","last_undegraded":"2024-02-10T07:54:22.715889+0000","last_fullsized":"2024-02-10T07:54:22.715889+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:22.715819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:54:22.715819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:31.365491+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"430'32","reported_seq":971,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662837+0000","last_change":"2024-02-10T07:49:16.249257+0000","last_active":"2024-02-10T07:54:22.662837+0000","last_peered":"2024-02-10T07:54:22.662837+0000","last_clean":"2024-02-10T07:54:22.662837+0000","last_became_active":"2024-02-10T07:48:35.062573+0000","last_became_peered":"2024-02-10T07:48:35.062573+0000","last_unstale":"2024-02-10T07:54:22.662837+0000","last_undegraded":"2024-02-10T07:54:22.662837+0000","last_fullsized":"2024-02-10T07:54:22.662837+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:49:16.248936+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.248936+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":134,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.714183+0000","last_change":"2024-02-10T07:50:43.295417+0000","last_active":"2024-02-10T07:54:22.714183+0000","last_peered":"2024-02-10T07:54:22.714183+0000","last_clean":"2024-02-10T07:54:22.714183+0000","last_became_active":"2024-02-10T07:50:43.290295+0000","last_became_peered":"2024-02-10T07:50:43.290295+0000","last_unstale":"2024-02-10T07:54:22.714183+0000","last_undegraded":"2024-02-10T07:54:22.714183+0000","last_fullsized":"2024-02-10T07:54:22.714183+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.153368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.153368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.254505+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":871,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.753776+0000","last_change":"2024-02-10T07:49:07.170105+0000","last_active":"2024-02-10T07:54:22.753776+0000","last_peered":"2024-02-10T07:54:22.753776+0000","last_clean":"2024-02-10T07:54:22.753776+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:54:22.753776+0000","last_undegraded":"2024-02-10T07:54:22.753776+0000","last_fullsized":"2024-02-10T07:54:22.753776+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:07.169962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:07.169962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":913,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.287742+0000","last_change":"2024-02-10T07:52:26.772420+0000","last_active":"2024-02-10T07:54:22.287742+0000","last_peered":"2024-02-10T07:54:22.287742+0000","last_clean":"2024-02-10T07:54:22.287742+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:54:22.287742+0000","last_undegraded":"2024-02-10T07:54:22.287742+0000","last_fullsized":"2024-02-10T07:54:22.287742+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:26.772322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:03.803361+0000","last_clean_scrub_stamp":"2024-02-10T07:52:26.772322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:23.207218+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"430'53","reported_seq":1019,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662882+0000","last_change":"2024-02-10T07:50:43.316983+0000","last_active":"2024-02-10T07:54:22.662882+0000","last_peered":"2024-02-10T07:54:22.662882+0000","last_clean":"2024-02-10T07:54:22.662882+0000","last_became_active":"2024-02-10T07:50:43.281879+0000","last_became_peered":"2024-02-10T07:50:43.281879+0000","last_unstale":"2024-02-10T07:54:22.662882+0000","last_undegraded":"2024-02-10T07:54:22.662882+0000","last_fullsized":"2024-02-10T07:54:22.662882+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T07:48:59.066256+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.066256+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":873,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662973+0000","last_change":"2024-02-10T07:50:43.290883+0000","last_active":"2024-02-10T07:54:22.662973+0000","last_peered":"2024-02-10T07:54:22.662973+0000","last_clean":"2024-02-10T07:54:22.662973+0000","last_became_active":"2024-02-10T07:50:43.290334+0000","last_became_peered":"2024-02-10T07:50:43.290334+0000","last_unstale":"2024-02-10T07:54:22.662973+0000","last_undegraded":"2024-02-10T07:54:22.662973+0000","last_fullsized":"2024-02-10T07:54:22.662973+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:47.275882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.275882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"405'12","reported_seq":183,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.714065+0000","last_change":"2024-02-10T07:50:43.294666+0000","last_active":"2024-02-10T07:54:22.714065+0000","last_peered":"2024-02-10T07:54:22.714065+0000","last_clean":"2024-02-10T07:54:22.714065+0000","last_became_active":"2024-02-10T07:50:43.280142+0000","last_became_peered":"2024-02-10T07:50:43.280142+0000","last_unstale":"2024-02-10T07:54:22.714065+0000","last_undegraded":"2024-02-10T07:54:22.714065+0000","last_fullsized":"2024-02-10T07:54:22.714065+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'11","last_scrub_stamp":"2024-02-10T07:49:14.335064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:14.335064+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:26.254557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":379,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.394140+0000","last_change":"2024-02-10T07:53:20.474601+0000","last_active":"2024-02-10T07:54:22.394140+0000","last_peered":"2024-02-10T07:54:22.394140+0000","last_clean":"2024-02-10T07:54:22.394140+0000","last_became_active":"2024-02-10T07:49:06.258926+0000","last_became_peered":"2024-02-10T07:49:06.258926+0000","last_unstale":"2024-02-10T07:54:22.394140+0000","last_undegraded":"2024-02-10T07:54:22.394140+0000","last_fullsized":"2024-02-10T07:54:22.394140+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:20.474497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_clean_scrub_stamp":"2024-02-10T07:53:20.474497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:48.442368+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"430'30","reported_seq":953,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-10T07:54:22.662515+0000","last_change":"2024-02-10T07:49:06.289632+0000","last_active":"2024-02-10T07:54:22.662515+0000","last_peered":"2024-02-10T07:54:22.662515+0000","last_clean":"2024-02-10T07:54:22.662515+0000","last_became_active":"2024-02-10T07:49:06.238858+0000","last_became_peered":"2024-02-10T07:49:06.238858+0000","last_unstale":"2024-02-10T07:54:22.662515+0000","last_undegraded":"2024-02-10T07:54:22.662515+0000","last_fullsized":"2024-02-10T07:54:22.662515+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:48:15.588199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.588199+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1345,"num_read_kb":1340,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1540,"num_read_kb":1297,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":6382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1060,"ondisk_log_size":1060,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038035,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1213036,"kb_used_data":397472,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92532116,"statfs":{"total":95995035648,"available":94752886784,"internally_reserved":0,"allocated":407011328,"data_stored":406452975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":835123658},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":3.0030000000000001,"5min":1.8480000000000001,"15min":1.6899999999999999},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.215},"max":{"1min":18.202999999999999,"5min":39.494,"15min":39.494},"last":0.60099999999999998},{"interface":"front","average":{"1min":2.968,"5min":1.8959999999999999,"15min":1.7030000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.23000000000000001},"max":{"1min":19.745000000000001,"5min":38.524000000000001,"15min":38.524000000000001},"last":0.56699999999999995}]},{"osd":1,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":6.6909999999999998,"5min":2.4009999999999998,"15min":2.069},"min":{"1min":0.33400000000000002,"5min":0.251,"15min":0.24399999999999999},"max":{"1min":50.149000000000001,"5min":50.149000000000001,"15min":50.149000000000001},"last":0.442},{"interface":"front","average":{"1min":6.5720000000000001,"5min":2.431,"15min":2.0190000000000001},"min":{"1min":0.33100000000000002,"5min":0.26300000000000001,"15min":0.24399999999999999},"max":{"1min":54.048000000000002,"5min":54.048000000000002,"15min":54.048000000000002},"last":0.70999999999999996}]},{"osd":2,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":1.696,"5min":1.25,"15min":1.141},"min":{"1min":0.28199999999999997,"5min":0.27700000000000002,"15min":0.255},"max":{"1min":7.0019999999999998,"5min":7.0019999999999998,"15min":18.57},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.4510000000000001,"15min":1.2989999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":7.8410000000000002,"5min":8.3949999999999996,"15min":18.521000000000001},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Feb 10 07:54:05 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.1950000000000001,"15min":1.099},"min":{"1min":0.35499999999999998,"5min":0.32400000000000001,"15min":0.307},"max":{"1min":1.6930000000000001,"5min":4.7590000000000003,"15min":4.7590000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.222,"15min":1.022},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.313},"max":{"1min":3.1859999999999999,"5min":6.2839999999999998,"15min":6.2839999999999998},"last":0.53300000000000003}]},{"osd":4,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":1.347,"15min":1.1719999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.22},"max":{"1min":1.5960000000000001,"5min":26.001000000000001,"15min":26.001000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":1.3089999999999999,"15min":1.1539999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.94199999999999995,"5min":27.09,"15min":27.09},"last":0.58599999999999997}]},{"osd":5,"last update":"Sat Feb 10 07:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.75900000000000001,"15min":1.0449999999999999},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.245},"max":{"1min":1.6479999999999999,"5min":3.5489999999999999,"15min":18.126999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.98099999999999998,"15min":1.1699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.6639999999999999,"5min":5.8490000000000002,"15min":19.548999999999999},"last":0.66800000000000004}]},{"osd":6,"last update":"Sat Feb 10 07:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":1.1599999999999999,"15min":1.159},"min":{"1min":0.30299999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.0409999999999999,"5min":7.4329999999999998,"15min":17.832000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":1.077,"15min":1.179},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.29799999999999999},"max":{"1min":1.1719999999999999,"5min":6.101,"15min":20.449999999999999},"last":0.65100000000000002}]}]},{"osd":6,"up_from":50,"seq":214748365080,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":646548,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93098604,"statfs":{"total":95995035648,"available":95332970496,"internally_reserved":0,"allocated":1773568,"data_stored":1242904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":660273606},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":1.9199999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.1580000000000004,"5min":19.484999999999999,"15min":24.727},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.4419999999999999,"15min":1.407},"min":{"1min":0.25700000000000001,"5min":0.25600000000000001,"15min":0.23100000000000001},"max":{"1min":6.0469999999999997,"5min":14.808,"15min":29.084},"last":0.66900000000000004}]},{"osd":1,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.7310000000000001,"15min":1.907},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.23799999999999999},"max":{"1min":9.9710000000000001,"5min":15.007999999999999,"15min":37.659999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.651,"5min":1.6759999999999999,"15min":1.948},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.253},"max":{"1min":10.029999999999999,"5min":19.524999999999999,"15min":37.75},"last":0.56599999999999995}]},{"osd":2,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":1.7929999999999999,"15min":1.409},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":11.839,"5min":15.768000000000001,"15min":15.768000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.853,"5min":1.5980000000000001,"15min":1.2430000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.013999999999999,"5min":15.728999999999999,"15min":15.728999999999999},"last":0.52600000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.284,"15min":1.5980000000000001},"min":{"1min":0.371,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.7779999999999996,"5min":5.6500000000000004,"15min":13.279},"last":0.58699999999999997},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.2470000000000001,"15min":1.2450000000000001},"min":{"1min":0.376,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.8929999999999998,"5min":7.7949999999999999,"15min":7.7949999999999999},"last":0.68300000000000005}]},{"osd":4,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.60699999999999998,"15min":0.85499999999999998},"min":{"1min":0.26800000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":3.6099999999999999,"5min":3.6099999999999999,"15min":7.6509999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.70099999999999996,"15min":0.88600000000000001},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.26000000000000001},"max":{"1min":2.3220000000000001,"5min":4.8600000000000003,"15min":7.0300000000000002},"last":0.61499999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:54:05 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.58899999999999997,"15min":1.0780000000000001},"min":{"1min":0.308,"5min":0.28799999999999998,"15min":0.24099999999999999},"max":{"1min":4.5720000000000001,"5min":4.5720000000000001,"15min":44.689},"last":0.63},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.748,"15min":1.21},"min":{"1min":0.32700000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.4460000000000002,"5min":4.8940000000000001,"15min":44.465000000000003},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Feb 10 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85499999999999998,"15min":1.3180000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.7509999999999999,"5min":4.7750000000000004,"15min":42.631999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.79200000000000004,"15min":1.228},"min":{"1min":0.38200000000000001,"5min":0.32500000000000001,"15min":0.315},"max":{"1min":2.407,"5min":4.9290000000000003,"15min":44.792999999999999},"last":0.65300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575154,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852848,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892304,"statfs":{"total":95995035648,"available":95121719296,"internally_reserved":0,"allocated":1216512,"data_stored":653263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":872085958},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:49 2024","interfaces":[{"interface":"back","average":{"1min":2.3170000000000002,"5min":2.117,"15min":1.589},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":9.0530000000000008,"5min":14.382999999999999,"15min":14.382999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":3.2719999999999998,"5min":2.109,"15min":1.5920000000000001},"min":{"1min":0.47699999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":7.6779999999999999,"5min":14.16,"15min":14.16},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:54:20 2024","interfaces":[{"interface":"back","average":{"1min":2.9950000000000001,"5min":2.6360000000000001,"15min":1.6140000000000001},"min":{"1min":0.25800000000000001,"5min":0.246,"15min":0.20399999999999999},"max":{"1min":20.129999999999999,"5min":20.129999999999999,"15min":20.129999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":2.8500000000000001,"5min":2.29,"15min":1.5609999999999999},"min":{"1min":0.29399999999999998,"5min":0.27000000000000002,"15min":0.224},"max":{"1min":20.047000000000001,"5min":20.047000000000001,"15min":20.047000000000001},"last":0.30099999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:54:02 2024","interfaces":[{"interface":"back","average":{"1min":2.714,"5min":2.1890000000000001,"15min":1.0549999999999999},"min":{"1min":0.314,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":9.0830000000000002,"5min":22.367999999999999,"15min":22.367999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.2269999999999999,"15min":1},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":7.6020000000000003,"5min":22.199999999999999,"15min":22.199999999999999},"last":0.40699999999999997}]},{"osd":4,"last update":"Sat Feb 10 07:54:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9319999999999999,"5min":1.5900000000000001,"15min":2.2370000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28399999999999997},"max":{"1min":9.2010000000000005,"5min":9.2010000000000005,"15min":207.45699999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2130000000000001,"15min":1.3160000000000001},"min":{"1min":0.35299999999999998,"5min":0.29599999999999999,"15min":0.28899999999999998},"max":{"1min":7.9749999999999996,"5min":10.327999999999999,"15min":32.201000000000001},"last":0.51400000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:54:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9970000000000001,"5min":1.9119999999999999,"15min":2.1269999999999998},"min":{"1min":0.35899999999999999,"5min":0.34200000000000003,"15min":0.308},"max":{"1min":9.1419999999999995,"5min":23.978999999999999,"15min":98.703000000000003},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.893,"5min":1.6850000000000001,"15min":2.0470000000000002},"min":{"1min":0.40999999999999998,"5min":0.32600000000000001,"15min":0.30199999999999999},"max":{"1min":7.4320000000000004,"5min":24.024999999999999,"15min":98.664000000000001},"last":0.497}]},{"osd":6,"last update":"Sat Feb 10 07:53:49 2024","interfaces":[{"interface":"back","average":{"1min":1.7270000000000001,"5min":1.514,"15min":1.47},"min":{"1min":0.40400000000000003,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":8.0600000000000005,"5min":10.291,"15min":36.808},"last":0.44700000000000001},{"interface":"front","average":{"1min":2.2629999999999999,"5min":1.837,"15min":1.7749999999999999},"min":{"1min":0.47399999999999998,"5min":0.40100000000000002,"15min":0.33300000000000002},"max":{"1min":9.1750000000000007,"5min":9.1750000000000007,"15min":33.774999999999999},"last":0.63}]},{"osd":7,"last update":"Sat Feb 10 07:54:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6699999999999999,"5min":1.8879999999999999,"15min":1.583},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":7.6699999999999999,"5min":24.099,"15min":24.099},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.242,"5min":1.3580000000000001,"15min":1.3140000000000001},"min":{"1min":0.42199999999999999,"5min":0.39300000000000002,"15min":0.34300000000000003},"max":{"1min":8.0180000000000007,"5min":10.356999999999999,"15min":10.356999999999999},"last":0.58299999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738679,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799280,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92945872,"statfs":{"total":95995035648,"available":95176572928,"internally_reserved":0,"allocated":1802240,"data_stored":1244284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":816642501},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:53:44 2024","interfaces":[{"interface":"back","average":{"1min":4.194,"5min":2.5979999999999999,"15min":2.625},"min":{"1min":0.23899999999999999,"5min":0.20499999999999999,"15min":0.182},"max":{"1min":11.933,"5min":16.942,"15min":156.16200000000001},"last":0.25},{"interface":"front","average":{"1min":3.29,"5min":1.9790000000000001,"15min":2.4220000000000002},"min":{"1min":0.42099999999999999,"5min":0.22,"15min":0.183},"max":{"1min":8.891,"5min":8.891,"15min":156.941},"last":0.33600000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":2.0899999999999999,"15min":1.629},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.20000000000000001},"max":{"1min":3.5299999999999998,"5min":19.733000000000001,"15min":34.603000000000002},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.5609999999999999,"5min":2.1150000000000002,"15min":1.4890000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.7800000000000002,"5min":10.116,"15min":34.718000000000004},"last":0.39900000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":4.2039999999999997,"5min":2.21,"15min":1.1579999999999999},"min":{"1min":0.45900000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":29.792999999999999,"5min":29.792999999999999,"15min":29.792999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":4.1929999999999996,"5min":2.444,"15min":1.23},"min":{"1min":0.38800000000000001,"5min":0.214,"15min":0.214},"max":{"1min":29.506,"5min":29.506,"15min":29.506},"last":0.31900000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:54:03 2024","interfaces":[{"interface":"back","average":{"1min":1.7170000000000001,"5min":1.825,"15min":1.4890000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27600000000000002},"max":{"1min":7.181,"5min":16.991,"15min":37.286000000000001},"last":0.505},{"interface":"front","average":{"1min":1.46,"5min":1.5980000000000001,"15min":1.4350000000000001},"min":{"1min":0.32400000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.0709999999999997,"5min":8.5939999999999994,"15min":34.683},"last":0.47799999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:54:03 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.925,"15min":2.0379999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":7.1120000000000001,"5min":17.062999999999999,"15min":41.429000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.625,"15min":1.899},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.308},"max":{"1min":6,"5min":8.6460000000000008,"15min":41.456000000000003},"last":0.58699999999999997}]},{"osd":6,"last update":"Sat Feb 10 07:53:44 2024","interfaces":[{"interface":"back","average":{"1min":2.1320000000000001,"5min":1.6100000000000001,"15min":1.6020000000000001},"min":{"1min":0.45400000000000001,"5min":0.34599999999999997,"15min":0.33600000000000002},"max":{"1min":8.2349999999999994,"5min":8.5630000000000006,"15min":41.587000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":2.694,"5min":2.0310000000000001,"15min":1.879},"min":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":7.1520000000000001,"5min":17.027999999999999,"15min":41.256},"last":0.54100000000000004}]},{"osd":7,"last update":"Sat Feb 10 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.33300000000000002},"max":{"1min":2.1709999999999998,"5min":8.6809999999999992,"15min":34.710999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.482,"15min":1.4019999999999999},"min":{"1min":0.46100000000000002,"5min":0.39200000000000002,"15min":0.379},"max":{"1min":5.742,"5min":8.5999999999999996,"15min":34.997999999999998},"last":0.64200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411628,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":694540,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":93050612,"statfs":{"total":95995035648,"available":95283826688,"internally_reserved":0,"allocated":1175552,"data_stored":653126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":710015429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:23 2024","interfaces":[{"interface":"back","average":{"1min":3.5510000000000002,"5min":2.46,"15min":1.8029999999999999},"min":{"1min":0.47499999999999998,"5min":0.20699999999999999,"15min":0.191},"max":{"1min":32.371000000000002,"5min":32.371000000000002,"15min":65.430999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":4.0869999999999997,"5min":2.6749999999999998,"15min":1.6990000000000001},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.20100000000000001},"max":{"1min":33.280000000000001,"5min":38.173000000000002,"15min":38.173000000000002},"last":0.52500000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:53:23 2024","interfaces":[{"interface":"back","average":{"1min":3.2490000000000001,"5min":1.9690000000000001,"15min":1.736},"min":{"1min":0.46700000000000003,"5min":0.21299999999999999,"15min":0.20599999999999999},"max":{"1min":13.071999999999999,"5min":27.606000000000002,"15min":65.795000000000002},"last":0.5},{"interface":"front","average":{"1min":4.0030000000000001,"5min":2.456,"15min":1.681},"min":{"1min":0.38500000000000001,"5min":0.219,"15min":0.20499999999999999},"max":{"1min":12.17,"5min":29.709,"15min":29.709},"last":0.46000000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:54:01 2024","interfaces":[{"interface":"back","average":{"1min":1.829,"5min":1.4139999999999999,"15min":0.85199999999999998},"min":{"1min":0.28199999999999997,"5min":0.217,"15min":0.217},"max":{"1min":6.5259999999999998,"5min":7.0819999999999999,"15min":7.0819999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":2.415,"5min":1.776,"15min":0.91700000000000004},"min":{"1min":0.29299999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":5.633,"5min":7.7530000000000001,"15min":7.7530000000000001},"last":0.51300000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:54:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1890000000000001,"15min":1.401},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29599999999999999},"max":{"1min":3.4820000000000002,"5min":11.395,"15min":65.494},"last":0.749},{"interface":"front","average":{"1min":1.881,"5min":1.6779999999999999,"15min":1.3169999999999999},"min":{"1min":0.40100000000000002,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":5.5890000000000004,"5min":7.7149999999999999,"15min":7.7149999999999999},"last":0.65600000000000003}]},{"osd":5,"last update":"Sat Feb 10 07:54:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.4339999999999999,"15min":1.4910000000000001},"min":{"1min":0.36099999999999999,"5min":0.33400000000000002,"15min":0.32600000000000001},"max":{"1min":4.7480000000000002,"5min":10.807,"15min":25.533000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.105,"15min":1.536},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.5310000000000001,"5min":4.9749999999999996,"15min":65.834000000000003},"last":0.72599999999999998}]},{"osd":6,"last update":"Sat Feb 10 07:53:41 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.2110000000000001,"15min":1.6890000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.35199999999999998},"max":{"1min":4.7400000000000002,"5min":5.069,"15min":65.748000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":2.4780000000000002,"5min":1.5,"15min":1.4550000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":10.893000000000001,"5min":10.893000000000001,"15min":25.901},"last":0.69499999999999995}]},{"osd":7,"last update":"Sat Feb 10 07:53:23 2024","interfaces":[{"interface":"back","average":{"1min":2.5030000000000001,"5min":1.702,"15min":1.3979999999999999},"min":{"1min":0.42299999999999999,"5min":0.36899999999999999,"15min":0.36699999999999999},"max":{"1min":10.859999999999999,"5min":10.859999999999999,"15min":10.859999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.2210000000000001,"15min":1.444},"min":{"1min":0.42799999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.609,"5min":5.0979999999999999,"15min":65.539000000000001},"last":0.79600000000000004}]}]},{"osd":3,"up_from":347,"seq":1490353651792,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":269028,"kb_used_data":1176,"kb_used_omap":5,"kb_used_meta":267834,"kb_avail":93476124,"statfs":{"total":95995035648,"available":95719550976,"internally_reserved":0,"allocated":1204224,"data_stored":645074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5625,"internal_metadata":274262535},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.8700000000000001,"5min":2.0329999999999999,"15min":1.028},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23699999999999999},"max":{"1min":6.694,"5min":28.247,"15min":28.247},"last":0.251},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0939999999999999,"15min":1.115},"min":{"1min":0.24399999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.9409999999999998,"5min":27.361999999999998,"15min":27.361999999999998},"last":0.27900000000000003}]},{"osd":1,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":3.0760000000000001,"5min":2.8860000000000001,"15min":1.226},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":15.547000000000001,"5min":30.718,"15min":30.718},"last":0.35499999999999998},{"interface":"front","average":{"1min":2.4940000000000002,"5min":2.5939999999999999,"15min":1.1499999999999999},"min":{"1min":0.32800000000000001,"5min":0.22,"15min":0.22},"max":{"1min":12.583,"5min":32.927999999999997,"15min":32.927999999999997},"last":0.31}]},{"osd":2,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":2.0350000000000001,"5min":1.7130000000000001,"15min":0.879},"min":{"1min":0.30599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.7430000000000003,"5min":10.686,"15min":10.686},"last":0.317},{"interface":"front","average":{"1min":2.3010000000000002,"5min":2.0390000000000001,"15min":0.996},"min":{"1min":0.39100000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.4209999999999994,"5min":11.25,"15min":11.25},"last":0.33400000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.637,"15min":2.383},"min":{"1min":0.34200000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":7.782,"5min":15.505000000000001,"15min":28.707999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.7849999999999999,"5min":1.514,"15min":2.5529999999999999},"min":{"1min":0.39100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":6.7759999999999998,"5min":15.551,"15min":28.760999999999999},"last":0.54500000000000004}]},{"osd":5,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.655,"15min":1.1160000000000001},"min":{"1min":0.371,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":6.343,"5min":15.525,"15min":15.525},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.4810000000000001,"15min":0.99199999999999999},"min":{"1min":0.36199999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":7.8280000000000003,"5min":10.298,"15min":10.298},"last":0.57999999999999996}]},{"osd":6,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":2.1850000000000001,"15min":2.444},"min":{"1min":0.40799999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":6.3920000000000003,"5min":55.750999999999998,"15min":55.750999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":2.0169999999999999,"5min":2.4100000000000001,"15min":2.956},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":8.5559999999999992,"5min":55.807000000000002,"15min":55.807000000000002},"last":0.61599999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":1.8720000000000001,"15min":1.367},"min":{"1min":0.433,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.8760000000000003,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.495},{"interface":"front","average":{"1min":2.0819999999999999,"5min":1.5920000000000001,"15min":1.3759999999999999},"min":{"1min":0.499,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.8049999999999997,"5min":7.8650000000000002,"15min":7.8650000000000002},"last":0.59799999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953762,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758444,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92986708,"statfs":{"total":95995035648,"available":95218388992,"internally_reserved":0,"allocated":1212416,"data_stored":653154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:54:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":1.341,"15min":1.7030000000000001},"min":{"1min":0.26300000000000001,"5min":0.251,"15min":0.20499999999999999},"max":{"1min":4.0339999999999998,"5min":20.004999999999999,"15min":74.533000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.7050000000000001,"15min":1.839},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.22700000000000001},"max":{"1min":4.8710000000000004,"5min":20.050000000000001,"15min":74.340999999999994},"last":0.65600000000000003}]},{"osd":1,"last update":"Sat Feb 10 07:54:04 2024","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.1030000000000002,"15min":2.5489999999999999},"min":{"1min":0.27400000000000002,"5min":0.245,"15min":0.23200000000000001},"max":{"1min":23.460000000000001,"5min":35.673999999999999,"15min":69.852000000000004},"last":0.61099999999999999},{"interface":"front","average":{"1min":2.117,"5min":2.3799999999999999,"15min":2.5},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":18.305,"5min":35.375,"15min":70.692999999999998},"last":0.45300000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:54:04 2024","interfaces":[{"interface":"back","average":{"1min":2.0710000000000002,"5min":1.4890000000000001,"15min":1.262},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.26600000000000001},"max":{"1min":25.32,"5min":25.32,"15min":25.32},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.841,"5min":1.2989999999999999,"15min":1.1970000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":20.535,"5min":20.535,"15min":20.535},"last":0.58399999999999996}]},{"osd":3,"last update":"Sat Feb 10 07:53:58 2024","interfaces":[{"interface":"back","average":{"1min":1.7609999999999999,"5min":1.3959999999999999,"15min":3.6280000000000001},"min":{"1min":0.30499999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":20.318000000000001,"5min":20.318000000000001,"15min":66.379999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.8009999999999999,"5min":1.3400000000000001,"15min":1.629},"min":{"1min":0.314,"5min":0.314,"15min":0.309},"max":{"1min":22.917000000000002,"5min":22.917000000000002,"15min":22.917000000000002},"last":0.63400000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:54:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.70099999999999996,"15min":0.98799999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.3109999999999999,"5min":5.9720000000000004,"15min":10.502000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60299999999999998,"15min":0.98499999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.247},"max":{"1min":2.0289999999999999,"5min":2.9820000000000002,"15min":9.2460000000000004},"last":0.67200000000000004}]},{"osd":6,"last update":"Sat Feb 10 07:53:49 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.77100000000000002,"15min":1.2070000000000001},"min":{"1min":0.27800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.2090000000000001,"5min":6.0039999999999996,"15min":67.433999999999997},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.79800000000000004,"15min":1.2849999999999999},"min":{"1min":0.29499999999999998,"5min":0.29199999999999998,"15min":0.26600000000000001},"max":{"1min":1.798,"5min":5.0090000000000003,"15min":66.876000000000005},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Feb 10 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.70799999999999996,"15min":0.97899999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.1220000000000001,"5min":4.5730000000000004,"15min":7.4930000000000003},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.67500000000000004,"15min":0.98599999999999999},"min":{"1min":0.314,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":1.9259999999999999,"5min":3.8479999999999999,"15min":10.919},"last":0.71799999999999997}]}]},{"osd":5,"up_from":156,"seq":670014898391,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":697748,"kb_used_data":1160,"kb_used_omap":4,"kb_used_meta":696571,"kb_avail":93047404,"statfs":{"total":95995035648,"available":95280541696,"internally_reserved":0,"allocated":1187840,"data_stored":646000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":713289378},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":1.6719999999999999,"15min":1.772},"min":{"1min":0.24299999999999999,"5min":0.22600000000000001,"15min":0.20899999999999999},"max":{"1min":3.0739999999999998,"5min":24.292999999999999,"15min":52.938000000000002},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.4970000000000001,"15min":1.974},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.22900000000000001},"max":{"1min":3.2589999999999999,"5min":23.908000000000001,"15min":52.805},"last":0.68600000000000005}]},{"osd":1,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0950000000000002,"5min":2.2309999999999999,"15min":2.1360000000000001},"min":{"1min":0.26700000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":12.362,"5min":28.780999999999999,"15min":54.298000000000002},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.843,"5min":2.2850000000000001,"15min":2.0569999999999999},"min":{"1min":0.27900000000000003,"5min":0.248,"15min":0.246},"max":{"1min":6.1630000000000003,"5min":28.356000000000002,"15min":52.969999999999999},"last":0.877}]},{"osd":2,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.329,"15min":1.244},"min":{"1min":0.29899999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.2709999999999999,"5min":10.039999999999999,"15min":10.039999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.587,"5min":1.347,"15min":1.2789999999999999},"min":{"1min":0.30199999999999999,"5min":0.28799999999999998,"15min":0.28100000000000003},"max":{"1min":5.2270000000000003,"5min":9.6069999999999993,"15min":13.662000000000001},"last":0.81799999999999995}]},{"osd":3,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":2.1280000000000001,"15min":1.8620000000000001},"min":{"1min":0.33400000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.4540000000000002,"5min":97.119,"15min":97.119},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.0449999999999999,"5min":2.4020000000000001,"15min":1.784},"min":{"1min":0.32700000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.1819999999999999,"5min":97.141999999999996,"15min":97.141999999999996},"last":0.66600000000000004}]},{"osd":4,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.81999999999999995,"15min":1.1439999999999999},"min":{"1min":0.27300000000000002,"5min":0.214,"15min":0.214},"max":{"1min":2.4830000000000001,"5min":6.7759999999999998,"15min":8.952},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.82999999999999996,"15min":1.0620000000000001},"min":{"1min":0.27500000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.411,"5min":6.202,"15min":10.180999999999999},"last":0.57499999999999996}]},{"osd":6,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":1.1200000000000001,"15min":1.1539999999999999},"min":{"1min":0.26900000000000002,"5min":0.26500000000000001,"15min":0.23999999999999999},"max":{"1min":2.508,"5min":21.748000000000001,"15min":21.748000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.69899999999999995,"5min":1.155,"15min":1.0960000000000001},"min":{"1min":0.32300000000000001,"5min":0.28599999999999998,"15min":0.26300000000000001},"max":{"1min":2.4420000000000002,"5min":22.187999999999999,"15min":22.187999999999999},"last":0.72499999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":1.0009999999999999,"15min":1.25},"min":{"1min":0.313,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.98099999999999998,"5min":22.302,"15min":22.302},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.133,"15min":1.1479999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.26300000000000001},"max":{"1min":2.4580000000000002,"5min":21.779,"15min":21.779},"last":0.70399999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:54:28.839 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-10T07:54:28.839 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-10T07:54:28.839 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:54:29.062 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:54:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:54:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:54:29.089 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:54:29.090 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:54:29.315 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:29.331 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:29 smithi008 ceph-mon[29816]: 4.15 scrub starts 2024-02-10T07:54:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:29 smithi008 ceph-mon[29816]: 4.15 scrub ok 2024-02-10T07:54:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:29 smithi008 ceph-mon[29816]: pgmap v1046: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-02-10T07:54:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:29 smithi008 ceph-mon[29816]: 4.a scrub starts 2024-02-10T07:54:29.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:29 smithi008 ceph-mon[29816]: 4.a scrub ok 2024-02-10T07:54:29.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:29 smithi008 ceph-mon[40743]: 4.15 scrub starts 2024-02-10T07:54:29.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:29 smithi008 ceph-mon[40743]: 4.15 scrub ok 2024-02-10T07:54:29.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:29 smithi008 ceph-mon[40743]: pgmap v1046: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-02-10T07:54:29.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:29 smithi008 ceph-mon[40743]: 4.a scrub starts 2024-02-10T07:54:29.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:29 smithi008 ceph-mon[40743]: 4.a scrub ok 2024-02-10T07:54:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:29 smithi170 ceph-mon[33394]: 4.15 scrub starts 2024-02-10T07:54:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:29 smithi170 ceph-mon[33394]: 4.15 scrub ok 2024-02-10T07:54:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:29 smithi170 ceph-mon[33394]: pgmap v1046: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-02-10T07:54:29.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:29 smithi170 ceph-mon[33394]: 4.a scrub starts 2024-02-10T07:54:29.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:29 smithi170 ceph-mon[33394]: 4.a scrub ok 2024-02-10T07:54:29.811 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-10T07:54:30.054 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:54:30.316 INFO:teuthology.orchestra.run.smithi008.stderr:nodeep-scrub is unset 2024-02-10T07:54:30.417 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:30 smithi170 ceph-mon[33394]: 3.1e scrub starts 2024-02-10T07:54:30.417 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:30 smithi170 ceph-mon[33394]: 3.1e scrub ok 2024-02-10T07:54:30.417 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:30 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3507036686' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:54:30.431 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:30 smithi008 ceph-mon[29816]: 3.1e scrub starts 2024-02-10T07:54:30.431 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:30 smithi008 ceph-mon[29816]: 3.1e scrub ok 2024-02-10T07:54:30.431 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:30 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3507036686' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:54:30.431 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:30 smithi008 ceph-mon[40743]: 3.1e scrub starts 2024-02-10T07:54:30.431 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:30 smithi008 ceph-mon[40743]: 3.1e scrub ok 2024-02-10T07:54:30.431 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:30 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3507036686' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-10T07:54:31.396 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[29816]: 4.1e scrub starts 2024-02-10T07:54:31.396 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[29816]: 4.1e scrub ok 2024-02-10T07:54:31.396 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[29816]: pgmap v1047: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:54:31.396 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[29816]: 4.10 scrub starts 2024-02-10T07:54:31.396 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[29816]: 4.10 scrub ok 2024-02-10T07:54:31.396 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[29816]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:54:31.396 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:54:31.396 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3507036686' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:54:31.396 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[29816]: osdmap e468: 8 total, 8 up, 8 in 2024-02-10T07:54:31.396 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[29816]: 5.a scrub starts 2024-02-10T07:54:31.397 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[29816]: 5.a scrub ok 2024-02-10T07:54:31.397 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[40743]: 4.1e scrub starts 2024-02-10T07:54:31.397 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[40743]: 4.1e scrub ok 2024-02-10T07:54:31.397 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[40743]: pgmap v1047: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:54:31.397 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[40743]: 4.10 scrub starts 2024-02-10T07:54:31.397 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[40743]: 4.10 scrub ok 2024-02-10T07:54:31.397 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[40743]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:54:31.397 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:54:31.397 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3507036686' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:54:31.397 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[40743]: osdmap e468: 8 total, 8 up, 8 in 2024-02-10T07:54:31.397 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[40743]: 5.a scrub starts 2024-02-10T07:54:31.398 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:31 smithi008 ceph-mon[40743]: 5.a scrub ok 2024-02-10T07:54:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:31 smithi170 ceph-mon[33394]: 4.1e scrub starts 2024-02-10T07:54:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:31 smithi170 ceph-mon[33394]: 4.1e scrub ok 2024-02-10T07:54:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:31 smithi170 ceph-mon[33394]: pgmap v1047: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:54:31.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:31 smithi170 ceph-mon[33394]: 4.10 scrub starts 2024-02-10T07:54:31.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:31 smithi170 ceph-mon[33394]: 4.10 scrub ok 2024-02-10T07:54:31.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:31 smithi170 ceph-mon[33394]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-10T07:54:31.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:31 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:54:31.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:31 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3507036686' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-10T07:54:31.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:31 smithi170 ceph-mon[33394]: osdmap e468: 8 total, 8 up, 8 in 2024-02-10T07:54:31.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:31 smithi170 ceph-mon[33394]: 5.a scrub starts 2024-02-10T07:54:31.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:31 smithi170 ceph-mon[33394]: 5.a scrub ok 2024-02-10T07:54:32.222 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:54:32.224 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:54:32.252 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:54:32.252 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:54:32.535 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:32 smithi008 ceph-mon[29816]: 4.8 scrub starts 2024-02-10T07:54:32.535 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:32 smithi008 ceph-mon[29816]: 4.8 scrub ok 2024-02-10T07:54:32.535 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:32 smithi008 ceph-mon[29816]: 5.1f scrub starts 2024-02-10T07:54:32.535 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:32 smithi008 ceph-mon[29816]: 5.1f scrub ok 2024-02-10T07:54:32.535 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:32 smithi008 ceph-mon[29816]: 4.13 scrub starts 2024-02-10T07:54:32.536 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:32 smithi008 ceph-mon[29816]: 4.13 scrub ok 2024-02-10T07:54:32.536 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:32 smithi008 ceph-mon[29816]: osdmap e469: 8 total, 8 up, 8 in 2024-02-10T07:54:32.536 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:32 smithi008 ceph-mon[40743]: 4.8 scrub starts 2024-02-10T07:54:32.536 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:32 smithi008 ceph-mon[40743]: 4.8 scrub ok 2024-02-10T07:54:32.536 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:32 smithi008 ceph-mon[40743]: 5.1f scrub starts 2024-02-10T07:54:32.536 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:32 smithi008 ceph-mon[40743]: 5.1f scrub ok 2024-02-10T07:54:32.536 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:32 smithi008 ceph-mon[40743]: 4.13 scrub starts 2024-02-10T07:54:32.536 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:32 smithi008 ceph-mon[40743]: 4.13 scrub ok 2024-02-10T07:54:32.536 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:32 smithi008 ceph-mon[40743]: osdmap e469: 8 total, 8 up, 8 in 2024-02-10T07:54:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:32 smithi170 ceph-mon[33394]: 4.8 scrub starts 2024-02-10T07:54:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:32 smithi170 ceph-mon[33394]: 4.8 scrub ok 2024-02-10T07:54:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:32 smithi170 ceph-mon[33394]: 5.1f scrub starts 2024-02-10T07:54:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:32 smithi170 ceph-mon[33394]: 5.1f scrub ok 2024-02-10T07:54:32.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:32 smithi170 ceph-mon[33394]: 4.13 scrub starts 2024-02-10T07:54:32.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:32 smithi170 ceph-mon[33394]: 4.13 scrub ok 2024-02-10T07:54:32.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:32 smithi170 ceph-mon[33394]: osdmap e469: 8 total, 8 up, 8 in 2024-02-10T07:54:32.997 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":1049,"stamp":"2024-02-10T07:54:32.138736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3126,"num_read_kb":2972,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":1780341,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1297,"ondisk_log_size":1297,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":398,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5912392,"kb_used_data":387752,"kb_used_omap":19,"kb_used_meta":5524524,"kb_avail":744048824,"statfs":{"total":767960285184,"available":761905995776,"internally_reserved":0,"allocated":397058048,"data_stored":392617178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19634,"internal_metadata":5657113422},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.922941"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":872,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139678+0000","last_change":"2024-02-10T07:49:56.315126+0000","last_active":"2024-02-10T07:54:27.139678+0000","last_peered":"2024-02-10T07:54:27.139678+0000","last_clean":"2024-02-10T07:54:27.139678+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:54:27.139678+0000","last_undegraded":"2024-02-10T07:54:27.139678+0000","last_fullsized":"2024-02-10T07:54:27.139678+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.315029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.315029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":882,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183441+0000","last_change":"2024-02-10T07:49:08.138553+0000","last_active":"2024-02-10T07:54:27.183441+0000","last_peered":"2024-02-10T07:54:27.183441+0000","last_clean":"2024-02-10T07:54:27.183441+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:54:27.183441+0000","last_undegraded":"2024-02-10T07:54:27.183441+0000","last_fullsized":"2024-02-10T07:54:27.183441+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:08.138442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.138442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"67'1","reported_seq":919,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184463+0000","last_change":"2024-02-10T07:54:25.476912+0000","last_active":"2024-02-10T07:54:27.184463+0000","last_peered":"2024-02-10T07:54:27.184463+0000","last_clean":"2024-02-10T07:54:27.184463+0000","last_became_active":"2024-02-10T07:50:43.281905+0000","last_became_peered":"2024-02-10T07:50:43.281905+0000","last_unstale":"2024-02-10T07:54:27.184463+0000","last_undegraded":"2024-02-10T07:54:27.184463+0000","last_fullsized":"2024-02-10T07:54:27.184463+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:54:25.476772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:54:25.476772+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:30.916677+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"430'25","reported_seq":949,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139581+0000","last_change":"2024-02-10T07:54:23.745030+0000","last_active":"2024-02-10T07:54:27.139581+0000","last_peered":"2024-02-10T07:54:27.139581+0000","last_clean":"2024-02-10T07:54:27.139581+0000","last_became_active":"2024-02-10T07:46:10.717475+0000","last_became_peered":"2024-02-10T07:46:10.717475+0000","last_unstale":"2024-02-10T07:54:27.139581+0000","last_undegraded":"2024-02-10T07:54:27.139581+0000","last_fullsized":"2024-02-10T07:54:27.139581+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'25","last_scrub_stamp":"2024-02-10T07:54:23.744907+0000","last_deep_scrub":"287'19","last_deep_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_clean_scrub_stamp":"2024-02-10T07:54:23.744907+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:34.286102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":213,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-02-10T07:54:30.311569+0000","last_change":"2024-02-10T07:53:26.218018+0000","last_active":"2024-02-10T07:54:30.311569+0000","last_peered":"2024-02-10T07:54:30.311569+0000","last_clean":"2024-02-10T07:54:30.311569+0000","last_became_active":"2024-02-10T07:49:06.236235+0000","last_became_peered":"2024-02-10T07:49:06.236235+0000","last_unstale":"2024-02-10T07:54:30.311569+0000","last_undegraded":"2024-02-10T07:54:30.311569+0000","last_fullsized":"2024-02-10T07:54:30.311569+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:26.217749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:15.079428+0000","last_clean_scrub_stamp":"2024-02-10T07:53:26.217749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:52.906075+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":875,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.138850+0000","last_change":"2024-02-10T07:50:06.335390+0000","last_active":"2024-02-10T07:54:27.138850+0000","last_peered":"2024-02-10T07:54:27.138850+0000","last_clean":"2024-02-10T07:54:27.138850+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:54:27.138850+0000","last_undegraded":"2024-02-10T07:54:27.138850+0000","last_fullsized":"2024-02-10T07:54:27.138850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:06.335262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:50:06.335262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":398,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183626+0000","last_change":"2024-02-10T07:51:05.208836+0000","last_active":"2024-02-10T07:54:27.183626+0000","last_peered":"2024-02-10T07:54:27.183626+0000","last_clean":"2024-02-10T07:54:27.183626+0000","last_became_active":"2024-02-10T07:50:43.282449+0000","last_became_peered":"2024-02-10T07:50:43.282449+0000","last_unstale":"2024-02-10T07:54:27.183626+0000","last_undegraded":"2024-02-10T07:54:27.183626+0000","last_fullsized":"2024-02-10T07:54:27.183626+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:51:05.208755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:51:05.208755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"430'39","reported_seq":170,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183889+0000","last_change":"2024-02-10T07:50:43.318008+0000","last_active":"2024-02-10T07:54:27.183889+0000","last_peered":"2024-02-10T07:54:27.183889+0000","last_clean":"2024-02-10T07:54:27.183889+0000","last_became_active":"2024-02-10T07:50:43.290681+0000","last_became_peered":"2024-02-10T07:50:43.290681+0000","last_unstale":"2024-02-10T07:54:27.183889+0000","last_undegraded":"2024-02-10T07:54:27.183889+0000","last_fullsized":"2024-02-10T07:54:27.183889+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:48:13.230908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.230908+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":871,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183247+0000","last_change":"2024-02-10T07:50:05.620136+0000","last_active":"2024-02-10T07:54:27.183247+0000","last_peered":"2024-02-10T07:54:27.183247+0000","last_clean":"2024-02-10T07:54:27.183247+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:54:27.183247+0000","last_undegraded":"2024-02-10T07:54:27.183247+0000","last_fullsized":"2024-02-10T07:54:27.183247+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:05.619958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:05.619958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":860,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.142613+0000","last_change":"2024-02-10T07:53:23.117713+0000","last_active":"2024-02-10T07:54:27.142613+0000","last_peered":"2024-02-10T07:54:27.142613+0000","last_clean":"2024-02-10T07:54:27.142613+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:54:27.142613+0000","last_undegraded":"2024-02-10T07:54:27.142613+0000","last_fullsized":"2024-02-10T07:54:27.142613+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:23.117597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:23.117597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:27.492528+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":412,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.140173+0000","last_change":"2024-02-10T07:49:13.218634+0000","last_active":"2024-02-10T07:54:27.140173+0000","last_peered":"2024-02-10T07:54:27.140173+0000","last_clean":"2024-02-10T07:54:27.140173+0000","last_became_active":"2024-02-10T07:48:02.332960+0000","last_became_peered":"2024-02-10T07:48:02.332960+0000","last_unstale":"2024-02-10T07:54:27.140173+0000","last_undegraded":"2024-02-10T07:54:27.140173+0000","last_fullsized":"2024-02-10T07:54:27.140173+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.218549+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"430'21","reported_seq":906,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139259+0000","last_change":"2024-02-10T07:47:49.553645+0000","last_active":"2024-02-10T07:54:27.139259+0000","last_peered":"2024-02-10T07:54:27.139259+0000","last_clean":"2024-02-10T07:54:27.139259+0000","last_became_active":"2024-02-10T07:47:47.965930+0000","last_became_peered":"2024-02-10T07:47:47.965930+0000","last_unstale":"2024-02-10T07:54:27.139259+0000","last_undegraded":"2024-02-10T07:54:27.139259+0000","last_fullsized":"2024-02-10T07:54:27.139259+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:47:49.553517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:47:49.553517+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":783,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183623+0000","last_change":"2024-02-10T07:49:56.718482+0000","last_active":"2024-02-10T07:54:27.183623+0000","last_peered":"2024-02-10T07:54:27.183623+0000","last_clean":"2024-02-10T07:54:27.183623+0000","last_became_active":"2024-02-10T07:49:06.266689+0000","last_became_peered":"2024-02-10T07:49:06.266689+0000","last_unstale":"2024-02-10T07:54:27.183623+0000","last_undegraded":"2024-02-10T07:54:27.183623+0000","last_fullsized":"2024-02-10T07:54:27.183623+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:56.718382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:56.718382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":875,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.138581+0000","last_change":"2024-02-10T07:48:18.069773+0000","last_active":"2024-02-10T07:54:27.138581+0000","last_peered":"2024-02-10T07:54:27.138581+0000","last_clean":"2024-02-10T07:54:27.138581+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:54:27.138581+0000","last_undegraded":"2024-02-10T07:54:27.138581+0000","last_fullsized":"2024-02-10T07:54:27.138581+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:18.069641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:18.069641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":400,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139110+0000","last_change":"2024-02-10T07:49:06.115547+0000","last_active":"2024-02-10T07:54:27.139110+0000","last_peered":"2024-02-10T07:54:27.139110+0000","last_clean":"2024-02-10T07:54:27.139110+0000","last_became_active":"2024-02-10T07:48:02.329859+0000","last_became_peered":"2024-02-10T07:48:02.329859+0000","last_unstale":"2024-02-10T07:54:27.139110+0000","last_undegraded":"2024-02-10T07:54:27.139110+0000","last_fullsized":"2024-02-10T07:54:27.139110+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.115466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.473686+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.115466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"430'9","reported_seq":888,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.140148+0000","last_change":"2024-02-10T07:48:11.611397+0000","last_active":"2024-02-10T07:54:27.140148+0000","last_peered":"2024-02-10T07:54:27.140148+0000","last_clean":"2024-02-10T07:54:27.140148+0000","last_became_active":"2024-02-10T07:47:47.966093+0000","last_became_peered":"2024-02-10T07:47:47.966093+0000","last_unstale":"2024-02-10T07:54:27.140148+0000","last_undegraded":"2024-02-10T07:54:27.140148+0000","last_fullsized":"2024-02-10T07:54:27.140148+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'7","last_scrub_stamp":"2024-02-10T07:48:11.608499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.608499+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":874,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139511+0000","last_change":"2024-02-10T07:48:13.621135+0000","last_active":"2024-02-10T07:54:27.139511+0000","last_peered":"2024-02-10T07:54:27.139511+0000","last_clean":"2024-02-10T07:54:27.139511+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:54:27.139511+0000","last_undegraded":"2024-02-10T07:54:27.139511+0000","last_fullsized":"2024-02-10T07:54:27.139511+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:13.620938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:13.620938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":881,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184749+0000","last_change":"2024-02-10T07:49:51.810158+0000","last_active":"2024-02-10T07:54:27.184749+0000","last_peered":"2024-02-10T07:54:27.184749+0000","last_clean":"2024-02-10T07:54:27.184749+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:54:27.184749+0000","last_undegraded":"2024-02-10T07:54:27.184749+0000","last_fullsized":"2024-02-10T07:54:27.184749+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:51.810062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:51.810062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":883,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183873+0000","last_change":"2024-02-10T07:49:16.313655+0000","last_active":"2024-02-10T07:54:27.183873+0000","last_peered":"2024-02-10T07:54:27.183873+0000","last_clean":"2024-02-10T07:54:27.183873+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:54:27.183873+0000","last_undegraded":"2024-02-10T07:54:27.183873+0000","last_fullsized":"2024-02-10T07:54:27.183873+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:16.313573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:47.723328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.313573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"430'46","reported_seq":170,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184688+0000","last_change":"2024-02-10T07:50:43.322322+0000","last_active":"2024-02-10T07:54:27.184688+0000","last_peered":"2024-02-10T07:54:27.184688+0000","last_clean":"2024-02-10T07:54:27.184688+0000","last_became_active":"2024-02-10T07:50:43.300153+0000","last_became_peered":"2024-02-10T07:50:43.300153+0000","last_unstale":"2024-02-10T07:54:27.184688+0000","last_undegraded":"2024-02-10T07:54:27.184688+0000","last_fullsized":"2024-02-10T07:54:27.184688+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:49:00.551605+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.551605+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":577,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183723+0000","last_change":"2024-02-10T07:52:11.043473+0000","last_active":"2024-02-10T07:54:27.183723+0000","last_peered":"2024-02-10T07:54:27.183723+0000","last_clean":"2024-02-10T07:54:27.183723+0000","last_became_active":"2024-02-10T07:49:06.263790+0000","last_became_peered":"2024-02-10T07:49:06.263790+0000","last_unstale":"2024-02-10T07:54:27.183723+0000","last_undegraded":"2024-02-10T07:54:27.183723+0000","last_fullsized":"2024-02-10T07:54:27.183723+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:11.043368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:11.043368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":892,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139490+0000","last_change":"2024-02-10T07:50:43.286623+0000","last_active":"2024-02-10T07:54:27.139490+0000","last_peered":"2024-02-10T07:54:27.139490+0000","last_clean":"2024-02-10T07:54:27.139490+0000","last_became_active":"2024-02-10T07:50:43.285201+0000","last_became_peered":"2024-02-10T07:50:43.285201+0000","last_unstale":"2024-02-10T07:54:27.139490+0000","last_undegraded":"2024-02-10T07:54:27.139490+0000","last_fullsized":"2024-02-10T07:54:27.139490+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.680331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.680331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":895,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139312+0000","last_change":"2024-02-10T07:50:43.282584+0000","last_active":"2024-02-10T07:54:27.139312+0000","last_peered":"2024-02-10T07:54:27.139312+0000","last_clean":"2024-02-10T07:54:27.139312+0000","last_became_active":"2024-02-10T07:50:43.281382+0000","last_became_peered":"2024-02-10T07:50:43.281382+0000","last_unstale":"2024-02-10T07:54:27.139312+0000","last_undegraded":"2024-02-10T07:54:27.139312+0000","last_fullsized":"2024-02-10T07:54:27.139312+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:04.396771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.396771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"430'24","reported_seq":850,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184526+0000","last_change":"2024-02-10T07:49:58.618564+0000","last_active":"2024-02-10T07:54:27.184526+0000","last_peered":"2024-02-10T07:54:27.184526+0000","last_clean":"2024-02-10T07:54:27.184526+0000","last_became_active":"2024-02-10T07:49:06.263261+0000","last_became_peered":"2024-02-10T07:49:06.263261+0000","last_unstale":"2024-02-10T07:54:27.184526+0000","last_undegraded":"2024-02-10T07:54:27.184526+0000","last_fullsized":"2024-02-10T07:54:27.184526+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'18","last_scrub_stamp":"2024-02-10T07:49:58.618464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:58.618464+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":915,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139092+0000","last_change":"2024-02-10T07:54:22.819331+0000","last_active":"2024-02-10T07:54:27.139092+0000","last_peered":"2024-02-10T07:54:27.139092+0000","last_clean":"2024-02-10T07:54:27.139092+0000","last_became_active":"2024-02-10T07:50:43.285254+0000","last_became_peered":"2024-02-10T07:50:43.285254+0000","last_unstale":"2024-02-10T07:54:27.139092+0000","last_undegraded":"2024-02-10T07:54:27.139092+0000","last_fullsized":"2024-02-10T07:54:27.139092+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:54:22.819236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:54:22.819236+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:42.843356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"414'9","reported_seq":1191,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183589+0000","last_change":"2024-02-10T07:51:13.307061+0000","last_active":"2024-02-10T07:54:27.183589+0000","last_peered":"2024-02-10T07:54:27.183589+0000","last_clean":"2024-02-10T07:54:27.183589+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:54:27.183589+0000","last_undegraded":"2024-02-10T07:54:27.183589+0000","last_fullsized":"2024-02-10T07:54:27.183589+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'9","last_scrub_stamp":"2024-02-10T07:51:13.306986+0000","last_deep_scrub":"414'9","last_deep_scrub_stamp":"2024-02-10T07:51:13.306986+0000","last_clean_scrub_stamp":"2024-02-10T07:51:13.306986+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":891,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183346+0000","last_change":"2024-02-10T07:48:30.384635+0000","last_active":"2024-02-10T07:54:27.183346+0000","last_peered":"2024-02-10T07:54:27.183346+0000","last_clean":"2024-02-10T07:54:27.183346+0000","last_became_active":"2024-02-10T07:47:47.966127+0000","last_became_peered":"2024-02-10T07:47:47.966127+0000","last_unstale":"2024-02-10T07:54:27.183346+0000","last_undegraded":"2024-02-10T07:54:27.183346+0000","last_fullsized":"2024-02-10T07:54:27.183346+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:48:30.384546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:30.384546+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"430'35","reported_seq":978,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:29.476914+0000","last_change":"2024-02-10T07:54:29.476969+0000","last_active":"2024-02-10T07:54:29.476914+0000","last_peered":"2024-02-10T07:54:29.476914+0000","last_clean":"2024-02-10T07:54:29.476914+0000","last_became_active":"2024-02-10T07:49:06.261487+0000","last_became_peered":"2024-02-10T07:49:06.261487+0000","last_unstale":"2024-02-10T07:54:29.476914+0000","last_undegraded":"2024-02-10T07:54:29.476914+0000","last_fullsized":"2024-02-10T07:54:29.476914+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'35","last_scrub_stamp":"2024-02-10T07:54:29.476823+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-02-10T07:42:17.820959+0000","last_clean_scrub_stamp":"2024-02-10T07:54:29.476823+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:57.498468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":878,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184885+0000","last_change":"2024-02-10T07:49:11.231819+0000","last_active":"2024-02-10T07:54:27.184885+0000","last_peered":"2024-02-10T07:54:27.184885+0000","last_clean":"2024-02-10T07:54:27.184885+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:54:27.184885+0000","last_undegraded":"2024-02-10T07:54:27.184885+0000","last_fullsized":"2024-02-10T07:54:27.184885+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.231704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.231704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":881,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184350+0000","last_change":"2024-02-10T07:50:43.289955+0000","last_active":"2024-02-10T07:54:27.184350+0000","last_peered":"2024-02-10T07:54:27.184350+0000","last_clean":"2024-02-10T07:54:27.184350+0000","last_became_active":"2024-02-10T07:50:43.289011+0000","last_became_peered":"2024-02-10T07:50:43.289011+0000","last_unstale":"2024-02-10T07:54:27.184350+0000","last_undegraded":"2024-02-10T07:54:27.184350+0000","last_fullsized":"2024-02-10T07:54:27.184350+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.208451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:06.200009+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.208451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":386,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183146+0000","last_change":"2024-02-10T07:53:21.437757+0000","last_active":"2024-02-10T07:54:27.183146+0000","last_peered":"2024-02-10T07:54:27.183146+0000","last_clean":"2024-02-10T07:54:27.183146+0000","last_became_active":"2024-02-10T07:49:06.258789+0000","last_became_peered":"2024-02-10T07:49:06.258789+0000","last_unstale":"2024-02-10T07:54:27.183146+0000","last_undegraded":"2024-02-10T07:54:27.183146+0000","last_fullsized":"2024-02-10T07:54:27.183146+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:21.437653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:53:21.437653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"430'27","reported_seq":947,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183670+0000","last_change":"2024-02-10T07:52:11.834589+0000","last_active":"2024-02-10T07:54:27.183670+0000","last_peered":"2024-02-10T07:54:27.183670+0000","last_clean":"2024-02-10T07:54:27.183670+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:54:27.183670+0000","last_undegraded":"2024-02-10T07:54:27.183670+0000","last_fullsized":"2024-02-10T07:54:27.183670+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'27","last_scrub_stamp":"2024-02-10T07:52:11.834490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:52:11.834490+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":893,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.138454+0000","last_change":"2024-02-10T07:50:43.284049+0000","last_active":"2024-02-10T07:54:27.138454+0000","last_peered":"2024-02-10T07:54:27.138454+0000","last_clean":"2024-02-10T07:54:27.138454+0000","last_became_active":"2024-02-10T07:50:43.283688+0000","last_became_peered":"2024-02-10T07:50:43.283688+0000","last_unstale":"2024-02-10T07:54:27.138454+0000","last_undegraded":"2024-02-10T07:54:27.138454+0000","last_fullsized":"2024-02-10T07:54:27.138454+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:28.087614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:48:28.087614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":898,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183161+0000","last_change":"2024-02-10T07:52:22.845994+0000","last_active":"2024-02-10T07:54:27.183161+0000","last_peered":"2024-02-10T07:54:27.183161+0000","last_clean":"2024-02-10T07:54:27.183161+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:54:27.183161+0000","last_undegraded":"2024-02-10T07:54:27.183161+0000","last_fullsized":"2024-02-10T07:54:27.183161+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:22.845773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:52:22.845773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":907,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184667+0000","last_change":"2024-02-10T07:49:06.259100+0000","last_active":"2024-02-10T07:54:27.184667+0000","last_peered":"2024-02-10T07:54:27.184667+0000","last_clean":"2024-02-10T07:54:27.184667+0000","last_became_active":"2024-02-10T07:49:06.258763+0000","last_became_peered":"2024-02-10T07:49:06.258763+0000","last_unstale":"2024-02-10T07:54:27.184667+0000","last_undegraded":"2024-02-10T07:54:27.184667+0000","last_fullsized":"2024-02-10T07:54:27.184667+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:29.188832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:48:29.188832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"430'14","reported_seq":233,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-02-10T07:54:30.311297+0000","last_change":"2024-02-10T07:54:30.274424+0000","last_active":"2024-02-10T07:54:30.311297+0000","last_peered":"2024-02-10T07:54:30.311297+0000","last_clean":"2024-02-10T07:54:30.311297+0000","last_became_active":"2024-02-10T07:49:06.250577+0000","last_became_peered":"2024-02-10T07:49:06.250577+0000","last_unstale":"2024-02-10T07:54:30.311297+0000","last_undegraded":"2024-02-10T07:54:30.311297+0000","last_fullsized":"2024-02-10T07:54:30.311297+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'14","last_scrub_stamp":"2024-02-10T07:54:30.274329+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:54:30.274329+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:54.003122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":381,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183495+0000","last_change":"2024-02-10T07:53:33.873865+0000","last_active":"2024-02-10T07:54:27.183495+0000","last_peered":"2024-02-10T07:54:27.183495+0000","last_clean":"2024-02-10T07:54:27.183495+0000","last_became_active":"2024-02-10T07:49:06.259341+0000","last_became_peered":"2024-02-10T07:49:06.259341+0000","last_unstale":"2024-02-10T07:54:27.183495+0000","last_undegraded":"2024-02-10T07:54:27.183495+0000","last_fullsized":"2024-02-10T07:54:27.183495+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:33.873724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:53:33.873724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:51.425277+0000","scrub_duration":324,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":875,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.138567+0000","last_change":"2024-02-10T07:48:24.171978+0000","last_active":"2024-02-10T07:54:27.138567+0000","last_peered":"2024-02-10T07:54:27.138567+0000","last_clean":"2024-02-10T07:54:27.138567+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:54:27.138567+0000","last_undegraded":"2024-02-10T07:54:27.138567+0000","last_fullsized":"2024-02-10T07:54:27.138567+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:24.171859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:24.171859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":840,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.142974+0000","last_change":"2024-02-10T07:54:23.178187+0000","last_active":"2024-02-10T07:54:27.142974+0000","last_peered":"2024-02-10T07:54:27.142974+0000","last_clean":"2024-02-10T07:54:27.142974+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:54:27.142974+0000","last_undegraded":"2024-02-10T07:54:27.142974+0000","last_fullsized":"2024-02-10T07:54:27.142974+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:23.178094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:54:23.178094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:53.106130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"430'27","reported_seq":854,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.138573+0000","last_change":"2024-02-10T07:49:53.245490+0000","last_active":"2024-02-10T07:54:27.138573+0000","last_peered":"2024-02-10T07:54:27.138573+0000","last_clean":"2024-02-10T07:54:27.138573+0000","last_became_active":"2024-02-10T07:49:06.239701+0000","last_became_peered":"2024-02-10T07:49:06.239701+0000","last_unstale":"2024-02-10T07:54:27.138573+0000","last_undegraded":"2024-02-10T07:54:27.138573+0000","last_fullsized":"2024-02-10T07:54:27.138573+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:53.245410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:53.245410+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":872,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.182935+0000","last_change":"2024-02-10T07:49:52.530010+0000","last_active":"2024-02-10T07:54:27.182935+0000","last_peered":"2024-02-10T07:54:27.182935+0000","last_clean":"2024-02-10T07:54:27.182935+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:54:27.182935+0000","last_undegraded":"2024-02-10T07:54:27.182935+0000","last_fullsized":"2024-02-10T07:54:27.182935+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:52.529908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:59.179342+0000","last_clean_scrub_stamp":"2024-02-10T07:49:52.529908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":381,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183780+0000","last_change":"2024-02-10T07:53:19.493460+0000","last_active":"2024-02-10T07:54:27.183780+0000","last_peered":"2024-02-10T07:54:27.183780+0000","last_clean":"2024-02-10T07:54:27.183780+0000","last_became_active":"2024-02-10T07:49:06.265267+0000","last_became_peered":"2024-02-10T07:49:06.265267+0000","last_unstale":"2024-02-10T07:54:27.183780+0000","last_undegraded":"2024-02-10T07:54:27.183780+0000","last_fullsized":"2024-02-10T07:54:27.183780+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:19.492989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:19.492989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:32.822823+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":892,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139345+0000","last_change":"2024-02-10T07:50:00.265308+0000","last_active":"2024-02-10T07:54:27.139345+0000","last_peered":"2024-02-10T07:54:27.139345+0000","last_clean":"2024-02-10T07:54:27.139345+0000","last_became_active":"2024-02-10T07:47:47.966472+0000","last_became_peered":"2024-02-10T07:47:47.966472+0000","last_unstale":"2024-02-10T07:54:27.139345+0000","last_undegraded":"2024-02-10T07:54:27.139345+0000","last_fullsized":"2024-02-10T07:54:27.139345+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.265208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.265208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"430'25","reported_seq":725,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183730+0000","last_change":"2024-02-10T07:50:43.310533+0000","last_active":"2024-02-10T07:54:27.183730+0000","last_peered":"2024-02-10T07:54:27.183730+0000","last_clean":"2024-02-10T07:54:27.183730+0000","last_became_active":"2024-02-10T07:50:43.296447+0000","last_became_peered":"2024-02-10T07:50:43.296447+0000","last_unstale":"2024-02-10T07:54:27.183730+0000","last_undegraded":"2024-02-10T07:54:27.183730+0000","last_fullsized":"2024-02-10T07:54:27.183730+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'19","last_scrub_stamp":"2024-02-10T07:49:09.116615+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-02-10T07:39:32.934269+0000","last_clean_scrub_stamp":"2024-02-10T07:49:09.116615+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":569,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183173+0000","last_change":"2024-02-10T07:53:35.716416+0000","last_active":"2024-02-10T07:54:27.183173+0000","last_peered":"2024-02-10T07:54:27.183173+0000","last_clean":"2024-02-10T07:54:27.183173+0000","last_became_active":"2024-02-10T07:49:06.260569+0000","last_became_peered":"2024-02-10T07:49:06.260569+0000","last_unstale":"2024-02-10T07:54:27.183173+0000","last_undegraded":"2024-02-10T07:54:27.183173+0000","last_fullsized":"2024-02-10T07:54:27.183173+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:35.715987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:53:35.715987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:50.372195+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":905,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.185157+0000","last_change":"2024-02-10T07:50:02.685941+0000","last_active":"2024-02-10T07:54:27.185157+0000","last_peered":"2024-02-10T07:54:27.185157+0000","last_clean":"2024-02-10T07:54:27.185157+0000","last_became_active":"2024-02-10T07:49:06.252380+0000","last_became_peered":"2024-02-10T07:49:06.252380+0000","last_unstale":"2024-02-10T07:54:27.185157+0000","last_undegraded":"2024-02-10T07:54:27.185157+0000","last_fullsized":"2024-02-10T07:54:27.185157+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_clean_scrub_stamp":"2024-02-10T07:50:02.685831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":883,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139204+0000","last_change":"2024-02-10T07:50:00.345087+0000","last_active":"2024-02-10T07:54:27.139204+0000","last_peered":"2024-02-10T07:54:27.139204+0000","last_clean":"2024-02-10T07:54:27.139204+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:54:27.139204+0000","last_undegraded":"2024-02-10T07:54:27.139204+0000","last_fullsized":"2024-02-10T07:54:27.139204+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:00.344993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.344993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"430'32","reported_seq":955,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139876+0000","last_change":"2024-02-10T07:47:43.494877+0000","last_active":"2024-02-10T07:54:27.139876+0000","last_peered":"2024-02-10T07:54:27.139876+0000","last_clean":"2024-02-10T07:54:27.139876+0000","last_became_active":"2024-02-10T07:46:10.721336+0000","last_became_peered":"2024-02-10T07:46:10.721336+0000","last_unstale":"2024-02-10T07:54:27.139876+0000","last_undegraded":"2024-02-10T07:54:27.139876+0000","last_fullsized":"2024-02-10T07:54:27.139876+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:47:43.494350+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:47:43.494350+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":886,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183155+0000","last_change":"2024-02-10T07:53:42.705767+0000","last_active":"2024-02-10T07:54:27.183155+0000","last_peered":"2024-02-10T07:54:27.183155+0000","last_clean":"2024-02-10T07:54:27.183155+0000","last_became_active":"2024-02-10T07:49:06.261570+0000","last_became_peered":"2024-02-10T07:49:06.261570+0000","last_unstale":"2024-02-10T07:54:27.183155+0000","last_undegraded":"2024-02-10T07:54:27.183155+0000","last_fullsized":"2024-02-10T07:54:27.183155+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:42.705250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:53:42.705250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:05.715285+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"356'11","reported_seq":399,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.736269+0000","last_change":"2024-02-10T07:54:27.736297+0000","last_active":"2024-02-10T07:54:27.736269+0000","last_peered":"2024-02-10T07:54:27.736269+0000","last_clean":"2024-02-10T07:54:27.736269+0000","last_became_active":"2024-02-10T07:48:02.371888+0000","last_became_peered":"2024-02-10T07:48:02.371888+0000","last_unstale":"2024-02-10T07:54:27.736269+0000","last_undegraded":"2024-02-10T07:54:27.736269+0000","last_fullsized":"2024-02-10T07:54:27.736269+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'11","last_scrub_stamp":"2024-02-10T07:54:27.736200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:54:27.736200+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:36.751496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":903,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.138526+0000","last_change":"2024-02-10T07:50:43.283263+0000","last_active":"2024-02-10T07:54:27.138526+0000","last_peered":"2024-02-10T07:54:27.138526+0000","last_clean":"2024-02-10T07:54:27.138526+0000","last_became_active":"2024-02-10T07:50:43.282925+0000","last_became_peered":"2024-02-10T07:50:43.282925+0000","last_unstale":"2024-02-10T07:54:27.138526+0000","last_undegraded":"2024-02-10T07:54:27.138526+0000","last_fullsized":"2024-02-10T07:54:27.138526+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.335455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.335455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"430'35","reported_seq":519,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184429+0000","last_change":"2024-02-10T07:49:12.150442+0000","last_active":"2024-02-10T07:54:27.184429+0000","last_peered":"2024-02-10T07:54:27.184429+0000","last_clean":"2024-02-10T07:54:27.184429+0000","last_became_active":"2024-02-10T07:48:02.329187+0000","last_became_peered":"2024-02-10T07:48:02.329187+0000","last_unstale":"2024-02-10T07:54:27.184429+0000","last_undegraded":"2024-02-10T07:54:27.184429+0000","last_fullsized":"2024-02-10T07:54:27.184429+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'25","last_scrub_stamp":"2024-02-10T07:49:12.150298+0000","last_deep_scrub":"96'15","last_deep_scrub_stamp":"2024-02-10T07:33:48.458885+0000","last_clean_scrub_stamp":"2024-02-10T07:49:12.150298+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":884,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.140139+0000","last_change":"2024-02-10T07:50:43.285390+0000","last_active":"2024-02-10T07:54:27.140139+0000","last_peered":"2024-02-10T07:54:27.140139+0000","last_clean":"2024-02-10T07:54:27.140139+0000","last_became_active":"2024-02-10T07:50:43.284846+0000","last_became_peered":"2024-02-10T07:50:43.284846+0000","last_unstale":"2024-02-10T07:54:27.140139+0000","last_undegraded":"2024-02-10T07:54:27.140139+0000","last_fullsized":"2024-02-10T07:54:27.140139+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:01.513972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:01.513972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"405'15","reported_seq":548,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.138958+0000","last_change":"2024-02-10T07:50:43.310228+0000","last_active":"2024-02-10T07:54:27.138958+0000","last_peered":"2024-02-10T07:54:27.138958+0000","last_clean":"2024-02-10T07:54:27.138958+0000","last_became_active":"2024-02-10T07:50:43.285213+0000","last_became_peered":"2024-02-10T07:50:43.285213+0000","last_unstale":"2024-02-10T07:54:27.138958+0000","last_undegraded":"2024-02-10T07:54:27.138958+0000","last_fullsized":"2024-02-10T07:54:27.138958+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'13","last_scrub_stamp":"2024-02-10T07:49:00.366610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.366610+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":898,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183227+0000","last_change":"2024-02-10T07:50:07.501014+0000","last_active":"2024-02-10T07:54:27.183227+0000","last_peered":"2024-02-10T07:54:27.183227+0000","last_clean":"2024-02-10T07:54:27.183227+0000","last_became_active":"2024-02-10T07:49:06.257043+0000","last_became_peered":"2024-02-10T07:49:06.257043+0000","last_unstale":"2024-02-10T07:54:27.183227+0000","last_undegraded":"2024-02-10T07:54:27.183227+0000","last_fullsized":"2024-02-10T07:54:27.183227+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:07.500890+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.500890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"430'21","reported_seq":653,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.830675+0000","last_change":"2024-02-10T07:54:27.830709+0000","last_active":"2024-02-10T07:54:27.830675+0000","last_peered":"2024-02-10T07:54:27.830675+0000","last_clean":"2024-02-10T07:54:27.830675+0000","last_became_active":"2024-02-10T07:49:06.255239+0000","last_became_peered":"2024-02-10T07:49:06.255239+0000","last_unstale":"2024-02-10T07:54:27.830675+0000","last_undegraded":"2024-02-10T07:54:27.830675+0000","last_fullsized":"2024-02-10T07:54:27.830675+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'21","last_scrub_stamp":"2024-02-10T07:54:27.830607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:54:27.830607+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:38.599485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":136,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184790+0000","last_change":"2024-02-10T07:50:43.282730+0000","last_active":"2024-02-10T07:54:27.184790+0000","last_peered":"2024-02-10T07:54:27.184790+0000","last_clean":"2024-02-10T07:54:27.184790+0000","last_became_active":"2024-02-10T07:50:43.281549+0000","last_became_peered":"2024-02-10T07:50:43.281549+0000","last_unstale":"2024-02-10T07:54:27.184790+0000","last_undegraded":"2024-02-10T07:54:27.184790+0000","last_fullsized":"2024-02-10T07:54:27.184790+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:44.324046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:10.120462+0000","last_clean_scrub_stamp":"2024-02-10T07:49:44.324046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":239,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183581+0000","last_change":"2024-02-10T07:52:34.860375+0000","last_active":"2024-02-10T07:54:27.183581+0000","last_peered":"2024-02-10T07:54:27.183581+0000","last_clean":"2024-02-10T07:54:27.183581+0000","last_became_active":"2024-02-10T07:49:06.262933+0000","last_became_peered":"2024-02-10T07:49:06.262933+0000","last_unstale":"2024-02-10T07:54:27.183581+0000","last_undegraded":"2024-02-10T07:54:27.183581+0000","last_fullsized":"2024-02-10T07:54:27.183581+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:34.860273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:17.587433+0000","last_clean_scrub_stamp":"2024-02-10T07:52:34.860273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":891,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.138486+0000","last_change":"2024-02-10T07:49:59.278666+0000","last_active":"2024-02-10T07:54:27.138486+0000","last_peered":"2024-02-10T07:54:27.138486+0000","last_clean":"2024-02-10T07:54:27.138486+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:54:27.138486+0000","last_undegraded":"2024-02-10T07:54:27.138486+0000","last_fullsized":"2024-02-10T07:54:27.138486+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.278574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.278574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"430'25","reported_seq":960,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184197+0000","last_change":"2024-02-10T07:53:44.569050+0000","last_active":"2024-02-10T07:54:27.184197+0000","last_peered":"2024-02-10T07:54:27.184197+0000","last_clean":"2024-02-10T07:54:27.184197+0000","last_became_active":"2024-02-10T07:49:06.260962+0000","last_became_peered":"2024-02-10T07:49:06.260962+0000","last_unstale":"2024-02-10T07:54:27.184197+0000","last_undegraded":"2024-02-10T07:54:27.184197+0000","last_fullsized":"2024-02-10T07:54:27.184197+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'25","last_scrub_stamp":"2024-02-10T07:53:44.568910+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:53:44.568910+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:59.428046+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":871,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139268+0000","last_change":"2024-02-10T07:48:58.511630+0000","last_active":"2024-02-10T07:54:27.139268+0000","last_peered":"2024-02-10T07:54:27.139268+0000","last_clean":"2024-02-10T07:54:27.139268+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:54:27.139268+0000","last_undegraded":"2024-02-10T07:54:27.139268+0000","last_fullsized":"2024-02-10T07:54:27.139268+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:58.511251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.511251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":213,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-02-10T07:54:30.312050+0000","last_change":"2024-02-10T07:53:42.214117+0000","last_active":"2024-02-10T07:54:30.312050+0000","last_peered":"2024-02-10T07:54:30.312050+0000","last_clean":"2024-02-10T07:54:30.312050+0000","last_became_active":"2024-02-10T07:49:06.250832+0000","last_became_peered":"2024-02-10T07:49:06.250832+0000","last_unstale":"2024-02-10T07:54:30.312050+0000","last_undegraded":"2024-02-10T07:54:30.312050+0000","last_fullsized":"2024-02-10T07:54:30.312050+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:42.213981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:42.213981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:57.611769+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":923,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.182995+0000","last_change":"2024-02-10T07:52:16.809094+0000","last_active":"2024-02-10T07:54:27.182995+0000","last_peered":"2024-02-10T07:54:27.182995+0000","last_clean":"2024-02-10T07:54:27.182995+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:54:27.182995+0000","last_undegraded":"2024-02-10T07:54:27.182995+0000","last_fullsized":"2024-02-10T07:54:27.182995+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:16.808978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:52:16.808978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"430'18","reported_seq":230,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-02-10T07:54:30.312631+0000","last_change":"2024-02-10T07:53:29.079571+0000","last_active":"2024-02-10T07:54:30.312631+0000","last_peered":"2024-02-10T07:54:30.312631+0000","last_clean":"2024-02-10T07:54:30.312631+0000","last_became_active":"2024-02-10T07:49:06.258281+0000","last_became_peered":"2024-02-10T07:49:06.258281+0000","last_unstale":"2024-02-10T07:54:30.312631+0000","last_undegraded":"2024-02-10T07:54:30.312631+0000","last_fullsized":"2024-02-10T07:54:30.312631+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'18","last_scrub_stamp":"2024-02-10T07:53:29.076952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:53:29.076952+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:30.380886+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":412,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183077+0000","last_change":"2024-02-10T07:52:21.875347+0000","last_active":"2024-02-10T07:54:27.183077+0000","last_peered":"2024-02-10T07:54:27.183077+0000","last_clean":"2024-02-10T07:54:27.183077+0000","last_became_active":"2024-02-10T07:48:02.366002+0000","last_became_peered":"2024-02-10T07:48:02.366002+0000","last_unstale":"2024-02-10T07:54:27.183077+0000","last_undegraded":"2024-02-10T07:54:27.183077+0000","last_fullsized":"2024-02-10T07:54:27.183077+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:21.875258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:21.875258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"245'4","reported_seq":906,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139347+0000","last_change":"2024-02-10T07:49:08.311056+0000","last_active":"2024-02-10T07:54:27.139347+0000","last_peered":"2024-02-10T07:54:27.139347+0000","last_clean":"2024-02-10T07:54:27.139347+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:54:27.139347+0000","last_undegraded":"2024-02-10T07:54:27.139347+0000","last_fullsized":"2024-02-10T07:54:27.139347+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'4","last_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_deep_scrub":"245'4","last_deep_scrub_stamp":"2024-02-10T07:49:08.310917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:08.310917+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"430'46","reported_seq":649,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.138676+0000","last_change":"2024-02-10T07:48:11.967227+0000","last_active":"2024-02-10T07:54:27.138676+0000","last_peered":"2024-02-10T07:54:27.138676+0000","last_clean":"2024-02-10T07:54:27.138676+0000","last_became_active":"2024-02-10T07:48:02.330130+0000","last_became_peered":"2024-02-10T07:48:02.330130+0000","last_unstale":"2024-02-10T07:54:27.138676+0000","last_undegraded":"2024-02-10T07:54:27.138676+0000","last_fullsized":"2024-02-10T07:54:27.138676+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'34","last_scrub_stamp":"2024-02-10T07:48:11.966957+0000","last_deep_scrub":"246'22","last_deep_scrub_stamp":"2024-02-10T07:41:26.771112+0000","last_clean_scrub_stamp":"2024-02-10T07:48:11.966957+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":1164,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.138597+0000","last_change":"2024-02-10T07:48:53.423840+0000","last_active":"2024-02-10T07:54:27.138597+0000","last_peered":"2024-02-10T07:54:27.138597+0000","last_clean":"2024-02-10T07:54:27.138597+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:54:27.138597+0000","last_undegraded":"2024-02-10T07:54:27.138597+0000","last_fullsized":"2024-02-10T07:54:27.138597+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:48:53.423613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:53.423613+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":720,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-02-10T07:54:30.311775+0000","last_change":"2024-02-10T07:53:32.243227+0000","last_active":"2024-02-10T07:54:30.311775+0000","last_peered":"2024-02-10T07:54:30.311775+0000","last_clean":"2024-02-10T07:54:30.311775+0000","last_became_active":"2024-02-10T07:47:47.966183+0000","last_became_peered":"2024-02-10T07:47:47.966183+0000","last_unstale":"2024-02-10T07:54:30.311775+0000","last_undegraded":"2024-02-10T07:54:30.311775+0000","last_fullsized":"2024-02-10T07:54:30.311775+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:32.242307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:53:32.242307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:47.454633+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":884,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139115+0000","last_change":"2024-02-10T07:48:56.416159+0000","last_active":"2024-02-10T07:54:27.139115+0000","last_peered":"2024-02-10T07:54:27.139115+0000","last_clean":"2024-02-10T07:54:27.139115+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:54:27.139115+0000","last_undegraded":"2024-02-10T07:54:27.139115+0000","last_fullsized":"2024-02-10T07:54:27.139115+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:56.416037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:48:56.416037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"430'29","reported_seq":999,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.140073+0000","last_change":"2024-02-10T07:49:43.339027+0000","last_active":"2024-02-10T07:54:27.140073+0000","last_peered":"2024-02-10T07:54:27.140073+0000","last_clean":"2024-02-10T07:54:27.140073+0000","last_became_active":"2024-02-10T07:49:06.243569+0000","last_became_peered":"2024-02-10T07:49:06.243569+0000","last_unstale":"2024-02-10T07:54:27.140073+0000","last_undegraded":"2024-02-10T07:54:27.140073+0000","last_fullsized":"2024-02-10T07:54:27.140073+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-02-10T07:49:43.338928+0000","last_deep_scrub":"245'17","last_deep_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_clean_scrub_stamp":"2024-02-10T07:49:43.338928+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":136,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183170+0000","last_change":"2024-02-10T07:50:43.285105+0000","last_active":"2024-02-10T07:54:27.183170+0000","last_peered":"2024-02-10T07:54:27.183170+0000","last_clean":"2024-02-10T07:54:27.183170+0000","last_became_active":"2024-02-10T07:50:43.284749+0000","last_became_peered":"2024-02-10T07:50:43.284749+0000","last_unstale":"2024-02-10T07:54:27.183170+0000","last_undegraded":"2024-02-10T07:54:27.183170+0000","last_fullsized":"2024-02-10T07:54:27.183170+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:04.308807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.308807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":905,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.138224+0000","last_change":"2024-02-10T07:50:43.281637+0000","last_active":"2024-02-10T07:54:27.138224+0000","last_peered":"2024-02-10T07:54:27.138224+0000","last_clean":"2024-02-10T07:54:27.138224+0000","last_became_active":"2024-02-10T07:50:43.281262+0000","last_became_peered":"2024-02-10T07:50:43.281262+0000","last_unstale":"2024-02-10T07:54:27.138224+0000","last_undegraded":"2024-02-10T07:54:27.138224+0000","last_fullsized":"2024-02-10T07:54:27.138224+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_clean_scrub_stamp":"2024-02-10T07:49:11.361139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":136,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183912+0000","last_change":"2024-02-10T07:50:43.281022+0000","last_active":"2024-02-10T07:54:27.183912+0000","last_peered":"2024-02-10T07:54:27.183912+0000","last_clean":"2024-02-10T07:54:27.183912+0000","last_became_active":"2024-02-10T07:50:43.280617+0000","last_became_peered":"2024-02-10T07:50:43.280617+0000","last_unstale":"2024-02-10T07:54:27.183912+0000","last_undegraded":"2024-02-10T07:54:27.183912+0000","last_fullsized":"2024-02-10T07:54:27.183912+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:21.172601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:48:21.172601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":901,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139540+0000","last_change":"2024-02-10T07:50:07.252577+0000","last_active":"2024-02-10T07:54:27.139540+0000","last_peered":"2024-02-10T07:54:27.139540+0000","last_clean":"2024-02-10T07:54:27.139540+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:54:27.139540+0000","last_undegraded":"2024-02-10T07:54:27.139540+0000","last_fullsized":"2024-02-10T07:54:27.139540+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.252458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.252458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"430'51","reported_seq":990,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:28.279871+0000","last_change":"2024-02-10T07:54:28.279902+0000","last_active":"2024-02-10T07:54:28.279871+0000","last_peered":"2024-02-10T07:54:28.279871+0000","last_clean":"2024-02-10T07:54:28.279871+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:54:28.279871+0000","last_undegraded":"2024-02-10T07:54:28.279871+0000","last_fullsized":"2024-02-10T07:54:28.279871+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'51","last_scrub_stamp":"2024-02-10T07:54:28.279807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:54:28.279807+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:52.528856+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":894,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183325+0000","last_change":"2024-02-10T07:50:43.284817+0000","last_active":"2024-02-10T07:54:27.183325+0000","last_peered":"2024-02-10T07:54:27.183325+0000","last_clean":"2024-02-10T07:54:27.183325+0000","last_became_active":"2024-02-10T07:50:43.284289+0000","last_became_peered":"2024-02-10T07:50:43.284289+0000","last_unstale":"2024-02-10T07:54:27.183325+0000","last_undegraded":"2024-02-10T07:54:27.183325+0000","last_fullsized":"2024-02-10T07:54:27.183325+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:00.336175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:00.336175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":136,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183711+0000","last_change":"2024-02-10T07:50:43.295279+0000","last_active":"2024-02-10T07:54:27.183711+0000","last_peered":"2024-02-10T07:54:27.183711+0000","last_clean":"2024-02-10T07:54:27.183711+0000","last_became_active":"2024-02-10T07:50:43.294972+0000","last_became_peered":"2024-02-10T07:50:43.294972+0000","last_unstale":"2024-02-10T07:54:27.183711+0000","last_undegraded":"2024-02-10T07:54:27.183711+0000","last_fullsized":"2024-02-10T07:54:27.183711+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:46:28.822700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:46:28.822700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":213,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-02-10T07:54:30.311779+0000","last_change":"2024-02-10T07:53:35.090294+0000","last_active":"2024-02-10T07:54:30.311779+0000","last_peered":"2024-02-10T07:54:30.311779+0000","last_clean":"2024-02-10T07:54:30.311779+0000","last_became_active":"2024-02-10T07:49:06.249431+0000","last_became_peered":"2024-02-10T07:49:06.249431+0000","last_unstale":"2024-02-10T07:54:30.311779+0000","last_undegraded":"2024-02-10T07:54:30.311779+0000","last_fullsized":"2024-02-10T07:54:30.311779+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:35.090175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:53:35.090175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:50.135166+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"430'21","reported_seq":914,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184131+0000","last_change":"2024-02-10T07:48:59.304277+0000","last_active":"2024-02-10T07:54:27.184131+0000","last_peered":"2024-02-10T07:54:27.184131+0000","last_clean":"2024-02-10T07:54:27.184131+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:54:27.184131+0000","last_undegraded":"2024-02-10T07:54:27.184131+0000","last_fullsized":"2024-02-10T07:54:27.184131+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'15","last_scrub_stamp":"2024-02-10T07:48:59.304199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.304199+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":213,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-02-10T07:54:30.312149+0000","last_change":"2024-02-10T07:53:41.173096+0000","last_active":"2024-02-10T07:54:30.312149+0000","last_peered":"2024-02-10T07:54:30.312149+0000","last_clean":"2024-02-10T07:54:30.312149+0000","last_became_active":"2024-02-10T07:49:06.238367+0000","last_became_peered":"2024-02-10T07:49:06.238367+0000","last_unstale":"2024-02-10T07:54:30.312149+0000","last_undegraded":"2024-02-10T07:54:30.312149+0000","last_fullsized":"2024-02-10T07:54:30.312149+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:41.172967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:53:41.172967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:48.222331+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":304,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184961+0000","last_change":"2024-02-10T07:49:10.087939+0000","last_active":"2024-02-10T07:54:27.184961+0000","last_peered":"2024-02-10T07:54:27.184961+0000","last_clean":"2024-02-10T07:54:27.184961+0000","last_became_active":"2024-02-10T07:48:02.328316+0000","last_became_peered":"2024-02-10T07:48:02.328316+0000","last_unstale":"2024-02-10T07:54:27.184961+0000","last_undegraded":"2024-02-10T07:54:27.184961+0000","last_fullsized":"2024-02-10T07:54:27.184961+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:49:10.087795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:49:10.087795+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":876,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139712+0000","last_change":"2024-02-10T07:49:19.198419+0000","last_active":"2024-02-10T07:54:27.139712+0000","last_peered":"2024-02-10T07:54:27.139712+0000","last_clean":"2024-02-10T07:54:27.139712+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:54:27.139712+0000","last_undegraded":"2024-02-10T07:54:27.139712+0000","last_fullsized":"2024-02-10T07:54:27.139712+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:19.198312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:04.220328+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.198312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"430'59","reported_seq":1352,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184700+0000","last_change":"2024-02-10T07:50:43.327112+0000","last_active":"2024-02-10T07:54:27.184700+0000","last_peered":"2024-02-10T07:54:27.184700+0000","last_clean":"2024-02-10T07:54:27.184700+0000","last_became_active":"2024-02-10T07:50:43.297262+0000","last_became_peered":"2024-02-10T07:50:43.297262+0000","last_unstale":"2024-02-10T07:54:27.184700+0000","last_undegraded":"2024-02-10T07:54:27.184700+0000","last_fullsized":"2024-02-10T07:54:27.184700+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'46","last_scrub_stamp":"2024-02-10T07:49:55.714323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:55.714323+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":884,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183511+0000","last_change":"2024-02-10T07:49:17.357664+0000","last_active":"2024-02-10T07:54:27.183511+0000","last_peered":"2024-02-10T07:54:27.183511+0000","last_clean":"2024-02-10T07:54:27.183511+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:54:27.183511+0000","last_undegraded":"2024-02-10T07:54:27.183511+0000","last_fullsized":"2024-02-10T07:54:27.183511+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.357533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:22.786949+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.357533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":881,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139703+0000","last_change":"2024-02-10T07:54:24.843973+0000","last_active":"2024-02-10T07:54:27.139703+0000","last_peered":"2024-02-10T07:54:27.139703+0000","last_clean":"2024-02-10T07:54:27.139703+0000","last_became_active":"2024-02-10T07:47:48.149256+0000","last_became_peered":"2024-02-10T07:47:48.149256+0000","last_unstale":"2024-02-10T07:54:27.139703+0000","last_undegraded":"2024-02-10T07:54:27.139703+0000","last_fullsized":"2024-02-10T07:54:27.139703+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:24.843861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:42.730875+0000","last_clean_scrub_stamp":"2024-02-10T07:54:24.843861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:41.517344+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":903,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184094+0000","last_change":"2024-02-10T07:50:43.288780+0000","last_active":"2024-02-10T07:54:27.184094+0000","last_peered":"2024-02-10T07:54:27.184094+0000","last_clean":"2024-02-10T07:54:27.184094+0000","last_became_active":"2024-02-10T07:50:43.288388+0000","last_became_peered":"2024-02-10T07:50:43.288388+0000","last_unstale":"2024-02-10T07:54:27.184094+0000","last_undegraded":"2024-02-10T07:54:27.184094+0000","last_fullsized":"2024-02-10T07:54:27.184094+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:49.886354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:49.886354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":136,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183804+0000","last_change":"2024-02-10T07:50:43.287356+0000","last_active":"2024-02-10T07:54:27.183804+0000","last_peered":"2024-02-10T07:54:27.183804+0000","last_clean":"2024-02-10T07:54:27.183804+0000","last_became_active":"2024-02-10T07:50:43.287043+0000","last_became_peered":"2024-02-10T07:50:43.287043+0000","last_unstale":"2024-02-10T07:54:27.183804+0000","last_undegraded":"2024-02-10T07:54:27.183804+0000","last_fullsized":"2024-02-10T07:54:27.183804+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:15.234112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.234112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"430'21","reported_seq":243,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139106+0000","last_change":"2024-02-10T07:49:04.340445+0000","last_active":"2024-02-10T07:54:27.139106+0000","last_peered":"2024-02-10T07:54:27.139106+0000","last_clean":"2024-02-10T07:54:27.139106+0000","last_became_active":"2024-02-10T07:48:35.057662+0000","last_became_peered":"2024-02-10T07:48:35.057662+0000","last_unstale":"2024-02-10T07:54:27.139106+0000","last_undegraded":"2024-02-10T07:54:27.139106+0000","last_fullsized":"2024-02-10T07:54:27.139106+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'17","last_scrub_stamp":"2024-02-10T07:49:04.340366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:04.340366+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":136,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184112+0000","last_change":"2024-02-10T07:50:43.280896+0000","last_active":"2024-02-10T07:54:27.184112+0000","last_peered":"2024-02-10T07:54:27.184112+0000","last_clean":"2024-02-10T07:54:27.184112+0000","last_became_active":"2024-02-10T07:50:43.279125+0000","last_became_peered":"2024-02-10T07:50:43.279125+0000","last_unstale":"2024-02-10T07:54:27.184112+0000","last_undegraded":"2024-02-10T07:54:27.184112+0000","last_fullsized":"2024-02-10T07:54:27.184112+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:50.232544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.232544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":300,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139956+0000","last_change":"2024-02-10T07:54:26.826207+0000","last_active":"2024-02-10T07:54:27.139956+0000","last_peered":"2024-02-10T07:54:27.139956+0000","last_clean":"2024-02-10T07:54:27.139956+0000","last_became_active":"2024-02-10T07:48:02.329687+0000","last_became_peered":"2024-02-10T07:48:02.329687+0000","last_unstale":"2024-02-10T07:54:27.139956+0000","last_undegraded":"2024-02-10T07:54:27.139956+0000","last_fullsized":"2024-02-10T07:54:27.139956+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:54:26.826096+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_clean_scrub_stamp":"2024-02-10T07:54:26.826096+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:36.692789+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1130,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.138334+0000","last_change":"2024-02-10T07:48:15.033077+0000","last_active":"2024-02-10T07:54:27.138334+0000","last_peered":"2024-02-10T07:54:27.138334+0000","last_clean":"2024-02-10T07:54:27.138334+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:54:27.138334+0000","last_undegraded":"2024-02-10T07:54:27.138334+0000","last_fullsized":"2024-02-10T07:54:27.138334+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:48:15.032944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.032944+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":530,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183332+0000","last_change":"2024-02-10T07:54:25.279368+0000","last_active":"2024-02-10T07:54:27.183332+0000","last_peered":"2024-02-10T07:54:27.183332+0000","last_clean":"2024-02-10T07:54:27.183332+0000","last_became_active":"2024-02-10T07:49:06.256223+0000","last_became_peered":"2024-02-10T07:49:06.256223+0000","last_unstale":"2024-02-10T07:54:27.183332+0000","last_undegraded":"2024-02-10T07:54:27.183332+0000","last_fullsized":"2024-02-10T07:54:27.183332+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:25.279268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:54:25.279268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:49.367909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"430'48","reported_seq":1241,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183483+0000","last_change":"2024-02-10T07:52:03.839258+0000","last_active":"2024-02-10T07:54:27.183483+0000","last_peered":"2024-02-10T07:54:27.183483+0000","last_clean":"2024-02-10T07:54:27.183483+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:54:27.183483+0000","last_undegraded":"2024-02-10T07:54:27.183483+0000","last_fullsized":"2024-02-10T07:54:27.183483+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'48","last_scrub_stamp":"2024-02-10T07:52:03.839174+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:52:03.839174+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":905,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139137+0000","last_change":"2024-02-10T07:50:43.285613+0000","last_active":"2024-02-10T07:54:27.139137+0000","last_peered":"2024-02-10T07:54:27.139137+0000","last_clean":"2024-02-10T07:54:27.139137+0000","last_became_active":"2024-02-10T07:50:43.284708+0000","last_became_peered":"2024-02-10T07:50:43.284708+0000","last_unstale":"2024-02-10T07:54:27.139137+0000","last_undegraded":"2024-02-10T07:54:27.139137+0000","last_fullsized":"2024-02-10T07:54:27.139137+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:13.369607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.736901+0000","last_clean_scrub_stamp":"2024-02-10T07:49:13.369607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":677,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184330+0000","last_change":"2024-02-10T07:50:07.589020+0000","last_active":"2024-02-10T07:54:27.184330+0000","last_peered":"2024-02-10T07:54:27.184330+0000","last_clean":"2024-02-10T07:54:27.184330+0000","last_became_active":"2024-02-10T07:49:06.270457+0000","last_became_peered":"2024-02-10T07:49:06.270457+0000","last_unstale":"2024-02-10T07:54:27.184330+0000","last_undegraded":"2024-02-10T07:54:27.184330+0000","last_fullsized":"2024-02-10T07:54:27.184330+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:50:07.588888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:50:07.588888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":899,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183083+0000","last_change":"2024-02-10T07:53:41.577827+0000","last_active":"2024-02-10T07:54:27.183083+0000","last_peered":"2024-02-10T07:54:27.183083+0000","last_clean":"2024-02-10T07:54:27.183083+0000","last_became_active":"2024-02-10T07:49:06.266625+0000","last_became_peered":"2024-02-10T07:49:06.266625+0000","last_unstale":"2024-02-10T07:54:27.183083+0000","last_undegraded":"2024-02-10T07:54:27.183083+0000","last_fullsized":"2024-02-10T07:54:27.183083+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:41.577586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_clean_scrub_stamp":"2024-02-10T07:53:41.577586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:59.035911+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2255,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.348182+0000","last_change":"2024-02-10T07:50:04.159154+0000","last_active":"2024-02-10T07:54:27.348182+0000","last_peered":"2024-02-10T07:54:27.348182+0000","last_clean":"2024-02-10T07:54:27.348182+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:54:27.348182+0000","last_undegraded":"2024-02-10T07:54:27.348182+0000","last_fullsized":"2024-02-10T07:54:27.348182+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:50:04.159046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:50:04.159046+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1350,"num_read_kb":1345,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"430'42","reported_seq":980,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183891+0000","last_change":"2024-02-10T07:50:43.324808+0000","last_active":"2024-02-10T07:54:27.183891+0000","last_peered":"2024-02-10T07:54:27.183891+0000","last_clean":"2024-02-10T07:54:27.183891+0000","last_became_active":"2024-02-10T07:50:43.291771+0000","last_became_peered":"2024-02-10T07:50:43.291771+0000","last_unstale":"2024-02-10T07:54:27.183891+0000","last_undegraded":"2024-02-10T07:54:27.183891+0000","last_fullsized":"2024-02-10T07:54:27.183891+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:47.417096+0000","last_deep_scrub":"237'18","last_deep_scrub_stamp":"2024-02-10T07:40:31.005376+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.417096+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":855,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.143122+0000","last_change":"2024-02-10T07:53:34.104503+0000","last_active":"2024-02-10T07:54:27.143122+0000","last_peered":"2024-02-10T07:54:27.143122+0000","last_clean":"2024-02-10T07:54:27.143122+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:54:27.143122+0000","last_undegraded":"2024-02-10T07:54:27.143122+0000","last_fullsized":"2024-02-10T07:54:27.143122+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:34.104386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:53:34.104386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:46.902238+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":898,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183526+0000","last_change":"2024-02-10T07:50:43.296823+0000","last_active":"2024-02-10T07:54:27.183526+0000","last_peered":"2024-02-10T07:54:27.183526+0000","last_clean":"2024-02-10T07:54:27.183526+0000","last_became_active":"2024-02-10T07:50:43.296488+0000","last_became_peered":"2024-02-10T07:50:43.296488+0000","last_unstale":"2024-02-10T07:54:27.183526+0000","last_undegraded":"2024-02-10T07:54:27.183526+0000","last_fullsized":"2024-02-10T07:54:27.183526+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:59.630083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:59.630083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":903,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184612+0000","last_change":"2024-02-10T07:50:43.298468+0000","last_active":"2024-02-10T07:54:27.184612+0000","last_peered":"2024-02-10T07:54:27.184612+0000","last_clean":"2024-02-10T07:54:27.184612+0000","last_became_active":"2024-02-10T07:50:43.297861+0000","last_became_peered":"2024-02-10T07:50:43.297861+0000","last_unstale":"2024-02-10T07:54:27.184612+0000","last_undegraded":"2024-02-10T07:54:27.184612+0000","last_fullsized":"2024-02-10T07:54:27.184612+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:05.193158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:49:05.193158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"430'47","reported_seq":963,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183868+0000","last_change":"2024-02-10T07:49:57.522503+0000","last_active":"2024-02-10T07:54:27.183868+0000","last_peered":"2024-02-10T07:54:27.183868+0000","last_clean":"2024-02-10T07:54:27.183868+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:54:27.183868+0000","last_undegraded":"2024-02-10T07:54:27.183868+0000","last_fullsized":"2024-02-10T07:54:27.183868+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'37","last_scrub_stamp":"2024-02-10T07:49:57.522391+0000","last_deep_scrub":"271'37","last_deep_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.522391+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":922,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183211+0000","last_change":"2024-02-10T07:54:23.265999+0000","last_active":"2024-02-10T07:54:27.183211+0000","last_peered":"2024-02-10T07:54:27.183211+0000","last_clean":"2024-02-10T07:54:27.183211+0000","last_became_active":"2024-02-10T07:48:35.062118+0000","last_became_peered":"2024-02-10T07:48:35.062118+0000","last_unstale":"2024-02-10T07:54:27.183211+0000","last_undegraded":"2024-02-10T07:54:27.183211+0000","last_fullsized":"2024-02-10T07:54:27.183211+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:23.265906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:00.292320+0000","last_clean_scrub_stamp":"2024-02-10T07:54:23.265906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:43.409955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":775,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.182930+0000","last_change":"2024-02-10T07:52:23.855559+0000","last_active":"2024-02-10T07:54:27.182930+0000","last_peered":"2024-02-10T07:54:27.182930+0000","last_clean":"2024-02-10T07:54:27.182930+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:54:27.182930+0000","last_undegraded":"2024-02-10T07:54:27.182930+0000","last_fullsized":"2024-02-10T07:54:27.182930+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:23.855477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:52:23.855477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":136,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184275+0000","last_change":"2024-02-10T07:50:43.289736+0000","last_active":"2024-02-10T07:54:27.184275+0000","last_peered":"2024-02-10T07:54:27.184275+0000","last_clean":"2024-02-10T07:54:27.184275+0000","last_became_active":"2024-02-10T07:50:43.289458+0000","last_became_peered":"2024-02-10T07:50:43.289458+0000","last_unstale":"2024-02-10T07:54:27.184275+0000","last_undegraded":"2024-02-10T07:54:27.184275+0000","last_fullsized":"2024-02-10T07:54:27.184275+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:48.809513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:49:48.809513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"430'35","reported_seq":709,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.185159+0000","last_change":"2024-02-10T07:49:19.084010+0000","last_active":"2024-02-10T07:54:27.185159+0000","last_peered":"2024-02-10T07:54:27.185159+0000","last_clean":"2024-02-10T07:54:27.185159+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:54:27.185159+0000","last_undegraded":"2024-02-10T07:54:27.185159+0000","last_fullsized":"2024-02-10T07:54:27.185159+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"382'25","last_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_deep_scrub":"382'25","last_deep_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_clean_scrub_stamp":"2024-02-10T07:49:19.083856+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"158'10","reported_seq":1166,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184192+0000","last_change":"2024-02-10T07:49:50.396822+0000","last_active":"2024-02-10T07:54:27.184192+0000","last_peered":"2024-02-10T07:54:27.184192+0000","last_clean":"2024-02-10T07:54:27.184192+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:54:27.184192+0000","last_undegraded":"2024-02-10T07:54:27.184192+0000","last_fullsized":"2024-02-10T07:54:27.184192+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'10","last_scrub_stamp":"2024-02-10T07:49:50.396730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:50.396730+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":884,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.138766+0000","last_change":"2024-02-10T07:50:43.283505+0000","last_active":"2024-02-10T07:54:27.138766+0000","last_peered":"2024-02-10T07:54:27.138766+0000","last_clean":"2024-02-10T07:54:27.138766+0000","last_became_active":"2024-02-10T07:50:43.283161+0000","last_became_peered":"2024-02-10T07:50:43.283161+0000","last_unstale":"2024-02-10T07:54:27.138766+0000","last_undegraded":"2024-02-10T07:54:27.138766+0000","last_fullsized":"2024-02-10T07:54:27.138766+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_clean_scrub_stamp":"2024-02-10T07:48:23.271454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"296'2","reported_seq":894,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183994+0000","last_change":"2024-02-10T07:50:03.643473+0000","last_active":"2024-02-10T07:54:27.183994+0000","last_peered":"2024-02-10T07:54:27.183994+0000","last_clean":"2024-02-10T07:54:27.183994+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:54:27.183994+0000","last_undegraded":"2024-02-10T07:54:27.183994+0000","last_fullsized":"2024-02-10T07:54:27.183994+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'2","last_scrub_stamp":"2024-02-10T07:50:03.643367+0000","last_deep_scrub":"296'2","last_deep_scrub_stamp":"2024-02-10T07:45:03.233375+0000","last_clean_scrub_stamp":"2024-02-10T07:50:03.643367+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"430'42","reported_seq":944,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139877+0000","last_change":"2024-02-10T07:49:20.216865+0000","last_active":"2024-02-10T07:54:27.139877+0000","last_peered":"2024-02-10T07:54:27.139877+0000","last_clean":"2024-02-10T07:54:27.139877+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:54:27.139877+0000","last_undegraded":"2024-02-10T07:54:27.139877+0000","last_fullsized":"2024-02-10T07:54:27.139877+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-02-10T07:49:20.216750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:20.216750+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"376'12","reported_seq":953,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183818+0000","last_change":"2024-02-10T07:49:06.282456+0000","last_active":"2024-02-10T07:54:27.183818+0000","last_peered":"2024-02-10T07:54:27.183818+0000","last_clean":"2024-02-10T07:54:27.183818+0000","last_became_active":"2024-02-10T07:49:06.255324+0000","last_became_peered":"2024-02-10T07:49:06.255324+0000","last_unstale":"2024-02-10T07:54:27.183818+0000","last_undegraded":"2024-02-10T07:54:27.183818+0000","last_fullsized":"2024-02-10T07:54:27.183818+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'11","last_scrub_stamp":"2024-02-10T07:48:58.368663+0000","last_deep_scrub":"227'9","last_deep_scrub_stamp":"2024-02-10T07:39:53.440221+0000","last_clean_scrub_stamp":"2024-02-10T07:48:58.368663+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":889,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139065+0000","last_change":"2024-02-10T07:54:21.727843+0000","last_active":"2024-02-10T07:54:27.139065+0000","last_peered":"2024-02-10T07:54:27.139065+0000","last_clean":"2024-02-10T07:54:27.139065+0000","last_became_active":"2024-02-10T07:50:43.281465+0000","last_became_peered":"2024-02-10T07:50:43.281465+0000","last_unstale":"2024-02-10T07:54:27.139065+0000","last_undegraded":"2024-02-10T07:54:27.139065+0000","last_fullsized":"2024-02-10T07:54:27.139065+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:21.727684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:54:21.727684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:26.036430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":366,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139206+0000","last_change":"2024-02-10T07:47:50.948710+0000","last_active":"2024-02-10T07:54:27.139206+0000","last_peered":"2024-02-10T07:54:27.139206+0000","last_clean":"2024-02-10T07:54:27.139206+0000","last_became_active":"2024-02-10T07:47:47.966443+0000","last_became_peered":"2024-02-10T07:47:47.966443+0000","last_unstale":"2024-02-10T07:54:27.139206+0000","last_undegraded":"2024-02-10T07:54:27.139206+0000","last_fullsized":"2024-02-10T07:54:27.139206+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:47:50.948624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:47:50.948624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"430'42","reported_seq":883,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.138471+0000","last_change":"2024-02-10T07:49:57.303310+0000","last_active":"2024-02-10T07:54:27.138471+0000","last_peered":"2024-02-10T07:54:27.138471+0000","last_clean":"2024-02-10T07:54:27.138471+0000","last_became_active":"2024-02-10T07:49:06.252935+0000","last_became_peered":"2024-02-10T07:49:06.252935+0000","last_unstale":"2024-02-10T07:54:27.138471+0000","last_undegraded":"2024-02-10T07:54:27.138471+0000","last_fullsized":"2024-02-10T07:54:27.138471+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'30","last_scrub_stamp":"2024-02-10T07:49:57.303200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.303200+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":891,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139635+0000","last_change":"2024-02-10T07:54:24.728577+0000","last_active":"2024-02-10T07:54:27.139635+0000","last_peered":"2024-02-10T07:54:27.139635+0000","last_clean":"2024-02-10T07:54:27.139635+0000","last_became_active":"2024-02-10T07:49:06.239190+0000","last_became_peered":"2024-02-10T07:49:06.239190+0000","last_unstale":"2024-02-10T07:54:27.139635+0000","last_undegraded":"2024-02-10T07:54:27.139635+0000","last_fullsized":"2024-02-10T07:54:27.139635+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:24.728480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:54:24.728480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:53.925678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":390,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184430+0000","last_change":"2024-02-10T07:49:06.274374+0000","last_active":"2024-02-10T07:54:27.184430+0000","last_peered":"2024-02-10T07:54:27.184430+0000","last_clean":"2024-02-10T07:54:27.184430+0000","last_became_active":"2024-02-10T07:48:02.371926+0000","last_became_peered":"2024-02-10T07:48:02.371926+0000","last_unstale":"2024-02-10T07:54:27.184430+0000","last_undegraded":"2024-02-10T07:54:27.184430+0000","last_fullsized":"2024-02-10T07:54:27.184430+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:06.274263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:49.704024+0000","last_clean_scrub_stamp":"2024-02-10T07:49:06.274263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"395'12","reported_seq":309,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183519+0000","last_change":"2024-02-10T07:52:24.850402+0000","last_active":"2024-02-10T07:54:27.183519+0000","last_peered":"2024-02-10T07:54:27.183519+0000","last_clean":"2024-02-10T07:54:27.183519+0000","last_became_active":"2024-02-10T07:48:35.071916+0000","last_became_peered":"2024-02-10T07:48:35.071916+0000","last_unstale":"2024-02-10T07:54:27.183519+0000","last_undegraded":"2024-02-10T07:54:27.183519+0000","last_fullsized":"2024-02-10T07:54:27.183519+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'12","last_scrub_stamp":"2024-02-10T07:52:24.850269+0000","last_deep_scrub":"395'12","last_deep_scrub_stamp":"2024-02-10T07:52:24.850269+0000","last_clean_scrub_stamp":"2024-02-10T07:52:24.850269+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":213,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-02-10T07:54:30.311415+0000","last_change":"2024-02-10T07:53:37.144276+0000","last_active":"2024-02-10T07:54:30.311415+0000","last_peered":"2024-02-10T07:54:30.311415+0000","last_clean":"2024-02-10T07:54:30.311415+0000","last_became_active":"2024-02-10T07:49:06.251038+0000","last_became_peered":"2024-02-10T07:49:06.251038+0000","last_unstale":"2024-02-10T07:54:30.311415+0000","last_undegraded":"2024-02-10T07:54:30.311415+0000","last_fullsized":"2024-02-10T07:54:30.311415+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:37.144142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:53:37.144142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:01.122427+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"430'39","reported_seq":956,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184965+0000","last_change":"2024-02-10T07:50:00.655466+0000","last_active":"2024-02-10T07:54:27.184965+0000","last_peered":"2024-02-10T07:54:27.184965+0000","last_clean":"2024-02-10T07:54:27.184965+0000","last_became_active":"2024-02-10T07:47:47.967427+0000","last_became_peered":"2024-02-10T07:47:47.967427+0000","last_unstale":"2024-02-10T07:54:27.184965+0000","last_undegraded":"2024-02-10T07:54:27.184965+0000","last_fullsized":"2024-02-10T07:54:27.184965+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'27","last_scrub_stamp":"2024-02-10T07:50:00.655377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:50:00.655377+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":585,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183327+0000","last_change":"2024-02-10T07:51:59.948039+0000","last_active":"2024-02-10T07:54:27.183327+0000","last_peered":"2024-02-10T07:54:27.183327+0000","last_clean":"2024-02-10T07:54:27.183327+0000","last_became_active":"2024-02-10T07:49:06.268874+0000","last_became_peered":"2024-02-10T07:49:06.268874+0000","last_unstale":"2024-02-10T07:54:27.183327+0000","last_undegraded":"2024-02-10T07:54:27.183327+0000","last_fullsized":"2024-02-10T07:54:27.183327+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:51:59.947908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_clean_scrub_stamp":"2024-02-10T07:51:59.947908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":364,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.140089+0000","last_change":"2024-02-10T07:49:57.310447+0000","last_active":"2024-02-10T07:54:27.140089+0000","last_peered":"2024-02-10T07:54:27.140089+0000","last_clean":"2024-02-10T07:54:27.140089+0000","last_became_active":"2024-02-10T07:48:02.333689+0000","last_became_peered":"2024-02-10T07:48:02.333689+0000","last_unstale":"2024-02-10T07:54:27.140089+0000","last_undegraded":"2024-02-10T07:54:27.140089+0000","last_fullsized":"2024-02-10T07:54:27.140089+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:57.310333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:57.310333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":896,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139724+0000","last_change":"2024-02-10T07:54:22.715912+0000","last_active":"2024-02-10T07:54:27.139724+0000","last_peered":"2024-02-10T07:54:27.139724+0000","last_clean":"2024-02-10T07:54:27.139724+0000","last_became_active":"2024-02-10T07:50:43.283104+0000","last_became_peered":"2024-02-10T07:50:43.283104+0000","last_unstale":"2024-02-10T07:54:27.139724+0000","last_undegraded":"2024-02-10T07:54:27.139724+0000","last_fullsized":"2024-02-10T07:54:27.139724+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:22.715819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:54:22.715819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:55:31.365491+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"430'32","reported_seq":974,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139452+0000","last_change":"2024-02-10T07:49:16.249257+0000","last_active":"2024-02-10T07:54:27.139452+0000","last_peered":"2024-02-10T07:54:27.139452+0000","last_clean":"2024-02-10T07:54:27.139452+0000","last_became_active":"2024-02-10T07:48:35.062573+0000","last_became_peered":"2024-02-10T07:48:35.062573+0000","last_unstale":"2024-02-10T07:54:27.139452+0000","last_undegraded":"2024-02-10T07:54:27.139452+0000","last_fullsized":"2024-02-10T07:54:27.139452+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'24","last_scrub_stamp":"2024-02-10T07:49:16.248936+0000","last_deep_scrub":"89'16","last_deep_scrub_stamp":"2024-02-10T07:33:21.286955+0000","last_clean_scrub_stamp":"2024-02-10T07:49:16.248936+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":136,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184494+0000","last_change":"2024-02-10T07:50:43.295417+0000","last_active":"2024-02-10T07:54:27.184494+0000","last_peered":"2024-02-10T07:54:27.184494+0000","last_clean":"2024-02-10T07:54:27.184494+0000","last_became_active":"2024-02-10T07:50:43.290295+0000","last_became_peered":"2024-02-10T07:50:43.290295+0000","last_unstale":"2024-02-10T07:54:27.184494+0000","last_undegraded":"2024-02-10T07:54:27.184494+0000","last_fullsized":"2024-02-10T07:54:27.184494+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:17.153368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:49:17.153368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":874,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184827+0000","last_change":"2024-02-10T07:49:07.170105+0000","last_active":"2024-02-10T07:54:27.184827+0000","last_peered":"2024-02-10T07:54:27.184827+0000","last_clean":"2024-02-10T07:54:27.184827+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:54:27.184827+0000","last_undegraded":"2024-02-10T07:54:27.184827+0000","last_fullsized":"2024-02-10T07:54:27.184827+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:07.169962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:07.169962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":915,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183296+0000","last_change":"2024-02-10T07:52:26.772420+0000","last_active":"2024-02-10T07:54:27.183296+0000","last_peered":"2024-02-10T07:54:27.183296+0000","last_clean":"2024-02-10T07:54:27.183296+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:54:27.183296+0000","last_undegraded":"2024-02-10T07:54:27.183296+0000","last_fullsized":"2024-02-10T07:54:27.183296+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:52:26.772322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:03.803361+0000","last_clean_scrub_stamp":"2024-02-10T07:52:26.772322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"430'53","reported_seq":1022,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.140299+0000","last_change":"2024-02-10T07:50:43.316983+0000","last_active":"2024-02-10T07:54:27.140299+0000","last_peered":"2024-02-10T07:54:27.140299+0000","last_clean":"2024-02-10T07:54:27.140299+0000","last_became_active":"2024-02-10T07:50:43.281879+0000","last_became_peered":"2024-02-10T07:50:43.281879+0000","last_unstale":"2024-02-10T07:54:27.140299+0000","last_undegraded":"2024-02-10T07:54:27.140299+0000","last_fullsized":"2024-02-10T07:54:27.140299+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'39","last_scrub_stamp":"2024-02-10T07:48:59.066256+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_clean_scrub_stamp":"2024-02-10T07:48:59.066256+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":876,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139666+0000","last_change":"2024-02-10T07:50:43.290883+0000","last_active":"2024-02-10T07:54:27.139666+0000","last_peered":"2024-02-10T07:54:27.139666+0000","last_clean":"2024-02-10T07:54:27.139666+0000","last_became_active":"2024-02-10T07:50:43.290334+0000","last_became_peered":"2024-02-10T07:50:43.290334+0000","last_unstale":"2024-02-10T07:54:27.139666+0000","last_undegraded":"2024-02-10T07:54:27.139666+0000","last_fullsized":"2024-02-10T07:54:27.139666+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:49:47.275882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:49:47.275882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"405'12","reported_seq":186,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.184359+0000","last_change":"2024-02-10T07:50:43.294666+0000","last_active":"2024-02-10T07:54:27.184359+0000","last_peered":"2024-02-10T07:54:27.184359+0000","last_clean":"2024-02-10T07:54:27.184359+0000","last_became_active":"2024-02-10T07:50:43.280142+0000","last_became_peered":"2024-02-10T07:50:43.280142+0000","last_unstale":"2024-02-10T07:54:27.184359+0000","last_undegraded":"2024-02-10T07:54:27.184359+0000","last_fullsized":"2024-02-10T07:54:27.184359+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"313'11","last_scrub_stamp":"2024-02-10T07:49:14.335064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:49:14.335064+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":381,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.183992+0000","last_change":"2024-02-10T07:53:20.474601+0000","last_active":"2024-02-10T07:54:27.183992+0000","last_peered":"2024-02-10T07:54:27.183992+0000","last_clean":"2024-02-10T07:54:27.183992+0000","last_became_active":"2024-02-10T07:49:06.258926+0000","last_became_peered":"2024-02-10T07:49:06.258926+0000","last_unstale":"2024-02-10T07:54:27.183992+0000","last_undegraded":"2024-02-10T07:54:27.183992+0000","last_fullsized":"2024-02-10T07:54:27.183992+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:53:20.474497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_clean_scrub_stamp":"2024-02-10T07:53:20.474497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:54:48.442368+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"430'30","reported_seq":956,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-10T07:54:27.139394+0000","last_change":"2024-02-10T07:49:06.289632+0000","last_active":"2024-02-10T07:54:27.139394+0000","last_peered":"2024-02-10T07:54:27.139394+0000","last_clean":"2024-02-10T07:54:27.139394+0000","last_became_active":"2024-02-10T07:49:06.238858+0000","last_became_peered":"2024-02-10T07:49:06.238858+0000","last_unstale":"2024-02-10T07:54:27.139394+0000","last_undegraded":"2024-02-10T07:54:27.139394+0000","last_fullsized":"2024-02-10T07:54:27.139394+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'20","last_scrub_stamp":"2024-02-10T07:48:15.588199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:48:15.588199+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1350,"num_read_kb":1345,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1540,"num_read_kb":1297,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":6382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1060,"ondisk_log_size":1060,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038036,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1193848,"kb_used_data":378284,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92551304,"statfs":{"total":95995035648,"available":94772535296,"internally_reserved":0,"allocated":387362816,"data_stored":386800093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":835123658},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:54:24 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":1.3899999999999999,"15min":1.6659999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.67400000000000004,"5min":18.202999999999999,"15min":39.494},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":1.52,"15min":1.712},"min":{"1min":0.32400000000000001,"5min":0.255,"15min":0.23000000000000001},"max":{"1min":2.367,"5min":19.745000000000001,"15min":38.524000000000001},"last":0.67300000000000004}]},{"osd":1,"last update":"Sat Feb 10 07:54:24 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":2.4580000000000002,"15min":2.0880000000000001},"min":{"1min":0.34200000000000003,"5min":0.251,"15min":0.24399999999999999},"max":{"1min":5.1299999999999999,"5min":50.149000000000001,"15min":50.149000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.1140000000000001,"5min":2.4449999999999998,"15min":2.032},"min":{"1min":0.28299999999999997,"5min":0.26300000000000001,"15min":0.24399999999999999},"max":{"1min":4.3310000000000004,"5min":54.048000000000002,"15min":54.048000000000002},"last":0.70599999999999996}]},{"osd":2,"last update":"Sat Feb 10 07:54:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.288,"15min":1.163},"min":{"1min":0.41799999999999998,"5min":0.28199999999999997,"15min":0.255},"max":{"1min":4.5970000000000004,"5min":7.0019999999999998,"15min":18.57},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.4370000000000001,"15min":1.2949999999999999},"min":{"1min":0.39600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":3.0819999999999999,"5min":8.3949999999999996,"15min":18.521000000000001},"last":0.745}]},{"osd":3,"last update":"Sat Feb 10 07:54:05 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.1950000000000001,"15min":1.099},"min":{"1min":0.35499999999999998,"5min":0.32400000000000001,"15min":0.307},"max":{"1min":1.6930000000000001,"5min":4.7590000000000003,"15min":4.7590000000000003},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.222,"15min":1.022},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.313},"max":{"1min":3.1859999999999999,"5min":6.2839999999999998,"15min":6.2839999999999998},"last":0.58199999999999996}]},{"osd":4,"last update":"Sat Feb 10 07:54:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.76500000000000001,"15min":1.1739999999999999},"min":{"1min":0.34699999999999998,"5min":0.24399999999999999,"15min":0.22},"max":{"1min":1.3080000000000001,"5min":5.0679999999999996,"15min":26.001000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.63200000000000001,"15min":1.145},"min":{"1min":0.33100000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.69199999999999995,"5min":3.1099999999999999,"15min":27.09},"last":0.72299999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:54:09 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.75900000000000001,"15min":1.0449999999999999},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.245},"max":{"1min":1.6479999999999999,"5min":3.5489999999999999,"15min":18.126999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.98099999999999998,"15min":1.1699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.6639999999999999,"5min":5.8490000000000002,"15min":19.548999999999999},"last":0.73399999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:54:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.90600000000000003,"15min":1.159},"min":{"1min":0.41699999999999998,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":1.446,"5min":7.4329999999999998,"15min":17.832000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.77400000000000002,"15min":1.179},"min":{"1min":0.36799999999999999,"5min":0.30399999999999999,"15min":0.29799999999999999},"max":{"1min":1.419,"5min":3.4129999999999998,"15min":20.449999999999999},"last":0.66200000000000003}]}]},{"osd":6,"up_from":50,"seq":214748365081,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":646556,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93098596,"statfs":{"total":95995035648,"available":95332962304,"internally_reserved":0,"allocated":1785856,"data_stored":1250822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":660273606},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":1.9199999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.1580000000000004,"5min":19.484999999999999,"15min":24.727},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.4419999999999999,"15min":1.407},"min":{"1min":0.25700000000000001,"5min":0.25600000000000001,"15min":0.23100000000000001},"max":{"1min":6.0469999999999997,"5min":14.808,"15min":29.084},"last":0.45700000000000002}]},{"osd":1,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.7310000000000001,"15min":1.907},"min":{"1min":0.29299999999999998,"5min":0.26200000000000001,"15min":0.23799999999999999},"max":{"1min":9.9710000000000001,"5min":15.007999999999999,"15min":37.659999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.651,"5min":1.6759999999999999,"15min":1.948},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.253},"max":{"1min":10.029999999999999,"5min":19.524999999999999,"15min":37.75},"last":0.39100000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":1.966,"5min":1.7929999999999999,"15min":1.409},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":11.839,"5min":15.768000000000001,"15min":15.768000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.853,"5min":1.5980000000000001,"15min":1.2430000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":12.013999999999999,"5min":15.728999999999999,"15min":15.728999999999999},"last":0.52300000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.284,"15min":1.5980000000000001},"min":{"1min":0.371,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.7779999999999996,"5min":5.6500000000000004,"15min":13.279},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.2470000000000001,"15min":1.2450000000000001},"min":{"1min":0.376,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.8929999999999998,"5min":7.7949999999999999,"15min":7.7949999999999999},"last":0.71799999999999997}]},{"osd":4,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.60699999999999998,"15min":0.85499999999999998},"min":{"1min":0.26800000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":3.6099999999999999,"5min":3.6099999999999999,"15min":7.6509999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.70099999999999996,"15min":0.88600000000000001},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.26000000000000001},"max":{"1min":2.3220000000000001,"5min":4.8600000000000003,"15min":7.0300000000000002},"last":0.66500000000000004}]},{"osd":5,"last update":"Sat Feb 10 07:54:05 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.58899999999999997,"15min":1.0780000000000001},"min":{"1min":0.308,"5min":0.28799999999999998,"15min":0.24099999999999999},"max":{"1min":4.5720000000000001,"5min":4.5720000000000001,"15min":44.689},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.748,"15min":1.21},"min":{"1min":0.32700000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.4460000000000002,"5min":4.8940000000000001,"15min":44.465000000000003},"last":0.749}]},{"osd":7,"last update":"Sat Feb 10 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.85499999999999998,"15min":1.3180000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.7509999999999999,"5min":4.7750000000000004,"15min":42.631999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.79200000000000004,"15min":1.228},"min":{"1min":0.38200000000000001,"5min":0.32500000000000001,"15min":0.315},"max":{"1min":2.407,"5min":4.9290000000000003,"15min":44.792999999999999},"last":0.77200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575155,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852860,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892292,"statfs":{"total":95995035648,"available":95121707008,"internally_reserved":0,"allocated":1228800,"data_stored":661181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":872085958},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:49 2024","interfaces":[{"interface":"back","average":{"1min":2.3170000000000002,"5min":2.117,"15min":1.589},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":9.0530000000000008,"5min":14.382999999999999,"15min":14.382999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":3.2719999999999998,"5min":2.109,"15min":1.5920000000000001},"min":{"1min":0.47699999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":7.6779999999999999,"5min":14.16,"15min":14.16},"last":0.33600000000000002}]},{"osd":2,"last update":"Sat Feb 10 07:54:20 2024","interfaces":[{"interface":"back","average":{"1min":2.9950000000000001,"5min":2.6360000000000001,"15min":1.6140000000000001},"min":{"1min":0.25800000000000001,"5min":0.246,"15min":0.20399999999999999},"max":{"1min":20.129999999999999,"5min":20.129999999999999,"15min":20.129999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":2.8500000000000001,"5min":2.29,"15min":1.5609999999999999},"min":{"1min":0.29399999999999998,"5min":0.27000000000000002,"15min":0.224},"max":{"1min":20.047000000000001,"5min":20.047000000000001,"15min":20.047000000000001},"last":0.379}]},{"osd":3,"last update":"Sat Feb 10 07:54:02 2024","interfaces":[{"interface":"back","average":{"1min":2.714,"5min":2.1890000000000001,"15min":1.0549999999999999},"min":{"1min":0.314,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":9.0830000000000002,"5min":22.367999999999999,"15min":22.367999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.2269999999999999,"15min":1},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":7.6020000000000003,"5min":22.199999999999999,"15min":22.199999999999999},"last":0.40799999999999997}]},{"osd":4,"last update":"Sat Feb 10 07:54:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9319999999999999,"5min":1.5900000000000001,"15min":2.2370000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.28399999999999997},"max":{"1min":9.2010000000000005,"5min":9.2010000000000005,"15min":207.45699999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2130000000000001,"15min":1.3160000000000001},"min":{"1min":0.35299999999999998,"5min":0.29599999999999999,"15min":0.28899999999999998},"max":{"1min":7.9749999999999996,"5min":10.327999999999999,"15min":32.201000000000001},"last":0.52500000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:54:11 2024","interfaces":[{"interface":"back","average":{"1min":1.9970000000000001,"5min":1.9119999999999999,"15min":2.1269999999999998},"min":{"1min":0.35899999999999999,"5min":0.34200000000000003,"15min":0.308},"max":{"1min":9.1419999999999995,"5min":23.978999999999999,"15min":98.703000000000003},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.893,"5min":1.6850000000000001,"15min":2.0470000000000002},"min":{"1min":0.40999999999999998,"5min":0.32600000000000001,"15min":0.30199999999999999},"max":{"1min":7.4320000000000004,"5min":24.024999999999999,"15min":98.664000000000001},"last":0.49399999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:53:49 2024","interfaces":[{"interface":"back","average":{"1min":1.7270000000000001,"5min":1.514,"15min":1.47},"min":{"1min":0.40400000000000003,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":8.0600000000000005,"5min":10.291,"15min":36.808},"last":0.47099999999999997},{"interface":"front","average":{"1min":2.2629999999999999,"5min":1.837,"15min":1.7749999999999999},"min":{"1min":0.47399999999999998,"5min":0.40100000000000002,"15min":0.33300000000000002},"max":{"1min":9.1750000000000007,"5min":9.1750000000000007,"15min":33.774999999999999},"last":0.48199999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:54:20 2024","interfaces":[{"interface":"back","average":{"1min":1.6699999999999999,"5min":1.8879999999999999,"15min":1.583},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":7.6699999999999999,"5min":24.099,"15min":24.099},"last":0.439},{"interface":"front","average":{"1min":1.242,"5min":1.3580000000000001,"15min":1.3140000000000001},"min":{"1min":0.42199999999999999,"5min":0.39300000000000002,"15min":0.34300000000000003},"max":{"1min":8.0180000000000007,"5min":10.356999999999999,"15min":10.356999999999999},"last":0.503}]}]},{"osd":0,"up_from":8,"seq":34359738680,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799288,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92945864,"statfs":{"total":95995035648,"available":95176564736,"internally_reserved":0,"allocated":1814528,"data_stored":1252202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":816642501},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:53:44 2024","interfaces":[{"interface":"back","average":{"1min":4.194,"5min":2.5979999999999999,"15min":2.625},"min":{"1min":0.23899999999999999,"5min":0.20499999999999999,"15min":0.182},"max":{"1min":11.933,"5min":16.942,"15min":156.16200000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":3.29,"5min":1.9790000000000001,"15min":2.4220000000000002},"min":{"1min":0.42099999999999999,"5min":0.22,"15min":0.183},"max":{"1min":8.891,"5min":8.891,"15min":156.941},"last":0.26200000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":2.0899999999999999,"15min":1.629},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.20000000000000001},"max":{"1min":3.5299999999999998,"5min":19.733000000000001,"15min":34.603000000000002},"last":0.373},{"interface":"front","average":{"1min":1.5609999999999999,"5min":2.1150000000000002,"15min":1.4890000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":5.7800000000000002,"5min":10.116,"15min":34.718000000000004},"last":0.33300000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:53:57 2024","interfaces":[{"interface":"back","average":{"1min":4.2039999999999997,"5min":2.21,"15min":1.1579999999999999},"min":{"1min":0.45900000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":29.792999999999999,"5min":29.792999999999999,"15min":29.792999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":4.1929999999999996,"5min":2.444,"15min":1.23},"min":{"1min":0.38800000000000001,"5min":0.214,"15min":0.214},"max":{"1min":29.506,"5min":29.506,"15min":29.506},"last":0.40500000000000003}]},{"osd":4,"last update":"Sat Feb 10 07:54:03 2024","interfaces":[{"interface":"back","average":{"1min":1.7170000000000001,"5min":1.825,"15min":1.4890000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27600000000000002},"max":{"1min":7.181,"5min":16.991,"15min":37.286000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.46,"5min":1.5980000000000001,"15min":1.4350000000000001},"min":{"1min":0.32400000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.0709999999999997,"5min":8.5939999999999994,"15min":34.683},"last":0.64600000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:54:03 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.925,"15min":2.0379999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":7.1120000000000001,"5min":17.062999999999999,"15min":41.429000000000002},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.625,"15min":1.899},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.308},"max":{"1min":6,"5min":8.6460000000000008,"15min":41.456000000000003},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Feb 10 07:53:44 2024","interfaces":[{"interface":"back","average":{"1min":2.1320000000000001,"5min":1.6100000000000001,"15min":1.6020000000000001},"min":{"1min":0.45400000000000001,"5min":0.34599999999999997,"15min":0.33600000000000002},"max":{"1min":8.2349999999999994,"5min":8.5630000000000006,"15min":41.587000000000003},"last":0.66300000000000003},{"interface":"front","average":{"1min":2.694,"5min":2.0310000000000001,"15min":1.879},"min":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":7.1520000000000001,"5min":17.027999999999999,"15min":41.256},"last":0.61299999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:54:23 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.5900000000000001,"15min":1.5900000000000001},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.33300000000000002},"max":{"1min":2.1709999999999998,"5min":8.6809999999999992,"15min":34.710999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.482,"15min":1.4019999999999999},"min":{"1min":0.46100000000000002,"5min":0.39200000000000002,"15min":0.379},"max":{"1min":5.742,"5min":8.5999999999999996,"15min":34.997999999999998},"last":0.57699999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411629,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":694548,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":93050604,"statfs":{"total":95995035648,"available":95283818496,"internally_reserved":0,"allocated":1187840,"data_stored":661044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":710015429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:54:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":2.3559999999999999,"15min":1.8580000000000001},"min":{"1min":0.20999999999999999,"5min":0.20699999999999999,"15min":0.191},"max":{"1min":6.569,"5min":32.371000000000002,"15min":65.430999999999997},"last":0.373},{"interface":"front","average":{"1min":1.139,"5min":2.5619999999999998,"15min":1.73},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":4.8449999999999998,"5min":38.173000000000002,"15min":38.173000000000002},"last":0.32900000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:54:24 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":2.077,"15min":1.758},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.4800000000000004,"5min":27.606000000000002,"15min":65.795000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.6479999999999999,"5min":2.6819999999999999,"15min":1.7490000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.20499999999999999},"max":{"1min":8.0980000000000008,"5min":29.709,"15min":29.709},"last":0.436}]},{"osd":3,"last update":"Sat Feb 10 07:54:01 2024","interfaces":[{"interface":"back","average":{"1min":1.829,"5min":1.4139999999999999,"15min":0.85199999999999998},"min":{"1min":0.28199999999999997,"5min":0.217,"15min":0.217},"max":{"1min":6.5259999999999998,"5min":7.0819999999999999,"15min":7.0819999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":2.415,"5min":1.776,"15min":0.91700000000000004},"min":{"1min":0.29299999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":5.633,"5min":7.7530000000000001,"15min":7.7530000000000001},"last":0.49199999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:54:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.1890000000000001,"15min":1.401},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.29599999999999999},"max":{"1min":3.4820000000000002,"5min":11.395,"15min":65.494},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.881,"5min":1.6779999999999999,"15min":1.3169999999999999},"min":{"1min":0.40100000000000002,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":5.5890000000000004,"5min":7.7149999999999999,"15min":7.7149999999999999},"last":0.71999999999999997}]},{"osd":5,"last update":"Sat Feb 10 07:54:02 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.4339999999999999,"15min":1.4910000000000001},"min":{"1min":0.36099999999999999,"5min":0.33400000000000002,"15min":0.32600000000000001},"max":{"1min":4.7480000000000002,"5min":10.807,"15min":25.533000000000001},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.105,"15min":1.536},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.5310000000000001,"5min":4.9749999999999996,"15min":65.834000000000003},"last":0.58999999999999997}]},{"osd":6,"last update":"Sat Feb 10 07:53:41 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.2110000000000001,"15min":1.6890000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.35199999999999998},"max":{"1min":4.7400000000000002,"5min":5.069,"15min":65.748000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":2.4780000000000002,"5min":1.5,"15min":1.4550000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":10.893000000000001,"5min":10.893000000000001,"15min":25.901},"last":0.81000000000000005}]},{"osd":7,"last update":"Sat Feb 10 07:54:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.5669999999999999,"15min":1.425},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":4.7880000000000003,"5min":10.859999999999999,"15min":10.859999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.1970000000000001,"15min":1.474},"min":{"1min":0.504,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.8050000000000002,"5min":5.0979999999999999,"15min":65.539000000000001},"last":0.64200000000000002}]}]},{"osd":3,"up_from":347,"seq":1490353651794,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":269064,"kb_used_data":1212,"kb_used_omap":5,"kb_used_meta":267834,"kb_avail":93476088,"statfs":{"total":95995035648,"available":95719514112,"internally_reserved":0,"allocated":1241088,"data_stored":668878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5625,"internal_metadata":274262535},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.8700000000000001,"5min":2.0329999999999999,"15min":1.028},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23699999999999999},"max":{"1min":6.694,"5min":28.247,"15min":28.247},"last":0.32200000000000001},{"interface":"front","average":{"1min":2.2269999999999999,"5min":2.0939999999999999,"15min":1.115},"min":{"1min":0.24399999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.9409999999999998,"5min":27.361999999999998,"15min":27.361999999999998},"last":0.22800000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":3.0760000000000001,"5min":2.8860000000000001,"15min":1.226},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":15.547000000000001,"5min":30.718,"15min":30.718},"last":0.252},{"interface":"front","average":{"1min":2.4940000000000002,"5min":2.5939999999999999,"15min":1.1499999999999999},"min":{"1min":0.32800000000000001,"5min":0.22,"15min":0.22},"max":{"1min":12.583,"5min":32.927999999999997,"15min":32.927999999999997},"last":0.31}]},{"osd":2,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":2.0350000000000001,"5min":1.7130000000000001,"15min":0.879},"min":{"1min":0.30599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.7430000000000003,"5min":10.686,"15min":10.686},"last":0.35599999999999998},{"interface":"front","average":{"1min":2.3010000000000002,"5min":2.0390000000000001,"15min":0.996},"min":{"1min":0.39100000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.4209999999999994,"5min":11.25,"15min":11.25},"last":0.28699999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.637,"15min":2.383},"min":{"1min":0.34200000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":7.782,"5min":15.505000000000001,"15min":28.707999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.7849999999999999,"5min":1.514,"15min":2.5529999999999999},"min":{"1min":0.39100000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":6.7759999999999998,"5min":15.551,"15min":28.760999999999999},"last":0.42399999999999999}]},{"osd":5,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.655,"15min":1.1160000000000001},"min":{"1min":0.371,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":6.343,"5min":15.525,"15min":15.525},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.4810000000000001,"15min":0.99199999999999999},"min":{"1min":0.36199999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":7.8280000000000003,"5min":10.298,"15min":10.298},"last":0.56499999999999995}]},{"osd":6,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":2.1850000000000001,"15min":2.444},"min":{"1min":0.40799999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":6.3920000000000003,"5min":55.750999999999998,"15min":55.750999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":2.0169999999999999,"5min":2.4100000000000001,"15min":2.956},"min":{"1min":0.39800000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":8.5559999999999992,"5min":55.807000000000002,"15min":55.807000000000002},"last":0.496}]},{"osd":7,"last update":"Sat Feb 10 07:53:54 2024","interfaces":[{"interface":"back","average":{"1min":1.8640000000000001,"5min":1.8720000000000001,"15min":1.367},"min":{"1min":0.433,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":7.8760000000000003,"5min":15.577999999999999,"15min":15.577999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":2.0819999999999999,"5min":1.5920000000000001,"15min":1.3759999999999999},"min":{"1min":0.499,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.8049999999999997,"5min":7.8650000000000002,"15min":7.8650000000000002},"last":0.48299999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953763,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758456,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92986696,"statfs":{"total":95995035648,"available":95218376704,"internally_reserved":0,"allocated":1224704,"data_stored":661072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:54:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":1.341,"15min":1.7030000000000001},"min":{"1min":0.26300000000000001,"5min":0.251,"15min":0.20499999999999999},"max":{"1min":4.0339999999999998,"5min":20.004999999999999,"15min":74.533000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.7050000000000001,"15min":1.839},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.22700000000000001},"max":{"1min":4.8710000000000004,"5min":20.050000000000001,"15min":74.340999999999994},"last":0.503}]},{"osd":1,"last update":"Sat Feb 10 07:54:04 2024","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.1030000000000002,"15min":2.5489999999999999},"min":{"1min":0.27400000000000002,"5min":0.245,"15min":0.23200000000000001},"max":{"1min":23.460000000000001,"5min":35.673999999999999,"15min":69.852000000000004},"last":0.59699999999999998},{"interface":"front","average":{"1min":2.117,"5min":2.3799999999999999,"15min":2.5},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":18.305,"5min":35.375,"15min":70.692999999999998},"last":0.57699999999999996}]},{"osd":2,"last update":"Sat Feb 10 07:54:04 2024","interfaces":[{"interface":"back","average":{"1min":2.0710000000000002,"5min":1.4890000000000001,"15min":1.262},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.26600000000000001},"max":{"1min":25.32,"5min":25.32,"15min":25.32},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.841,"5min":1.2989999999999999,"15min":1.1970000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":20.535,"5min":20.535,"15min":20.535},"last":0.53100000000000003}]},{"osd":3,"last update":"Sat Feb 10 07:53:58 2024","interfaces":[{"interface":"back","average":{"1min":1.7609999999999999,"5min":1.3959999999999999,"15min":3.6280000000000001},"min":{"1min":0.30499999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":20.318000000000001,"5min":20.318000000000001,"15min":66.379999999999995},"last":0.63},{"interface":"front","average":{"1min":1.8009999999999999,"5min":1.3400000000000001,"15min":1.629},"min":{"1min":0.314,"5min":0.314,"15min":0.309},"max":{"1min":22.917000000000002,"5min":22.917000000000002,"15min":22.917000000000002},"last":0.54800000000000004}]},{"osd":5,"last update":"Sat Feb 10 07:54:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.70099999999999996,"15min":0.98799999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.3109999999999999,"5min":5.9720000000000004,"15min":10.502000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60299999999999998,"15min":0.98499999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.247},"max":{"1min":2.0289999999999999,"5min":2.9820000000000002,"15min":9.2460000000000004},"last":0.36599999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:53:49 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.77100000000000002,"15min":1.2070000000000001},"min":{"1min":0.27800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.2090000000000001,"5min":6.0039999999999996,"15min":67.433999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.79800000000000004,"15min":1.2849999999999999},"min":{"1min":0.29499999999999998,"5min":0.29199999999999998,"15min":0.26600000000000001},"max":{"1min":1.798,"5min":5.0090000000000003,"15min":66.876000000000005},"last":0.47299999999999998}]},{"osd":7,"last update":"Sat Feb 10 07:54:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.70799999999999996,"15min":0.97899999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.1220000000000001,"5min":4.5730000000000004,"15min":7.4930000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.67500000000000004,"15min":0.98599999999999999},"min":{"1min":0.314,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":1.9259999999999999,"5min":3.8479999999999999,"15min":10.919},"last":0.56200000000000006}]}]},{"osd":5,"up_from":156,"seq":670014898392,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":697772,"kb_used_data":1184,"kb_used_omap":4,"kb_used_meta":696571,"kb_avail":93047380,"statfs":{"total":95995035648,"available":95280517120,"internally_reserved":0,"allocated":1212416,"data_stored":661886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":713289378},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":1.6719999999999999,"15min":1.772},"min":{"1min":0.24299999999999999,"5min":0.22600000000000001,"15min":0.20899999999999999},"max":{"1min":3.0739999999999998,"5min":24.292999999999999,"15min":52.938000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.4970000000000001,"15min":1.974},"min":{"1min":0.25800000000000001,"5min":0.23899999999999999,"15min":0.22900000000000001},"max":{"1min":3.2589999999999999,"5min":23.908000000000001,"15min":52.805},"last":0.309}]},{"osd":1,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0950000000000002,"5min":2.2309999999999999,"15min":2.1360000000000001},"min":{"1min":0.26700000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":12.362,"5min":28.780999999999999,"15min":54.298000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":1.843,"5min":2.2850000000000001,"15min":2.0569999999999999},"min":{"1min":0.27900000000000003,"5min":0.248,"15min":0.246},"max":{"1min":6.1630000000000003,"5min":28.356000000000002,"15min":52.969999999999999},"last":0.53400000000000003}]},{"osd":2,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.329,"15min":1.244},"min":{"1min":0.29899999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.2709999999999999,"5min":10.039999999999999,"15min":10.039999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":1.587,"5min":1.347,"15min":1.2789999999999999},"min":{"1min":0.30199999999999999,"5min":0.28799999999999998,"15min":0.28100000000000003},"max":{"1min":5.2270000000000003,"5min":9.6069999999999993,"15min":13.662000000000001},"last":0.48899999999999999}]},{"osd":3,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":2.1280000000000001,"15min":1.8620000000000001},"min":{"1min":0.33400000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.4540000000000002,"5min":97.119,"15min":97.119},"last":0.628},{"interface":"front","average":{"1min":1.0449999999999999,"5min":2.4020000000000001,"15min":1.784},"min":{"1min":0.32700000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.1819999999999999,"5min":97.141999999999996,"15min":97.141999999999996},"last":0.54900000000000004}]},{"osd":4,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.81999999999999995,"15min":1.1439999999999999},"min":{"1min":0.27300000000000002,"5min":0.214,"15min":0.214},"max":{"1min":2.4830000000000001,"5min":6.7759999999999998,"15min":8.952},"last":0.44},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.82999999999999996,"15min":1.0620000000000001},"min":{"1min":0.27500000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.411,"5min":6.202,"15min":10.180999999999999},"last":0.61199999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":1.1200000000000001,"15min":1.1539999999999999},"min":{"1min":0.26900000000000002,"5min":0.26500000000000001,"15min":0.23999999999999999},"max":{"1min":2.508,"5min":21.748000000000001,"15min":21.748000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":1.155,"15min":1.0960000000000001},"min":{"1min":0.32300000000000001,"5min":0.28599999999999998,"15min":0.26300000000000001},"max":{"1min":2.4420000000000002,"5min":22.187999999999999,"15min":22.187999999999999},"last":0.58299999999999996}]},{"osd":7,"last update":"Sat Feb 10 07:53:52 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":1.0009999999999999,"15min":1.25},"min":{"1min":0.313,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.98099999999999998,"5min":22.302,"15min":22.302},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.133,"15min":1.1479999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.26300000000000001},"max":{"1min":2.4580000000000002,"5min":21.779,"15min":21.779},"last":0.56399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:54:33.167 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-10T07:54:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:33 smithi170 ceph-mon[33394]: 3.17 deep-scrub starts 2024-02-10T07:54:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:33 smithi170 ceph-mon[33394]: 3.17 deep-scrub ok 2024-02-10T07:54:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:33 smithi170 ceph-mon[33394]: 3.7 scrub starts 2024-02-10T07:54:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:33 smithi170 ceph-mon[33394]: 3.7 scrub ok 2024-02-10T07:54:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:33 smithi170 ceph-mon[33394]: from='client.26090 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:54:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:33 smithi170 ceph-mon[33394]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:54:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:33 smithi170 ceph-mon[33394]: 6.1f scrub starts 2024-02-10T07:54:33.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:33 smithi170 ceph-mon[33394]: 6.1f scrub ok 2024-02-10T07:54:33.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:33 smithi170 ceph-mon[33394]: 4.1a scrub starts 2024-02-10T07:54:33.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:33 smithi170 ceph-mon[33394]: 4.1a scrub ok 2024-02-10T07:54:33.422 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:54:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[29816]: 3.17 deep-scrub starts 2024-02-10T07:54:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[29816]: 3.17 deep-scrub ok 2024-02-10T07:54:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[29816]: 3.7 scrub starts 2024-02-10T07:54:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[29816]: 3.7 scrub ok 2024-02-10T07:54:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[29816]: from='client.26090 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:54:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[29816]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:54:33.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[29816]: 6.1f scrub starts 2024-02-10T07:54:33.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[29816]: 6.1f scrub ok 2024-02-10T07:54:33.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[29816]: 4.1a scrub starts 2024-02-10T07:54:33.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[29816]: 4.1a scrub ok 2024-02-10T07:54:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[40743]: 3.17 deep-scrub starts 2024-02-10T07:54:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[40743]: 3.17 deep-scrub ok 2024-02-10T07:54:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[40743]: 3.7 scrub starts 2024-02-10T07:54:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[40743]: 3.7 scrub ok 2024-02-10T07:54:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[40743]: from='client.26090 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:54:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[40743]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:54:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[40743]: 6.1f scrub starts 2024-02-10T07:54:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[40743]: 6.1f scrub ok 2024-02-10T07:54:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[40743]: 4.1a scrub starts 2024-02-10T07:54:33.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:33 smithi008 ceph-mon[40743]: 4.1a scrub ok 2024-02-10T07:54:34.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:54:34 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:54:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[29816]: 5.1b scrub starts 2024-02-10T07:54:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[29816]: 5.1b scrub ok 2024-02-10T07:54:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[29816]: 4.17 scrub starts 2024-02-10T07:54:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[29816]: 4.17 scrub ok 2024-02-10T07:54:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:54:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:54:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[29816]: 3.19 scrub starts 2024-02-10T07:54:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[29816]: 3.19 scrub ok 2024-02-10T07:54:34.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-02-10T07:54:34.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[40743]: 5.1b scrub starts 2024-02-10T07:54:34.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[40743]: 5.1b scrub ok 2024-02-10T07:54:34.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[40743]: 4.17 scrub starts 2024-02-10T07:54:34.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[40743]: 4.17 scrub ok 2024-02-10T07:54:34.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:54:34.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:54:34.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[40743]: 3.19 scrub starts 2024-02-10T07:54:34.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[40743]: 3.19 scrub ok 2024-02-10T07:54:34.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-02-10T07:54:34.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:34 smithi170 ceph-mon[33394]: 5.1b scrub starts 2024-02-10T07:54:34.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:34 smithi170 ceph-mon[33394]: 5.1b scrub ok 2024-02-10T07:54:34.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:34 smithi170 ceph-mon[33394]: 4.17 scrub starts 2024-02-10T07:54:34.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:34 smithi170 ceph-mon[33394]: 4.17 scrub ok 2024-02-10T07:54:34.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:54:34.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:54:34.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:34 smithi170 ceph-mon[33394]: 3.19 scrub starts 2024-02-10T07:54:34.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:34 smithi170 ceph-mon[33394]: 3.19 scrub ok 2024-02-10T07:54:34.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-02-10T07:54:35.208 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:54:35.209 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:54:35.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[29816]: 6.3 scrub starts 2024-02-10T07:54:35.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[29816]: 6.3 scrub ok 2024-02-10T07:54:35.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:35.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[29816]: pgmap v1051: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:54:35.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[29816]: 4.19 scrub starts 2024-02-10T07:54:35.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[29816]: 4.19 scrub ok 2024-02-10T07:54:35.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-02-10T07:54:35.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[29816]: osdmap e470: 8 total, 8 up, 8 in 2024-02-10T07:54:35.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[40743]: 6.3 scrub starts 2024-02-10T07:54:35.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[40743]: 6.3 scrub ok 2024-02-10T07:54:35.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:35.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[40743]: pgmap v1051: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:54:35.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[40743]: 4.19 scrub starts 2024-02-10T07:54:35.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[40743]: 4.19 scrub ok 2024-02-10T07:54:35.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-02-10T07:54:35.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:35 smithi008 ceph-mon[40743]: osdmap e470: 8 total, 8 up, 8 in 2024-02-10T07:54:35.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:35 smithi170 ceph-mon[33394]: 6.3 scrub starts 2024-02-10T07:54:35.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:35 smithi170 ceph-mon[33394]: 6.3 scrub ok 2024-02-10T07:54:35.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:35 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:35.602 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:35 smithi170 ceph-mon[33394]: pgmap v1051: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:54:35.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:35 smithi170 ceph-mon[33394]: 4.19 scrub starts 2024-02-10T07:54:35.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:35 smithi170 ceph-mon[33394]: 4.19 scrub ok 2024-02-10T07:54:35.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:35 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-02-10T07:54:35.603 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:35 smithi170 ceph-mon[33394]: osdmap e470: 8 total, 8 up, 8 in 2024-02-10T07:54:35.886 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-10T07:54:36.141 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:54:36.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:36 smithi170 ceph-mon[33394]: 6.9 scrub starts 2024-02-10T07:54:36.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:36 smithi170 ceph-mon[33394]: 6.9 scrub ok 2024-02-10T07:54:36.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:36 smithi170 ceph-mon[33394]: 5.10 scrub starts 2024-02-10T07:54:36.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:36 smithi170 ceph-mon[33394]: 5.10 scrub ok 2024-02-10T07:54:36.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:36 smithi170 ceph-mon[33394]: osdmap e471: 8 total, 8 up, 8 in 2024-02-10T07:54:36.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:36 smithi170 ceph-mon[33394]: 6.1e scrub starts 2024-02-10T07:54:36.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:36 smithi170 ceph-mon[33394]: 6.1e scrub ok 2024-02-10T07:54:36.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:36 smithi170 ceph-mon[33394]: 6.d scrub starts 2024-02-10T07:54:36.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:36 smithi170 ceph-mon[33394]: 6.d scrub ok 2024-02-10T07:54:36.551 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:36 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:54:36.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[29816]: 6.9 scrub starts 2024-02-10T07:54:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[29816]: 6.9 scrub ok 2024-02-10T07:54:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[29816]: 5.10 scrub starts 2024-02-10T07:54:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[29816]: 5.10 scrub ok 2024-02-10T07:54:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[29816]: osdmap e471: 8 total, 8 up, 8 in 2024-02-10T07:54:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[29816]: 6.1e scrub starts 2024-02-10T07:54:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[29816]: 6.1e scrub ok 2024-02-10T07:54:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[29816]: 6.d scrub starts 2024-02-10T07:54:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[29816]: 6.d scrub ok 2024-02-10T07:54:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:54:36.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[40743]: 6.9 scrub starts 2024-02-10T07:54:36.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[40743]: 6.9 scrub ok 2024-02-10T07:54:36.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[40743]: 5.10 scrub starts 2024-02-10T07:54:36.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[40743]: 5.10 scrub ok 2024-02-10T07:54:36.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[40743]: osdmap e471: 8 total, 8 up, 8 in 2024-02-10T07:54:36.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[40743]: 6.1e scrub starts 2024-02-10T07:54:36.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[40743]: 6.1e scrub ok 2024-02-10T07:54:36.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[40743]: 6.d scrub starts 2024-02-10T07:54:36.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[40743]: 6.d scrub ok 2024-02-10T07:54:36.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:36 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:54:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:37 smithi170 ceph-mon[33394]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-02-10T07:54:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:37 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:54:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:37 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:54:37.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:37 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:54:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:37 smithi008 ceph-mon[29816]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-02-10T07:54:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:37 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:54:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:37 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:54:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:37 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:54:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:37 smithi008 ceph-mon[40743]: pgmap v1054: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 866 B/s rd, 0 op/s 2024-02-10T07:54:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:37 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:54:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:37 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:54:37.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:37 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:54:38.004 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-02-10T07:54:38.074 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:54:38.074 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:54:38.264 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:38.352 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:38 smithi170 ceph-mon[33394]: 5.1e scrub starts 2024-02-10T07:54:38.352 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:38 smithi170 ceph-mon[33394]: 5.1e scrub ok 2024-02-10T07:54:38.353 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:38 smithi170 ceph-mon[33394]: 1.0 scrub starts 2024-02-10T07:54:38.353 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:38 smithi170 ceph-mon[33394]: 1.0 scrub ok 2024-02-10T07:54:38.353 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:38 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:54:38.353 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:38 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:54:38.353 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:38 smithi170 ceph-mon[33394]: 3.4 scrub starts 2024-02-10T07:54:38.353 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:38 smithi170 ceph-mon[33394]: 3.4 scrub ok 2024-02-10T07:54:38.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[29816]: 5.1e scrub starts 2024-02-10T07:54:38.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[29816]: 5.1e scrub ok 2024-02-10T07:54:38.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[29816]: 1.0 scrub starts 2024-02-10T07:54:38.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[29816]: 1.0 scrub ok 2024-02-10T07:54:38.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:54:38.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:54:38.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[29816]: 3.4 scrub starts 2024-02-10T07:54:38.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[29816]: 3.4 scrub ok 2024-02-10T07:54:38.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[40743]: 5.1e scrub starts 2024-02-10T07:54:38.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[40743]: 5.1e scrub ok 2024-02-10T07:54:38.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[40743]: 1.0 scrub starts 2024-02-10T07:54:38.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[40743]: 1.0 scrub ok 2024-02-10T07:54:38.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:54:38.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:54:38.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[40743]: 3.4 scrub starts 2024-02-10T07:54:38.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-mon[40743]: 3.4 scrub ok 2024-02-10T07:54:38.647 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-10T07:54:38.891 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:54:39.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:54:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:54:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:54:39.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:39 smithi008 ceph-mon[29816]: 3.1b scrub starts 2024-02-10T07:54:39.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:39 smithi008 ceph-mon[29816]: 3.1b scrub ok 2024-02-10T07:54:39.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:39 smithi008 ceph-mon[29816]: pgmap v1055: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-10T07:54:39.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:39 smithi008 ceph-mon[29816]: 2.1 scrub starts 2024-02-10T07:54:39.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:39 smithi008 ceph-mon[29816]: 2.1 scrub ok 2024-02-10T07:54:39.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:39 smithi008 ceph-mon[29816]: osdmap e472: 8 total, 8 up, 8 in 2024-02-10T07:54:39.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:39 smithi008 ceph-mon[40743]: 3.1b scrub starts 2024-02-10T07:54:39.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:39 smithi008 ceph-mon[40743]: 3.1b scrub ok 2024-02-10T07:54:39.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:39 smithi008 ceph-mon[40743]: pgmap v1055: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-10T07:54:39.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:39 smithi008 ceph-mon[40743]: 2.1 scrub starts 2024-02-10T07:54:39.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:39 smithi008 ceph-mon[40743]: 2.1 scrub ok 2024-02-10T07:54:39.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:39 smithi008 ceph-mon[40743]: osdmap e472: 8 total, 8 up, 8 in 2024-02-10T07:54:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:39 smithi170 ceph-mon[33394]: 3.1b scrub starts 2024-02-10T07:54:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:39 smithi170 ceph-mon[33394]: 3.1b scrub ok 2024-02-10T07:54:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:39 smithi170 ceph-mon[33394]: pgmap v1055: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-10T07:54:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:39 smithi170 ceph-mon[33394]: 2.1 scrub starts 2024-02-10T07:54:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:39 smithi170 ceph-mon[33394]: 2.1 scrub ok 2024-02-10T07:54:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:39 smithi170 ceph-mon[33394]: osdmap e472: 8 total, 8 up, 8 in 2024-02-10T07:54:40.049 INFO:teuthology.orchestra.run.smithi008.stdout:mon.a: {} 2024-02-10T07:54:40.060 INFO:teuthology.orchestra.run.smithi008.stdout:mon.b: {} 2024-02-10T07:54:40.069 INFO:teuthology.orchestra.run.smithi008.stdout:mon.c: {} 2024-02-10T07:54:40.071 INFO:teuthology.orchestra.run.smithi008.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-10T07:54:40.071 INFO:teuthology.orchestra.run.smithi008.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-10T07:54:40.071 INFO:teuthology.orchestra.run.smithi008.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-10T07:54:40.522 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:40 smithi170 ceph-mon[33394]: 4.c scrub starts 2024-02-10T07:54:40.522 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:40 smithi170 ceph-mon[33394]: 4.c scrub ok 2024-02-10T07:54:40.522 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:40 smithi170 ceph-mon[33394]: 5.1a scrub starts 2024-02-10T07:54:40.522 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:40 smithi170 ceph-mon[33394]: 5.1a scrub ok 2024-02-10T07:54:40.522 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:40 smithi170 ceph-mon[33394]: 6.c scrub starts 2024-02-10T07:54:40.522 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:40 smithi170 ceph-mon[33394]: 6.c scrub ok 2024-02-10T07:54:40.522 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:40 smithi170 ceph-mon[33394]: 2.2 scrub starts 2024-02-10T07:54:40.522 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:40 smithi170 ceph-mon[33394]: 2.2 scrub ok 2024-02-10T07:54:40.523 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:40 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3904937312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:54:40.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[29816]: 4.c scrub starts 2024-02-10T07:54:40.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[29816]: 4.c scrub ok 2024-02-10T07:54:40.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[29816]: 5.1a scrub starts 2024-02-10T07:54:40.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[29816]: 5.1a scrub ok 2024-02-10T07:54:40.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[29816]: 6.c scrub starts 2024-02-10T07:54:40.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[29816]: 6.c scrub ok 2024-02-10T07:54:40.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[29816]: 2.2 scrub starts 2024-02-10T07:54:40.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[29816]: 2.2 scrub ok 2024-02-10T07:54:40.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3904937312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:54:40.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[40743]: 4.c scrub starts 2024-02-10T07:54:40.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[40743]: 4.c scrub ok 2024-02-10T07:54:40.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[40743]: 5.1a scrub starts 2024-02-10T07:54:40.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[40743]: 5.1a scrub ok 2024-02-10T07:54:40.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[40743]: 6.c scrub starts 2024-02-10T07:54:40.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[40743]: 6.c scrub ok 2024-02-10T07:54:40.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[40743]: 2.2 scrub starts 2024-02-10T07:54:40.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[40743]: 2.2 scrub ok 2024-02-10T07:54:40.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:40 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3904937312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T07:54:40.645 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:54:40.645 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:54:40.676 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 0 1 2024-02-10T07:54:40.926 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:41.262 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-10T07:54:41.504 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:54:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:41 smithi008 ceph-mon[29816]: pgmap v1057: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-02-10T07:54:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:41 smithi008 ceph-mon[29816]: 5.11 scrub starts 2024-02-10T07:54:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:41 smithi008 ceph-mon[29816]: 5.11 scrub ok 2024-02-10T07:54:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:41 smithi008 ceph-mon[29816]: 5.b scrub starts 2024-02-10T07:54:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:41 smithi008 ceph-mon[29816]: 5.b scrub ok 2024-02-10T07:54:41.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:41 smithi008 ceph-mon[29816]: 4.12 scrub starts 2024-02-10T07:54:41.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:41 smithi008 ceph-mon[29816]: 4.12 scrub ok 2024-02-10T07:54:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:41 smithi008 ceph-mon[40743]: pgmap v1057: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-02-10T07:54:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:41 smithi008 ceph-mon[40743]: 5.11 scrub starts 2024-02-10T07:54:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:41 smithi008 ceph-mon[40743]: 5.11 scrub ok 2024-02-10T07:54:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:41 smithi008 ceph-mon[40743]: 5.b scrub starts 2024-02-10T07:54:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:41 smithi008 ceph-mon[40743]: 5.b scrub ok 2024-02-10T07:54:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:41 smithi008 ceph-mon[40743]: 4.12 scrub starts 2024-02-10T07:54:41.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:41 smithi008 ceph-mon[40743]: 4.12 scrub ok 2024-02-10T07:54:41.626 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:41 smithi170 ceph-mon[33394]: pgmap v1057: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-02-10T07:54:41.626 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:41 smithi170 ceph-mon[33394]: 5.11 scrub starts 2024-02-10T07:54:41.626 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:41 smithi170 ceph-mon[33394]: 5.11 scrub ok 2024-02-10T07:54:41.626 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:41 smithi170 ceph-mon[33394]: 5.b scrub starts 2024-02-10T07:54:41.626 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:41 smithi170 ceph-mon[33394]: 5.b scrub ok 2024-02-10T07:54:41.626 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:41 smithi170 ceph-mon[33394]: 4.12 scrub starts 2024-02-10T07:54:41.626 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:41 smithi170 ceph-mon[33394]: 4.12 scrub ok 2024-02-10T07:54:42.358 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:42 smithi008 ceph-mon[29816]: 4.1b scrub starts 2024-02-10T07:54:42.358 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:42 smithi008 ceph-mon[29816]: 4.1b scrub ok 2024-02-10T07:54:42.359 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:42 smithi008 ceph-mon[29816]: osdmap e473: 8 total, 8 up, 8 in 2024-02-10T07:54:42.359 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:42 smithi008 ceph-mon[40743]: 4.1b scrub starts 2024-02-10T07:54:42.359 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:42 smithi008 ceph-mon[40743]: 4.1b scrub ok 2024-02-10T07:54:42.359 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:42 smithi008 ceph-mon[40743]: osdmap e473: 8 total, 8 up, 8 in 2024-02-10T07:54:42.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:42 smithi170 ceph-mon[33394]: 4.1b scrub starts 2024-02-10T07:54:42.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:42 smithi170 ceph-mon[33394]: 4.1b scrub ok 2024-02-10T07:54:42.600 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:42 smithi170 ceph-mon[33394]: osdmap e473: 8 total, 8 up, 8 in 2024-02-10T07:54:43.135 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:54:43.135 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:54:43.335 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.0 to 1 (10000) 2024-02-10T07:54:43.632 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:43 smithi008 ceph-mon[29816]: pgmap v1059: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 623 B/s rd, 0 op/s; 33 B/s, 1 objects/s recovering 2024-02-10T07:54:43.632 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:43 smithi008 ceph-mon[29816]: 4.14 scrub starts 2024-02-10T07:54:43.632 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:43 smithi008 ceph-mon[29816]: 4.14 scrub ok 2024-02-10T07:54:43.632 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:43 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3061625469' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:54:43.632 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:43 smithi008 ceph-mon[40743]: pgmap v1059: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 623 B/s rd, 0 op/s; 33 B/s, 1 objects/s recovering 2024-02-10T07:54:43.632 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:43 smithi008 ceph-mon[40743]: 4.14 scrub starts 2024-02-10T07:54:43.632 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:43 smithi008 ceph-mon[40743]: 4.14 scrub ok 2024-02-10T07:54:43.633 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:43 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3061625469' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:54:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:43 smithi170 ceph-mon[33394]: pgmap v1059: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 623 B/s rd, 0 op/s; 33 B/s, 1 objects/s recovering 2024-02-10T07:54:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:43 smithi170 ceph-mon[33394]: 4.14 scrub starts 2024-02-10T07:54:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:43 smithi170 ceph-mon[33394]: 4.14 scrub ok 2024-02-10T07:54:43.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:43 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3061625469' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:54:43.810 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-10T07:54:44.017 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 0 1 2024-02-10T07:54:44.049 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:54:44.268 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:44.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:54:44 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:54:44.614 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[29816]: 4.b scrub starts 2024-02-10T07:54:44.614 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[29816]: 4.b scrub ok 2024-02-10T07:54:44.614 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3061625469' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:54:44.614 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[29816]: osdmap e474: 8 total, 8 up, 8 in 2024-02-10T07:54:44.614 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[29816]: 5.0 deep-scrub starts 2024-02-10T07:54:44.614 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[29816]: 6.12 scrub starts 2024-02-10T07:54:44.615 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[29816]: 5.0 deep-scrub ok 2024-02-10T07:54:44.615 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[29816]: 6.12 scrub ok 2024-02-10T07:54:44.615 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[40743]: 4.b scrub starts 2024-02-10T07:54:44.615 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[40743]: 4.b scrub ok 2024-02-10T07:54:44.615 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3061625469' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:54:44.615 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[40743]: osdmap e474: 8 total, 8 up, 8 in 2024-02-10T07:54:44.615 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[40743]: 5.0 deep-scrub starts 2024-02-10T07:54:44.615 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[40743]: 6.12 scrub starts 2024-02-10T07:54:44.615 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[40743]: 5.0 deep-scrub ok 2024-02-10T07:54:44.615 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:44 smithi008 ceph-mon[40743]: 6.12 scrub ok 2024-02-10T07:54:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:44 smithi170 ceph-mon[33394]: 4.b scrub starts 2024-02-10T07:54:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:44 smithi170 ceph-mon[33394]: 4.b scrub ok 2024-02-10T07:54:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:44 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3061625469' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:54:44.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:44 smithi170 ceph-mon[33394]: osdmap e474: 8 total, 8 up, 8 in 2024-02-10T07:54:44.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:44 smithi170 ceph-mon[33394]: 5.0 deep-scrub starts 2024-02-10T07:54:44.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:44 smithi170 ceph-mon[33394]: 6.12 scrub starts 2024-02-10T07:54:44.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:44 smithi170 ceph-mon[33394]: 5.0 deep-scrub ok 2024-02-10T07:54:44.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:44 smithi170 ceph-mon[33394]: 6.12 scrub ok 2024-02-10T07:54:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:45 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:45.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:45 smithi170 ceph-mon[33394]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 36 B/s, 1 objects/s recovering 2024-02-10T07:54:45.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:45 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:45.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:45 smithi008 ceph-mon[29816]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 36 B/s, 1 objects/s recovering 2024-02-10T07:54:45.468 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:45 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:45.468 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:45 smithi008 ceph-mon[40743]: pgmap v1061: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 36 B/s, 1 objects/s recovering 2024-02-10T07:54:45.683 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:54:45.684 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:54:46.351 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-02-10T07:54:46.372 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-10T07:54:46.620 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:54:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:46 smithi170 ceph-mon[33394]: 3.e scrub starts 2024-02-10T07:54:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:46 smithi170 ceph-mon[33394]: 3.e scrub ok 2024-02-10T07:54:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:46 smithi170 ceph-mon[33394]: 5.1c scrub starts 2024-02-10T07:54:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:46 smithi170 ceph-mon[33394]: 5.1c scrub ok 2024-02-10T07:54:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:46 smithi170 ceph-mon[33394]: 4.5 scrub starts 2024-02-10T07:54:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:46 smithi170 ceph-mon[33394]: 4.5 scrub ok 2024-02-10T07:54:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:46 smithi170 ceph-mon[33394]: 5.c scrub starts 2024-02-10T07:54:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:46 smithi170 ceph-mon[33394]: 5.c scrub ok 2024-02-10T07:54:46.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:46 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/333943777' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:54:46.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:46 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:54:46.696 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[29816]: 3.e scrub starts 2024-02-10T07:54:46.697 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[29816]: 3.e scrub ok 2024-02-10T07:54:46.697 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[29816]: 5.1c scrub starts 2024-02-10T07:54:46.697 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[29816]: 5.1c scrub ok 2024-02-10T07:54:46.697 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[29816]: 4.5 scrub starts 2024-02-10T07:54:46.697 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[29816]: 4.5 scrub ok 2024-02-10T07:54:46.697 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[29816]: 5.c scrub starts 2024-02-10T07:54:46.697 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[29816]: 5.c scrub ok 2024-02-10T07:54:46.697 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/333943777' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:54:46.697 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:54:46.697 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[40743]: 3.e scrub starts 2024-02-10T07:54:46.698 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[40743]: 3.e scrub ok 2024-02-10T07:54:46.698 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[40743]: 5.1c scrub starts 2024-02-10T07:54:46.698 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[40743]: 5.1c scrub ok 2024-02-10T07:54:46.698 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[40743]: 4.5 scrub starts 2024-02-10T07:54:46.698 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[40743]: 4.5 scrub ok 2024-02-10T07:54:46.698 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[40743]: 5.c scrub starts 2024-02-10T07:54:46.698 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[40743]: 5.c scrub ok 2024-02-10T07:54:46.698 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/333943777' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:54:46.698 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:46 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-10T07:54:46.991 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 1 1 2024-02-10T07:54:47.239 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:47 smithi170 ceph-mon[33394]: 5.9 scrub starts 2024-02-10T07:54:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:47 smithi170 ceph-mon[33394]: 5.9 scrub ok 2024-02-10T07:54:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:47 smithi170 ceph-mon[33394]: pgmap v1062: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T07:54:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:47 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:54:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:47 smithi170 ceph-mon[33394]: osdmap e475: 8 total, 8 up, 8 in 2024-02-10T07:54:47.693 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:47 smithi008 ceph-mon[29816]: 5.9 scrub starts 2024-02-10T07:54:47.693 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:47 smithi008 ceph-mon[29816]: 5.9 scrub ok 2024-02-10T07:54:47.693 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:47 smithi008 ceph-mon[29816]: pgmap v1062: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T07:54:47.693 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:47 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:54:47.693 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:47 smithi008 ceph-mon[29816]: osdmap e475: 8 total, 8 up, 8 in 2024-02-10T07:54:47.694 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:47 smithi008 ceph-mon[40743]: 5.9 scrub starts 2024-02-10T07:54:47.694 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:47 smithi008 ceph-mon[40743]: 5.9 scrub ok 2024-02-10T07:54:47.694 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:47 smithi008 ceph-mon[40743]: pgmap v1062: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-02-10T07:54:47.694 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:47 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-10T07:54:47.694 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:47 smithi008 ceph-mon[40743]: osdmap e475: 8 total, 8 up, 8 in 2024-02-10T07:54:48.248 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:54:48.248 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:54:48.612 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:48 smithi170 ceph-mon[33394]: 6.f scrub starts 2024-02-10T07:54:48.612 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:48 smithi170 ceph-mon[33394]: 6.f scrub ok 2024-02-10T07:54:48.612 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:48 smithi170 ceph-mon[33394]: 5.14 scrub starts 2024-02-10T07:54:48.612 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:48 smithi170 ceph-mon[33394]: 5.14 scrub ok 2024-02-10T07:54:48.612 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:48 smithi170 ceph-mon[33394]: 4.d scrub starts 2024-02-10T07:54:48.612 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:48 smithi170 ceph-mon[33394]: 4.d scrub ok 2024-02-10T07:54:48.612 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:48 smithi170 ceph-mon[33394]: 4.1f scrub starts 2024-02-10T07:54:48.612 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:48 smithi170 ceph-mon[33394]: 4.1f scrub ok 2024-02-10T07:54:48.616 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[29816]: 6.f scrub starts 2024-02-10T07:54:48.616 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[29816]: 6.f scrub ok 2024-02-10T07:54:48.616 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[29816]: 5.14 scrub starts 2024-02-10T07:54:48.616 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[29816]: 5.14 scrub ok 2024-02-10T07:54:48.616 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[29816]: 4.d scrub starts 2024-02-10T07:54:48.617 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[29816]: 4.d scrub ok 2024-02-10T07:54:48.617 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[29816]: 4.1f scrub starts 2024-02-10T07:54:48.617 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[29816]: 4.1f scrub ok 2024-02-10T07:54:48.617 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[40743]: 6.f scrub starts 2024-02-10T07:54:48.617 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[40743]: 6.f scrub ok 2024-02-10T07:54:48.617 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[40743]: 5.14 scrub starts 2024-02-10T07:54:48.617 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[40743]: 5.14 scrub ok 2024-02-10T07:54:48.617 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[40743]: 4.d scrub starts 2024-02-10T07:54:48.617 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[40743]: 4.d scrub ok 2024-02-10T07:54:48.617 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[40743]: 4.1f scrub starts 2024-02-10T07:54:48.617 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-mon[40743]: 4.1f scrub ok 2024-02-10T07:54:48.909 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:54:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:54:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:54:49.002 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-10T07:54:49.248 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:54:49.359 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.1 to 1 (10000) 2024-02-10T07:54:49.360 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[40743]: 5.5 scrub starts 2024-02-10T07:54:49.361 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[40743]: 5.5 scrub ok 2024-02-10T07:54:49.361 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[40743]: 5.7 scrub starts 2024-02-10T07:54:49.361 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[40743]: 5.7 scrub ok 2024-02-10T07:54:49.361 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[40743]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-10T07:54:49.361 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[40743]: 5.8 scrub starts 2024-02-10T07:54:49.361 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[40743]: 5.8 scrub ok 2024-02-10T07:54:49.361 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:54:49.361 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/865791618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:54:49.361 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:54:49.363 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[29816]: 5.5 scrub starts 2024-02-10T07:54:49.363 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[29816]: 5.5 scrub ok 2024-02-10T07:54:49.363 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[29816]: 5.7 scrub starts 2024-02-10T07:54:49.363 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[29816]: 5.7 scrub ok 2024-02-10T07:54:49.363 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[29816]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-10T07:54:49.363 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[29816]: 5.8 scrub starts 2024-02-10T07:54:49.364 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[29816]: 5.8 scrub ok 2024-02-10T07:54:49.364 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:54:49.364 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/865791618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:54:49.364 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:49 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:54:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:49 smithi170 ceph-mon[33394]: 5.5 scrub starts 2024-02-10T07:54:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:49 smithi170 ceph-mon[33394]: 5.5 scrub ok 2024-02-10T07:54:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:49 smithi170 ceph-mon[33394]: 5.7 scrub starts 2024-02-10T07:54:49.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:49 smithi170 ceph-mon[33394]: 5.7 scrub ok 2024-02-10T07:54:49.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:49 smithi170 ceph-mon[33394]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-10T07:54:49.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:49 smithi170 ceph-mon[33394]: 5.8 scrub starts 2024-02-10T07:54:49.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:49 smithi170 ceph-mon[33394]: 5.8 scrub ok 2024-02-10T07:54:49.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:54:49.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:49 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/865791618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:54:49.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:49 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:54:50.053 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 1 1 2024-02-10T07:54:50.303 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:50 smithi170 ceph-mon[33394]: 4.7 scrub starts 2024-02-10T07:54:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:50 smithi170 ceph-mon[33394]: 4.7 scrub ok 2024-02-10T07:54:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:50 smithi170 ceph-mon[33394]: 6.0 deep-scrub starts 2024-02-10T07:54:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:50 smithi170 ceph-mon[33394]: 6.0 deep-scrub ok 2024-02-10T07:54:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:50 smithi170 ceph-mon[33394]: 6.b deep-scrub starts 2024-02-10T07:54:50.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:50 smithi170 ceph-mon[33394]: 4.1c scrub starts 2024-02-10T07:54:50.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:50 smithi170 ceph-mon[33394]: 4.1c scrub ok 2024-02-10T07:54:50.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:50 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:54:50.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:50 smithi170 ceph-mon[33394]: osdmap e476: 8 total, 8 up, 8 in 2024-02-10T07:54:50.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:50 smithi170 ceph-mon[33394]: 6.b deep-scrub ok 2024-02-10T07:54:50.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:50 smithi170 ceph-mon[33394]: 5.d scrub starts 2024-02-10T07:54:50.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:50 smithi170 ceph-mon[33394]: 5.d scrub ok 2024-02-10T07:54:50.694 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[29816]: 4.7 scrub starts 2024-02-10T07:54:50.694 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[29816]: 4.7 scrub ok 2024-02-10T07:54:50.694 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[29816]: 6.0 deep-scrub starts 2024-02-10T07:54:50.694 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[29816]: 6.0 deep-scrub ok 2024-02-10T07:54:50.694 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[29816]: 6.b deep-scrub starts 2024-02-10T07:54:50.694 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[29816]: 4.1c scrub starts 2024-02-10T07:54:50.694 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[29816]: 4.1c scrub ok 2024-02-10T07:54:50.694 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:54:50.694 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[29816]: osdmap e476: 8 total, 8 up, 8 in 2024-02-10T07:54:50.694 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[29816]: 6.b deep-scrub ok 2024-02-10T07:54:50.694 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[29816]: 5.d scrub starts 2024-02-10T07:54:50.694 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[29816]: 5.d scrub ok 2024-02-10T07:54:50.695 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[40743]: 4.7 scrub starts 2024-02-10T07:54:50.695 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[40743]: 4.7 scrub ok 2024-02-10T07:54:50.695 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[40743]: 6.0 deep-scrub starts 2024-02-10T07:54:50.695 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[40743]: 6.0 deep-scrub ok 2024-02-10T07:54:50.695 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[40743]: 6.b deep-scrub starts 2024-02-10T07:54:50.695 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[40743]: 4.1c scrub starts 2024-02-10T07:54:50.695 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[40743]: 4.1c scrub ok 2024-02-10T07:54:50.695 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:54:50.695 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[40743]: osdmap e476: 8 total, 8 up, 8 in 2024-02-10T07:54:50.695 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[40743]: 6.b deep-scrub ok 2024-02-10T07:54:50.695 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[40743]: 5.d scrub starts 2024-02-10T07:54:50.695 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:50 smithi008 ceph-mon[40743]: 5.d scrub ok 2024-02-10T07:54:50.972 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:54:50.972 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:54:51.526 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[29816]: 5.3 deep-scrub starts 2024-02-10T07:54:51.526 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[29816]: 5.3 deep-scrub ok 2024-02-10T07:54:51.526 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[29816]: 6.8 scrub starts 2024-02-10T07:54:51.526 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[29816]: 6.8 scrub ok 2024-02-10T07:54:51.526 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[29816]: 3.10 scrub starts 2024-02-10T07:54:51.526 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[29816]: 3.10 scrub ok 2024-02-10T07:54:51.526 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[29816]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T07:54:51.527 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[29816]: 6.1d scrub starts 2024-02-10T07:54:51.527 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[29816]: 6.1d scrub ok 2024-02-10T07:54:51.527 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[40743]: 5.3 deep-scrub starts 2024-02-10T07:54:51.527 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[40743]: 5.3 deep-scrub ok 2024-02-10T07:54:51.527 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[40743]: 6.8 scrub starts 2024-02-10T07:54:51.527 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[40743]: 6.8 scrub ok 2024-02-10T07:54:51.527 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[40743]: 3.10 scrub starts 2024-02-10T07:54:51.527 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[40743]: 3.10 scrub ok 2024-02-10T07:54:51.527 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[40743]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T07:54:51.527 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[40743]: 6.1d scrub starts 2024-02-10T07:54:51.527 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:51 smithi008 ceph-mon[40743]: 6.1d scrub ok 2024-02-10T07:54:51.595 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-10T07:54:51.620 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:51 smithi170 ceph-mon[33394]: 5.3 deep-scrub starts 2024-02-10T07:54:51.620 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:51 smithi170 ceph-mon[33394]: 5.3 deep-scrub ok 2024-02-10T07:54:51.620 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:51 smithi170 ceph-mon[33394]: 6.8 scrub starts 2024-02-10T07:54:51.620 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:51 smithi170 ceph-mon[33394]: 6.8 scrub ok 2024-02-10T07:54:51.621 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:51 smithi170 ceph-mon[33394]: 3.10 scrub starts 2024-02-10T07:54:51.621 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:51 smithi170 ceph-mon[33394]: 3.10 scrub ok 2024-02-10T07:54:51.621 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:51 smithi170 ceph-mon[33394]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-10T07:54:51.621 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:51 smithi170 ceph-mon[33394]: 6.1d scrub starts 2024-02-10T07:54:51.621 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:51 smithi170 ceph-mon[33394]: 6.1d scrub ok 2024-02-10T07:54:51.834 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:54:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: 6.10 scrub starts 2024-02-10T07:54:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: 6.10 scrub ok 2024-02-10T07:54:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: 4.1 scrub starts 2024-02-10T07:54:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: 4.1 scrub ok 2024-02-10T07:54:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: 5.4 deep-scrub starts 2024-02-10T07:54:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: 5.4 deep-scrub ok 2024-02-10T07:54:52.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: 5.e scrub starts 2024-02-10T07:54:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: 5.e scrub ok 2024-02-10T07:54:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: 6.6 scrub starts 2024-02-10T07:54:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: 6.6 scrub ok 2024-02-10T07:54:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: osdmap e477: 8 total, 8 up, 8 in 2024-02-10T07:54:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: 5.6 scrub starts 2024-02-10T07:54:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: 5.6 scrub ok 2024-02-10T07:54:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4138698922' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:54:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:52 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:54:52.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: 6.10 scrub starts 2024-02-10T07:54:52.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: 6.10 scrub ok 2024-02-10T07:54:52.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: 4.1 scrub starts 2024-02-10T07:54:52.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: 4.1 scrub ok 2024-02-10T07:54:52.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: 5.4 deep-scrub starts 2024-02-10T07:54:52.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: 5.4 deep-scrub ok 2024-02-10T07:54:52.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: 5.e scrub starts 2024-02-10T07:54:52.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: 5.e scrub ok 2024-02-10T07:54:52.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: 6.6 scrub starts 2024-02-10T07:54:52.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: 6.6 scrub ok 2024-02-10T07:54:52.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: osdmap e477: 8 total, 8 up, 8 in 2024-02-10T07:54:52.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: 5.6 scrub starts 2024-02-10T07:54:52.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: 5.6 scrub ok 2024-02-10T07:54:52.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4138698922' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:54:52.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:54:52.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: 6.10 scrub starts 2024-02-10T07:54:52.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: 6.10 scrub ok 2024-02-10T07:54:52.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: 4.1 scrub starts 2024-02-10T07:54:52.833 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: 4.1 scrub ok 2024-02-10T07:54:52.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: 5.4 deep-scrub starts 2024-02-10T07:54:52.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: 5.4 deep-scrub ok 2024-02-10T07:54:52.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: 5.e scrub starts 2024-02-10T07:54:52.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: 5.e scrub ok 2024-02-10T07:54:52.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: 6.6 scrub starts 2024-02-10T07:54:52.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: 6.6 scrub ok 2024-02-10T07:54:52.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: osdmap e477: 8 total, 8 up, 8 in 2024-02-10T07:54:52.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: 5.6 scrub starts 2024-02-10T07:54:52.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: 5.6 scrub ok 2024-02-10T07:54:52.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4138698922' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:54:52.834 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:52 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-10T07:54:53.157 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-02-10T07:54:53.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[29816]: 3.f scrub starts 2024-02-10T07:54:53.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[29816]: 3.f scrub ok 2024-02-10T07:54:53.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[29816]: 3.c scrub starts 2024-02-10T07:54:53.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[29816]: 3.c scrub ok 2024-02-10T07:54:53.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[29816]: pgmap v1068: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:54:53.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[29816]: 4.2 scrub starts 2024-02-10T07:54:53.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[29816]: 4.2 scrub ok 2024-02-10T07:54:53.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:54:53.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[29816]: osdmap e478: 8 total, 8 up, 8 in 2024-02-10T07:54:53.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[29816]: 3.5 scrub starts 2024-02-10T07:54:53.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[29816]: 3.5 scrub ok 2024-02-10T07:54:53.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[40743]: 3.f scrub starts 2024-02-10T07:54:53.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[40743]: 3.f scrub ok 2024-02-10T07:54:53.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[40743]: 3.c scrub starts 2024-02-10T07:54:53.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[40743]: 3.c scrub ok 2024-02-10T07:54:53.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[40743]: pgmap v1068: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:54:53.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[40743]: 4.2 scrub starts 2024-02-10T07:54:53.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[40743]: 4.2 scrub ok 2024-02-10T07:54:53.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:54:53.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[40743]: osdmap e478: 8 total, 8 up, 8 in 2024-02-10T07:54:53.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[40743]: 3.5 scrub starts 2024-02-10T07:54:53.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:53 smithi008 ceph-mon[40743]: 3.5 scrub ok 2024-02-10T07:54:53.629 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:54:53.629 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:54:53.629 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:53 smithi170 ceph-mon[33394]: 3.f scrub starts 2024-02-10T07:54:53.630 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:53 smithi170 ceph-mon[33394]: 3.f scrub ok 2024-02-10T07:54:53.630 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:53 smithi170 ceph-mon[33394]: 3.c scrub starts 2024-02-10T07:54:53.630 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:53 smithi170 ceph-mon[33394]: 3.c scrub ok 2024-02-10T07:54:53.630 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:53 smithi170 ceph-mon[33394]: pgmap v1068: 132 pgs: 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:54:53.630 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:53 smithi170 ceph-mon[33394]: 4.2 scrub starts 2024-02-10T07:54:53.630 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:53 smithi170 ceph-mon[33394]: 4.2 scrub ok 2024-02-10T07:54:53.630 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:53 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-10T07:54:53.630 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:53 smithi170 ceph-mon[33394]: osdmap e478: 8 total, 8 up, 8 in 2024-02-10T07:54:53.630 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:53 smithi170 ceph-mon[33394]: 3.5 scrub starts 2024-02-10T07:54:53.630 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:53 smithi170 ceph-mon[33394]: 3.5 scrub ok 2024-02-10T07:54:53.929 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 2 1 2024-02-10T07:54:54.182 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:54.284 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:54:54 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:54:54.315 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-10T07:54:54.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:54 smithi008 ceph-mon[29816]: 4.6 scrub starts 2024-02-10T07:54:54.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:54 smithi008 ceph-mon[29816]: 4.6 scrub ok 2024-02-10T07:54:54.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:54 smithi008 ceph-mon[29816]: 6.15 scrub starts 2024-02-10T07:54:54.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:54 smithi008 ceph-mon[29816]: 6.15 scrub ok 2024-02-10T07:54:54.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:54 smithi008 ceph-mon[40743]: 4.6 scrub starts 2024-02-10T07:54:54.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:54 smithi008 ceph-mon[40743]: 4.6 scrub ok 2024-02-10T07:54:54.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:54 smithi008 ceph-mon[40743]: 6.15 scrub starts 2024-02-10T07:54:54.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:54 smithi008 ceph-mon[40743]: 6.15 scrub ok 2024-02-10T07:54:54.601 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:54:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:54 smithi170 ceph-mon[33394]: 4.6 scrub starts 2024-02-10T07:54:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:54 smithi170 ceph-mon[33394]: 4.6 scrub ok 2024-02-10T07:54:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:54 smithi170 ceph-mon[33394]: 6.15 scrub starts 2024-02-10T07:54:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:54 smithi170 ceph-mon[33394]: 6.15 scrub ok 2024-02-10T07:54:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:55 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:55 smithi170 ceph-mon[33394]: pgmap v1070: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:54:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:55 smithi170 ceph-mon[33394]: 5.19 scrub starts 2024-02-10T07:54:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:55 smithi170 ceph-mon[33394]: 5.19 scrub ok 2024-02-10T07:54:55.714 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:55 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:55.714 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:55 smithi008 ceph-mon[29816]: pgmap v1070: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:54:55.714 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:55 smithi008 ceph-mon[29816]: 5.19 scrub starts 2024-02-10T07:54:55.714 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:55 smithi008 ceph-mon[29816]: 5.19 scrub ok 2024-02-10T07:54:55.715 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:55 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:54:55.715 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:55 smithi008 ceph-mon[40743]: pgmap v1070: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:54:55.715 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:55 smithi008 ceph-mon[40743]: 5.19 scrub starts 2024-02-10T07:54:55.715 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:55 smithi008 ceph-mon[40743]: 5.19 scrub ok 2024-02-10T07:54:56.208 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:54:56.232 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:54:56.378 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.2 to 1 (10000) 2024-02-10T07:54:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:56 smithi170 ceph-mon[33394]: 6.18 scrub starts 2024-02-10T07:54:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:56 smithi170 ceph-mon[33394]: 6.18 scrub ok 2024-02-10T07:54:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:56 smithi170 ceph-mon[33394]: 4.e deep-scrub starts 2024-02-10T07:54:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:56 smithi170 ceph-mon[33394]: 4.e deep-scrub ok 2024-02-10T07:54:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:56 smithi170 ceph-mon[33394]: 4.0 scrub starts 2024-02-10T07:54:56.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:56 smithi170 ceph-mon[33394]: 4.0 scrub ok 2024-02-10T07:54:56.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:56 smithi170 ceph-mon[33394]: 4.1d scrub starts 2024-02-10T07:54:56.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:56 smithi170 ceph-mon[33394]: 4.1d scrub ok 2024-02-10T07:54:56.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:56 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2114477963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:54:56.744 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[29816]: 6.18 scrub starts 2024-02-10T07:54:56.744 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[29816]: 6.18 scrub ok 2024-02-10T07:54:56.744 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[29816]: 4.e deep-scrub starts 2024-02-10T07:54:56.744 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[29816]: 4.e deep-scrub ok 2024-02-10T07:54:56.744 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[29816]: 4.0 scrub starts 2024-02-10T07:54:56.744 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[29816]: 4.0 scrub ok 2024-02-10T07:54:56.745 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[29816]: 4.1d scrub starts 2024-02-10T07:54:56.745 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[29816]: 4.1d scrub ok 2024-02-10T07:54:56.745 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2114477963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:54:56.745 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[40743]: 6.18 scrub starts 2024-02-10T07:54:56.745 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[40743]: 6.18 scrub ok 2024-02-10T07:54:56.745 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[40743]: 4.e deep-scrub starts 2024-02-10T07:54:56.745 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[40743]: 4.e deep-scrub ok 2024-02-10T07:54:56.745 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[40743]: 4.0 scrub starts 2024-02-10T07:54:56.745 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[40743]: 4.0 scrub ok 2024-02-10T07:54:56.745 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[40743]: 4.1d scrub starts 2024-02-10T07:54:56.745 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[40743]: 4.1d scrub ok 2024-02-10T07:54:56.745 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:56 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2114477963' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:54:56.869 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-10T07:54:57.048 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 2 1 2024-02-10T07:54:57.110 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T07:54:57.297 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:54:57.564 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:57 smithi170 ceph-mon[33394]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 745 B/s rd, 0 op/s 2024-02-10T07:54:57.564 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:57 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2114477963' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:54:57.564 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:57 smithi170 ceph-mon[33394]: osdmap e479: 8 total, 8 up, 8 in 2024-02-10T07:54:57.689 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:57 smithi008 ceph-mon[29816]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 745 B/s rd, 0 op/s 2024-02-10T07:54:57.689 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:57 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2114477963' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:54:57.689 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:57 smithi008 ceph-mon[29816]: osdmap e479: 8 total, 8 up, 8 in 2024-02-10T07:54:57.689 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:57 smithi008 ceph-mon[40743]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 745 B/s rd, 0 op/s 2024-02-10T07:54:57.689 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:57 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2114477963' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:54:57.690 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:57 smithi008 ceph-mon[40743]: osdmap e479: 8 total, 8 up, 8 in 2024-02-10T07:54:58.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:58 smithi170 ceph-mon[33394]: 6.14 scrub starts 2024-02-10T07:54:58.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:58 smithi170 ceph-mon[33394]: 6.14 scrub ok 2024-02-10T07:54:58.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:58 smithi008 ceph-mon[40743]: 6.14 scrub starts 2024-02-10T07:54:58.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:58 smithi008 ceph-mon[40743]: 6.14 scrub ok 2024-02-10T07:54:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:58 smithi008 ceph-mon[29816]: 6.14 scrub starts 2024-02-10T07:54:58.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:58 smithi008 ceph-mon[29816]: 6.14 scrub ok 2024-02-10T07:54:58.882 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:54:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:54:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:54:59.181 INFO:teuthology.orchestra.run.smithi170.stdout:ERROR: (22) Invalid argument 2024-02-10T07:54:59.182 INFO:teuthology.orchestra.run.smithi170.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:54:59.384 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-02-10T07:54:59.485 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:59 smithi170 ceph-mon[33394]: 3.6 scrub starts 2024-02-10T07:54:59.485 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:59 smithi170 ceph-mon[33394]: 3.6 scrub ok 2024-02-10T07:54:59.485 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:59 smithi170 ceph-mon[33394]: 3.3 deep-scrub starts 2024-02-10T07:54:59.485 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:59 smithi170 ceph-mon[33394]: 3.3 deep-scrub ok 2024-02-10T07:54:59.485 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:59 smithi170 ceph-mon[33394]: 6.1b scrub starts 2024-02-10T07:54:59.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:59 smithi170 ceph-mon[33394]: 6.1b scrub ok 2024-02-10T07:54:59.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:59 smithi170 ceph-mon[33394]: 5.17 deep-scrub starts 2024-02-10T07:54:59.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:59 smithi170 ceph-mon[33394]: 5.17 deep-scrub ok 2024-02-10T07:54:59.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:59 smithi170 ceph-mon[33394]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-10T07:54:59.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:54:59 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2065840253' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:54:59.736 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[29816]: 3.6 scrub starts 2024-02-10T07:54:59.736 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[29816]: 3.6 scrub ok 2024-02-10T07:54:59.736 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[29816]: 3.3 deep-scrub starts 2024-02-10T07:54:59.736 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[29816]: 3.3 deep-scrub ok 2024-02-10T07:54:59.736 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[29816]: 6.1b scrub starts 2024-02-10T07:54:59.736 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[29816]: 6.1b scrub ok 2024-02-10T07:54:59.737 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[29816]: 5.17 deep-scrub starts 2024-02-10T07:54:59.737 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[29816]: 5.17 deep-scrub ok 2024-02-10T07:54:59.737 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[29816]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-10T07:54:59.737 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2065840253' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:54:59.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[40743]: 3.6 scrub starts 2024-02-10T07:54:59.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[40743]: 3.6 scrub ok 2024-02-10T07:54:59.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[40743]: 3.3 deep-scrub starts 2024-02-10T07:54:59.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[40743]: 3.3 deep-scrub ok 2024-02-10T07:54:59.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[40743]: 6.1b scrub starts 2024-02-10T07:54:59.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[40743]: 6.1b scrub ok 2024-02-10T07:54:59.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[40743]: 5.17 deep-scrub starts 2024-02-10T07:54:59.737 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[40743]: 5.17 deep-scrub ok 2024-02-10T07:54:59.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[40743]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-10T07:54:59.738 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:54:59 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2065840253' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-10T07:54:59.863 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-10T07:55:00.093 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 4 1 2024-02-10T07:55:00.133 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:00.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:00 smithi008 ceph-mon[29816]: 4.3 scrub starts 2024-02-10T07:55:00.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:00 smithi008 ceph-mon[29816]: 4.3 scrub ok 2024-02-10T07:55:00.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:00 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2065840253' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:55:00.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:00 smithi008 ceph-mon[29816]: osdmap e480: 8 total, 8 up, 8 in 2024-02-10T07:55:00.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:00 smithi008 ceph-mon[29816]: 5.1d scrub starts 2024-02-10T07:55:00.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:00 smithi008 ceph-mon[29816]: 5.1d scrub ok 2024-02-10T07:55:00.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:00 smithi008 ceph-mon[40743]: 4.3 scrub starts 2024-02-10T07:55:00.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:00 smithi008 ceph-mon[40743]: 4.3 scrub ok 2024-02-10T07:55:00.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:00 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2065840253' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:55:00.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:00 smithi008 ceph-mon[40743]: osdmap e480: 8 total, 8 up, 8 in 2024-02-10T07:55:00.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:00 smithi008 ceph-mon[40743]: 5.1d scrub starts 2024-02-10T07:55:00.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:00 smithi008 ceph-mon[40743]: 5.1d scrub ok 2024-02-10T07:55:00.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:00 smithi170 ceph-mon[33394]: 4.3 scrub starts 2024-02-10T07:55:00.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:00 smithi170 ceph-mon[33394]: 4.3 scrub ok 2024-02-10T07:55:00.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:00 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2065840253' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-10T07:55:00.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:00 smithi170 ceph-mon[33394]: osdmap e480: 8 total, 8 up, 8 in 2024-02-10T07:55:00.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:00 smithi170 ceph-mon[33394]: 5.1d scrub starts 2024-02-10T07:55:00.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:00 smithi170 ceph-mon[33394]: 5.1d scrub ok 2024-02-10T07:55:01.312 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:01.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:01 smithi008 ceph-mon[29816]: 6.a scrub starts 2024-02-10T07:55:01.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:01 smithi008 ceph-mon[29816]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:55:01.560 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:01 smithi008 ceph-mon[29816]: 6.a scrub ok 2024-02-10T07:55:01.560 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:01 smithi008 ceph-mon[40743]: 6.a scrub starts 2024-02-10T07:55:01.561 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:01 smithi008 ceph-mon[40743]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:55:01.561 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:01 smithi008 ceph-mon[40743]: 6.a scrub ok 2024-02-10T07:55:01.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:01 smithi170 ceph-mon[33394]: 6.a scrub starts 2024-02-10T07:55:01.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:01 smithi170 ceph-mon[33394]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:55:01.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:01 smithi170 ceph-mon[33394]: 6.a scrub ok 2024-02-10T07:55:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:02 smithi008 ceph-mon[29816]: 3.12 scrub starts 2024-02-10T07:55:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:02 smithi008 ceph-mon[29816]: 3.12 scrub ok 2024-02-10T07:55:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:02 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:55:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:02 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:55:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:02 smithi008 ceph-mon[29816]: 5.16 scrub starts 2024-02-10T07:55:02.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:02 smithi008 ceph-mon[29816]: 5.16 scrub ok 2024-02-10T07:55:02.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:02 smithi008 ceph-mon[29816]: osdmap e481: 8 total, 8 up, 8 in 2024-02-10T07:55:02.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:02 smithi008 ceph-mon[40743]: 3.12 scrub starts 2024-02-10T07:55:02.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:02 smithi008 ceph-mon[40743]: 3.12 scrub ok 2024-02-10T07:55:02.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:02 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:55:02.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:02 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:55:02.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:02 smithi008 ceph-mon[40743]: 5.16 scrub starts 2024-02-10T07:55:02.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:02 smithi008 ceph-mon[40743]: 5.16 scrub ok 2024-02-10T07:55:02.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:02 smithi008 ceph-mon[40743]: osdmap e481: 8 total, 8 up, 8 in 2024-02-10T07:55:02.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:02 smithi170 ceph-mon[33394]: 3.12 scrub starts 2024-02-10T07:55:02.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:02 smithi170 ceph-mon[33394]: 3.12 scrub ok 2024-02-10T07:55:02.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:02 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:55:02.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:02 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:55:02.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:02 smithi170 ceph-mon[33394]: 5.16 scrub starts 2024-02-10T07:55:02.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:02 smithi170 ceph-mon[33394]: 5.16 scrub ok 2024-02-10T07:55:02.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:02 smithi170 ceph-mon[33394]: osdmap e481: 8 total, 8 up, 8 in 2024-02-10T07:55:03.445 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:55:03.446 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:55:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[29816]: 3.18 deep-scrub starts 2024-02-10T07:55:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[29816]: 3.18 deep-scrub ok 2024-02-10T07:55:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[29816]: 3.1c scrub starts 2024-02-10T07:55:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[29816]: 3.1c scrub ok 2024-02-10T07:55:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[29816]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[29816]: 5.13 scrub starts 2024-02-10T07:55:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[29816]: 5.13 scrub ok 2024-02-10T07:55:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[29816]: 3.0 scrub starts 2024-02-10T07:55:03.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[29816]: 3.0 scrub ok 2024-02-10T07:55:03.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[40743]: 3.18 deep-scrub starts 2024-02-10T07:55:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[40743]: 3.18 deep-scrub ok 2024-02-10T07:55:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[40743]: 3.1c scrub starts 2024-02-10T07:55:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[40743]: 3.1c scrub ok 2024-02-10T07:55:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[40743]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[40743]: 5.13 scrub starts 2024-02-10T07:55:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[40743]: 5.13 scrub ok 2024-02-10T07:55:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[40743]: 3.0 scrub starts 2024-02-10T07:55:03.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:03 smithi008 ceph-mon[40743]: 3.0 scrub ok 2024-02-10T07:55:03.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:03 smithi170 ceph-mon[33394]: 3.18 deep-scrub starts 2024-02-10T07:55:03.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:03 smithi170 ceph-mon[33394]: 3.18 deep-scrub ok 2024-02-10T07:55:03.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:03 smithi170 ceph-mon[33394]: 3.1c scrub starts 2024-02-10T07:55:03.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:03 smithi170 ceph-mon[33394]: 3.1c scrub ok 2024-02-10T07:55:03.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:03 smithi170 ceph-mon[33394]: pgmap v1077: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:03.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:03 smithi170 ceph-mon[33394]: 5.13 scrub starts 2024-02-10T07:55:03.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:03 smithi170 ceph-mon[33394]: 5.13 scrub ok 2024-02-10T07:55:03.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:03 smithi170 ceph-mon[33394]: 3.0 scrub starts 2024-02-10T07:55:03.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:03 smithi170 ceph-mon[33394]: 3.0 scrub ok 2024-02-10T07:55:04.077 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-10T07:55:04.362 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:04.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:55:04 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:55:04.407 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.4 to 1 (10000) 2024-02-10T07:55:04.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[29816]: 3.1d scrub starts 2024-02-10T07:55:04.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[29816]: 3.1d scrub ok 2024-02-10T07:55:04.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[29816]: 3.8 scrub starts 2024-02-10T07:55:04.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[29816]: 3.8 scrub ok 2024-02-10T07:55:04.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/77880621' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:55:04.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:55:04.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[29816]: 3.13 scrub starts 2024-02-10T07:55:04.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[29816]: 3.13 scrub ok 2024-02-10T07:55:04.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[40743]: 3.1d scrub starts 2024-02-10T07:55:04.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[40743]: 3.1d scrub ok 2024-02-10T07:55:04.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[40743]: 3.8 scrub starts 2024-02-10T07:55:04.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[40743]: 3.8 scrub ok 2024-02-10T07:55:04.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/77880621' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:55:04.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:55:04.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[40743]: 3.13 scrub starts 2024-02-10T07:55:04.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:04 smithi008 ceph-mon[40743]: 3.13 scrub ok 2024-02-10T07:55:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:04 smithi170 ceph-mon[33394]: 3.1d scrub starts 2024-02-10T07:55:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:04 smithi170 ceph-mon[33394]: 3.1d scrub ok 2024-02-10T07:55:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:04 smithi170 ceph-mon[33394]: 3.8 scrub starts 2024-02-10T07:55:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:04 smithi170 ceph-mon[33394]: 3.8 scrub ok 2024-02-10T07:55:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:04 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/77880621' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:55:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:55:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:04 smithi170 ceph-mon[33394]: 3.13 scrub starts 2024-02-10T07:55:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:04 smithi170 ceph-mon[33394]: 3.13 scrub ok 2024-02-10T07:55:05.295 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 4 1 2024-02-10T07:55:05.558 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[29816]: 6.e scrub starts 2024-02-10T07:55:05.559 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[29816]: 6.e scrub ok 2024-02-10T07:55:05.559 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:05.559 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[29816]: pgmap v1078: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:55:05.559 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/77880621' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:55:05.559 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[29816]: osdmap e482: 8 total, 8 up, 8 in 2024-02-10T07:55:05.559 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[29816]: 5.12 scrub starts 2024-02-10T07:55:05.559 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[29816]: 5.12 scrub ok 2024-02-10T07:55:05.559 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[40743]: 6.e scrub starts 2024-02-10T07:55:05.559 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[40743]: 6.e scrub ok 2024-02-10T07:55:05.560 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:05.560 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[40743]: pgmap v1078: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:55:05.560 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/77880621' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:55:05.560 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[40743]: osdmap e482: 8 total, 8 up, 8 in 2024-02-10T07:55:05.560 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[40743]: 5.12 scrub starts 2024-02-10T07:55:05.560 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:05 smithi008 ceph-mon[40743]: 5.12 scrub ok 2024-02-10T07:55:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:05 smithi170 ceph-mon[33394]: 6.e scrub starts 2024-02-10T07:55:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:05 smithi170 ceph-mon[33394]: 6.e scrub ok 2024-02-10T07:55:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:05 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:05 smithi170 ceph-mon[33394]: pgmap v1078: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:55:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:05 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/77880621' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:55:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:05 smithi170 ceph-mon[33394]: osdmap e482: 8 total, 8 up, 8 in 2024-02-10T07:55:05.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:05 smithi170 ceph-mon[33394]: 5.12 scrub starts 2024-02-10T07:55:05.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:05 smithi170 ceph-mon[33394]: 5.12 scrub ok 2024-02-10T07:55:06.313 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:06.569 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:06 smithi008 ceph-mon[29816]: 3.11 scrub starts 2024-02-10T07:55:06.569 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:06 smithi008 ceph-mon[29816]: 3.11 scrub ok 2024-02-10T07:55:06.569 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:06 smithi008 ceph-mon[40743]: 3.11 scrub starts 2024-02-10T07:55:06.569 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:06 smithi008 ceph-mon[40743]: 3.11 scrub ok 2024-02-10T07:55:06.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:06 smithi170 ceph-mon[33394]: 3.11 scrub starts 2024-02-10T07:55:06.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:06 smithi170 ceph-mon[33394]: 3.11 scrub ok 2024-02-10T07:55:07.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:07 smithi008 ceph-mon[29816]: pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 747 B/s rd, 0 op/s 2024-02-10T07:55:07.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:07 smithi008 ceph-mon[29816]: osdmap e483: 8 total, 8 up, 8 in 2024-02-10T07:55:07.766 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:07 smithi008 ceph-mon[40743]: pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 747 B/s rd, 0 op/s 2024-02-10T07:55:07.766 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:07 smithi008 ceph-mon[40743]: osdmap e483: 8 total, 8 up, 8 in 2024-02-10T07:55:07.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:07 smithi170 ceph-mon[33394]: pgmap v1080: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 747 B/s rd, 0 op/s 2024-02-10T07:55:07.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:07 smithi170 ceph-mon[33394]: osdmap e483: 8 total, 8 up, 8 in 2024-02-10T07:55:08.055 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:55:08.055 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:55:08.791 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:08 smithi008 ceph-mon[29816]: 4.9 scrub starts 2024-02-10T07:55:08.791 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:08 smithi008 ceph-mon[29816]: 4.9 scrub ok 2024-02-10T07:55:08.791 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:08 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:55:08.791 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:08 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:55:08.791 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:08 smithi008 ceph-mon[29816]: osdmap e484: 8 total, 8 up, 8 in 2024-02-10T07:55:08.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:08 smithi008 ceph-mon[40743]: 4.9 scrub starts 2024-02-10T07:55:08.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:08 smithi008 ceph-mon[40743]: 4.9 scrub ok 2024-02-10T07:55:08.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:08 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:55:08.791 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:08 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:55:08.792 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:08 smithi008 ceph-mon[40743]: osdmap e484: 8 total, 8 up, 8 in 2024-02-10T07:55:08.830 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-10T07:55:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:08 smithi170 ceph-mon[33394]: 4.9 scrub starts 2024-02-10T07:55:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:08 smithi170 ceph-mon[33394]: 4.9 scrub ok 2024-02-10T07:55:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:08 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:55:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:08 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:55:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:08 smithi170 ceph-mon[33394]: osdmap e484: 8 total, 8 up, 8 in 2024-02-10T07:55:09.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:55:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:55:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:55:09.447 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:09.569 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:09 smithi008 ceph-mon[29816]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:09.569 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:09 smithi008 ceph-mon[29816]: 3.d scrub starts 2024-02-10T07:55:09.569 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:09 smithi008 ceph-mon[29816]: 3.d scrub ok 2024-02-10T07:55:09.569 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:09 smithi008 ceph-mon[40743]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:09.569 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:09 smithi008 ceph-mon[40743]: 3.d scrub starts 2024-02-10T07:55:09.570 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:09 smithi008 ceph-mon[40743]: 3.d scrub ok 2024-02-10T07:55:09.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:09 smithi170 ceph-mon[33394]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:09.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:09 smithi170 ceph-mon[33394]: 3.d scrub starts 2024-02-10T07:55:09.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:09 smithi170 ceph-mon[33394]: 3.d scrub ok 2024-02-10T07:55:10.426 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-02-10T07:55:10.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:10 smithi008 ceph-mon[40743]: 6.17 scrub starts 2024-02-10T07:55:10.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:10 smithi008 ceph-mon[40743]: 6.17 scrub ok 2024-02-10T07:55:10.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:10 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1404491107' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:55:10.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:10 smithi008 ceph-mon[29816]: 6.17 scrub starts 2024-02-10T07:55:10.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:10 smithi008 ceph-mon[29816]: 6.17 scrub ok 2024-02-10T07:55:10.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:10 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1404491107' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:55:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:10 smithi170 ceph-mon[33394]: 6.17 scrub starts 2024-02-10T07:55:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:10 smithi170 ceph-mon[33394]: 6.17 scrub ok 2024-02-10T07:55:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:10 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1404491107' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-10T07:55:11.267 INFO:teuthology.orchestra.run.smithi008.stdout:ERROR: (22) Invalid argument 2024-02-10T07:55:11.267 INFO:teuthology.orchestra.run.smithi008.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-10T07:55:11.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:11 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:55:11.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:11 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:55:11.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:11 smithi008 ceph-mon[40743]: pgmap v1084: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:11.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:11 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1404491107' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:55:11.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:11 smithi008 ceph-mon[40743]: osdmap e485: 8 total, 8 up, 8 in 2024-02-10T07:55:11.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:11 smithi008 ceph-mon[40743]: 2.0 scrub starts 2024-02-10T07:55:11.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:11 smithi008 ceph-mon[40743]: 2.0 scrub ok 2024-02-10T07:55:11.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:11 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:55:11.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:11 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:55:11.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:11 smithi008 ceph-mon[29816]: pgmap v1084: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:11.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:11 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1404491107' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:55:11.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:11 smithi008 ceph-mon[29816]: osdmap e485: 8 total, 8 up, 8 in 2024-02-10T07:55:11.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:11 smithi008 ceph-mon[29816]: 2.0 scrub starts 2024-02-10T07:55:11.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:11 smithi008 ceph-mon[29816]: 2.0 scrub ok 2024-02-10T07:55:11.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:11 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:55:11.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:11 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:55:11.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:11 smithi170 ceph-mon[33394]: pgmap v1084: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:11.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:11 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1404491107' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-10T07:55:11.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:11 smithi170 ceph-mon[33394]: osdmap e485: 8 total, 8 up, 8 in 2024-02-10T07:55:11.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:11 smithi170 ceph-mon[33394]: 2.0 scrub starts 2024-02-10T07:55:11.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:11 smithi170 ceph-mon[33394]: 2.0 scrub ok 2024-02-10T07:55:12.640 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 6 1 2024-02-10T07:55:12.894 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:13.276 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:13 smithi008 ceph-mon[29816]: 3.1a scrub starts 2024-02-10T07:55:13.276 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:13 smithi008 ceph-mon[29816]: 3.1a scrub ok 2024-02-10T07:55:13.277 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:13 smithi008 ceph-mon[29816]: pgmap v1086: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:13.277 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:13 smithi008 ceph-mon[29816]: 4.4 scrub starts 2024-02-10T07:55:13.277 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:13 smithi008 ceph-mon[29816]: 4.4 scrub ok 2024-02-10T07:55:13.277 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:13 smithi008 ceph-mon[40743]: 3.1a scrub starts 2024-02-10T07:55:13.277 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:13 smithi008 ceph-mon[40743]: 3.1a scrub ok 2024-02-10T07:55:13.277 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:13 smithi008 ceph-mon[40743]: pgmap v1086: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:13.277 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:13 smithi008 ceph-mon[40743]: 4.4 scrub starts 2024-02-10T07:55:13.277 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:13 smithi008 ceph-mon[40743]: 4.4 scrub ok 2024-02-10T07:55:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:13 smithi170 ceph-mon[33394]: 3.1a scrub starts 2024-02-10T07:55:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:13 smithi170 ceph-mon[33394]: 3.1a scrub ok 2024-02-10T07:55:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:13 smithi170 ceph-mon[33394]: pgmap v1086: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:13.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:13 smithi170 ceph-mon[33394]: 4.4 scrub starts 2024-02-10T07:55:13.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:13 smithi170 ceph-mon[33394]: 4.4 scrub ok 2024-02-10T07:55:14.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:14 smithi008 ceph-mon[29816]: 6.1a scrub starts 2024-02-10T07:55:14.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:14 smithi008 ceph-mon[29816]: 6.1a scrub ok 2024-02-10T07:55:14.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:14 smithi008 ceph-mon[40743]: 6.1a scrub starts 2024-02-10T07:55:14.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:14 smithi008 ceph-mon[40743]: 6.1a scrub ok 2024-02-10T07:55:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:14 smithi170 ceph-mon[33394]: 6.1a scrub starts 2024-02-10T07:55:14.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:14 smithi170 ceph-mon[33394]: 6.1a scrub ok 2024-02-10T07:55:14.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:55:14 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:55:15.177 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.6 to 1 (10000) 2024-02-10T07:55:15.547 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:55:15.547 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:55:15.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:15.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[29816]: osdmap e486: 8 total, 8 up, 8 in 2024-02-10T07:55:15.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[29816]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-10T07:55:15.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:55:15.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:55:15.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[29816]: 3.b scrub starts 2024-02-10T07:55:15.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[29816]: 3.b scrub ok 2024-02-10T07:55:15.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[29816]: 4.11 scrub starts 2024-02-10T07:55:15.548 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[29816]: 4.11 scrub ok 2024-02-10T07:55:15.549 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/27283905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:55:15.549 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:55:15.549 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:55:15.549 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:55:15.549 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:15.549 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[40743]: osdmap e486: 8 total, 8 up, 8 in 2024-02-10T07:55:15.549 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[40743]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-10T07:55:15.549 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:55:15.550 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:55:15.550 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[40743]: 3.b scrub starts 2024-02-10T07:55:15.550 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[40743]: 3.b scrub ok 2024-02-10T07:55:15.550 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[40743]: 4.11 scrub starts 2024-02-10T07:55:15.550 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[40743]: 4.11 scrub ok 2024-02-10T07:55:15.550 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/27283905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:55:15.550 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:15 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:55:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:15 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:55:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:15 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:55:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:15 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:15 smithi170 ceph-mon[33394]: osdmap e486: 8 total, 8 up, 8 in 2024-02-10T07:55:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:15 smithi170 ceph-mon[33394]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-10T07:55:15.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:15 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:55:15.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:15 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:55:15.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:15 smithi170 ceph-mon[33394]: 3.b scrub starts 2024-02-10T07:55:15.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:15 smithi170 ceph-mon[33394]: 3.b scrub ok 2024-02-10T07:55:15.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:15 smithi170 ceph-mon[33394]: 4.11 scrub starts 2024-02-10T07:55:15.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:15 smithi170 ceph-mon[33394]: 4.11 scrub ok 2024-02-10T07:55:15.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:15 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/27283905' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:55:15.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:15 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:55:15.936 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 6 1 2024-02-10T07:55:16.186 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:16.439 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:16 smithi008 ceph-mon[29816]: 5.f scrub starts 2024-02-10T07:55:16.439 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:16 smithi008 ceph-mon[29816]: 5.f scrub ok 2024-02-10T07:55:16.439 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:16 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:55:16.439 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:16 smithi008 ceph-mon[29816]: osdmap e487: 8 total, 8 up, 8 in 2024-02-10T07:55:16.439 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:16 smithi008 ceph-mon[29816]: 6.11 scrub starts 2024-02-10T07:55:16.439 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:16 smithi008 ceph-mon[29816]: 3.15 scrub starts 2024-02-10T07:55:16.439 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:16 smithi008 ceph-mon[29816]: 3.15 scrub ok 2024-02-10T07:55:16.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:16 smithi008 ceph-mon[40743]: 5.f scrub starts 2024-02-10T07:55:16.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:16 smithi008 ceph-mon[40743]: 5.f scrub ok 2024-02-10T07:55:16.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:16 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:55:16.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:16 smithi008 ceph-mon[40743]: osdmap e487: 8 total, 8 up, 8 in 2024-02-10T07:55:16.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:16 smithi008 ceph-mon[40743]: 6.11 scrub starts 2024-02-10T07:55:16.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:16 smithi008 ceph-mon[40743]: 3.15 scrub starts 2024-02-10T07:55:16.440 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:16 smithi008 ceph-mon[40743]: 3.15 scrub ok 2024-02-10T07:55:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:16 smithi170 ceph-mon[33394]: 5.f scrub starts 2024-02-10T07:55:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:16 smithi170 ceph-mon[33394]: 5.f scrub ok 2024-02-10T07:55:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:16 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:55:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:16 smithi170 ceph-mon[33394]: osdmap e487: 8 total, 8 up, 8 in 2024-02-10T07:55:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:16 smithi170 ceph-mon[33394]: 6.11 scrub starts 2024-02-10T07:55:16.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:16 smithi170 ceph-mon[33394]: 3.15 scrub starts 2024-02-10T07:55:16.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:16 smithi170 ceph-mon[33394]: 3.15 scrub ok 2024-02-10T07:55:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:17 smithi008 ceph-mon[29816]: 6.11 scrub ok 2024-02-10T07:55:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:17 smithi008 ceph-mon[29816]: pgmap v1090: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:17 smithi008 ceph-mon[29816]: 3.a scrub starts 2024-02-10T07:55:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:17 smithi008 ceph-mon[29816]: 3.a scrub ok 2024-02-10T07:55:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:17 smithi008 ceph-mon[29816]: 6.19 scrub starts 2024-02-10T07:55:17.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:17 smithi008 ceph-mon[29816]: 6.19 scrub ok 2024-02-10T07:55:17.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:17 smithi008 ceph-mon[40743]: 6.11 scrub ok 2024-02-10T07:55:17.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:17 smithi008 ceph-mon[40743]: pgmap v1090: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:17.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:17 smithi008 ceph-mon[40743]: 3.a scrub starts 2024-02-10T07:55:17.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:17 smithi008 ceph-mon[40743]: 3.a scrub ok 2024-02-10T07:55:17.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:17 smithi008 ceph-mon[40743]: 6.19 scrub starts 2024-02-10T07:55:17.333 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:17 smithi008 ceph-mon[40743]: 6.19 scrub ok 2024-02-10T07:55:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:17 smithi170 ceph-mon[33394]: 6.11 scrub ok 2024-02-10T07:55:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:17 smithi170 ceph-mon[33394]: pgmap v1090: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:17 smithi170 ceph-mon[33394]: 3.a scrub starts 2024-02-10T07:55:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:17 smithi170 ceph-mon[33394]: 3.a scrub ok 2024-02-10T07:55:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:17 smithi170 ceph-mon[33394]: 6.19 scrub starts 2024-02-10T07:55:17.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:17 smithi170 ceph-mon[33394]: 6.19 scrub ok 2024-02-10T07:55:18.179 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-02-10T07:55:18.540 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:18 smithi008 ceph-mon[29816]: 5.18 scrub starts 2024-02-10T07:55:18.540 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:18 smithi008 ceph-mon[29816]: 5.18 scrub ok 2024-02-10T07:55:18.540 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:18 smithi008 ceph-mon[29816]: 6.13 scrub starts 2024-02-10T07:55:18.540 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:18 smithi008 ceph-mon[29816]: 6.4 scrub starts 2024-02-10T07:55:18.541 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:18 smithi008 ceph-mon[29816]: 6.4 scrub ok 2024-02-10T07:55:18.541 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:18 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1230261156' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:55:18.541 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:18 smithi008 ceph-mon[40743]: 5.18 scrub starts 2024-02-10T07:55:18.541 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:18 smithi008 ceph-mon[40743]: 5.18 scrub ok 2024-02-10T07:55:18.541 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:18 smithi008 ceph-mon[40743]: 6.13 scrub starts 2024-02-10T07:55:18.541 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:18 smithi008 ceph-mon[40743]: 6.4 scrub starts 2024-02-10T07:55:18.541 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:18 smithi008 ceph-mon[40743]: 6.4 scrub ok 2024-02-10T07:55:18.541 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:18 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1230261156' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:55:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:18 smithi170 ceph-mon[33394]: 5.18 scrub starts 2024-02-10T07:55:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:18 smithi170 ceph-mon[33394]: 5.18 scrub ok 2024-02-10T07:55:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:18 smithi170 ceph-mon[33394]: 6.13 scrub starts 2024-02-10T07:55:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:18 smithi170 ceph-mon[33394]: 6.4 scrub starts 2024-02-10T07:55:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:18 smithi170 ceph-mon[33394]: 6.4 scrub ok 2024-02-10T07:55:18.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:18 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1230261156' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-10T07:55:18.791 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:55:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:55:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:55:18.817 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 7 1 2024-02-10T07:55:19.063 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:19.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:19 smithi008 ceph-mon[29816]: 6.13 scrub ok 2024-02-10T07:55:19.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:19 smithi008 ceph-mon[29816]: 3.14 scrub starts 2024-02-10T07:55:19.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:19 smithi008 ceph-mon[29816]: 3.14 scrub ok 2024-02-10T07:55:19.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:19 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1230261156' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:55:19.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:19 smithi008 ceph-mon[29816]: osdmap e488: 8 total, 8 up, 8 in 2024-02-10T07:55:19.495 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:19 smithi008 ceph-mon[29816]: pgmap v1092: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:19.496 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:19 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:55:19.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:19 smithi008 ceph-mon[40743]: 6.13 scrub ok 2024-02-10T07:55:19.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:19 smithi008 ceph-mon[40743]: 3.14 scrub starts 2024-02-10T07:55:19.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:19 smithi008 ceph-mon[40743]: 3.14 scrub ok 2024-02-10T07:55:19.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:19 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1230261156' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:55:19.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:19 smithi008 ceph-mon[40743]: osdmap e488: 8 total, 8 up, 8 in 2024-02-10T07:55:19.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:19 smithi008 ceph-mon[40743]: pgmap v1092: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:19.496 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:19 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:55:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:19 smithi170 ceph-mon[33394]: 6.13 scrub ok 2024-02-10T07:55:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:19 smithi170 ceph-mon[33394]: 3.14 scrub starts 2024-02-10T07:55:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:19 smithi170 ceph-mon[33394]: 3.14 scrub ok 2024-02-10T07:55:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:19 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1230261156' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-10T07:55:19.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:19 smithi170 ceph-mon[33394]: osdmap e488: 8 total, 8 up, 8 in 2024-02-10T07:55:19.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:19 smithi170 ceph-mon[33394]: pgmap v1092: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:19.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:19 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:55:20.463 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:20 smithi008 ceph-mon[29816]: 3.1 scrub starts 2024-02-10T07:55:20.463 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:20 smithi008 ceph-mon[29816]: 3.1 scrub ok 2024-02-10T07:55:20.463 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:20 smithi008 ceph-mon[40743]: 3.1 scrub starts 2024-02-10T07:55:20.464 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:20 smithi008 ceph-mon[40743]: 3.1 scrub ok 2024-02-10T07:55:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:20 smithi170 ceph-mon[33394]: 3.1 scrub starts 2024-02-10T07:55:20.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:20 smithi170 ceph-mon[33394]: 3.1 scrub ok 2024-02-10T07:55:21.195 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.7 to 1 (10000) 2024-02-10T07:55:21.532 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:21 smithi008 ceph-mon[29816]: pgmap v1093: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 844 B/s rd, 0 op/s 2024-02-10T07:55:21.532 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:21 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2165251414' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:55:21.532 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:21 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:55:21.532 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:21 smithi008 ceph-mon[40743]: pgmap v1093: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 844 B/s rd, 0 op/s 2024-02-10T07:55:21.533 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:21 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2165251414' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:55:21.533 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:21 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:55:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:21 smithi170 ceph-mon[33394]: pgmap v1093: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 844 B/s rd, 0 op/s 2024-02-10T07:55:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:21 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2165251414' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:55:21.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:21 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:55:21.925 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 7 1 2024-02-10T07:55:22.180 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:22.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:22 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:55:22.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:22 smithi008 ceph-mon[29816]: osdmap e489: 8 total, 8 up, 8 in 2024-02-10T07:55:22.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:22 smithi008 ceph-mon[29816]: 4.16 scrub starts 2024-02-10T07:55:22.452 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:22 smithi008 ceph-mon[29816]: 4.16 scrub ok 2024-02-10T07:55:22.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:22 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:55:22.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:22 smithi008 ceph-mon[40743]: osdmap e489: 8 total, 8 up, 8 in 2024-02-10T07:55:22.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:22 smithi008 ceph-mon[40743]: 4.16 scrub starts 2024-02-10T07:55:22.453 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:22 smithi008 ceph-mon[40743]: 4.16 scrub ok 2024-02-10T07:55:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:22 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:55:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:22 smithi170 ceph-mon[33394]: osdmap e489: 8 total, 8 up, 8 in 2024-02-10T07:55:22.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:22 smithi170 ceph-mon[33394]: 4.16 scrub starts 2024-02-10T07:55:22.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:22 smithi170 ceph-mon[33394]: 4.16 scrub ok 2024-02-10T07:55:23.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:23 smithi008 ceph-mon[29816]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 725 B/s rd, 0 op/s 2024-02-10T07:55:23.332 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:23 smithi008 ceph-mon[40743]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 725 B/s rd, 0 op/s 2024-02-10T07:55:23.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:23 smithi170 ceph-mon[33394]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 725 B/s rd, 0 op/s 2024-02-10T07:55:24.209 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-02-10T07:55:24.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:24 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1021821524' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:55:24.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:55:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:55:24.483 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:24 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1021821524' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:55:24.483 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:24 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1021821524' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-10T07:55:24.873 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 5 1 2024-02-10T07:55:25.126 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:25.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:25 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:25.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:25 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1021821524' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:55:25.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:25 smithi008 ceph-mon[29816]: osdmap e490: 8 total, 8 up, 8 in 2024-02-10T07:55:25.488 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:25 smithi008 ceph-mon[29816]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:55:25.488 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:25 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:25.488 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:25 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1021821524' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:55:25.488 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:25 smithi008 ceph-mon[40743]: osdmap e490: 8 total, 8 up, 8 in 2024-02-10T07:55:25.488 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:25 smithi008 ceph-mon[40743]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:55:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:25 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:25 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1021821524' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-10T07:55:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:25 smithi170 ceph-mon[33394]: osdmap e490: 8 total, 8 up, 8 in 2024-02-10T07:55:25.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:25 smithi170 ceph-mon[33394]: pgmap v1097: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-10T07:55:27.160 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.5 to 1 (10000) 2024-02-10T07:55:27.499 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:27 smithi008 ceph-mon[29816]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:27.500 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:27 smithi008 ceph-mon[29816]: 4.f scrub starts 2024-02-10T07:55:27.500 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:27 smithi008 ceph-mon[29816]: 4.f scrub ok 2024-02-10T07:55:27.500 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:27 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/261793268' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:55:27.500 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:27 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:55:27.500 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:27 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:55:27.500 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:27 smithi008 ceph-mon[29816]: osdmap e491: 8 total, 8 up, 8 in 2024-02-10T07:55:27.500 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:27 smithi008 ceph-mon[40743]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:27.500 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:27 smithi008 ceph-mon[40743]: 4.f scrub starts 2024-02-10T07:55:27.500 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:27 smithi008 ceph-mon[40743]: 4.f scrub ok 2024-02-10T07:55:27.501 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:27 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/261793268' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:55:27.501 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:27 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:55:27.501 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:27 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:55:27.501 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:27 smithi008 ceph-mon[40743]: osdmap e491: 8 total, 8 up, 8 in 2024-02-10T07:55:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:27 smithi170 ceph-mon[33394]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:27 smithi170 ceph-mon[33394]: 4.f scrub starts 2024-02-10T07:55:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:27 smithi170 ceph-mon[33394]: 4.f scrub ok 2024-02-10T07:55:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:27 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/261793268' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:55:27.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:27 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:55:27.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:27 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:55:27.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:27 smithi170 ceph-mon[33394]: osdmap e491: 8 total, 8 up, 8 in 2024-02-10T07:55:27.771 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 5 1 2024-02-10T07:55:28.020 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:28.404 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:28 smithi008 ceph-mon[29816]: 6.1 scrub starts 2024-02-10T07:55:28.404 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:28 smithi008 ceph-mon[29816]: 6.1 scrub ok 2024-02-10T07:55:28.405 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:28 smithi008 ceph-mon[40743]: 6.1 scrub starts 2024-02-10T07:55:28.405 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:28 smithi008 ceph-mon[40743]: 6.1 scrub ok 2024-02-10T07:55:28.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:28 smithi170 ceph-mon[33394]: 6.1 scrub starts 2024-02-10T07:55:28.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:28 smithi170 ceph-mon[33394]: 6.1 scrub ok 2024-02-10T07:55:29.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:55:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:55:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:55:29.402 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:29 smithi008 ceph-mon[29816]: 6.2 scrub starts 2024-02-10T07:55:29.402 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:29 smithi008 ceph-mon[29816]: 6.2 scrub ok 2024-02-10T07:55:29.403 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:29 smithi008 ceph-mon[29816]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-02-10T07:55:29.403 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:29 smithi008 ceph-mon[40743]: 6.2 scrub starts 2024-02-10T07:55:29.403 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:29 smithi008 ceph-mon[40743]: 6.2 scrub ok 2024-02-10T07:55:29.403 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:29 smithi008 ceph-mon[40743]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-02-10T07:55:29.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:29 smithi170 ceph-mon[33394]: 6.2 scrub starts 2024-02-10T07:55:29.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:29 smithi170 ceph-mon[33394]: 6.2 scrub ok 2024-02-10T07:55:29.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:29 smithi170 ceph-mon[33394]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-02-10T07:55:30.304 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-02-10T07:55:30.549 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:30 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3730685041' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:55:30.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:30 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3730685041' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:55:30.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:30 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3730685041' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-10T07:55:31.075 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd reweight 3 1 2024-02-10T07:55:31.303 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:31 smithi008 ceph-mon[40743]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:31.303 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:31 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3730685041' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:55:31.303 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:31 smithi008 ceph-mon[40743]: osdmap e492: 8 total, 8 up, 8 in 2024-02-10T07:55:31.348 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:31.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:31 smithi008 ceph-mon[29816]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:31 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3730685041' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:55:31.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:31 smithi008 ceph-mon[29816]: osdmap e492: 8 total, 8 up, 8 in 2024-02-10T07:55:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:31 smithi170 ceph-mon[33394]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:31 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3730685041' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-10T07:55:31.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:31 smithi170 ceph-mon[33394]: osdmap e492: 8 total, 8 up, 8 in 2024-02-10T07:55:32.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:32 smithi008 ceph-mon[29816]: osdmap e493: 8 total, 8 up, 8 in 2024-02-10T07:55:32.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:32 smithi008 ceph-mon[40743]: osdmap e493: 8 total, 8 up, 8 in 2024-02-10T07:55:32.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:32 smithi170 ceph-mon[33394]: osdmap e493: 8 total, 8 up, 8 in 2024-02-10T07:55:33.312 INFO:teuthology.orchestra.run.smithi008.stderr:reweighted osd.3 to 1 (10000) 2024-02-10T07:55:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:33 smithi008 ceph-mon[29816]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:33 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1191232245' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:55:33.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:33 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:55:33.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:33 smithi008 ceph-mon[40743]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:33 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1191232245' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:55:33.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:33 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:55:33.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:33 smithi170 ceph-mon[33394]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T07:55:33.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:33 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1191232245' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:55:33.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:33 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:55:34.042 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd primary-affinity 3 1 2024-02-10T07:55:34.293 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:34.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:55:34 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:55:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:34 smithi170 ceph-mon[33394]: 3.9 scrub starts 2024-02-10T07:55:34.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:34 smithi170 ceph-mon[33394]: 3.9 scrub ok 2024-02-10T07:55:34.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:34 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:55:34.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:34 smithi170 ceph-mon[33394]: osdmap e494: 8 total, 8 up, 8 in 2024-02-10T07:55:34.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:34 smithi170 ceph-mon[33394]: 4.15 scrub starts 2024-02-10T07:55:34.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:34 smithi170 ceph-mon[33394]: 4.15 scrub ok 2024-02-10T07:55:34.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:34 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:55:34.570 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:34 smithi008 ceph-mon[29816]: 3.9 scrub starts 2024-02-10T07:55:34.570 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:34 smithi008 ceph-mon[29816]: 3.9 scrub ok 2024-02-10T07:55:34.570 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:34 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:55:34.570 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:34 smithi008 ceph-mon[29816]: osdmap e494: 8 total, 8 up, 8 in 2024-02-10T07:55:34.570 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:34 smithi008 ceph-mon[29816]: 4.15 scrub starts 2024-02-10T07:55:34.570 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:34 smithi008 ceph-mon[29816]: 4.15 scrub ok 2024-02-10T07:55:34.570 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:34 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:55:34.571 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:34 smithi008 ceph-mon[40743]: 3.9 scrub starts 2024-02-10T07:55:34.571 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:34 smithi008 ceph-mon[40743]: 3.9 scrub ok 2024-02-10T07:55:34.571 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:34 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:55:34.571 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:34 smithi008 ceph-mon[40743]: osdmap e494: 8 total, 8 up, 8 in 2024-02-10T07:55:34.571 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:34 smithi008 ceph-mon[40743]: 4.15 scrub starts 2024-02-10T07:55:34.571 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:34 smithi008 ceph-mon[40743]: 4.15 scrub ok 2024-02-10T07:55:34.571 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:34 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:55:35.509 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:35 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:35.510 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:35 smithi008 ceph-mon[29816]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:35.510 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:35 smithi008 ceph-mon[29816]: 6.7 scrub starts 2024-02-10T07:55:35.510 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:35 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:35.510 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:35 smithi008 ceph-mon[40743]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:35.510 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:35 smithi008 ceph-mon[40743]: 6.7 scrub starts 2024-02-10T07:55:35.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:35 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:35.635 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:35 smithi170 ceph-mon[33394]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:35.636 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:35 smithi170 ceph-mon[33394]: 6.7 scrub starts 2024-02-10T07:55:36.330 INFO:teuthology.orchestra.run.smithi008.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-02-10T07:55:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:36 smithi008 ceph-mon[29816]: 6.7 scrub ok 2024-02-10T07:55:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:36 smithi008 ceph-mon[29816]: 3.19 scrub starts 2024-02-10T07:55:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:36 smithi008 ceph-mon[29816]: 3.19 scrub ok 2024-02-10T07:55:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:36 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2177632978' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:55:36.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:36 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:55:36.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:36 smithi008 ceph-mon[40743]: 6.7 scrub ok 2024-02-10T07:55:36.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:36 smithi008 ceph-mon[40743]: 3.19 scrub starts 2024-02-10T07:55:36.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:36 smithi008 ceph-mon[40743]: 3.19 scrub ok 2024-02-10T07:55:36.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:36 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2177632978' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:55:36.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:36 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:55:36.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:36 smithi170 ceph-mon[33394]: 6.7 scrub ok 2024-02-10T07:55:36.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:36 smithi170 ceph-mon[33394]: 3.19 scrub starts 2024-02-10T07:55:36.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:36 smithi170 ceph-mon[33394]: 3.19 scrub ok 2024-02-10T07:55:36.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:36 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2177632978' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:55:36.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:36 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-10T07:55:36.949 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-02-10T07:55:36.949 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-02-10T07:55:36.949 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-02-10T07:55:36.949 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-02-10T07:55:36.949 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-10T07:55:36.949 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T07:55:37.207 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[29816]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:55:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:55:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[29816]: osdmap e495: 8 total, 8 up, 8 in 2024-02-10T07:55:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:55:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:55:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:55:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:55:37.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[29816]: osdmap e496: 8 total, 8 up, 8 in 2024-02-10T07:55:37.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[40743]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:55:37.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:55:37.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[40743]: osdmap e495: 8 total, 8 up, 8 in 2024-02-10T07:55:37.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:55:37.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:55:37.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:55:37.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:55:37.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:37 smithi008 ceph-mon[40743]: osdmap e496: 8 total, 8 up, 8 in 2024-02-10T07:55:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:37 smithi170 ceph-mon[33394]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:55:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:37 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-10T07:55:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:37 smithi170 ceph-mon[33394]: osdmap e495: 8 total, 8 up, 8 in 2024-02-10T07:55:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:37 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:55:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:37 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:55:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:37 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:55:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:37 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:55:37.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:37 smithi170 ceph-mon[33394]: osdmap e496: 8 total, 8 up, 8 in 2024-02-10T07:55:38.631 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:38 smithi008 ceph-mon[29816]: 6.16 scrub starts 2024-02-10T07:55:38.632 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:38 smithi008 ceph-mon[29816]: 6.16 scrub ok 2024-02-10T07:55:38.632 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:38 smithi008 ceph-mon[40743]: 6.16 scrub starts 2024-02-10T07:55:38.632 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:38 smithi008 ceph-mon[40743]: 6.16 scrub ok 2024-02-10T07:55:38.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:38 smithi170 ceph-mon[33394]: 6.16 scrub starts 2024-02-10T07:55:38.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:38 smithi170 ceph-mon[33394]: 6.16 scrub ok 2024-02-10T07:55:38.891 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:55:38 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:55:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:55:39.076 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:55:39.076 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":496,"fsid":"8c8244e6-c7e5-11ee-95b8-87774f69a715","created":"2024-02-10T07:26:21.316239+0000","modified":"2024-02-10T07:55:37.318014+0000","last_up_change":"2024-02-10T07:47:46.947584+0000","last_in_change":"2024-02-10T07:50:42.235603+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T07:29:32.704872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-10T07:31:49.150741+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-10T07:31:49.831683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-10T07:31:51.622544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-10T07:31:53.550827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-10T07:31:55.600117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"63abc4f1-1c83-4e03-8592-959c51d10847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":445,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6803","nonce":1442795455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6805","nonce":1442795455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6809","nonce":1442795455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6807","nonce":1442795455}]},"public_addr":"172.21.15.8:6803/1442795455","cluster_addr":"172.21.15.8:6805/1442795455","heartbeat_back_addr":"172.21.15.8:6809/1442795455","heartbeat_front_addr":"172.21.15.8:6807/1442795455","state":["exists","up"]},{"osd":1,"uuid":"88307389-bfa8-402e-92b9-9fa5094262c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":461,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6811","nonce":3278758893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6813","nonce":3278758893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6817","nonce":3278758893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6815","nonce":3278758893}]},"public_addr":"172.21.15.8:6811/3278758893","cluster_addr":"172.21.15.8:6813/3278758893","heartbeat_back_addr":"172.21.15.8:6817/3278758893","heartbeat_front_addr":"172.21.15.8:6815/3278758893","state":["exists","up"]},{"osd":2,"uuid":"c6120003-9ab5-4cf5-87ac-6351ed2c6226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":470,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6819","nonce":481314802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6821","nonce":481314802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6825","nonce":481314802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6823","nonce":481314802}]},"public_addr":"172.21.15.8:6819/481314802","cluster_addr":"172.21.15.8:6821/481314802","heartbeat_back_addr":"172.21.15.8:6825/481314802","heartbeat_front_addr":"172.21.15.8:6823/481314802","state":["exists","up"]},{"osd":3,"uuid":"6ca8f6f3-013a-48cb-9083-be37c1767489","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":295,"up_from":347,"up_thru":495,"down_at":296,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":4266121075},{"type":"v1","addr":"172.21.15.8:6827","nonce":4266121075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":4266121075},{"type":"v1","addr":"172.21.15.8:6829","nonce":4266121075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":4266121075},{"type":"v1","addr":"172.21.15.8:6833","nonce":4266121075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":4266121075},{"type":"v1","addr":"172.21.15.8:6831","nonce":4266121075}]},"public_addr":"172.21.15.8:6827/4266121075","cluster_addr":"172.21.15.8:6829/4266121075","heartbeat_back_addr":"172.21.15.8:6833/4266121075","heartbeat_front_addr":"172.21.15.8:6831/4266121075","state":["exists","up"]},{"osd":4,"uuid":"aa667766-e57f-447b-9a31-e3f3e61ac346","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":404,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6801","nonce":3997132504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6803","nonce":3997132504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6807","nonce":3997132504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6805","nonce":3997132504}]},"public_addr":"172.21.15.170:6801/3997132504","cluster_addr":"172.21.15.170:6803/3997132504","heartbeat_back_addr":"172.21.15.170:6807/3997132504","heartbeat_front_addr":"172.21.15.170:6805/3997132504","state":["exists","up"]},{"osd":5,"uuid":"2e9ab14d-1aff-4638-9f41-311f6b5f6770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":139,"up_from":156,"up_thru":492,"down_at":140,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":3225334381},{"type":"v1","addr":"172.21.15.170:6809","nonce":3225334381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":3225334381},{"type":"v1","addr":"172.21.15.170:6811","nonce":3225334381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":3225334381},{"type":"v1","addr":"172.21.15.170:6815","nonce":3225334381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":3225334381},{"type":"v1","addr":"172.21.15.170:6813","nonce":3225334381}]},"public_addr":"172.21.15.170:6809/3225334381","cluster_addr":"172.21.15.170:6811/3225334381","heartbeat_back_addr":"172.21.15.170:6815/3225334381","heartbeat_front_addr":"172.21.15.170:6813/3225334381","state":["exists","up"]},{"osd":6,"uuid":"61970c81-4b79-4743-a379-50d5d102b373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":404,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764436984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764436984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764436984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764436984}]},"public_addr":"172.21.15.170:6817/1764436984","cluster_addr":"172.21.15.170:6819/1764436984","heartbeat_back_addr":"172.21.15.170:6823/1764436984","heartbeat_front_addr":"172.21.15.170:6821/1764436984","state":["exists","up"]},{"osd":7,"uuid":"2e303441-b454-4750-8c5a-96c5348129a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":460,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6825","nonce":4048577717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6827","nonce":4048577717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6831","nonce":4048577717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6829","nonce":4048577717}]},"public_addr":"172.21.15.170:6825/4048577717","cluster_addr":"172.21.15.170:6827/4048577717","heartbeat_back_addr":"172.21.15.170:6831/4048577717","heartbeat_front_addr":"172.21.15.170:6829/4048577717","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:55:13.822897+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:54:37.536584+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:54:18.114397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:54:21.886325+0000","dead_epoch":295},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:55:09.891959+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:55:14.261045+0000","dead_epoch":139},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:55:00.742277+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:55:07.251538+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"6.0","osds":[6,0,4]}],"pg_upmap_items":[{"pgid":"4.0","mappings":[{"from":3,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":7}]},{"pgid":"4.d","mappings":[{"from":1,"to":0}]},{"pgid":"4.e","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":3,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]},{"pgid":"5.2","mappings":[{"from":0,"to":3}]},{"pgid":"5.e","mappings":[{"from":4,"to":6}]},{"pgid":"5.17","mappings":[{"from":1,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.8:0/2210906095":"2024-02-11T07:32:31.651879+0000","172.21.15.8:6800/1412156979":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/2515892714":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6801/4001702773":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6801/1412156979":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/1462056527":"2024-02-11T07:27:16.371197+0000","172.21.15.8:6800/4001702773":"2024-02-11T07:26:49.500037+0000","172.21.15.8:0/1905866501":"2024-02-11T07:26:49.500037+0000","172.21.15.8:0/2381650878":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/802075678":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6801/2677792951":"2024-02-11T07:32:31.651879+0000","172.21.15.8:0/3905205183":"2024-02-11T07:32:31.651879+0000","172.21.15.8:0/3314827874":"2024-02-11T07:32:31.651879+0000","172.21.15.8:0/1624888680":"2024-02-11T07:32:31.651879+0000","172.21.15.8:0/2789652913":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/4074987889":"2024-02-11T07:32:31.651879+0000","172.21.15.8:6800/2677792951":"2024-02-11T07:32:31.651879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T07:55:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:39 smithi170 ceph-mon[33394]: 4.18 scrub starts 2024-02-10T07:55:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:39 smithi170 ceph-mon[33394]: 4.18 scrub ok 2024-02-10T07:55:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:39 smithi170 ceph-mon[33394]: 5.a scrub starts 2024-02-10T07:55:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:39 smithi170 ceph-mon[33394]: 5.a scrub ok 2024-02-10T07:55:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:39 smithi170 ceph-mon[33394]: pgmap v1110: 132 pgs: 16 peering, 116 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail 2024-02-10T07:55:39.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:39 smithi170 ceph-mon[33394]: Health check failed: Reduced data availability: 13 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-02-10T07:55:39.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:39 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1728674443' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:55:39.763 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:39 smithi008 ceph-mon[29816]: 4.18 scrub starts 2024-02-10T07:55:39.764 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:39 smithi008 ceph-mon[29816]: 4.18 scrub ok 2024-02-10T07:55:39.764 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:39 smithi008 ceph-mon[29816]: 5.a scrub starts 2024-02-10T07:55:39.764 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:39 smithi008 ceph-mon[29816]: 5.a scrub ok 2024-02-10T07:55:39.764 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:39 smithi008 ceph-mon[29816]: pgmap v1110: 132 pgs: 16 peering, 116 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail 2024-02-10T07:55:39.764 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:39 smithi008 ceph-mon[29816]: Health check failed: Reduced data availability: 13 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-02-10T07:55:39.764 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:39 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1728674443' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:55:39.764 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:39 smithi008 ceph-mon[40743]: 4.18 scrub starts 2024-02-10T07:55:39.764 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:39 smithi008 ceph-mon[40743]: 4.18 scrub ok 2024-02-10T07:55:39.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:39 smithi008 ceph-mon[40743]: 5.a scrub starts 2024-02-10T07:55:39.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:39 smithi008 ceph-mon[40743]: 5.a scrub ok 2024-02-10T07:55:39.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:39 smithi008 ceph-mon[40743]: pgmap v1110: 132 pgs: 16 peering, 116 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail 2024-02-10T07:55:39.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:39 smithi008 ceph-mon[40743]: Health check failed: Reduced data availability: 13 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-02-10T07:55:39.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:39 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1728674443' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:55:39.791 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-10T07:55:39.791 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T07:55:40.037 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:40.467 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:40 smithi008 ceph-mon[29816]: 3.2 scrub starts 2024-02-10T07:55:40.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:40 smithi008 ceph-mon[29816]: 3.2 scrub ok 2024-02-10T07:55:40.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:40 smithi008 ceph-mon[29816]: 3.1f deep-scrub starts 2024-02-10T07:55:40.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:40 smithi008 ceph-mon[29816]: 3.1f deep-scrub ok 2024-02-10T07:55:40.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:40 smithi008 ceph-mon[29816]: 5.1b scrub starts 2024-02-10T07:55:40.468 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:40 smithi008 ceph-mon[29816]: 5.1b scrub ok 2024-02-10T07:55:40.468 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:40 smithi008 ceph-mon[40743]: 3.2 scrub starts 2024-02-10T07:55:40.468 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:40 smithi008 ceph-mon[40743]: 3.2 scrub ok 2024-02-10T07:55:40.468 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:40 smithi008 ceph-mon[40743]: 3.1f deep-scrub starts 2024-02-10T07:55:40.468 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:40 smithi008 ceph-mon[40743]: 3.1f deep-scrub ok 2024-02-10T07:55:40.468 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:40 smithi008 ceph-mon[40743]: 5.1b scrub starts 2024-02-10T07:55:40.468 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:40 smithi008 ceph-mon[40743]: 5.1b scrub ok 2024-02-10T07:55:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:40 smithi170 ceph-mon[33394]: 3.2 scrub starts 2024-02-10T07:55:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:40 smithi170 ceph-mon[33394]: 3.2 scrub ok 2024-02-10T07:55:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:40 smithi170 ceph-mon[33394]: 3.1f deep-scrub starts 2024-02-10T07:55:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:40 smithi170 ceph-mon[33394]: 3.1f deep-scrub ok 2024-02-10T07:55:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:40 smithi170 ceph-mon[33394]: 5.1b scrub starts 2024-02-10T07:55:40.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:40 smithi170 ceph-mon[33394]: 5.1b scrub ok 2024-02-10T07:55:41.446 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:41 smithi008 ceph-mon[29816]: pgmap v1111: 132 pgs: 16 peering, 116 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:55:41.446 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:41 smithi008 ceph-mon[29816]: 5.10 scrub starts 2024-02-10T07:55:41.446 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:41 smithi008 ceph-mon[29816]: 5.10 scrub ok 2024-02-10T07:55:41.446 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:41 smithi008 ceph-mon[40743]: pgmap v1111: 132 pgs: 16 peering, 116 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:55:41.446 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:41 smithi008 ceph-mon[40743]: 5.10 scrub starts 2024-02-10T07:55:41.446 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:41 smithi008 ceph-mon[40743]: 5.10 scrub ok 2024-02-10T07:55:41.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:41 smithi170 ceph-mon[33394]: pgmap v1111: 132 pgs: 16 peering, 116 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:55:41.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:41 smithi170 ceph-mon[33394]: 5.10 scrub starts 2024-02-10T07:55:41.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:41 smithi170 ceph-mon[33394]: 5.10 scrub ok 2024-02-10T07:55:41.886 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:55:41.886 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":496,"fsid":"8c8244e6-c7e5-11ee-95b8-87774f69a715","created":"2024-02-10T07:26:21.316239+0000","modified":"2024-02-10T07:55:37.318014+0000","last_up_change":"2024-02-10T07:47:46.947584+0000","last_in_change":"2024-02-10T07:50:42.235603+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T07:29:32.704872+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-10T07:31:49.150741+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-10T07:31:49.831683+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-10T07:31:51.622544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-10T07:31:53.550827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-10T07:31:55.600117+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"63abc4f1-1c83-4e03-8592-959c51d10847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":445,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6803","nonce":1442795455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6805","nonce":1442795455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6809","nonce":1442795455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":1442795455},{"type":"v1","addr":"172.21.15.8:6807","nonce":1442795455}]},"public_addr":"172.21.15.8:6803/1442795455","cluster_addr":"172.21.15.8:6805/1442795455","heartbeat_back_addr":"172.21.15.8:6809/1442795455","heartbeat_front_addr":"172.21.15.8:6807/1442795455","state":["exists","up"]},{"osd":1,"uuid":"88307389-bfa8-402e-92b9-9fa5094262c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":461,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6811","nonce":3278758893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6813","nonce":3278758893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6817","nonce":3278758893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":3278758893},{"type":"v1","addr":"172.21.15.8:6815","nonce":3278758893}]},"public_addr":"172.21.15.8:6811/3278758893","cluster_addr":"172.21.15.8:6813/3278758893","heartbeat_back_addr":"172.21.15.8:6817/3278758893","heartbeat_front_addr":"172.21.15.8:6815/3278758893","state":["exists","up"]},{"osd":2,"uuid":"c6120003-9ab5-4cf5-87ac-6351ed2c6226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":470,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6819","nonce":481314802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6820","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6821","nonce":481314802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6824","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6825","nonce":481314802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":481314802},{"type":"v1","addr":"172.21.15.8:6823","nonce":481314802}]},"public_addr":"172.21.15.8:6819/481314802","cluster_addr":"172.21.15.8:6821/481314802","heartbeat_back_addr":"172.21.15.8:6825/481314802","heartbeat_front_addr":"172.21.15.8:6823/481314802","state":["exists","up"]},{"osd":3,"uuid":"6ca8f6f3-013a-48cb-9083-be37c1767489","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":295,"up_from":347,"up_thru":495,"down_at":296,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":4266121075},{"type":"v1","addr":"172.21.15.8:6827","nonce":4266121075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6828","nonce":4266121075},{"type":"v1","addr":"172.21.15.8:6829","nonce":4266121075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":4266121075},{"type":"v1","addr":"172.21.15.8:6833","nonce":4266121075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":4266121075},{"type":"v1","addr":"172.21.15.8:6831","nonce":4266121075}]},"public_addr":"172.21.15.8:6827/4266121075","cluster_addr":"172.21.15.8:6829/4266121075","heartbeat_back_addr":"172.21.15.8:6833/4266121075","heartbeat_front_addr":"172.21.15.8:6831/4266121075","state":["exists","up"]},{"osd":4,"uuid":"aa667766-e57f-447b-9a31-e3f3e61ac346","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":404,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6801","nonce":3997132504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6803","nonce":3997132504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6807","nonce":3997132504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":3997132504},{"type":"v1","addr":"172.21.15.170:6805","nonce":3997132504}]},"public_addr":"172.21.15.170:6801/3997132504","cluster_addr":"172.21.15.170:6803/3997132504","heartbeat_back_addr":"172.21.15.170:6807/3997132504","heartbeat_front_addr":"172.21.15.170:6805/3997132504","state":["exists","up"]},{"osd":5,"uuid":"2e9ab14d-1aff-4638-9f41-311f6b5f6770","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":40,"last_clean_end":139,"up_from":156,"up_thru":492,"down_at":140,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":3225334381},{"type":"v1","addr":"172.21.15.170:6809","nonce":3225334381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":3225334381},{"type":"v1","addr":"172.21.15.170:6811","nonce":3225334381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":3225334381},{"type":"v1","addr":"172.21.15.170:6815","nonce":3225334381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":3225334381},{"type":"v1","addr":"172.21.15.170:6813","nonce":3225334381}]},"public_addr":"172.21.15.170:6809/3225334381","cluster_addr":"172.21.15.170:6811/3225334381","heartbeat_back_addr":"172.21.15.170:6815/3225334381","heartbeat_front_addr":"172.21.15.170:6813/3225334381","state":["exists","up"]},{"osd":6,"uuid":"61970c81-4b79-4743-a379-50d5d102b373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":404,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764436984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764436984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764436984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764436984},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764436984}]},"public_addr":"172.21.15.170:6817/1764436984","cluster_addr":"172.21.15.170:6819/1764436984","heartbeat_back_addr":"172.21.15.170:6823/1764436984","heartbeat_front_addr":"172.21.15.170:6821/1764436984","state":["exists","up"]},{"osd":7,"uuid":"2e303441-b454-4750-8c5a-96c5348129a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":460,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6825","nonce":4048577717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6827","nonce":4048577717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6831","nonce":4048577717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":4048577717},{"type":"v1","addr":"172.21.15.170:6829","nonce":4048577717}]},"public_addr":"172.21.15.170:6825/4048577717","cluster_addr":"172.21.15.170:6827/4048577717","heartbeat_back_addr":"172.21.15.170:6831/4048577717","heartbeat_front_addr":"172.21.15.170:6829/4048577717","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:55:13.822897+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:54:37.536584+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:54:18.114397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:54:21.886325+0000","dead_epoch":295},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:55:09.891959+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:55:14.261045+0000","dead_epoch":139},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:55:00.742277+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T07:55:07.251538+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"6.0","osds":[6,0,4]}],"pg_upmap_items":[{"pgid":"4.0","mappings":[{"from":3,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":7}]},{"pgid":"4.d","mappings":[{"from":1,"to":0}]},{"pgid":"4.e","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":3,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]},{"pgid":"5.2","mappings":[{"from":0,"to":3}]},{"pgid":"5.e","mappings":[{"from":4,"to":6}]},{"pgid":"5.17","mappings":[{"from":1,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.8:0/2210906095":"2024-02-11T07:32:31.651879+0000","172.21.15.8:6800/1412156979":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/2515892714":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6801/4001702773":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6801/1412156979":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/1462056527":"2024-02-11T07:27:16.371197+0000","172.21.15.8:6800/4001702773":"2024-02-11T07:26:49.500037+0000","172.21.15.8:0/1905866501":"2024-02-11T07:26:49.500037+0000","172.21.15.8:0/2381650878":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/802075678":"2024-02-11T07:26:49.500037+0000","172.21.15.8:6801/2677792951":"2024-02-11T07:32:31.651879+0000","172.21.15.8:0/3905205183":"2024-02-11T07:32:31.651879+0000","172.21.15.8:0/3314827874":"2024-02-11T07:32:31.651879+0000","172.21.15.8:0/1624888680":"2024-02-11T07:32:31.651879+0000","172.21.15.8:0/2789652913":"2024-02-11T07:27:16.371197+0000","172.21.15.8:0/4074987889":"2024-02-11T07:32:31.651879+0000","172.21.15.8:6800/2677792951":"2024-02-11T07:32:31.651879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T07:55:42.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:42 smithi008 ceph-mon[29816]: 3.7 scrub starts 2024-02-10T07:55:42.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:42 smithi008 ceph-mon[29816]: 3.7 scrub ok 2024-02-10T07:55:42.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:42 smithi008 ceph-mon[29816]: 6.5 scrub starts 2024-02-10T07:55:42.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:42 smithi008 ceph-mon[29816]: 6.5 scrub ok 2024-02-10T07:55:42.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:42 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1578140455' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:55:42.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:42 smithi008 ceph-mon[29816]: osdmap e497: 8 total, 8 up, 8 in 2024-02-10T07:55:42.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:42 smithi008 ceph-mon[40743]: 3.7 scrub starts 2024-02-10T07:55:42.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:42 smithi008 ceph-mon[40743]: 3.7 scrub ok 2024-02-10T07:55:42.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:42 smithi008 ceph-mon[40743]: 6.5 scrub starts 2024-02-10T07:55:42.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:42 smithi008 ceph-mon[40743]: 6.5 scrub ok 2024-02-10T07:55:42.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:42 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1578140455' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:55:42.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:42 smithi008 ceph-mon[40743]: osdmap e497: 8 total, 8 up, 8 in 2024-02-10T07:55:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:42 smithi170 ceph-mon[33394]: 3.7 scrub starts 2024-02-10T07:55:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:42 smithi170 ceph-mon[33394]: 3.7 scrub ok 2024-02-10T07:55:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:42 smithi170 ceph-mon[33394]: 6.5 scrub starts 2024-02-10T07:55:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:42 smithi170 ceph-mon[33394]: 6.5 scrub ok 2024-02-10T07:55:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:42 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1578140455' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T07:55:42.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:42 smithi170 ceph-mon[33394]: osdmap e497: 8 total, 8 up, 8 in 2024-02-10T07:55:42.930 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-10T07:55:42.931 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-10T07:55:42.931 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-10T07:55:42.931 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-10T07:55:42.932 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-10T07:55:42.932 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-10T07:55:42.933 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-10T07:55:42.933 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-10T07:55:43.483 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:43.483 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:43.484 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:43.484 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:43.485 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:43.486 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:43.487 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:43.492 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:55:43.632 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:43 smithi008 ceph-mon[29816]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:55:43.633 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:43 smithi008 ceph-mon[29816]: 3.17 deep-scrub starts 2024-02-10T07:55:43.633 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:43 smithi008 ceph-mon[29816]: 3.17 deep-scrub ok 2024-02-10T07:55:43.633 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:43 smithi008 ceph-mon[29816]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 16 pgs peering) 2024-02-10T07:55:43.633 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:43 smithi008 ceph-mon[29816]: Cluster is now healthy 2024-02-10T07:55:43.633 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:43 smithi008 ceph-mon[40743]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:55:43.633 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:43 smithi008 ceph-mon[40743]: 3.17 deep-scrub starts 2024-02-10T07:55:43.633 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:43 smithi008 ceph-mon[40743]: 3.17 deep-scrub ok 2024-02-10T07:55:43.633 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:43 smithi008 ceph-mon[40743]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 16 pgs peering) 2024-02-10T07:55:43.633 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:43 smithi008 ceph-mon[40743]: Cluster is now healthy 2024-02-10T07:55:43.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:43 smithi170 ceph-mon[33394]: pgmap v1113: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-02-10T07:55:43.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:43 smithi170 ceph-mon[33394]: 3.17 deep-scrub starts 2024-02-10T07:55:43.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:43 smithi170 ceph-mon[33394]: 3.17 deep-scrub ok 2024-02-10T07:55:43.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:43 smithi170 ceph-mon[33394]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 16 pgs peering) 2024-02-10T07:55:43.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:43 smithi170 ceph-mon[33394]: Cluster is now healthy 2024-02-10T07:55:44.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:55:44 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:55:44.383 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:44 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:55:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:44 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:55:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:44 smithi170 ceph-mon[33394]: 2.1 scrub starts 2024-02-10T07:55:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:44 smithi170 ceph-mon[33394]: 2.1 scrub ok 2024-02-10T07:55:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:44 smithi170 ceph-mon[33394]: 4.17 scrub starts 2024-02-10T07:55:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:44 smithi170 ceph-mon[33394]: 4.17 scrub ok 2024-02-10T07:55:44.384 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:44 smithi170 ceph-mon[33394]: osdmap e498: 8 total, 8 up, 8 in 2024-02-10T07:55:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:44 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:55:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:44 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:55:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:44 smithi008 ceph-mon[40743]: 2.1 scrub starts 2024-02-10T07:55:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:44 smithi008 ceph-mon[40743]: 2.1 scrub ok 2024-02-10T07:55:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:44 smithi008 ceph-mon[40743]: 4.17 scrub starts 2024-02-10T07:55:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:44 smithi008 ceph-mon[40743]: 4.17 scrub ok 2024-02-10T07:55:44.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:44 smithi008 ceph-mon[40743]: osdmap e498: 8 total, 8 up, 8 in 2024-02-10T07:55:44.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:44 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:55:44.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:44 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:55:44.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:44 smithi008 ceph-mon[29816]: 2.1 scrub starts 2024-02-10T07:55:44.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:44 smithi008 ceph-mon[29816]: 2.1 scrub ok 2024-02-10T07:55:44.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:44 smithi008 ceph-mon[29816]: 4.17 scrub starts 2024-02-10T07:55:44.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:44 smithi008 ceph-mon[29816]: 4.17 scrub ok 2024-02-10T07:55:44.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:44 smithi008 ceph-mon[29816]: osdmap e498: 8 total, 8 up, 8 in 2024-02-10T07:55:45.625 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:45 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:45.625 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:45 smithi008 ceph-mon[29816]: 5.2 scrub starts 2024-02-10T07:55:45.625 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:45 smithi008 ceph-mon[29816]: 5.2 scrub ok 2024-02-10T07:55:45.625 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:45 smithi008 ceph-mon[29816]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:55:45.625 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:45 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:45.625 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:45 smithi008 ceph-mon[40743]: 5.2 scrub starts 2024-02-10T07:55:45.625 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:45 smithi008 ceph-mon[40743]: 5.2 scrub ok 2024-02-10T07:55:45.625 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:45 smithi008 ceph-mon[40743]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:55:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:45 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:45 smithi170 ceph-mon[33394]: 5.2 scrub starts 2024-02-10T07:55:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:45 smithi170 ceph-mon[33394]: 5.2 scrub ok 2024-02-10T07:55:45.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:45 smithi170 ceph-mon[33394]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:55:47.397 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:47 smithi008 ceph-mon[29816]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:47.397 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:47 smithi008 ceph-mon[29816]: osdmap e499: 8 total, 8 up, 8 in 2024-02-10T07:55:47.397 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:47 smithi008 ceph-mon[40743]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:47.397 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:47 smithi008 ceph-mon[40743]: osdmap e499: 8 total, 8 up, 8 in 2024-02-10T07:55:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:47 smithi170 ceph-mon[33394]: pgmap v1116: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:47.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:47 smithi170 ceph-mon[33394]: osdmap e499: 8 total, 8 up, 8 in 2024-02-10T07:55:49.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:55:48 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:55:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:55:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:49 smithi170 ceph-mon[33394]: 4.c scrub starts 2024-02-10T07:55:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:49 smithi170 ceph-mon[33394]: 4.c scrub ok 2024-02-10T07:55:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:49 smithi170 ceph-mon[33394]: 5.1f scrub starts 2024-02-10T07:55:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:49 smithi170 ceph-mon[33394]: 5.1f scrub ok 2024-02-10T07:55:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:49 smithi170 ceph-mon[33394]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-10T07:55:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:49 smithi170 ceph-mon[33394]: 5.1e deep-scrub starts 2024-02-10T07:55:49.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:49 smithi170 ceph-mon[33394]: 5.1e deep-scrub ok 2024-02-10T07:55:49.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:49 smithi170 ceph-mon[33394]: 6.1c scrub starts 2024-02-10T07:55:49.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:49 smithi170 ceph-mon[33394]: 6.1c scrub ok 2024-02-10T07:55:49.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:49 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:55:49.713 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[29816]: 4.c scrub starts 2024-02-10T07:55:49.713 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[29816]: 4.c scrub ok 2024-02-10T07:55:49.713 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[29816]: 5.1f scrub starts 2024-02-10T07:55:49.713 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[29816]: 5.1f scrub ok 2024-02-10T07:55:49.713 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[29816]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-10T07:55:49.713 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[29816]: 5.1e deep-scrub starts 2024-02-10T07:55:49.713 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[29816]: 5.1e deep-scrub ok 2024-02-10T07:55:49.713 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[29816]: 6.1c scrub starts 2024-02-10T07:55:49.713 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[29816]: 6.1c scrub ok 2024-02-10T07:55:49.713 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:55:49.714 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[40743]: 4.c scrub starts 2024-02-10T07:55:49.714 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[40743]: 4.c scrub ok 2024-02-10T07:55:49.714 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[40743]: 5.1f scrub starts 2024-02-10T07:55:49.714 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[40743]: 5.1f scrub ok 2024-02-10T07:55:49.714 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[40743]: pgmap v1118: 132 pgs: 132 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-10T07:55:49.714 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[40743]: 5.1e deep-scrub starts 2024-02-10T07:55:49.714 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[40743]: 5.1e deep-scrub ok 2024-02-10T07:55:49.714 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[40743]: 6.1c scrub starts 2024-02-10T07:55:49.714 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[40743]: 6.1c scrub ok 2024-02-10T07:55:49.714 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:49 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:55:50.616 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:50 smithi008 ceph-mon[29816]: 4.8 scrub starts 2024-02-10T07:55:50.616 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:50 smithi008 ceph-mon[29816]: 4.8 scrub ok 2024-02-10T07:55:50.616 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:50 smithi008 ceph-mon[29816]: 1.0 scrub starts 2024-02-10T07:55:50.616 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:50 smithi008 ceph-mon[29816]: 1.0 scrub ok 2024-02-10T07:55:50.616 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:50 smithi008 ceph-mon[40743]: 4.8 scrub starts 2024-02-10T07:55:50.616 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:50 smithi008 ceph-mon[40743]: 4.8 scrub ok 2024-02-10T07:55:50.616 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:50 smithi008 ceph-mon[40743]: 1.0 scrub starts 2024-02-10T07:55:50.617 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:50 smithi008 ceph-mon[40743]: 1.0 scrub ok 2024-02-10T07:55:50.617 INFO:teuthology.orchestra.run.smithi008.stdout:55834575171 2024-02-10T07:55:50.618 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-10T07:55:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:50 smithi170 ceph-mon[33394]: 4.8 scrub starts 2024-02-10T07:55:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:50 smithi170 ceph-mon[33394]: 4.8 scrub ok 2024-02-10T07:55:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:50 smithi170 ceph-mon[33394]: 1.0 scrub starts 2024-02-10T07:55:50.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:50 smithi170 ceph-mon[33394]: 1.0 scrub ok 2024-02-10T07:55:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:51 smithi170 ceph-mon[33394]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:51 smithi170 ceph-mon[33394]: 6.3 scrub starts 2024-02-10T07:55:51.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:51 smithi170 ceph-mon[33394]: 6.3 scrub ok 2024-02-10T07:55:51.646 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:51 smithi008 ceph-mon[29816]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:51.646 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:51 smithi008 ceph-mon[29816]: 6.3 scrub starts 2024-02-10T07:55:51.646 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:51 smithi008 ceph-mon[29816]: 6.3 scrub ok 2024-02-10T07:55:51.646 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:51 smithi008 ceph-mon[40743]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:51.646 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:51 smithi008 ceph-mon[40743]: 6.3 scrub starts 2024-02-10T07:55:51.646 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:51 smithi008 ceph-mon[40743]: 6.3 scrub ok 2024-02-10T07:55:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:52 smithi008 ceph-mon[29816]: 5.b deep-scrub starts 2024-02-10T07:55:52.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:52 smithi008 ceph-mon[29816]: 5.b deep-scrub ok 2024-02-10T07:55:52.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:52 smithi008 ceph-mon[40743]: 5.b deep-scrub starts 2024-02-10T07:55:52.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:52 smithi008 ceph-mon[40743]: 5.b deep-scrub ok 2024-02-10T07:55:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:52 smithi170 ceph-mon[33394]: 5.b deep-scrub starts 2024-02-10T07:55:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:52 smithi170 ceph-mon[33394]: 5.b deep-scrub ok 2024-02-10T07:55:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:53 smithi170 ceph-mon[33394]: 4.1f deep-scrub starts 2024-02-10T07:55:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:53 smithi170 ceph-mon[33394]: 4.1f deep-scrub ok 2024-02-10T07:55:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:53 smithi170 ceph-mon[33394]: 4.d scrub starts 2024-02-10T07:55:53.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:53 smithi170 ceph-mon[33394]: 4.d scrub ok 2024-02-10T07:55:53.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:53 smithi170 ceph-mon[33394]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:55:53.791 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:53 smithi008 ceph-mon[29816]: 4.1f deep-scrub starts 2024-02-10T07:55:53.791 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:53 smithi008 ceph-mon[29816]: 4.1f deep-scrub ok 2024-02-10T07:55:53.791 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:53 smithi008 ceph-mon[29816]: 4.d scrub starts 2024-02-10T07:55:53.791 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:53 smithi008 ceph-mon[29816]: 4.d scrub ok 2024-02-10T07:55:53.792 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:53 smithi008 ceph-mon[29816]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:55:53.792 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:53 smithi008 ceph-mon[40743]: 4.1f deep-scrub starts 2024-02-10T07:55:53.792 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:53 smithi008 ceph-mon[40743]: 4.1f deep-scrub ok 2024-02-10T07:55:53.792 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:53 smithi008 ceph-mon[40743]: 4.d scrub starts 2024-02-10T07:55:53.792 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:53 smithi008 ceph-mon[40743]: 4.d scrub ok 2024-02-10T07:55:53.792 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:53 smithi008 ceph-mon[40743]: pgmap v1120: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:55:54.437 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:54 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:55:54.437 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:54 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:55:54.437 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:54 smithi008 ceph-mon[29816]: 4.a scrub starts 2024-02-10T07:55:54.438 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:54 smithi008 ceph-mon[29816]: 4.a scrub ok 2024-02-10T07:55:54.438 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:54 smithi008 ceph-mon[29816]: osdmap e500: 8 total, 8 up, 8 in 2024-02-10T07:55:54.438 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:54 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:55:54.438 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:54 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:55:54.438 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:54 smithi008 ceph-mon[40743]: 4.a scrub starts 2024-02-10T07:55:54.438 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:54 smithi008 ceph-mon[40743]: 4.a scrub ok 2024-02-10T07:55:54.438 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:54 smithi008 ceph-mon[40743]: osdmap e500: 8 total, 8 up, 8 in 2024-02-10T07:55:54.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:55:54 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:55:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:54 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:55:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:54 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:55:54.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:54 smithi170 ceph-mon[33394]: 4.a scrub starts 2024-02-10T07:55:54.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:54 smithi170 ceph-mon[33394]: 4.a scrub ok 2024-02-10T07:55:54.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:54 smithi170 ceph-mon[33394]: osdmap e500: 8 total, 8 up, 8 in 2024-02-10T07:55:54.874 INFO:teuthology.orchestra.run.smithi008.stdout:137438953780 2024-02-10T07:55:54.874 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-10T07:55:55.382 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:55 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:55.382 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:55 smithi008 ceph-mon[29816]: 5.1 scrub starts 2024-02-10T07:55:55.382 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:55 smithi008 ceph-mon[29816]: 5.1 scrub ok 2024-02-10T07:55:55.382 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:55 smithi008 ceph-mon[29816]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:55.383 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:55 smithi008 ceph-mon[29816]: 6.9 deep-scrub starts 2024-02-10T07:55:55.383 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:55 smithi008 ceph-mon[29816]: 6.9 deep-scrub ok 2024-02-10T07:55:55.383 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:55 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:55.383 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:55 smithi008 ceph-mon[40743]: 5.1 scrub starts 2024-02-10T07:55:55.383 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:55 smithi008 ceph-mon[40743]: 5.1 scrub ok 2024-02-10T07:55:55.383 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:55 smithi008 ceph-mon[40743]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:55.383 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:55 smithi008 ceph-mon[40743]: 6.9 deep-scrub starts 2024-02-10T07:55:55.383 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:55 smithi008 ceph-mon[40743]: 6.9 deep-scrub ok 2024-02-10T07:55:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:55 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:55:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:55 smithi170 ceph-mon[33394]: 5.1 scrub starts 2024-02-10T07:55:55.633 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:55 smithi170 ceph-mon[33394]: 5.1 scrub ok 2024-02-10T07:55:55.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:55 smithi170 ceph-mon[33394]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T07:55:55.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:55 smithi170 ceph-mon[33394]: 6.9 deep-scrub starts 2024-02-10T07:55:55.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:55 smithi170 ceph-mon[33394]: 6.9 deep-scrub ok 2024-02-10T07:55:56.432 INFO:teuthology.orchestra.run.smithi008.stdout:77309411647 2024-02-10T07:55:56.432 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-10T07:55:56.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:56 smithi008 ceph-mon[29816]: 5.15 scrub starts 2024-02-10T07:55:56.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:56 smithi008 ceph-mon[29816]: 5.15 scrub ok 2024-02-10T07:55:56.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:56 smithi008 ceph-mon[29816]: 4.1a scrub starts 2024-02-10T07:55:56.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:56 smithi008 ceph-mon[29816]: 4.1a scrub ok 2024-02-10T07:55:56.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:56 smithi008 ceph-mon[29816]: 3.e scrub starts 2024-02-10T07:55:56.524 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:56 smithi008 ceph-mon[29816]: 3.e scrub ok 2024-02-10T07:55:56.525 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:56 smithi008 ceph-mon[40743]: 5.15 scrub starts 2024-02-10T07:55:56.525 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:56 smithi008 ceph-mon[40743]: 5.15 scrub ok 2024-02-10T07:55:56.525 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:56 smithi008 ceph-mon[40743]: 4.1a scrub starts 2024-02-10T07:55:56.525 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:56 smithi008 ceph-mon[40743]: 4.1a scrub ok 2024-02-10T07:55:56.525 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:56 smithi008 ceph-mon[40743]: 3.e scrub starts 2024-02-10T07:55:56.525 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:56 smithi008 ceph-mon[40743]: 3.e scrub ok 2024-02-10T07:55:56.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:56 smithi170 ceph-mon[33394]: 5.15 scrub starts 2024-02-10T07:55:56.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:56 smithi170 ceph-mon[33394]: 5.15 scrub ok 2024-02-10T07:55:56.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:56 smithi170 ceph-mon[33394]: 4.1a scrub starts 2024-02-10T07:55:56.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:56 smithi170 ceph-mon[33394]: 4.1a scrub ok 2024-02-10T07:55:56.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:56 smithi170 ceph-mon[33394]: 3.e scrub starts 2024-02-10T07:55:56.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:56 smithi170 ceph-mon[33394]: 3.e scrub ok 2024-02-10T07:55:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:57 smithi008 ceph-mon[29816]: 6.1f scrub starts 2024-02-10T07:55:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:57 smithi008 ceph-mon[29816]: pgmap v1123: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:55:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:57 smithi008 ceph-mon[29816]: 6.1f scrub ok 2024-02-10T07:55:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:57 smithi008 ceph-mon[29816]: 6.12 scrub starts 2024-02-10T07:55:57.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:57 smithi008 ceph-mon[29816]: 6.12 scrub ok 2024-02-10T07:55:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:57 smithi008 ceph-mon[40743]: 6.1f scrub starts 2024-02-10T07:55:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:57 smithi008 ceph-mon[40743]: pgmap v1123: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:55:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:57 smithi008 ceph-mon[40743]: 6.1f scrub ok 2024-02-10T07:55:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:57 smithi008 ceph-mon[40743]: 6.12 scrub starts 2024-02-10T07:55:57.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:57 smithi008 ceph-mon[40743]: 6.12 scrub ok 2024-02-10T07:55:57.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:57 smithi170 ceph-mon[33394]: 6.1f scrub starts 2024-02-10T07:55:57.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:57 smithi170 ceph-mon[33394]: pgmap v1123: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:55:57.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:57 smithi170 ceph-mon[33394]: 6.1f scrub ok 2024-02-10T07:55:57.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:57 smithi170 ceph-mon[33394]: 6.12 scrub starts 2024-02-10T07:55:57.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:57 smithi170 ceph-mon[33394]: 6.12 scrub ok 2024-02-10T07:55:57.909 INFO:teuthology.orchestra.run.smithi008.stdout:34359738697 2024-02-10T07:55:57.910 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-10T07:55:58.290 INFO:teuthology.orchestra.run.smithi008.stdout:1490353651812 2024-02-10T07:55:58.290 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-10T07:55:58.420 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:58 smithi008 ceph-mon[29816]: 3.16 scrub starts 2024-02-10T07:55:58.420 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:58 smithi008 ceph-mon[29816]: 3.16 scrub ok 2024-02-10T07:55:58.420 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:58 smithi008 ceph-mon[40743]: 3.16 scrub starts 2024-02-10T07:55:58.420 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:58 smithi008 ceph-mon[40743]: 3.16 scrub ok 2024-02-10T07:55:58.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:58 smithi170 ceph-mon[33394]: 3.16 scrub starts 2024-02-10T07:55:58.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:58 smithi170 ceph-mon[33394]: 3.16 scrub ok 2024-02-10T07:55:59.081 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:55:58 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:55:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:55:59.406 INFO:teuthology.orchestra.run.smithi008.stdout:670014898411 2024-02-10T07:55:59.406 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-10T07:55:59.431 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:59 smithi008 ceph-mon[29816]: 3.1e scrub starts 2024-02-10T07:55:59.432 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:59 smithi008 ceph-mon[29816]: 3.1e scrub ok 2024-02-10T07:55:59.432 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:59 smithi008 ceph-mon[29816]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:55:59.432 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:59 smithi008 ceph-mon[29816]: 5.11 scrub starts 2024-02-10T07:55:59.432 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:55:59 smithi008 ceph-mon[29816]: 5.11 scrub ok 2024-02-10T07:55:59.432 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:59 smithi008 ceph-mon[40743]: 3.1e scrub starts 2024-02-10T07:55:59.432 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:59 smithi008 ceph-mon[40743]: 3.1e scrub ok 2024-02-10T07:55:59.432 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:59 smithi008 ceph-mon[40743]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:55:59.432 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:59 smithi008 ceph-mon[40743]: 5.11 scrub starts 2024-02-10T07:55:59.432 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:55:59 smithi008 ceph-mon[40743]: 5.11 scrub ok 2024-02-10T07:55:59.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:59 smithi170 ceph-mon[33394]: 3.1e scrub starts 2024-02-10T07:55:59.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:59 smithi170 ceph-mon[33394]: 3.1e scrub ok 2024-02-10T07:55:59.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:59 smithi170 ceph-mon[33394]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-10T07:55:59.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:59 smithi170 ceph-mon[33394]: 5.11 scrub starts 2024-02-10T07:55:59.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:55:59 smithi170 ceph-mon[33394]: 5.11 scrub ok 2024-02-10T07:55:59.998 INFO:teuthology.orchestra.run.smithi008.stdout:214748365100 2024-02-10T07:55:59.999 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-10T07:56:00.463 INFO:teuthology.orchestra.run.smithi008.stdout:257698038054 2024-02-10T07:56:00.463 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-10T07:56:00.476 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:56:00.485 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:56:00.496 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:56:00.536 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:56:00.738 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[29816]: 2.2 scrub starts 2024-02-10T07:56:00.738 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[29816]: 2.2 scrub ok 2024-02-10T07:56:00.738 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[29816]: 4.13 scrub starts 2024-02-10T07:56:00.739 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[29816]: 4.13 scrub ok 2024-02-10T07:56:00.739 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[29816]: 4.1e scrub starts 2024-02-10T07:56:00.739 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[29816]: 4.1e scrub ok 2024-02-10T07:56:00.739 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[29816]: 3.10 scrub starts 2024-02-10T07:56:00.739 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[29816]: 3.10 scrub ok 2024-02-10T07:56:00.739 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[29816]: 4.7 scrub starts 2024-02-10T07:56:00.739 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[29816]: 4.7 scrub ok 2024-02-10T07:56:00.739 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[29816]: 5.19 scrub starts 2024-02-10T07:56:00.739 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[29816]: 5.19 scrub ok 2024-02-10T07:56:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[40743]: 2.2 scrub starts 2024-02-10T07:56:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[40743]: 2.2 scrub ok 2024-02-10T07:56:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[40743]: 4.13 scrub starts 2024-02-10T07:56:00.739 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[40743]: 4.13 scrub ok 2024-02-10T07:56:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[40743]: 4.1e scrub starts 2024-02-10T07:56:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[40743]: 4.1e scrub ok 2024-02-10T07:56:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[40743]: 3.10 scrub starts 2024-02-10T07:56:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[40743]: 3.10 scrub ok 2024-02-10T07:56:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[40743]: 4.7 scrub starts 2024-02-10T07:56:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[40743]: 4.7 scrub ok 2024-02-10T07:56:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[40743]: 5.19 scrub starts 2024-02-10T07:56:00.740 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:00 smithi008 ceph-mon[40743]: 5.19 scrub ok 2024-02-10T07:56:00.784 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:00 smithi170 ceph-mon[33394]: 2.2 scrub starts 2024-02-10T07:56:00.784 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:00 smithi170 ceph-mon[33394]: 2.2 scrub ok 2024-02-10T07:56:00.785 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:00 smithi170 ceph-mon[33394]: 4.13 scrub starts 2024-02-10T07:56:00.785 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:00 smithi170 ceph-mon[33394]: 4.13 scrub ok 2024-02-10T07:56:00.785 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:00 smithi170 ceph-mon[33394]: 4.1e scrub starts 2024-02-10T07:56:00.785 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:00 smithi170 ceph-mon[33394]: 4.1e scrub ok 2024-02-10T07:56:00.785 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:00 smithi170 ceph-mon[33394]: 3.10 scrub starts 2024-02-10T07:56:00.785 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:00 smithi170 ceph-mon[33394]: 3.10 scrub ok 2024-02-10T07:56:00.785 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:00 smithi170 ceph-mon[33394]: 4.7 scrub starts 2024-02-10T07:56:00.785 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:00 smithi170 ceph-mon[33394]: 4.7 scrub ok 2024-02-10T07:56:00.785 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:00 smithi170 ceph-mon[33394]: 5.19 scrub starts 2024-02-10T07:56:00.785 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:00 smithi170 ceph-mon[33394]: 5.19 scrub ok 2024-02-10T07:56:01.777 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:56:01.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:01 smithi008 ceph-mon[29816]: 4.14 deep-scrub starts 2024-02-10T07:56:01.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:01 smithi008 ceph-mon[29816]: 4.14 deep-scrub ok 2024-02-10T07:56:01.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:01 smithi008 ceph-mon[29816]: pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:01.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:01 smithi008 ceph-mon[40743]: 4.14 deep-scrub starts 2024-02-10T07:56:01.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:01 smithi008 ceph-mon[40743]: 4.14 deep-scrub ok 2024-02-10T07:56:01.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:01 smithi008 ceph-mon[40743]: pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:01.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:01 smithi170 ceph-mon[33394]: 4.14 deep-scrub starts 2024-02-10T07:56:01.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:01 smithi170 ceph-mon[33394]: 4.14 deep-scrub ok 2024-02-10T07:56:01.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:01 smithi170 ceph-mon[33394]: pgmap v1125: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:02 smithi008 ceph-mon[29816]: 4.2 scrub starts 2024-02-10T07:56:02.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:02 smithi008 ceph-mon[29816]: 4.2 scrub ok 2024-02-10T07:56:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:02 smithi008 ceph-mon[40743]: 4.2 scrub starts 2024-02-10T07:56:02.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:02 smithi008 ceph-mon[40743]: 4.2 scrub ok 2024-02-10T07:56:02.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:02 smithi170 ceph-mon[33394]: 4.2 scrub starts 2024-02-10T07:56:02.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:02 smithi170 ceph-mon[33394]: 4.2 scrub ok 2024-02-10T07:56:03.443 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:03 smithi008 ceph-mon[29816]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:03.444 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:03 smithi008 ceph-mon[40743]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:03.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:03 smithi170 ceph-mon[33394]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:04.411 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:56:04 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:56:04.534 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:04 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:56:04.535 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:04 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:56:04.535 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:04 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:56:04.535 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:04 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:56:04.535 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:04 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:56:04.535 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:04 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:56:04.825 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:56:04.826 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:56:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:04 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:56:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:04 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:56:04.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:04 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:56:05.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:05 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:56:05.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:05 smithi008 ceph-mon[29816]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:56:05.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:05 smithi008 ceph-mon[29816]: osdmap e501: 8 total, 8 up, 8 in 2024-02-10T07:56:05.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:05 smithi008 ceph-mon[29816]: 3.4 scrub starts 2024-02-10T07:56:05.474 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:05 smithi008 ceph-mon[29816]: 3.4 scrub ok 2024-02-10T07:56:05.475 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:05 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:56:05.475 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:05 smithi008 ceph-mon[40743]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:56:05.475 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:05 smithi008 ceph-mon[40743]: osdmap e501: 8 total, 8 up, 8 in 2024-02-10T07:56:05.475 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:05 smithi008 ceph-mon[40743]: 3.4 scrub starts 2024-02-10T07:56:05.475 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:05 smithi008 ceph-mon[40743]: 3.4 scrub ok 2024-02-10T07:56:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:05 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:56:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:05 smithi170 ceph-mon[33394]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-10T07:56:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:05 smithi170 ceph-mon[33394]: osdmap e501: 8 total, 8 up, 8 in 2024-02-10T07:56:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:05 smithi170 ceph-mon[33394]: 3.4 scrub starts 2024-02-10T07:56:05.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:05 smithi170 ceph-mon[33394]: 3.4 scrub ok 2024-02-10T07:56:06.657 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:06 smithi008 ceph-mon[29816]: 5.c scrub starts 2024-02-10T07:56:06.658 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:06 smithi008 ceph-mon[29816]: 5.c scrub ok 2024-02-10T07:56:06.658 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:06 smithi008 ceph-mon[40743]: 5.c scrub starts 2024-02-10T07:56:06.658 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:06 smithi008 ceph-mon[40743]: 5.c scrub ok 2024-02-10T07:56:06.772 INFO:teuthology.orchestra.run.smithi008.stdout:34359738700 2024-02-10T07:56:06.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:06 smithi170 ceph-mon[33394]: 5.c scrub starts 2024-02-10T07:56:06.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:06 smithi170 ceph-mon[33394]: 5.c scrub ok 2024-02-10T07:56:07.307 INFO:teuthology.orchestra.run.smithi008.stdout:670014898412 2024-02-10T07:56:07.717 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:07 smithi008 ceph-mon[29816]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:07.717 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:07 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2204639153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T07:56:07.717 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:07 smithi008 ceph-mon[29816]: 3.0 deep-scrub starts 2024-02-10T07:56:07.717 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:07 smithi008 ceph-mon[29816]: 3.0 deep-scrub ok 2024-02-10T07:56:07.717 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:07 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/1551055775' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T07:56:07.717 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:07 smithi008 ceph-mon[40743]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:07.717 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:07 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2204639153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T07:56:07.717 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:07 smithi008 ceph-mon[40743]: 3.0 deep-scrub starts 2024-02-10T07:56:07.718 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:07 smithi008 ceph-mon[40743]: 3.0 deep-scrub ok 2024-02-10T07:56:07.718 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:07 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/1551055775' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T07:56:07.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:07 smithi170 ceph-mon[33394]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:07.932 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:07 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2204639153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T07:56:07.932 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:07 smithi170 ceph-mon[33394]: 3.0 deep-scrub starts 2024-02-10T07:56:07.932 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:07 smithi170 ceph-mon[33394]: 3.0 deep-scrub ok 2024-02-10T07:56:07.932 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:07 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/1551055775' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T07:56:08.142 INFO:teuthology.orchestra.run.smithi008.stdout:214748365101 2024-02-10T07:56:08.326 INFO:teuthology.orchestra.run.smithi008.stdout:77309411649 2024-02-10T07:56:08.581 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:08 smithi008 ceph-mon[29816]: 5.7 scrub starts 2024-02-10T07:56:08.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:08 smithi008 ceph-mon[29816]: 5.7 scrub ok 2024-02-10T07:56:08.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:08 smithi008 ceph-mon[29816]: 3.f deep-scrub starts 2024-02-10T07:56:08.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:08 smithi008 ceph-mon[29816]: 3.f deep-scrub ok 2024-02-10T07:56:08.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:08 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2398485889' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T07:56:08.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:08 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/3977730257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T07:56:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:08 smithi008 ceph-mon[40743]: 5.7 scrub starts 2024-02-10T07:56:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:08 smithi008 ceph-mon[40743]: 5.7 scrub ok 2024-02-10T07:56:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:08 smithi008 ceph-mon[40743]: 3.f deep-scrub starts 2024-02-10T07:56:08.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:08 smithi008 ceph-mon[40743]: 3.f deep-scrub ok 2024-02-10T07:56:08.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:08 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2398485889' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T07:56:08.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:08 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/3977730257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T07:56:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:08 smithi170 ceph-mon[33394]: 5.7 scrub starts 2024-02-10T07:56:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:08 smithi170 ceph-mon[33394]: 5.7 scrub ok 2024-02-10T07:56:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:08 smithi170 ceph-mon[33394]: 3.f deep-scrub starts 2024-02-10T07:56:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:08 smithi170 ceph-mon[33394]: 3.f deep-scrub ok 2024-02-10T07:56:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:08 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2398485889' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T07:56:08.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:08 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/3977730257' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T07:56:09.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:08 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:56:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:56:09.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:09 smithi008 ceph-mon[29816]: 6.d scrub starts 2024-02-10T07:56:09.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:09 smithi008 ceph-mon[29816]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-02-10T07:56:09.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:09 smithi008 ceph-mon[29816]: 6.d scrub ok 2024-02-10T07:56:09.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:09 smithi008 ceph-mon[40743]: 6.d scrub starts 2024-02-10T07:56:09.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:09 smithi008 ceph-mon[40743]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-02-10T07:56:09.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:09 smithi008 ceph-mon[40743]: 6.d scrub ok 2024-02-10T07:56:09.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:09 smithi170 ceph-mon[33394]: 6.d scrub starts 2024-02-10T07:56:09.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:09 smithi170 ceph-mon[33394]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-02-10T07:56:09.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:09 smithi170 ceph-mon[33394]: 6.d scrub ok 2024-02-10T07:56:09.989 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411647 got 77309411649 for osd.2 2024-02-10T07:56:09.989 DEBUG:teuthology.parallel:result is None 2024-02-10T07:56:10.493 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:10 smithi008 ceph-mon[29816]: 4.5 scrub starts 2024-02-10T07:56:10.493 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:10 smithi008 ceph-mon[29816]: 4.5 scrub ok 2024-02-10T07:56:10.493 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:10 smithi008 ceph-mon[29816]: 5.1a scrub starts 2024-02-10T07:56:10.493 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:10 smithi008 ceph-mon[29816]: 5.1a scrub ok 2024-02-10T07:56:10.493 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:10 smithi008 ceph-mon[29816]: 5.8 scrub starts 2024-02-10T07:56:10.493 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:10 smithi008 ceph-mon[29816]: 5.8 scrub ok 2024-02-10T07:56:10.493 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:10 smithi008 ceph-mon[40743]: 4.5 scrub starts 2024-02-10T07:56:10.494 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:10 smithi008 ceph-mon[40743]: 4.5 scrub ok 2024-02-10T07:56:10.494 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:10 smithi008 ceph-mon[40743]: 5.1a scrub starts 2024-02-10T07:56:10.494 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:10 smithi008 ceph-mon[40743]: 5.1a scrub ok 2024-02-10T07:56:10.494 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:10 smithi008 ceph-mon[40743]: 5.8 scrub starts 2024-02-10T07:56:10.494 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:10 smithi008 ceph-mon[40743]: 5.8 scrub ok 2024-02-10T07:56:10.519 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738697 got 34359738700 for osd.0 2024-02-10T07:56:10.519 DEBUG:teuthology.parallel:result is None 2024-02-10T07:56:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:10 smithi170 ceph-mon[33394]: 4.5 scrub starts 2024-02-10T07:56:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:10 smithi170 ceph-mon[33394]: 4.5 scrub ok 2024-02-10T07:56:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:10 smithi170 ceph-mon[33394]: 5.1a scrub starts 2024-02-10T07:56:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:10 smithi170 ceph-mon[33394]: 5.1a scrub ok 2024-02-10T07:56:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:10 smithi170 ceph-mon[33394]: 5.8 scrub starts 2024-02-10T07:56:10.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:10 smithi170 ceph-mon[33394]: 5.8 scrub ok 2024-02-10T07:56:11.191 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:56:11.398 INFO:teuthology.orchestra.run.smithi008.stdout:137438953784 2024-02-10T07:56:11.681 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:11 smithi008 ceph-mon[29816]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:11.681 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:11 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/803124922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T07:56:11.682 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:11 smithi008 ceph-mon[40743]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:11.682 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:11 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/803124922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T07:56:11.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:11 smithi170 ceph-mon[33394]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:11.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:11 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/803124922' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T07:56:12.718 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:56:12.718 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:56:12.718 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[29816]: 5.16 scrub starts 2024-02-10T07:56:12.718 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[29816]: 5.16 scrub ok 2024-02-10T07:56:12.718 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[29816]: 3.12 deep-scrub starts 2024-02-10T07:56:12.718 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[29816]: 3.12 deep-scrub ok 2024-02-10T07:56:12.719 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[29816]: osdmap e502: 8 total, 8 up, 8 in 2024-02-10T07:56:12.719 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[29816]: 4.1c scrub starts 2024-02-10T07:56:12.719 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[29816]: 4.1c scrub ok 2024-02-10T07:56:12.719 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:56:12.719 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:56:12.719 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[40743]: 5.16 scrub starts 2024-02-10T07:56:12.719 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[40743]: 5.16 scrub ok 2024-02-10T07:56:12.719 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[40743]: 3.12 deep-scrub starts 2024-02-10T07:56:12.719 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[40743]: 3.12 deep-scrub ok 2024-02-10T07:56:12.719 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[40743]: osdmap e502: 8 total, 8 up, 8 in 2024-02-10T07:56:12.719 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[40743]: 4.1c scrub starts 2024-02-10T07:56:12.719 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:12 smithi008 ceph-mon[40743]: 4.1c scrub ok 2024-02-10T07:56:12.748 INFO:tasks.cephadm.ceph_manager.ceph:need seq 670014898411 got 670014898412 for osd.5 2024-02-10T07:56:12.748 DEBUG:teuthology.parallel:result is None 2024-02-10T07:56:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:12 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:56:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:12 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:56:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:12 smithi170 ceph-mon[33394]: 5.16 scrub starts 2024-02-10T07:56:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:12 smithi170 ceph-mon[33394]: 5.16 scrub ok 2024-02-10T07:56:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:12 smithi170 ceph-mon[33394]: 3.12 deep-scrub starts 2024-02-10T07:56:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:12 smithi170 ceph-mon[33394]: 3.12 deep-scrub ok 2024-02-10T07:56:12.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:12 smithi170 ceph-mon[33394]: osdmap e502: 8 total, 8 up, 8 in 2024-02-10T07:56:12.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:12 smithi170 ceph-mon[33394]: 4.1c scrub starts 2024-02-10T07:56:12.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:12 smithi170 ceph-mon[33394]: 4.1c scrub ok 2024-02-10T07:56:13.599 INFO:teuthology.orchestra.run.smithi008.stdout:55834575176 2024-02-10T07:56:13.775 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:13 smithi008 ceph-mon[29816]: 6.a scrub starts 2024-02-10T07:56:13.775 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:13 smithi008 ceph-mon[29816]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:56:13.775 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:13 smithi008 ceph-mon[29816]: 6.a scrub ok 2024-02-10T07:56:13.775 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:13 smithi008 ceph-mon[29816]: 6.15 scrub starts 2024-02-10T07:56:13.775 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:13 smithi008 ceph-mon[29816]: 6.15 scrub ok 2024-02-10T07:56:13.775 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:13 smithi008 ceph-mon[29816]: 6.1d scrub starts 2024-02-10T07:56:13.775 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:13 smithi008 ceph-mon[29816]: 6.1d scrub ok 2024-02-10T07:56:13.775 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:13 smithi008 ceph-mon[40743]: 6.a scrub starts 2024-02-10T07:56:13.775 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:13 smithi008 ceph-mon[40743]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:56:13.776 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:13 smithi008 ceph-mon[40743]: 6.a scrub ok 2024-02-10T07:56:13.776 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:13 smithi008 ceph-mon[40743]: 6.15 scrub starts 2024-02-10T07:56:13.776 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:13 smithi008 ceph-mon[40743]: 6.15 scrub ok 2024-02-10T07:56:13.776 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:13 smithi008 ceph-mon[40743]: 6.1d scrub starts 2024-02-10T07:56:13.776 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:13 smithi008 ceph-mon[40743]: 6.1d scrub ok 2024-02-10T07:56:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:13 smithi170 ceph-mon[33394]: 6.a scrub starts 2024-02-10T07:56:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:13 smithi170 ceph-mon[33394]: pgmap v1133: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-10T07:56:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:13 smithi170 ceph-mon[33394]: 6.a scrub ok 2024-02-10T07:56:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:13 smithi170 ceph-mon[33394]: 6.15 scrub starts 2024-02-10T07:56:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:13 smithi170 ceph-mon[33394]: 6.15 scrub ok 2024-02-10T07:56:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:13 smithi170 ceph-mon[33394]: 6.1d scrub starts 2024-02-10T07:56:13.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:13 smithi170 ceph-mon[33394]: 6.1d scrub ok 2024-02-10T07:56:14.441 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:56:14 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:56:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:14 smithi008 ceph-mon[29816]: 4.1 scrub starts 2024-02-10T07:56:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:14 smithi008 ceph-mon[29816]: 4.1 scrub ok 2024-02-10T07:56:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:14 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/178277035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T07:56:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:14 smithi008 ceph-mon[29816]: 4.12 scrub starts 2024-02-10T07:56:14.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:14 smithi008 ceph-mon[29816]: 4.12 scrub ok 2024-02-10T07:56:14.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:14 smithi008 ceph-mon[40743]: 4.1 scrub starts 2024-02-10T07:56:14.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:14 smithi008 ceph-mon[40743]: 4.1 scrub ok 2024-02-10T07:56:14.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:14 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/178277035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T07:56:14.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:14 smithi008 ceph-mon[40743]: 4.12 scrub starts 2024-02-10T07:56:14.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:14 smithi008 ceph-mon[40743]: 4.12 scrub ok 2024-02-10T07:56:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:14 smithi170 ceph-mon[33394]: 4.1 scrub starts 2024-02-10T07:56:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:14 smithi170 ceph-mon[33394]: 4.1 scrub ok 2024-02-10T07:56:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:14 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/178277035' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T07:56:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:14 smithi170 ceph-mon[33394]: 4.12 scrub starts 2024-02-10T07:56:14.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:14 smithi170 ceph-mon[33394]: 4.12 scrub ok 2024-02-10T07:56:15.455 INFO:teuthology.orchestra.run.smithi008.stdout:1490353651815 2024-02-10T07:56:15.818 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:15 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:56:15.818 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:15 smithi008 ceph-mon[29816]: 3.1c scrub starts 2024-02-10T07:56:15.818 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:15 smithi008 ceph-mon[29816]: 3.1c scrub ok 2024-02-10T07:56:15.818 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:15 smithi008 ceph-mon[29816]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:56:15.819 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:15 smithi008 ceph-mon[29816]: 4.10 scrub starts 2024-02-10T07:56:15.819 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:15 smithi008 ceph-mon[29816]: 4.10 scrub ok 2024-02-10T07:56:15.819 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:15 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:56:15.819 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:15 smithi008 ceph-mon[40743]: 3.1c scrub starts 2024-02-10T07:56:15.819 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:15 smithi008 ceph-mon[40743]: 3.1c scrub ok 2024-02-10T07:56:15.819 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:15 smithi008 ceph-mon[40743]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:56:15.819 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:15 smithi008 ceph-mon[40743]: 4.10 scrub starts 2024-02-10T07:56:15.819 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:15 smithi008 ceph-mon[40743]: 4.10 scrub ok 2024-02-10T07:56:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:15 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:56:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:15 smithi170 ceph-mon[33394]: 3.1c scrub starts 2024-02-10T07:56:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:15 smithi170 ceph-mon[33394]: 3.1c scrub ok 2024-02-10T07:56:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:15 smithi170 ceph-mon[33394]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-10T07:56:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:15 smithi170 ceph-mon[33394]: 4.10 scrub starts 2024-02-10T07:56:15.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:15 smithi170 ceph-mon[33394]: 4.10 scrub ok 2024-02-10T07:56:16.133 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1490353651812 got 1490353651815 for osd.3 2024-02-10T07:56:16.133 DEBUG:teuthology.parallel:result is None 2024-02-10T07:56:16.586 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575171 got 55834575176 for osd.1 2024-02-10T07:56:16.586 DEBUG:teuthology.parallel:result is None 2024-02-10T07:56:16.698 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:16 smithi008 ceph-mon[29816]: 3.1b deep-scrub starts 2024-02-10T07:56:16.698 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:16 smithi008 ceph-mon[29816]: 3.1b deep-scrub ok 2024-02-10T07:56:16.698 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:16 smithi008 ceph-mon[29816]: 4.19 scrub starts 2024-02-10T07:56:16.698 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:16 smithi008 ceph-mon[29816]: 4.19 scrub ok 2024-02-10T07:56:16.698 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:16 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/219074106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T07:56:16.698 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:16 smithi008 ceph-mon[29816]: 5.9 scrub starts 2024-02-10T07:56:16.698 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:16 smithi008 ceph-mon[29816]: 5.9 scrub ok 2024-02-10T07:56:16.698 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:16 smithi008 ceph-mon[40743]: 3.1b deep-scrub starts 2024-02-10T07:56:16.699 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:16 smithi008 ceph-mon[40743]: 3.1b deep-scrub ok 2024-02-10T07:56:16.699 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:16 smithi008 ceph-mon[40743]: 4.19 scrub starts 2024-02-10T07:56:16.699 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:16 smithi008 ceph-mon[40743]: 4.19 scrub ok 2024-02-10T07:56:16.699 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:16 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/219074106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T07:56:16.699 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:16 smithi008 ceph-mon[40743]: 5.9 scrub starts 2024-02-10T07:56:16.699 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:16 smithi008 ceph-mon[40743]: 5.9 scrub ok 2024-02-10T07:56:16.725 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748365100 got 214748365101 for osd.6 2024-02-10T07:56:16.725 DEBUG:teuthology.parallel:result is None 2024-02-10T07:56:16.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:16 smithi170 ceph-mon[33394]: 3.1b deep-scrub starts 2024-02-10T07:56:16.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:16 smithi170 ceph-mon[33394]: 3.1b deep-scrub ok 2024-02-10T07:56:16.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:16 smithi170 ceph-mon[33394]: 4.19 scrub starts 2024-02-10T07:56:16.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:16 smithi170 ceph-mon[33394]: 4.19 scrub ok 2024-02-10T07:56:16.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:16 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/219074106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T07:56:16.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:16 smithi170 ceph-mon[33394]: 5.9 scrub starts 2024-02-10T07:56:16.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:16 smithi170 ceph-mon[33394]: 5.9 scrub ok 2024-02-10T07:56:17.165 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953780 got 137438953784 for osd.4 2024-02-10T07:56:17.165 DEBUG:teuthology.parallel:result is None 2024-02-10T07:56:17.669 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:17 smithi008 ceph-mon[29816]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:17.669 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:17 smithi008 ceph-mon[29816]: 6.14 deep-scrub starts 2024-02-10T07:56:17.669 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:17 smithi008 ceph-mon[29816]: 6.14 deep-scrub ok 2024-02-10T07:56:17.669 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:17 smithi008 ceph-mon[40743]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:17.670 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:17 smithi008 ceph-mon[40743]: 6.14 deep-scrub starts 2024-02-10T07:56:17.670 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:17 smithi008 ceph-mon[40743]: 6.14 deep-scrub ok 2024-02-10T07:56:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:17 smithi170 ceph-mon[33394]: pgmap v1135: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:17 smithi170 ceph-mon[33394]: 6.14 deep-scrub starts 2024-02-10T07:56:17.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:17 smithi170 ceph-mon[33394]: 6.14 deep-scrub ok 2024-02-10T07:56:17.887 INFO:teuthology.orchestra.run.smithi008.stdout:257698038058 2024-02-10T07:56:18.565 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:18 smithi008 ceph-mon[29816]: 6.0 scrub starts 2024-02-10T07:56:18.566 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:18 smithi008 ceph-mon[29816]: 6.0 scrub ok 2024-02-10T07:56:18.566 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:18 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/4273485793' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T07:56:18.566 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:18 smithi008 ceph-mon[29816]: 6.f scrub starts 2024-02-10T07:56:18.566 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:18 smithi008 ceph-mon[29816]: 6.f scrub ok 2024-02-10T07:56:18.566 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:18 smithi008 ceph-mon[40743]: 6.0 scrub starts 2024-02-10T07:56:18.566 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:18 smithi008 ceph-mon[40743]: 6.0 scrub ok 2024-02-10T07:56:18.566 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:18 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/4273485793' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T07:56:18.566 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:18 smithi008 ceph-mon[40743]: 6.f scrub starts 2024-02-10T07:56:18.566 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:18 smithi008 ceph-mon[40743]: 6.f scrub ok 2024-02-10T07:56:18.592 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698038054 got 257698038058 for osd.7 2024-02-10T07:56:18.592 DEBUG:teuthology.parallel:result is None 2024-02-10T07:56:18.593 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-10T07:56:18.593 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:56:18.832 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:18 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:56:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:56:18.845 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:56:18.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:18 smithi170 ceph-mon[33394]: 6.0 scrub starts 2024-02-10T07:56:18.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:18 smithi170 ceph-mon[33394]: 6.0 scrub ok 2024-02-10T07:56:18.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:18 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/4273485793' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T07:56:18.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:18 smithi170 ceph-mon[33394]: 6.f scrub starts 2024-02-10T07:56:18.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:18 smithi170 ceph-mon[33394]: 6.f scrub ok 2024-02-10T07:56:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[29816]: pgmap v1136: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[29816]: 6.1e scrub starts 2024-02-10T07:56:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[29816]: 6.1e scrub ok 2024-02-10T07:56:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[29816]: 4.b scrub starts 2024-02-10T07:56:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[29816]: 4.b scrub ok 2024-02-10T07:56:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[29816]: 4.1d scrub starts 2024-02-10T07:56:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[29816]: 4.1d scrub ok 2024-02-10T07:56:19.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:56:19.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[40743]: pgmap v1136: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[40743]: 6.1e scrub starts 2024-02-10T07:56:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[40743]: 6.1e scrub ok 2024-02-10T07:56:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[40743]: 4.b scrub starts 2024-02-10T07:56:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[40743]: 4.b scrub ok 2024-02-10T07:56:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[40743]: 4.1d scrub starts 2024-02-10T07:56:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[40743]: 4.1d scrub ok 2024-02-10T07:56:19.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:19 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:56:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:19 smithi170 ceph-mon[33394]: pgmap v1136: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:19 smithi170 ceph-mon[33394]: 6.1e scrub starts 2024-02-10T07:56:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:19 smithi170 ceph-mon[33394]: 6.1e scrub ok 2024-02-10T07:56:19.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:19 smithi170 ceph-mon[33394]: 4.b scrub starts 2024-02-10T07:56:19.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:19 smithi170 ceph-mon[33394]: 4.b scrub ok 2024-02-10T07:56:19.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:19 smithi170 ceph-mon[33394]: 4.1d scrub starts 2024-02-10T07:56:19.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:19 smithi170 ceph-mon[33394]: 4.1d scrub ok 2024-02-10T07:56:19.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:19 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T07:56:20.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[29816]: 3.c deep-scrub starts 2024-02-10T07:56:20.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[29816]: 3.c deep-scrub ok 2024-02-10T07:56:20.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[29816]: 5.1c scrub starts 2024-02-10T07:56:20.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[29816]: 5.1c scrub ok 2024-02-10T07:56:20.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[29816]: 4.6 scrub starts 2024-02-10T07:56:20.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[29816]: 4.6 scrub ok 2024-02-10T07:56:20.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[29816]: purged_snaps scrub starts 2024-02-10T07:56:20.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[29816]: purged_snaps scrub ok 2024-02-10T07:56:20.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[29816]: osdmap e503: 8 total, 8 up, 8 in 2024-02-10T07:56:20.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[29816]: 5.12 scrub starts 2024-02-10T07:56:20.583 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[29816]: 5.12 scrub ok 2024-02-10T07:56:20.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[40743]: 3.c deep-scrub starts 2024-02-10T07:56:20.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[40743]: 3.c deep-scrub ok 2024-02-10T07:56:20.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[40743]: 5.1c scrub starts 2024-02-10T07:56:20.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[40743]: 5.1c scrub ok 2024-02-10T07:56:20.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[40743]: 4.6 scrub starts 2024-02-10T07:56:20.583 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[40743]: 4.6 scrub ok 2024-02-10T07:56:20.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[40743]: purged_snaps scrub starts 2024-02-10T07:56:20.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[40743]: purged_snaps scrub ok 2024-02-10T07:56:20.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[40743]: osdmap e503: 8 total, 8 up, 8 in 2024-02-10T07:56:20.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[40743]: 5.12 scrub starts 2024-02-10T07:56:20.584 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:20 smithi008 ceph-mon[40743]: 5.12 scrub ok 2024-02-10T07:56:20.732 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:56:20.735 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:56:20.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:20 smithi170 ceph-mon[33394]: 3.c deep-scrub starts 2024-02-10T07:56:20.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:20 smithi170 ceph-mon[33394]: 3.c deep-scrub ok 2024-02-10T07:56:20.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:20 smithi170 ceph-mon[33394]: 5.1c scrub starts 2024-02-10T07:56:20.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:20 smithi170 ceph-mon[33394]: 5.1c scrub ok 2024-02-10T07:56:20.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:20 smithi170 ceph-mon[33394]: 4.6 scrub starts 2024-02-10T07:56:20.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:20 smithi170 ceph-mon[33394]: 4.6 scrub ok 2024-02-10T07:56:20.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:20 smithi170 ceph-mon[33394]: purged_snaps scrub starts 2024-02-10T07:56:20.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:20 smithi170 ceph-mon[33394]: purged_snaps scrub ok 2024-02-10T07:56:20.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:20 smithi170 ceph-mon[33394]: osdmap e503: 8 total, 8 up, 8 in 2024-02-10T07:56:20.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:20 smithi170 ceph-mon[33394]: 5.12 scrub starts 2024-02-10T07:56:20.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:20 smithi170 ceph-mon[33394]: 5.12 scrub ok 2024-02-10T07:56:21.426 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":1138,"stamp":"2024-02-10T07:56:20.246087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3229,"num_read_kb":3075,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":1780561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1307,"ondisk_log_size":1307,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":397,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5541572,"kb_used_data":28956,"kb_used_omap":19,"kb_used_meta":5512044,"kb_avail":744419644,"statfs":{"total":767960285184,"available":762285715456,"internally_reserved":0,"allocated":29650944,"data_stored":26362052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19634,"internal_metadata":5644333902},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.811265"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":950,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436958+0000","last_change":"2024-02-10T07:55:12.844519+0000","last_active":"2024-02-10T07:56:11.436958+0000","last_peered":"2024-02-10T07:56:11.436958+0000","last_clean":"2024-02-10T07:56:11.436958+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:56:11.436958+0000","last_undegraded":"2024-02-10T07:56:11.436958+0000","last_fullsized":"2024-02-10T07:56:11.436958+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:12.844419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:12.844419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:15.833463+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":971,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642227+0000","last_change":"2024-02-10T07:55:59.815245+0000","last_active":"2024-02-10T07:56:19.642227+0000","last_peered":"2024-02-10T07:56:19.642227+0000","last_clean":"2024-02-10T07:56:19.642227+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:56:19.642227+0000","last_undegraded":"2024-02-10T07:56:19.642227+0000","last_fullsized":"2024-02-10T07:56:19.642227+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:59.815161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:59.815161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:11.379146+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"496'2","reported_seq":999,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675655+0000","last_change":"2024-02-10T07:55:39.332198+0000","last_active":"2024-02-10T07:56:11.675655+0000","last_peered":"2024-02-10T07:56:11.675655+0000","last_clean":"2024-02-10T07:56:11.675655+0000","last_became_active":"2024-02-10T07:50:43.281905+0000","last_became_peered":"2024-02-10T07:50:43.281905+0000","last_unstale":"2024-02-10T07:56:11.675655+0000","last_undegraded":"2024-02-10T07:56:11.675655+0000","last_fullsized":"2024-02-10T07:56:11.675655+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'2","last_scrub_stamp":"2024-02-10T07:55:39.332082+0000","last_deep_scrub":"496'2","last_deep_scrub_stamp":"2024-02-10T07:55:39.332082+0000","last_clean_scrub_stamp":"2024-02-10T07:55:39.332082+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:42.221152+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"430'25","reported_seq":1028,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436414+0000","last_change":"2024-02-10T07:55:38.061236+0000","last_active":"2024-02-10T07:56:11.436414+0000","last_peered":"2024-02-10T07:56:11.436414+0000","last_clean":"2024-02-10T07:56:11.436414+0000","last_became_active":"2024-02-10T07:46:10.717475+0000","last_became_peered":"2024-02-10T07:46:10.717475+0000","last_unstale":"2024-02-10T07:56:11.436414+0000","last_undegraded":"2024-02-10T07:56:11.436414+0000","last_fullsized":"2024-02-10T07:56:11.436414+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'25","last_scrub_stamp":"2024-02-10T07:55:38.061140+0000","last_deep_scrub":"287'19","last_deep_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_clean_scrub_stamp":"2024-02-10T07:55:38.061140+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:51.039567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":289,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436617+0000","last_change":"2024-02-10T07:54:58.107439+0000","last_active":"2024-02-10T07:56:11.436617+0000","last_peered":"2024-02-10T07:56:11.436617+0000","last_clean":"2024-02-10T07:56:11.436617+0000","last_became_active":"2024-02-10T07:49:06.236235+0000","last_became_peered":"2024-02-10T07:49:06.236235+0000","last_unstale":"2024-02-10T07:56:11.436617+0000","last_undegraded":"2024-02-10T07:56:11.436617+0000","last_fullsized":"2024-02-10T07:56:11.436617+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:58.107337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:15.079428+0000","last_clean_scrub_stamp":"2024-02-10T07:54:58.107337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":954,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436308+0000","last_change":"2024-02-10T07:55:16.854169+0000","last_active":"2024-02-10T07:56:11.436308+0000","last_peered":"2024-02-10T07:56:11.436308+0000","last_clean":"2024-02-10T07:56:11.436308+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:56:11.436308+0000","last_undegraded":"2024-02-10T07:56:11.436308+0000","last_fullsized":"2024-02-10T07:56:11.436308+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:16.854068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:16.854068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:22.589436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":287,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437514+0000","last_change":"2024-02-10T07:55:58.207446+0000","last_active":"2024-02-10T07:56:11.437514+0000","last_peered":"2024-02-10T07:56:11.437514+0000","last_clean":"2024-02-10T07:56:11.437514+0000","last_became_active":"2024-02-10T07:55:37.333014+0000","last_became_peered":"2024-02-10T07:55:37.333014+0000","last_unstale":"2024-02-10T07:56:11.437514+0000","last_undegraded":"2024-02-10T07:56:11.437514+0000","last_fullsized":"2024-02-10T07:56:11.437514+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:58.207336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:55:58.207336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:08.282823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"430'39","reported_seq":125,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:15.354488+0000","last_change":"2024-02-10T07:56:15.354517+0000","last_active":"2024-02-10T07:56:15.354488+0000","last_peered":"2024-02-10T07:56:15.354488+0000","last_clean":"2024-02-10T07:56:15.354488+0000","last_became_active":"2024-02-10T07:55:37.330931+0000","last_became_peered":"2024-02-10T07:55:37.330931+0000","last_unstale":"2024-02-10T07:56:15.354488+0000","last_undegraded":"2024-02-10T07:56:15.354488+0000","last_fullsized":"2024-02-10T07:56:15.354488+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'39","last_scrub_stamp":"2024-02-10T07:56:15.354439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:56:15.354439+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:16.256553+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":950,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675294+0000","last_change":"2024-02-10T07:54:54.543925+0000","last_active":"2024-02-10T07:56:11.675294+0000","last_peered":"2024-02-10T07:56:11.675294+0000","last_clean":"2024-02-10T07:56:11.675294+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:56:11.675294+0000","last_undegraded":"2024-02-10T07:56:11.675294+0000","last_fullsized":"2024-02-10T07:56:11.675294+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:54.543830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:54:54.543830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":946,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675865+0000","last_change":"2024-02-10T07:55:39.378371+0000","last_active":"2024-02-10T07:56:11.675865+0000","last_peered":"2024-02-10T07:56:11.675865+0000","last_clean":"2024-02-10T07:56:11.675865+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:56:11.675865+0000","last_undegraded":"2024-02-10T07:56:11.675865+0000","last_fullsized":"2024-02-10T07:56:11.675865+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:39.378240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:39.378240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:56.084564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":609,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676520+0000","last_change":"2024-02-10T07:55:31.314516+0000","last_active":"2024-02-10T07:56:11.676520+0000","last_peered":"2024-02-10T07:56:11.676520+0000","last_clean":"2024-02-10T07:56:11.676520+0000","last_became_active":"2024-02-10T07:55:31.314182+0000","last_became_peered":"2024-02-10T07:55:31.314182+0000","last_unstale":"2024-02-10T07:56:11.676520+0000","last_undegraded":"2024-02-10T07:56:11.676520+0000","last_fullsized":"2024-02-10T07:56:11.676520+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:55:02.912660+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_clean_scrub_stamp":"2024-02-10T07:55:02.912660+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:19.640092+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"430'21","reported_seq":992,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437894+0000","last_change":"2024-02-10T07:55:55.145845+0000","last_active":"2024-02-10T07:56:11.437894+0000","last_peered":"2024-02-10T07:56:11.437894+0000","last_clean":"2024-02-10T07:56:11.437894+0000","last_became_active":"2024-02-10T07:47:47.965930+0000","last_became_peered":"2024-02-10T07:47:47.965930+0000","last_unstale":"2024-02-10T07:56:11.437894+0000","last_undegraded":"2024-02-10T07:56:11.437894+0000","last_fullsized":"2024-02-10T07:56:11.437894+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'21","last_scrub_stamp":"2024-02-10T07:55:55.145674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:55.145674+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:17.552998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":637,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675623+0000","last_change":"2024-02-10T07:55:31.309229+0000","last_active":"2024-02-10T07:56:11.675623+0000","last_peered":"2024-02-10T07:56:11.675623+0000","last_clean":"2024-02-10T07:56:11.675623+0000","last_became_active":"2024-02-10T07:55:31.308892+0000","last_became_peered":"2024-02-10T07:55:31.308892+0000","last_unstale":"2024-02-10T07:56:11.675623+0000","last_undegraded":"2024-02-10T07:56:11.675623+0000","last_fullsized":"2024-02-10T07:56:11.675623+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:16.806296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:16.806296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:41.616615+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":961,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436586+0000","last_change":"2024-02-10T07:56:09.521122+0000","last_active":"2024-02-10T07:56:11.436586+0000","last_peered":"2024-02-10T07:56:11.436586+0000","last_clean":"2024-02-10T07:56:11.436586+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:56:11.436586+0000","last_undegraded":"2024-02-10T07:56:11.436586+0000","last_fullsized":"2024-02-10T07:56:11.436586+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:09.520986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:56:09.520986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:22.476454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":640,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:14.222707+0000","last_change":"2024-02-10T07:56:14.222738+0000","last_active":"2024-02-10T07:56:14.222707+0000","last_peered":"2024-02-10T07:56:14.222707+0000","last_clean":"2024-02-10T07:56:14.222707+0000","last_became_active":"2024-02-10T07:55:31.309628+0000","last_became_peered":"2024-02-10T07:55:31.309628+0000","last_unstale":"2024-02-10T07:56:14.222707+0000","last_undegraded":"2024-02-10T07:56:14.222707+0000","last_fullsized":"2024-02-10T07:56:14.222707+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:14.222627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.473686+0000","last_clean_scrub_stamp":"2024-02-10T07:56:14.222627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:42.470351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"430'9","reported_seq":966,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437559+0000","last_change":"2024-02-10T07:54:40.819181+0000","last_active":"2024-02-10T07:56:11.437559+0000","last_peered":"2024-02-10T07:56:11.437559+0000","last_clean":"2024-02-10T07:56:11.437559+0000","last_became_active":"2024-02-10T07:47:47.966093+0000","last_became_peered":"2024-02-10T07:47:47.966093+0000","last_unstale":"2024-02-10T07:56:11.437559+0000","last_undegraded":"2024-02-10T07:56:11.437559+0000","last_fullsized":"2024-02-10T07:56:11.437559+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'9","last_scrub_stamp":"2024-02-10T07:54:40.819097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:54:40.819097+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":961,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:18.253463+0000","last_change":"2024-02-10T07:56:18.253486+0000","last_active":"2024-02-10T07:56:18.253463+0000","last_peered":"2024-02-10T07:56:18.253463+0000","last_clean":"2024-02-10T07:56:18.253463+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:56:18.253463+0000","last_undegraded":"2024-02-10T07:56:18.253463+0000","last_fullsized":"2024-02-10T07:56:18.253463+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:18.253405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:56:18.253405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:33.380146+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":961,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.641981+0000","last_change":"2024-02-10T07:54:59.789284+0000","last_active":"2024-02-10T07:56:19.641981+0000","last_peered":"2024-02-10T07:56:19.641981+0000","last_clean":"2024-02-10T07:56:19.641981+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:56:19.641981+0000","last_undegraded":"2024-02-10T07:56:19.641981+0000","last_fullsized":"2024-02-10T07:56:19.641981+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:59.789208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:54:59.789208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":969,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:15.342693+0000","last_change":"2024-02-10T07:56:15.342712+0000","last_active":"2024-02-10T07:56:15.342693+0000","last_peered":"2024-02-10T07:56:15.342693+0000","last_clean":"2024-02-10T07:56:15.342693+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:56:15.342693+0000","last_undegraded":"2024-02-10T07:56:15.342693+0000","last_fullsized":"2024-02-10T07:56:15.342693+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:15.342620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:56:15.342620+0000","last_clean_scrub_stamp":"2024-02-10T07:56:15.342620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:42.289567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"430'46","reported_seq":257,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:12.213289+0000","last_change":"2024-02-10T07:56:12.213323+0000","last_active":"2024-02-10T07:56:12.213289+0000","last_peered":"2024-02-10T07:56:12.213289+0000","last_clean":"2024-02-10T07:56:12.213289+0000","last_became_active":"2024-02-10T07:50:43.300153+0000","last_became_peered":"2024-02-10T07:50:43.300153+0000","last_unstale":"2024-02-10T07:56:12.213289+0000","last_undegraded":"2024-02-10T07:56:12.213289+0000","last_fullsized":"2024-02-10T07:56:12.213289+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'46","last_scrub_stamp":"2024-02-10T07:56:12.213228+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:56:12.213228+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:23.851735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":399,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437912+0000","last_change":"2024-02-10T07:55:56.318846+0000","last_active":"2024-02-10T07:56:11.437912+0000","last_peered":"2024-02-10T07:56:11.437912+0000","last_clean":"2024-02-10T07:56:11.437912+0000","last_became_active":"2024-02-10T07:55:37.329874+0000","last_became_peered":"2024-02-10T07:55:37.329874+0000","last_unstale":"2024-02-10T07:56:11.437912+0000","last_undegraded":"2024-02-10T07:56:11.437912+0000","last_fullsized":"2024-02-10T07:56:11.437912+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:56.318750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:56.318750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:19.181640+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":970,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437552+0000","last_change":"2024-02-10T07:54:44.867195+0000","last_active":"2024-02-10T07:56:11.437552+0000","last_peered":"2024-02-10T07:56:11.437552+0000","last_clean":"2024-02-10T07:56:11.437552+0000","last_became_active":"2024-02-10T07:50:43.285201+0000","last_became_peered":"2024-02-10T07:50:43.285201+0000","last_unstale":"2024-02-10T07:56:11.437552+0000","last_undegraded":"2024-02-10T07:56:11.437552+0000","last_fullsized":"2024-02-10T07:56:11.437552+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:44.867113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_clean_scrub_stamp":"2024-02-10T07:54:44.867113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":973,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437984+0000","last_change":"2024-02-10T07:55:11.841026+0000","last_active":"2024-02-10T07:56:11.437984+0000","last_peered":"2024-02-10T07:56:11.437984+0000","last_clean":"2024-02-10T07:56:11.437984+0000","last_became_active":"2024-02-10T07:50:43.281382+0000","last_became_peered":"2024-02-10T07:50:43.281382+0000","last_unstale":"2024-02-10T07:56:11.437984+0000","last_undegraded":"2024-02-10T07:56:11.437984+0000","last_fullsized":"2024-02-10T07:56:11.437984+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:11.840927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:11.840927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:14.450537+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"430'24","reported_seq":797,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436352+0000","last_change":"2024-02-10T07:55:37.326144+0000","last_active":"2024-02-10T07:56:11.436352+0000","last_peered":"2024-02-10T07:56:11.436352+0000","last_clean":"2024-02-10T07:56:11.436352+0000","last_became_active":"2024-02-10T07:55:37.325786+0000","last_became_peered":"2024-02-10T07:55:37.325786+0000","last_unstale":"2024-02-10T07:56:11.436352+0000","last_undegraded":"2024-02-10T07:56:11.436352+0000","last_fullsized":"2024-02-10T07:56:11.436352+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'24","last_scrub_stamp":"2024-02-10T07:54:55.819607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:54:55.819607+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":993,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437387+0000","last_change":"2024-02-10T07:55:48.558565+0000","last_active":"2024-02-10T07:56:11.437387+0000","last_peered":"2024-02-10T07:56:11.437387+0000","last_clean":"2024-02-10T07:56:11.437387+0000","last_became_active":"2024-02-10T07:50:43.285254+0000","last_became_peered":"2024-02-10T07:50:43.285254+0000","last_unstale":"2024-02-10T07:56:11.437387+0000","last_undegraded":"2024-02-10T07:56:11.437387+0000","last_fullsized":"2024-02-10T07:56:11.437387+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:55:48.558470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:48.558470+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:15.581859+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"414'9","reported_seq":1299,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676554+0000","last_change":"2024-02-10T07:55:48.183624+0000","last_active":"2024-02-10T07:56:11.676554+0000","last_peered":"2024-02-10T07:56:11.676554+0000","last_clean":"2024-02-10T07:56:11.676554+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:56:11.676554+0000","last_undegraded":"2024-02-10T07:56:11.676554+0000","last_fullsized":"2024-02-10T07:56:11.676554+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'9","last_scrub_stamp":"2024-02-10T07:55:48.183527+0000","last_deep_scrub":"414'9","last_deep_scrub_stamp":"2024-02-10T07:51:13.306986+0000","last_clean_scrub_stamp":"2024-02-10T07:55:48.183527+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:11.823285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":980,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642179+0000","last_change":"2024-02-10T07:55:35.756308+0000","last_active":"2024-02-10T07:56:19.642179+0000","last_peered":"2024-02-10T07:56:19.642179+0000","last_clean":"2024-02-10T07:56:19.642179+0000","last_became_active":"2024-02-10T07:47:47.966127+0000","last_became_peered":"2024-02-10T07:47:47.966127+0000","last_unstale":"2024-02-10T07:56:19.642179+0000","last_undegraded":"2024-02-10T07:56:19.642179+0000","last_fullsized":"2024-02-10T07:56:19.642179+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:55:35.756214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:35.756214+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:58.483732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"430'35","reported_seq":1057,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675985+0000","last_change":"2024-02-10T07:55:59.214996+0000","last_active":"2024-02-10T07:56:11.675985+0000","last_peered":"2024-02-10T07:56:11.675985+0000","last_clean":"2024-02-10T07:56:11.675985+0000","last_became_active":"2024-02-10T07:49:06.261487+0000","last_became_peered":"2024-02-10T07:49:06.261487+0000","last_unstale":"2024-02-10T07:56:11.675985+0000","last_undegraded":"2024-02-10T07:56:11.675985+0000","last_fullsized":"2024-02-10T07:56:11.675985+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'35","last_scrub_stamp":"2024-02-10T07:55:59.214906+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-02-10T07:42:17.820959+0000","last_clean_scrub_stamp":"2024-02-10T07:55:59.214906+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:21.584520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":966,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642846+0000","last_change":"2024-02-10T07:56:12.918823+0000","last_active":"2024-02-10T07:56:19.642846+0000","last_peered":"2024-02-10T07:56:19.642846+0000","last_clean":"2024-02-10T07:56:19.642846+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:56:19.642846+0000","last_undegraded":"2024-02-10T07:56:19.642846+0000","last_fullsized":"2024-02-10T07:56:19.642846+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:12.918725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_clean_scrub_stamp":"2024-02-10T07:56:12.918725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:34.344392+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":968,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676241+0000","last_change":"2024-02-10T07:55:48.243563+0000","last_active":"2024-02-10T07:56:11.676241+0000","last_peered":"2024-02-10T07:56:11.676241+0000","last_clean":"2024-02-10T07:56:11.676241+0000","last_became_active":"2024-02-10T07:50:43.289011+0000","last_became_peered":"2024-02-10T07:50:43.289011+0000","last_unstale":"2024-02-10T07:56:11.676241+0000","last_undegraded":"2024-02-10T07:56:11.676241+0000","last_fullsized":"2024-02-10T07:56:11.676241+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:48.243485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:55:48.243485+0000","last_clean_scrub_stamp":"2024-02-10T07:55:48.243485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:56.752501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":209,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.435930+0000","last_change":"2024-02-10T07:55:37.330511+0000","last_active":"2024-02-10T07:56:11.435930+0000","last_peered":"2024-02-10T07:56:11.435930+0000","last_clean":"2024-02-10T07:56:11.435930+0000","last_became_active":"2024-02-10T07:55:37.330208+0000","last_became_peered":"2024-02-10T07:55:37.330208+0000","last_unstale":"2024-02-10T07:56:11.435930+0000","last_undegraded":"2024-02-10T07:56:11.435930+0000","last_fullsized":"2024-02-10T07:56:11.435930+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:01.490384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:55:01.490384+0000","last_clean_scrub_stamp":"2024-02-10T07:55:01.490384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:24.405129+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"499'32","reported_seq":1038,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676230+0000","last_change":"2024-02-10T07:55:52.183996+0000","last_active":"2024-02-10T07:56:11.676230+0000","last_peered":"2024-02-10T07:56:11.676230+0000","last_clean":"2024-02-10T07:56:11.676230+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:56:11.676230+0000","last_undegraded":"2024-02-10T07:56:11.676230+0000","last_fullsized":"2024-02-10T07:56:11.676230+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'32","last_scrub_stamp":"2024-02-10T07:55:52.183856+0000","last_deep_scrub":"499'32","last_deep_scrub_stamp":"2024-02-10T07:55:52.183856+0000","last_clean_scrub_stamp":"2024-02-10T07:55:52.183856+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:05.009917+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":980,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436474+0000","last_change":"2024-02-10T07:55:56.533465+0000","last_active":"2024-02-10T07:56:11.436474+0000","last_peered":"2024-02-10T07:56:11.436474+0000","last_clean":"2024-02-10T07:56:11.436474+0000","last_became_active":"2024-02-10T07:50:43.283688+0000","last_became_peered":"2024-02-10T07:50:43.283688+0000","last_unstale":"2024-02-10T07:56:11.436474+0000","last_undegraded":"2024-02-10T07:56:11.436474+0000","last_fullsized":"2024-02-10T07:56:11.436474+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:56.533379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:55:56.533379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:59.462630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":984,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675599+0000","last_change":"2024-02-10T07:55:58.252182+0000","last_active":"2024-02-10T07:56:11.675599+0000","last_peered":"2024-02-10T07:56:11.675599+0000","last_clean":"2024-02-10T07:56:11.675599+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:56:11.675599+0000","last_undegraded":"2024-02-10T07:56:11.675599+0000","last_fullsized":"2024-02-10T07:56:11.675599+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:58.252063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:58.252063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:18.787601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":994,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675892+0000","last_change":"2024-02-10T07:55:42.252188+0000","last_active":"2024-02-10T07:56:11.675892+0000","last_peered":"2024-02-10T07:56:11.675892+0000","last_clean":"2024-02-10T07:56:11.675892+0000","last_became_active":"2024-02-10T07:49:06.258763+0000","last_became_peered":"2024-02-10T07:49:06.258763+0000","last_unstale":"2024-02-10T07:56:11.675892+0000","last_undegraded":"2024-02-10T07:56:11.675892+0000","last_fullsized":"2024-02-10T07:56:11.675892+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:42.252112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:55:42.252112+0000","last_clean_scrub_stamp":"2024-02-10T07:55:42.252112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:06.348116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"430'14","reported_seq":310,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:14.318457+0000","last_change":"2024-02-10T07:56:14.318484+0000","last_active":"2024-02-10T07:56:14.318457+0000","last_peered":"2024-02-10T07:56:14.318457+0000","last_clean":"2024-02-10T07:56:14.318457+0000","last_became_active":"2024-02-10T07:49:06.250577+0000","last_became_peered":"2024-02-10T07:49:06.250577+0000","last_unstale":"2024-02-10T07:56:14.318457+0000","last_undegraded":"2024-02-10T07:56:14.318457+0000","last_fullsized":"2024-02-10T07:56:14.318457+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'14","last_scrub_stamp":"2024-02-10T07:56:14.318387+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:56:14.318387+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:38.626302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":203,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436456+0000","last_change":"2024-02-10T07:55:37.331956+0000","last_active":"2024-02-10T07:56:11.436456+0000","last_peered":"2024-02-10T07:56:11.436456+0000","last_clean":"2024-02-10T07:56:11.436456+0000","last_became_active":"2024-02-10T07:55:37.331369+0000","last_became_peered":"2024-02-10T07:55:37.331369+0000","last_unstale":"2024-02-10T07:56:11.436456+0000","last_undegraded":"2024-02-10T07:56:11.436456+0000","last_fullsized":"2024-02-10T07:56:11.436456+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:17.615043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:17.615043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:41.267467+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":962,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436702+0000","last_change":"2024-02-10T07:55:40.553297+0000","last_active":"2024-02-10T07:56:11.436702+0000","last_peered":"2024-02-10T07:56:11.436702+0000","last_clean":"2024-02-10T07:56:11.436702+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:56:11.436702+0000","last_undegraded":"2024-02-10T07:56:11.436702+0000","last_fullsized":"2024-02-10T07:56:11.436702+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:40.553198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:40.553198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:01.172275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":919,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676690+0000","last_change":"2024-02-10T07:55:57.070134+0000","last_active":"2024-02-10T07:56:11.676690+0000","last_peered":"2024-02-10T07:56:11.676690+0000","last_clean":"2024-02-10T07:56:11.676690+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:56:11.676690+0000","last_undegraded":"2024-02-10T07:56:11.676690+0000","last_fullsized":"2024-02-10T07:56:11.676690+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:57.069994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:55:57.069994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:01.803000+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"430'27","reported_seq":800,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436325+0000","last_change":"2024-02-10T07:55:37.328415+0000","last_active":"2024-02-10T07:56:11.436325+0000","last_peered":"2024-02-10T07:56:11.436325+0000","last_clean":"2024-02-10T07:56:11.436325+0000","last_became_active":"2024-02-10T07:55:37.327995+0000","last_became_peered":"2024-02-10T07:55:37.327995+0000","last_unstale":"2024-02-10T07:56:11.436325+0000","last_undegraded":"2024-02-10T07:56:11.436325+0000","last_fullsized":"2024-02-10T07:56:11.436325+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'27","last_scrub_stamp":"2024-02-10T07:55:14.660812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:14.660812+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:31.873409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":951,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675015+0000","last_change":"2024-02-10T07:54:50.614286+0000","last_active":"2024-02-10T07:56:11.675015+0000","last_peered":"2024-02-10T07:56:11.675015+0000","last_clean":"2024-02-10T07:56:11.675015+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:56:11.675015+0000","last_undegraded":"2024-02-10T07:56:11.675015+0000","last_fullsized":"2024-02-10T07:56:11.675015+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:50.614178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:59.179342+0000","last_clean_scrub_stamp":"2024-02-10T07:54:50.614178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:14.321428+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":203,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436349+0000","last_change":"2024-02-10T07:55:37.331959+0000","last_active":"2024-02-10T07:56:11.436349+0000","last_peered":"2024-02-10T07:56:11.436349+0000","last_clean":"2024-02-10T07:56:11.436349+0000","last_became_active":"2024-02-10T07:55:37.331684+0000","last_became_peered":"2024-02-10T07:55:37.331684+0000","last_unstale":"2024-02-10T07:56:11.436349+0000","last_undegraded":"2024-02-10T07:56:11.436349+0000","last_fullsized":"2024-02-10T07:56:11.436349+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:02.441727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:02.441727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:30.783948+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":971,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.438129+0000","last_change":"2024-02-10T07:55:15.657294+0000","last_active":"2024-02-10T07:56:11.438129+0000","last_peered":"2024-02-10T07:56:11.438129+0000","last_clean":"2024-02-10T07:56:11.438129+0000","last_became_active":"2024-02-10T07:47:47.966472+0000","last_became_peered":"2024-02-10T07:47:47.966472+0000","last_unstale":"2024-02-10T07:56:11.438129+0000","last_undegraded":"2024-02-10T07:56:11.438129+0000","last_fullsized":"2024-02-10T07:56:11.438129+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:15.657153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:15.657153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:44.322928+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"430'25","reported_seq":814,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642222+0000","last_change":"2024-02-10T07:56:13.682987+0000","last_active":"2024-02-10T07:56:19.642222+0000","last_peered":"2024-02-10T07:56:19.642222+0000","last_clean":"2024-02-10T07:56:19.642222+0000","last_became_active":"2024-02-10T07:50:43.296447+0000","last_became_peered":"2024-02-10T07:50:43.296447+0000","last_unstale":"2024-02-10T07:56:19.642222+0000","last_undegraded":"2024-02-10T07:56:19.642222+0000","last_fullsized":"2024-02-10T07:56:19.642222+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'25","last_scrub_stamp":"2024-02-10T07:56:13.682903+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-02-10T07:39:32.934269+0000","last_clean_scrub_stamp":"2024-02-10T07:56:13.682903+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:18.120906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":391,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436212+0000","last_change":"2024-02-10T07:55:37.331174+0000","last_active":"2024-02-10T07:56:11.436212+0000","last_peered":"2024-02-10T07:56:11.436212+0000","last_clean":"2024-02-10T07:56:11.436212+0000","last_became_active":"2024-02-10T07:55:37.330870+0000","last_became_peered":"2024-02-10T07:55:37.330870+0000","last_unstale":"2024-02-10T07:56:11.436212+0000","last_undegraded":"2024-02-10T07:56:11.436212+0000","last_fullsized":"2024-02-10T07:56:11.436212+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:15.492475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:15.492475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:29.933780+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":994,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.644097+0000","last_change":"2024-02-10T07:56:19.644118+0000","last_active":"2024-02-10T07:56:19.644097+0000","last_peered":"2024-02-10T07:56:19.644097+0000","last_clean":"2024-02-10T07:56:19.644097+0000","last_became_active":"2024-02-10T07:49:06.252380+0000","last_became_peered":"2024-02-10T07:49:06.252380+0000","last_unstale":"2024-02-10T07:56:19.644097+0000","last_undegraded":"2024-02-10T07:56:19.644097+0000","last_fullsized":"2024-02-10T07:56:19.644097+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:19.644032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_clean_scrub_stamp":"2024-02-10T07:56:19.644032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:35.020948+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":962,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437296+0000","last_change":"2024-02-10T07:55:17.821620+0000","last_active":"2024-02-10T07:56:11.437296+0000","last_peered":"2024-02-10T07:56:11.437296+0000","last_clean":"2024-02-10T07:56:11.437296+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:56:11.437296+0000","last_undegraded":"2024-02-10T07:56:11.437296+0000","last_fullsized":"2024-02-10T07:56:11.437296+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:17.821524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:17.821524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:26.982295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"430'32","reported_seq":1042,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437309+0000","last_change":"2024-02-10T07:55:59.165358+0000","last_active":"2024-02-10T07:56:11.437309+0000","last_peered":"2024-02-10T07:56:11.437309+0000","last_clean":"2024-02-10T07:56:11.437309+0000","last_became_active":"2024-02-10T07:46:10.721336+0000","last_became_peered":"2024-02-10T07:46:10.721336+0000","last_unstale":"2024-02-10T07:56:11.437309+0000","last_undegraded":"2024-02-10T07:56:11.437309+0000","last_fullsized":"2024-02-10T07:56:11.437309+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'32","last_scrub_stamp":"2024-02-10T07:55:59.165240+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:55:59.165240+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:19.426052+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":965,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676327+0000","last_change":"2024-02-10T07:55:37.461141+0000","last_active":"2024-02-10T07:56:11.676327+0000","last_peered":"2024-02-10T07:56:11.676327+0000","last_clean":"2024-02-10T07:56:11.676327+0000","last_became_active":"2024-02-10T07:49:06.261570+0000","last_became_peered":"2024-02-10T07:49:06.261570+0000","last_unstale":"2024-02-10T07:56:11.676327+0000","last_undegraded":"2024-02-10T07:56:11.676327+0000","last_fullsized":"2024-02-10T07:56:11.676327+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:37.461030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:55:37.461030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:57.234558+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"493'12","reported_seq":797,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.677110+0000","last_change":"2024-02-10T07:55:55.083284+0000","last_active":"2024-02-10T07:56:11.677110+0000","last_peered":"2024-02-10T07:56:11.677110+0000","last_clean":"2024-02-10T07:56:11.677110+0000","last_became_active":"2024-02-10T07:55:31.311897+0000","last_became_peered":"2024-02-10T07:55:31.311897+0000","last_unstale":"2024-02-10T07:56:11.677110+0000","last_undegraded":"2024-02-10T07:56:11.677110+0000","last_fullsized":"2024-02-10T07:56:11.677110+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'12","last_scrub_stamp":"2024-02-10T07:55:55.083179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:55.083179+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:04.202185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":982,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436537+0000","last_change":"2024-02-10T07:55:03.764447+0000","last_active":"2024-02-10T07:56:11.436537+0000","last_peered":"2024-02-10T07:56:11.436537+0000","last_clean":"2024-02-10T07:56:11.436537+0000","last_became_active":"2024-02-10T07:50:43.282925+0000","last_became_peered":"2024-02-10T07:50:43.282925+0000","last_unstale":"2024-02-10T07:56:11.436537+0000","last_undegraded":"2024-02-10T07:56:11.436537+0000","last_fullsized":"2024-02-10T07:56:11.436537+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:03.764354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:55:03.764354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:19.195051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"430'35","reported_seq":480,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676106+0000","last_change":"2024-02-10T07:56:00.101243+0000","last_active":"2024-02-10T07:56:11.676106+0000","last_peered":"2024-02-10T07:56:11.676106+0000","last_clean":"2024-02-10T07:56:11.676106+0000","last_became_active":"2024-02-10T07:55:31.309201+0000","last_became_peered":"2024-02-10T07:55:31.309201+0000","last_unstale":"2024-02-10T07:56:11.676106+0000","last_undegraded":"2024-02-10T07:56:11.676106+0000","last_fullsized":"2024-02-10T07:56:11.676106+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'35","last_scrub_stamp":"2024-02-10T07:56:00.101117+0000","last_deep_scrub":"430'35","last_deep_scrub_stamp":"2024-02-10T07:56:00.101117+0000","last_clean_scrub_stamp":"2024-02-10T07:56:00.101117+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:22.636831+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":963,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.438129+0000","last_change":"2024-02-10T07:55:09.058032+0000","last_active":"2024-02-10T07:56:11.438129+0000","last_peered":"2024-02-10T07:56:11.438129+0000","last_clean":"2024-02-10T07:56:11.438129+0000","last_became_active":"2024-02-10T07:50:43.284846+0000","last_became_peered":"2024-02-10T07:50:43.284846+0000","last_unstale":"2024-02-10T07:56:11.438129+0000","last_undegraded":"2024-02-10T07:56:11.438129+0000","last_fullsized":"2024-02-10T07:56:11.438129+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:09.057934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:09.057934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:24.605074+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"496'16","reported_seq":306,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437113+0000","last_change":"2024-02-10T07:55:37.330498+0000","last_active":"2024-02-10T07:56:11.437113+0000","last_peered":"2024-02-10T07:56:11.437113+0000","last_clean":"2024-02-10T07:56:11.437113+0000","last_became_active":"2024-02-10T07:55:37.329477+0000","last_became_peered":"2024-02-10T07:55:37.329477+0000","last_unstale":"2024-02-10T07:56:11.437113+0000","last_undegraded":"2024-02-10T07:56:11.437113+0000","last_fullsized":"2024-02-10T07:56:11.437113+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'15","last_scrub_stamp":"2024-02-10T07:54:46.837590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:54:46.837590+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":985,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676241+0000","last_change":"2024-02-10T07:56:11.336996+0000","last_active":"2024-02-10T07:56:11.676241+0000","last_peered":"2024-02-10T07:56:11.676241+0000","last_clean":"2024-02-10T07:56:11.676241+0000","last_became_active":"2024-02-10T07:49:06.257043+0000","last_became_peered":"2024-02-10T07:49:06.257043+0000","last_unstale":"2024-02-10T07:56:11.676241+0000","last_undegraded":"2024-02-10T07:56:11.676241+0000","last_fullsized":"2024-02-10T07:56:11.676241+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:11.336921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:56:11.336921+0000","last_clean_scrub_stamp":"2024-02-10T07:56:11.336921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:27.915986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"430'21","reported_seq":606,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676906+0000","last_change":"2024-02-10T07:55:33.321702+0000","last_active":"2024-02-10T07:56:11.676906+0000","last_peered":"2024-02-10T07:56:11.676906+0000","last_clean":"2024-02-10T07:56:11.676906+0000","last_became_active":"2024-02-10T07:55:31.312535+0000","last_became_peered":"2024-02-10T07:55:31.312535+0000","last_unstale":"2024-02-10T07:56:11.676906+0000","last_undegraded":"2024-02-10T07:56:11.676906+0000","last_fullsized":"2024-02-10T07:56:11.676906+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'21","last_scrub_stamp":"2024-02-10T07:55:33.321610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:33.321610+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:02.787894+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":223,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:17.041837+0000","last_change":"2024-02-10T07:56:17.041861+0000","last_active":"2024-02-10T07:56:17.041837+0000","last_peered":"2024-02-10T07:56:17.041837+0000","last_clean":"2024-02-10T07:56:17.041837+0000","last_became_active":"2024-02-10T07:50:43.281549+0000","last_became_peered":"2024-02-10T07:50:43.281549+0000","last_unstale":"2024-02-10T07:56:17.041837+0000","last_undegraded":"2024-02-10T07:56:17.041837+0000","last_fullsized":"2024-02-10T07:56:17.041837+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:17.041759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:56:17.041759+0000","last_clean_scrub_stamp":"2024-02-10T07:56:17.041759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:20.596782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":54,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437761+0000","last_change":"2024-02-10T07:55:37.334058+0000","last_active":"2024-02-10T07:56:11.437761+0000","last_peered":"2024-02-10T07:56:11.437761+0000","last_clean":"2024-02-10T07:56:11.437761+0000","last_became_active":"2024-02-10T07:55:37.333766+0000","last_became_peered":"2024-02-10T07:55:37.333766+0000","last_unstale":"2024-02-10T07:56:11.437761+0000","last_undegraded":"2024-02-10T07:56:11.437761+0000","last_fullsized":"2024-02-10T07:56:11.437761+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:58.125605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:54:58.125605+0000","last_clean_scrub_stamp":"2024-02-10T07:54:58.125605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":970,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436672+0000","last_change":"2024-02-10T07:55:04.811613+0000","last_active":"2024-02-10T07:56:11.436672+0000","last_peered":"2024-02-10T07:56:11.436672+0000","last_clean":"2024-02-10T07:56:11.436672+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:56:11.436672+0000","last_undegraded":"2024-02-10T07:56:11.436672+0000","last_fullsized":"2024-02-10T07:56:11.436672+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:04.811514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:04.811514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:30.672774+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"430'25","reported_seq":1039,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676084+0000","last_change":"2024-02-10T07:55:21.440664+0000","last_active":"2024-02-10T07:56:11.676084+0000","last_peered":"2024-02-10T07:56:11.676084+0000","last_clean":"2024-02-10T07:56:11.676084+0000","last_became_active":"2024-02-10T07:49:06.260962+0000","last_became_peered":"2024-02-10T07:49:06.260962+0000","last_unstale":"2024-02-10T07:56:11.676084+0000","last_undegraded":"2024-02-10T07:56:11.676084+0000","last_fullsized":"2024-02-10T07:56:11.676084+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'25","last_scrub_stamp":"2024-02-10T07:55:21.440575+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:55:21.440575+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:48.008486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":958,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:12.603510+0000","last_change":"2024-02-10T07:56:12.603535+0000","last_active":"2024-02-10T07:56:12.603510+0000","last_peered":"2024-02-10T07:56:12.603510+0000","last_clean":"2024-02-10T07:56:12.603510+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:56:12.603510+0000","last_undegraded":"2024-02-10T07:56:12.603510+0000","last_fullsized":"2024-02-10T07:56:12.603510+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:12.603415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:56:12.603415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:33.929706+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":641,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676657+0000","last_change":"2024-02-10T07:56:11.265087+0000","last_active":"2024-02-10T07:56:11.676657+0000","last_peered":"2024-02-10T07:56:11.676657+0000","last_clean":"2024-02-10T07:56:11.676657+0000","last_became_active":"2024-02-10T07:55:31.311924+0000","last_became_peered":"2024-02-10T07:55:31.311924+0000","last_unstale":"2024-02-10T07:56:11.676657+0000","last_undegraded":"2024-02-10T07:56:11.676657+0000","last_fullsized":"2024-02-10T07:56:11.676657+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:11.264989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:56:11.264989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:35.015152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1009,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675236+0000","last_change":"2024-02-10T07:55:59.227276+0000","last_active":"2024-02-10T07:56:11.675236+0000","last_peered":"2024-02-10T07:56:11.675236+0000","last_clean":"2024-02-10T07:56:11.675236+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:56:11.675236+0000","last_undegraded":"2024-02-10T07:56:11.675236+0000","last_fullsized":"2024-02-10T07:56:11.675236+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:59.227189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:59.227189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:12.790306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"430'18","reported_seq":314,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437465+0000","last_change":"2024-02-10T07:55:43.231671+0000","last_active":"2024-02-10T07:56:11.437465+0000","last_peered":"2024-02-10T07:56:11.437465+0000","last_clean":"2024-02-10T07:56:11.437465+0000","last_became_active":"2024-02-10T07:49:06.258281+0000","last_became_peered":"2024-02-10T07:49:06.258281+0000","last_unstale":"2024-02-10T07:56:11.437465+0000","last_undegraded":"2024-02-10T07:56:11.437465+0000","last_fullsized":"2024-02-10T07:56:11.437465+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'18","last_scrub_stamp":"2024-02-10T07:55:43.231207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:43.231207+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:07.365676+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":637,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:12.249068+0000","last_change":"2024-02-10T07:56:12.249110+0000","last_active":"2024-02-10T07:56:12.249068+0000","last_peered":"2024-02-10T07:56:12.249068+0000","last_clean":"2024-02-10T07:56:12.249068+0000","last_became_active":"2024-02-10T07:55:31.310282+0000","last_became_peered":"2024-02-10T07:55:31.310282+0000","last_unstale":"2024-02-10T07:56:12.249068+0000","last_undegraded":"2024-02-10T07:56:12.249068+0000","last_fullsized":"2024-02-10T07:56:12.249068+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:12.248988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:56:12.248988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:37.161675+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"245'4","reported_seq":992,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437869+0000","last_change":"2024-02-10T07:56:07.572440+0000","last_active":"2024-02-10T07:56:11.437869+0000","last_peered":"2024-02-10T07:56:11.437869+0000","last_clean":"2024-02-10T07:56:11.437869+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:56:11.437869+0000","last_undegraded":"2024-02-10T07:56:11.437869+0000","last_fullsized":"2024-02-10T07:56:11.437869+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'4","last_scrub_stamp":"2024-02-10T07:56:07.572335+0000","last_deep_scrub":"245'4","last_deep_scrub_stamp":"2024-02-10T07:56:07.572335+0000","last_clean_scrub_stamp":"2024-02-10T07:56:07.572335+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:27.261312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"430'46","reported_seq":610,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675368+0000","last_change":"2024-02-10T07:55:49.141082+0000","last_active":"2024-02-10T07:56:11.675368+0000","last_peered":"2024-02-10T07:56:11.675368+0000","last_clean":"2024-02-10T07:56:11.675368+0000","last_became_active":"2024-02-10T07:55:31.309045+0000","last_became_peered":"2024-02-10T07:55:31.309045+0000","last_unstale":"2024-02-10T07:56:11.675368+0000","last_undegraded":"2024-02-10T07:56:11.675368+0000","last_fullsized":"2024-02-10T07:56:11.675368+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'46","last_scrub_stamp":"2024-02-10T07:55:49.140951+0000","last_deep_scrub":"246'22","last_deep_scrub_stamp":"2024-02-10T07:41:26.771112+0000","last_clean_scrub_stamp":"2024-02-10T07:55:49.140951+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:11.343668+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":1273,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:15.550604+0000","last_change":"2024-02-10T07:56:15.550604+0000","last_active":"2024-02-10T07:56:15.550604+0000","last_peered":"2024-02-10T07:56:15.550604+0000","last_clean":"2024-02-10T07:56:15.550604+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:56:15.550604+0000","last_undegraded":"2024-02-10T07:56:15.550604+0000","last_fullsized":"2024-02-10T07:56:15.550604+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:56:15.550501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:56:15.550501+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:27.495869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":796,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436749+0000","last_change":"2024-02-10T07:54:49.385794+0000","last_active":"2024-02-10T07:56:11.436749+0000","last_peered":"2024-02-10T07:56:11.436749+0000","last_clean":"2024-02-10T07:56:11.436749+0000","last_became_active":"2024-02-10T07:47:47.966183+0000","last_became_peered":"2024-02-10T07:47:47.966183+0000","last_unstale":"2024-02-10T07:56:11.436749+0000","last_undegraded":"2024-02-10T07:56:11.436749+0000","last_fullsized":"2024-02-10T07:56:11.436749+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:49.385673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:54:49.385673+0000","last_clean_scrub_stamp":"2024-02-10T07:54:49.385673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":970,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.438569+0000","last_change":"2024-02-10T07:55:55.559149+0000","last_active":"2024-02-10T07:56:11.438569+0000","last_peered":"2024-02-10T07:56:11.438569+0000","last_clean":"2024-02-10T07:56:11.438569+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:56:11.438569+0000","last_undegraded":"2024-02-10T07:56:11.438569+0000","last_fullsized":"2024-02-10T07:56:11.438569+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:55.559059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:55:55.559059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:19.165727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"430'29","reported_seq":1078,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437414+0000","last_change":"2024-02-10T07:55:06.937722+0000","last_active":"2024-02-10T07:56:11.437414+0000","last_peered":"2024-02-10T07:56:11.437414+0000","last_clean":"2024-02-10T07:56:11.437414+0000","last_became_active":"2024-02-10T07:49:06.243569+0000","last_became_peered":"2024-02-10T07:49:06.243569+0000","last_unstale":"2024-02-10T07:56:11.437414+0000","last_undegraded":"2024-02-10T07:56:11.437414+0000","last_fullsized":"2024-02-10T07:56:11.437414+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'29","last_scrub_stamp":"2024-02-10T07:55:06.937614+0000","last_deep_scrub":"245'17","last_deep_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_clean_scrub_stamp":"2024-02-10T07:55:06.937614+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:12.612438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":223,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.674966+0000","last_change":"2024-02-10T07:56:10.186999+0000","last_active":"2024-02-10T07:56:11.674966+0000","last_peered":"2024-02-10T07:56:11.674966+0000","last_clean":"2024-02-10T07:56:11.674966+0000","last_became_active":"2024-02-10T07:50:43.284749+0000","last_became_peered":"2024-02-10T07:50:43.284749+0000","last_unstale":"2024-02-10T07:56:11.674966+0000","last_undegraded":"2024-02-10T07:56:11.674966+0000","last_fullsized":"2024-02-10T07:56:11.674966+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:10.186920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:56:10.186920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:27.968852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":984,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436261+0000","last_change":"2024-02-10T07:54:49.873554+0000","last_active":"2024-02-10T07:56:11.436261+0000","last_peered":"2024-02-10T07:56:11.436261+0000","last_clean":"2024-02-10T07:56:11.436261+0000","last_became_active":"2024-02-10T07:50:43.281262+0000","last_became_peered":"2024-02-10T07:50:43.281262+0000","last_unstale":"2024-02-10T07:56:11.436261+0000","last_undegraded":"2024-02-10T07:56:11.436261+0000","last_fullsized":"2024-02-10T07:56:11.436261+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:49.873476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_clean_scrub_stamp":"2024-02-10T07:54:49.873476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:19.596744+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":223,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675232+0000","last_change":"2024-02-10T07:55:51.304237+0000","last_active":"2024-02-10T07:56:11.675232+0000","last_peered":"2024-02-10T07:56:11.675232+0000","last_clean":"2024-02-10T07:56:11.675232+0000","last_became_active":"2024-02-10T07:50:43.280617+0000","last_became_peered":"2024-02-10T07:50:43.280617+0000","last_unstale":"2024-02-10T07:56:11.675232+0000","last_undegraded":"2024-02-10T07:56:11.675232+0000","last_fullsized":"2024-02-10T07:56:11.675232+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:51.304158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:55:51.304158+0000","last_clean_scrub_stamp":"2024-02-10T07:55:51.304158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:00.959208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":979,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.438341+0000","last_change":"2024-02-10T07:55:08.700382+0000","last_active":"2024-02-10T07:56:11.438341+0000","last_peered":"2024-02-10T07:56:11.438341+0000","last_clean":"2024-02-10T07:56:11.438341+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:56:11.438341+0000","last_undegraded":"2024-02-10T07:56:11.438341+0000","last_fullsized":"2024-02-10T07:56:11.438341+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:08.700256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:08.700256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:25.597166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"430'51","reported_seq":1068,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675551+0000","last_change":"2024-02-10T07:55:53.162057+0000","last_active":"2024-02-10T07:56:11.675551+0000","last_peered":"2024-02-10T07:56:11.675551+0000","last_clean":"2024-02-10T07:56:11.675551+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:56:11.675551+0000","last_undegraded":"2024-02-10T07:56:11.675551+0000","last_fullsized":"2024-02-10T07:56:11.675551+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'51","last_scrub_stamp":"2024-02-10T07:55:53.161924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:53.161924+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:57.866005+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":981,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676064+0000","last_change":"2024-02-10T07:55:54.391918+0000","last_active":"2024-02-10T07:56:11.676064+0000","last_peered":"2024-02-10T07:56:11.676064+0000","last_clean":"2024-02-10T07:56:11.676064+0000","last_became_active":"2024-02-10T07:50:43.284289+0000","last_became_peered":"2024-02-10T07:50:43.284289+0000","last_unstale":"2024-02-10T07:56:11.676064+0000","last_undegraded":"2024-02-10T07:56:11.676064+0000","last_fullsized":"2024-02-10T07:56:11.676064+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:54.391813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:55:54.391813+0000","last_clean_scrub_stamp":"2024-02-10T07:55:54.391813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:55.249764+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":223,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675346+0000","last_change":"2024-02-10T07:55:38.144855+0000","last_active":"2024-02-10T07:56:11.675346+0000","last_peered":"2024-02-10T07:56:11.675346+0000","last_clean":"2024-02-10T07:56:11.675346+0000","last_became_active":"2024-02-10T07:50:43.294972+0000","last_became_peered":"2024-02-10T07:50:43.294972+0000","last_unstale":"2024-02-10T07:56:11.675346+0000","last_undegraded":"2024-02-10T07:56:11.675346+0000","last_fullsized":"2024-02-10T07:56:11.675346+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:38.144764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:38.144764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:53.577603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":642,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676614+0000","last_change":"2024-02-10T07:55:31.310852+0000","last_active":"2024-02-10T07:56:11.676614+0000","last_peered":"2024-02-10T07:56:11.676614+0000","last_clean":"2024-02-10T07:56:11.676614+0000","last_became_active":"2024-02-10T07:55:31.310032+0000","last_became_peered":"2024-02-10T07:55:31.310032+0000","last_unstale":"2024-02-10T07:56:11.676614+0000","last_undegraded":"2024-02-10T07:56:11.676614+0000","last_fullsized":"2024-02-10T07:56:11.676614+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:52.184663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:54:52.184663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"430'21","reported_seq":992,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676657+0000","last_change":"2024-02-10T07:54:42.585310+0000","last_active":"2024-02-10T07:56:11.676657+0000","last_peered":"2024-02-10T07:56:11.676657+0000","last_clean":"2024-02-10T07:56:11.676657+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:56:11.676657+0000","last_undegraded":"2024-02-10T07:56:11.676657+0000","last_fullsized":"2024-02-10T07:56:11.676657+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'21","last_scrub_stamp":"2024-02-10T07:54:42.585196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:54:42.585196+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":290,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437715+0000","last_change":"2024-02-10T07:54:51.220395+0000","last_active":"2024-02-10T07:56:11.437715+0000","last_peered":"2024-02-10T07:56:11.437715+0000","last_clean":"2024-02-10T07:56:11.437715+0000","last_became_active":"2024-02-10T07:49:06.238367+0000","last_became_peered":"2024-02-10T07:49:06.238367+0000","last_unstale":"2024-02-10T07:56:11.437715+0000","last_undegraded":"2024-02-10T07:56:11.437715+0000","last_fullsized":"2024-02-10T07:56:11.437715+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:51.220315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:54:51.220315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:12.604639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":624,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676085+0000","last_change":"2024-02-10T07:55:59.062555+0000","last_active":"2024-02-10T07:56:11.676085+0000","last_peered":"2024-02-10T07:56:11.676085+0000","last_clean":"2024-02-10T07:56:11.676085+0000","last_became_active":"2024-02-10T07:55:31.310657+0000","last_became_peered":"2024-02-10T07:55:31.310657+0000","last_unstale":"2024-02-10T07:56:11.676085+0000","last_undegraded":"2024-02-10T07:56:11.676085+0000","last_fullsized":"2024-02-10T07:56:11.676085+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:55:59.062446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:55:59.062446+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:20.920685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":955,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437175+0000","last_change":"2024-02-10T07:54:57.888563+0000","last_active":"2024-02-10T07:56:11.437175+0000","last_peered":"2024-02-10T07:56:11.437175+0000","last_clean":"2024-02-10T07:56:11.437175+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:56:11.437175+0000","last_undegraded":"2024-02-10T07:56:11.437175+0000","last_fullsized":"2024-02-10T07:56:11.437175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:57.888457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:54:57.888457+0000","last_clean_scrub_stamp":"2024-02-10T07:54:57.888457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:19.409712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"430'59","reported_seq":1455,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642625+0000","last_change":"2024-02-10T07:55:12.746702+0000","last_active":"2024-02-10T07:56:19.642625+0000","last_peered":"2024-02-10T07:56:19.642625+0000","last_clean":"2024-02-10T07:56:19.642625+0000","last_became_active":"2024-02-10T07:50:43.297262+0000","last_became_peered":"2024-02-10T07:50:43.297262+0000","last_unstale":"2024-02-10T07:56:19.642625+0000","last_undegraded":"2024-02-10T07:56:19.642625+0000","last_fullsized":"2024-02-10T07:56:19.642625+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'59","last_scrub_stamp":"2024-02-10T07:55:12.746601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:12.746601+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:40.616807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":963,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675673+0000","last_change":"2024-02-10T07:54:47.531544+0000","last_active":"2024-02-10T07:56:11.675673+0000","last_peered":"2024-02-10T07:56:11.675673+0000","last_clean":"2024-02-10T07:56:11.675673+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:56:11.675673+0000","last_undegraded":"2024-02-10T07:56:11.675673+0000","last_fullsized":"2024-02-10T07:56:11.675673+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:47.531464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:22.786949+0000","last_clean_scrub_stamp":"2024-02-10T07:54:47.531464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":960,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.438453+0000","last_change":"2024-02-10T07:55:41.578626+0000","last_active":"2024-02-10T07:56:11.438453+0000","last_peered":"2024-02-10T07:56:11.438453+0000","last_clean":"2024-02-10T07:56:11.438453+0000","last_became_active":"2024-02-10T07:47:48.149256+0000","last_became_peered":"2024-02-10T07:47:48.149256+0000","last_unstale":"2024-02-10T07:56:11.438453+0000","last_undegraded":"2024-02-10T07:56:11.438453+0000","last_fullsized":"2024-02-10T07:56:11.438453+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:41.578516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:42.730875+0000","last_clean_scrub_stamp":"2024-02-10T07:55:41.578516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:01.514055+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":992,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642468+0000","last_change":"2024-02-10T07:56:06.806801+0000","last_active":"2024-02-10T07:56:19.642468+0000","last_peered":"2024-02-10T07:56:19.642468+0000","last_clean":"2024-02-10T07:56:19.642468+0000","last_became_active":"2024-02-10T07:50:43.288388+0000","last_became_peered":"2024-02-10T07:50:43.288388+0000","last_unstale":"2024-02-10T07:56:19.642468+0000","last_undegraded":"2024-02-10T07:56:19.642468+0000","last_fullsized":"2024-02-10T07:56:19.642468+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:06.806699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:56:06.806699+0000","last_clean_scrub_stamp":"2024-02-10T07:56:06.806699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:29.965184+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":223,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675039+0000","last_change":"2024-02-10T07:55:43.182777+0000","last_active":"2024-02-10T07:56:11.675039+0000","last_peered":"2024-02-10T07:56:11.675039+0000","last_clean":"2024-02-10T07:56:11.675039+0000","last_became_active":"2024-02-10T07:50:43.287043+0000","last_became_peered":"2024-02-10T07:50:43.287043+0000","last_unstale":"2024-02-10T07:56:11.675039+0000","last_undegraded":"2024-02-10T07:56:11.675039+0000","last_fullsized":"2024-02-10T07:56:11.675039+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:43.182633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:55:43.182633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:03.841170+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"430'21","reported_seq":330,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437624+0000","last_change":"2024-02-10T07:55:59.454998+0000","last_active":"2024-02-10T07:56:11.437624+0000","last_peered":"2024-02-10T07:56:11.437624+0000","last_clean":"2024-02-10T07:56:11.437624+0000","last_became_active":"2024-02-10T07:48:35.057662+0000","last_became_peered":"2024-02-10T07:48:35.057662+0000","last_unstale":"2024-02-10T07:56:11.437624+0000","last_undegraded":"2024-02-10T07:56:11.437624+0000","last_fullsized":"2024-02-10T07:56:11.437624+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'21","last_scrub_stamp":"2024-02-10T07:55:59.454887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:59.454887+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:04.200114+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":215,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676483+0000","last_change":"2024-02-10T07:54:52.289867+0000","last_active":"2024-02-10T07:56:11.676483+0000","last_peered":"2024-02-10T07:56:11.676483+0000","last_clean":"2024-02-10T07:56:11.676483+0000","last_became_active":"2024-02-10T07:50:43.279125+0000","last_became_peered":"2024-02-10T07:50:43.279125+0000","last_unstale":"2024-02-10T07:56:11.676483+0000","last_undegraded":"2024-02-10T07:56:11.676483+0000","last_fullsized":"2024-02-10T07:56:11.676483+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:52.289795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:54:52.289795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:16.210220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":638,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.677014+0000","last_change":"2024-02-10T07:55:34.499797+0000","last_active":"2024-02-10T07:56:11.677014+0000","last_peered":"2024-02-10T07:56:11.677014+0000","last_clean":"2024-02-10T07:56:11.677014+0000","last_became_active":"2024-02-10T07:55:31.309335+0000","last_became_peered":"2024-02-10T07:55:31.309335+0000","last_unstale":"2024-02-10T07:56:11.677014+0000","last_undegraded":"2024-02-10T07:56:11.677014+0000","last_fullsized":"2024-02-10T07:56:11.677014+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:55:34.499623+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_clean_scrub_stamp":"2024-02-10T07:55:34.499623+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:48.456336+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1217,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436379+0000","last_change":"2024-02-10T07:55:49.600394+0000","last_active":"2024-02-10T07:56:11.436379+0000","last_peered":"2024-02-10T07:56:11.436379+0000","last_clean":"2024-02-10T07:56:11.436379+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:56:11.436379+0000","last_undegraded":"2024-02-10T07:56:11.436379+0000","last_fullsized":"2024-02-10T07:56:11.436379+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:55:49.600288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:55:49.600288+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:03.438354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":364,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436244+0000","last_change":"2024-02-10T07:55:39.252525+0000","last_active":"2024-02-10T07:56:11.436244+0000","last_peered":"2024-02-10T07:56:11.436244+0000","last_clean":"2024-02-10T07:56:11.436244+0000","last_became_active":"2024-02-10T07:55:37.329851+0000","last_became_peered":"2024-02-10T07:55:37.329851+0000","last_unstale":"2024-02-10T07:56:11.436244+0000","last_undegraded":"2024-02-10T07:56:11.436244+0000","last_fullsized":"2024-02-10T07:56:11.436244+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:39.252450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:39.252450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:54.582626+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"430'48","reported_seq":1349,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676829+0000","last_change":"2024-02-10T07:56:09.333014+0000","last_active":"2024-02-10T07:56:11.676829+0000","last_peered":"2024-02-10T07:56:11.676829+0000","last_clean":"2024-02-10T07:56:11.676829+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:56:11.676829+0000","last_undegraded":"2024-02-10T07:56:11.676829+0000","last_fullsized":"2024-02-10T07:56:11.676829+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'48","last_scrub_stamp":"2024-02-10T07:56:09.332879+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:56:09.332879+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:12.229760+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":984,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437516+0000","last_change":"2024-02-10T07:54:50.914395+0000","last_active":"2024-02-10T07:56:11.437516+0000","last_peered":"2024-02-10T07:56:11.437516+0000","last_clean":"2024-02-10T07:56:11.437516+0000","last_became_active":"2024-02-10T07:50:43.284708+0000","last_became_peered":"2024-02-10T07:50:43.284708+0000","last_unstale":"2024-02-10T07:56:11.437516+0000","last_undegraded":"2024-02-10T07:56:11.437516+0000","last_fullsized":"2024-02-10T07:56:11.437516+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:50.914315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:54:50.914315+0000","last_clean_scrub_stamp":"2024-02-10T07:54:50.914315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:18.725665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":758,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642367+0000","last_change":"2024-02-10T07:55:17.639799+0000","last_active":"2024-02-10T07:56:19.642367+0000","last_peered":"2024-02-10T07:56:19.642367+0000","last_clean":"2024-02-10T07:56:19.642367+0000","last_became_active":"2024-02-10T07:49:06.270457+0000","last_became_peered":"2024-02-10T07:49:06.270457+0000","last_unstale":"2024-02-10T07:56:19.642367+0000","last_undegraded":"2024-02-10T07:56:19.642367+0000","last_fullsized":"2024-02-10T07:56:19.642367+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:17.639725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:17.639725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:21.835287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":978,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675815+0000","last_change":"2024-02-10T07:55:19.449559+0000","last_active":"2024-02-10T07:56:11.675815+0000","last_peered":"2024-02-10T07:56:11.675815+0000","last_clean":"2024-02-10T07:56:11.675815+0000","last_became_active":"2024-02-10T07:49:06.266625+0000","last_became_peered":"2024-02-10T07:49:06.266625+0000","last_unstale":"2024-02-10T07:56:11.675815+0000","last_undegraded":"2024-02-10T07:56:11.675815+0000","last_fullsized":"2024-02-10T07:56:11.675815+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:19.449485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_clean_scrub_stamp":"2024-02-10T07:55:19.449485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:21.930276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2444,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:17.499887+0000","last_change":"2024-02-10T07:55:10.665617+0000","last_active":"2024-02-10T07:56:17.499887+0000","last_peered":"2024-02-10T07:56:17.499887+0000","last_clean":"2024-02-10T07:56:17.499887+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:56:17.499887+0000","last_undegraded":"2024-02-10T07:56:17.499887+0000","last_fullsized":"2024-02-10T07:56:17.499887+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:55:10.665441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:55:10.665441+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:23.246477+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1460,"num_read_kb":1455,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"430'42","reported_seq":1059,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675651+0000","last_change":"2024-02-10T07:54:52.481546+0000","last_active":"2024-02-10T07:56:11.675651+0000","last_peered":"2024-02-10T07:56:11.675651+0000","last_clean":"2024-02-10T07:56:11.675651+0000","last_became_active":"2024-02-10T07:50:43.291771+0000","last_became_peered":"2024-02-10T07:50:43.291771+0000","last_unstale":"2024-02-10T07:56:11.675651+0000","last_undegraded":"2024-02-10T07:56:11.675651+0000","last_fullsized":"2024-02-10T07:56:11.675651+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'42","last_scrub_stamp":"2024-02-10T07:54:52.481450+0000","last_deep_scrub":"237'18","last_deep_scrub_stamp":"2024-02-10T07:40:31.005376+0000","last_clean_scrub_stamp":"2024-02-10T07:54:52.481450+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":942,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.677224+0000","last_change":"2024-02-10T07:56:07.225072+0000","last_active":"2024-02-10T07:56:11.677224+0000","last_peered":"2024-02-10T07:56:11.677224+0000","last_clean":"2024-02-10T07:56:11.677224+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:56:11.677224+0000","last_undegraded":"2024-02-10T07:56:11.677224+0000","last_fullsized":"2024-02-10T07:56:11.677224+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:07.224956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:56:07.224956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:25.829280+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":979,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642307+0000","last_change":"2024-02-10T07:55:27.841804+0000","last_active":"2024-02-10T07:56:19.642307+0000","last_peered":"2024-02-10T07:56:19.642307+0000","last_clean":"2024-02-10T07:56:19.642307+0000","last_became_active":"2024-02-10T07:50:43.296488+0000","last_became_peered":"2024-02-10T07:50:43.296488+0000","last_unstale":"2024-02-10T07:56:19.642307+0000","last_undegraded":"2024-02-10T07:56:19.642307+0000","last_fullsized":"2024-02-10T07:56:19.642307+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:27.841703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:27.841703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:38.780615+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":992,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642511+0000","last_change":"2024-02-10T07:56:04.855805+0000","last_active":"2024-02-10T07:56:19.642511+0000","last_peered":"2024-02-10T07:56:19.642511+0000","last_clean":"2024-02-10T07:56:19.642511+0000","last_became_active":"2024-02-10T07:50:43.297861+0000","last_became_peered":"2024-02-10T07:50:43.297861+0000","last_unstale":"2024-02-10T07:56:19.642511+0000","last_undegraded":"2024-02-10T07:56:19.642511+0000","last_fullsized":"2024-02-10T07:56:19.642511+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:04.855726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:56:04.855726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:33.028779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"430'47","reported_seq":1042,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676544+0000","last_change":"2024-02-10T07:54:58.487627+0000","last_active":"2024-02-10T07:56:11.676544+0000","last_peered":"2024-02-10T07:56:11.676544+0000","last_clean":"2024-02-10T07:56:11.676544+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:56:11.676544+0000","last_undegraded":"2024-02-10T07:56:11.676544+0000","last_fullsized":"2024-02-10T07:56:11.676544+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'47","last_scrub_stamp":"2024-02-10T07:54:58.487485+0000","last_deep_scrub":"271'37","last_deep_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_clean_scrub_stamp":"2024-02-10T07:54:58.487485+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:15.589043+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1000,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675652+0000","last_change":"2024-02-10T07:55:44.154328+0000","last_active":"2024-02-10T07:56:11.675652+0000","last_peered":"2024-02-10T07:56:11.675652+0000","last_clean":"2024-02-10T07:56:11.675652+0000","last_became_active":"2024-02-10T07:48:35.062118+0000","last_became_peered":"2024-02-10T07:48:35.062118+0000","last_unstale":"2024-02-10T07:56:11.675652+0000","last_undegraded":"2024-02-10T07:56:11.675652+0000","last_fullsized":"2024-02-10T07:56:11.675652+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:44.154228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:00.292320+0000","last_clean_scrub_stamp":"2024-02-10T07:55:44.154228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:53.412497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":861,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:17.423496+0000","last_change":"2024-02-10T07:56:17.423521+0000","last_active":"2024-02-10T07:56:17.423496+0000","last_peered":"2024-02-10T07:56:17.423496+0000","last_clean":"2024-02-10T07:56:17.423496+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:56:17.423496+0000","last_undegraded":"2024-02-10T07:56:17.423496+0000","last_fullsized":"2024-02-10T07:56:17.423496+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:17.423386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:54:49.148869+0000","last_clean_scrub_stamp":"2024-02-10T07:56:17.423386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:45.221256+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":650,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675203+0000","last_change":"2024-02-10T07:55:31.495075+0000","last_active":"2024-02-10T07:56:11.675203+0000","last_peered":"2024-02-10T07:56:11.675203+0000","last_clean":"2024-02-10T07:56:11.675203+0000","last_became_active":"2024-02-10T07:55:31.494836+0000","last_became_peered":"2024-02-10T07:55:31.494836+0000","last_unstale":"2024-02-10T07:56:11.675203+0000","last_undegraded":"2024-02-10T07:56:11.675203+0000","last_fullsized":"2024-02-10T07:56:11.675203+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:53.334334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:54:53.334334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"430'35","reported_seq":798,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642725+0000","last_change":"2024-02-10T07:56:01.753567+0000","last_active":"2024-02-10T07:56:19.642725+0000","last_peered":"2024-02-10T07:56:19.642725+0000","last_clean":"2024-02-10T07:56:19.642725+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:56:19.642725+0000","last_undegraded":"2024-02-10T07:56:19.642725+0000","last_fullsized":"2024-02-10T07:56:19.642725+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'35","last_scrub_stamp":"2024-02-10T07:56:01.753456+0000","last_deep_scrub":"382'25","last_deep_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_clean_scrub_stamp":"2024-02-10T07:56:01.753456+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:02.056304+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"476'11","reported_seq":1267,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676749+0000","last_change":"2024-02-10T07:54:49.551910+0000","last_active":"2024-02-10T07:56:11.676749+0000","last_peered":"2024-02-10T07:56:11.676749+0000","last_clean":"2024-02-10T07:56:11.676749+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:56:11.676749+0000","last_undegraded":"2024-02-10T07:56:11.676749+0000","last_fullsized":"2024-02-10T07:56:11.676749+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'11","last_scrub_stamp":"2024-02-10T07:54:49.551831+0000","last_deep_scrub":"476'11","last_deep_scrub_stamp":"2024-02-10T07:54:49.551831+0000","last_clean_scrub_stamp":"2024-02-10T07:54:49.551831+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:19.380676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":971,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436917+0000","last_change":"2024-02-10T07:55:50.743012+0000","last_active":"2024-02-10T07:56:11.436917+0000","last_peered":"2024-02-10T07:56:11.436917+0000","last_clean":"2024-02-10T07:56:11.436917+0000","last_became_active":"2024-02-10T07:50:43.283161+0000","last_became_peered":"2024-02-10T07:50:43.283161+0000","last_unstale":"2024-02-10T07:56:11.436917+0000","last_undegraded":"2024-02-10T07:56:11.436917+0000","last_fullsized":"2024-02-10T07:56:11.436917+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:50.742905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_clean_scrub_stamp":"2024-02-10T07:55:50.742905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:03.864495+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"296'2","reported_seq":973,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675890+0000","last_change":"2024-02-10T07:54:57.480478+0000","last_active":"2024-02-10T07:56:11.675890+0000","last_peered":"2024-02-10T07:56:11.675890+0000","last_clean":"2024-02-10T07:56:11.675890+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:56:11.675890+0000","last_undegraded":"2024-02-10T07:56:11.675890+0000","last_fullsized":"2024-02-10T07:56:11.675890+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'2","last_scrub_stamp":"2024-02-10T07:54:57.480405+0000","last_deep_scrub":"296'2","last_deep_scrub_stamp":"2024-02-10T07:45:03.233375+0000","last_clean_scrub_stamp":"2024-02-10T07:54:57.480405+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:21.737320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"430'42","reported_seq":1031,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:13.141553+0000","last_change":"2024-02-10T07:56:13.141623+0000","last_active":"2024-02-10T07:56:13.141553+0000","last_peered":"2024-02-10T07:56:13.141553+0000","last_clean":"2024-02-10T07:56:13.141553+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:56:13.141553+0000","last_undegraded":"2024-02-10T07:56:13.141553+0000","last_fullsized":"2024-02-10T07:56:13.141553+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'42","last_scrub_stamp":"2024-02-10T07:56:13.141459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:56:13.141459+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:39.109637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"496'13","reported_seq":723,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.435987+0000","last_change":"2024-02-10T07:55:37.326057+0000","last_active":"2024-02-10T07:56:11.435987+0000","last_peered":"2024-02-10T07:56:11.435987+0000","last_clean":"2024-02-10T07:56:11.435987+0000","last_became_active":"2024-02-10T07:55:37.325698+0000","last_became_peered":"2024-02-10T07:55:37.325698+0000","last_unstale":"2024-02-10T07:56:11.435987+0000","last_undegraded":"2024-02-10T07:56:11.435987+0000","last_fullsized":"2024-02-10T07:56:11.435987+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'12","last_scrub_stamp":"2024-02-10T07:54:43.773982+0000","last_deep_scrub":"376'12","last_deep_scrub_stamp":"2024-02-10T07:54:43.773982+0000","last_clean_scrub_stamp":"2024-02-10T07:54:43.773982+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":968,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437067+0000","last_change":"2024-02-10T07:55:27.965230+0000","last_active":"2024-02-10T07:56:11.437067+0000","last_peered":"2024-02-10T07:56:11.437067+0000","last_clean":"2024-02-10T07:56:11.437067+0000","last_became_active":"2024-02-10T07:50:43.281465+0000","last_became_peered":"2024-02-10T07:50:43.281465+0000","last_unstale":"2024-02-10T07:56:11.437067+0000","last_undegraded":"2024-02-10T07:56:11.437067+0000","last_fullsized":"2024-02-10T07:56:11.437067+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:27.965137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:55:27.965137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:31.429452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":525,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437680+0000","last_change":"2024-02-10T07:55:41.250763+0000","last_active":"2024-02-10T07:56:11.437680+0000","last_peered":"2024-02-10T07:56:11.437680+0000","last_clean":"2024-02-10T07:56:11.437680+0000","last_became_active":"2024-02-10T07:55:37.330301+0000","last_became_peered":"2024-02-10T07:55:37.330301+0000","last_unstale":"2024-02-10T07:56:11.437680+0000","last_undegraded":"2024-02-10T07:56:11.437680+0000","last_fullsized":"2024-02-10T07:56:11.437680+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:41.250667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:55:41.250667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:44.143151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"430'42","reported_seq":99,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.674717+0000","last_change":"2024-02-10T07:54:55.373700+0000","last_active":"2024-02-10T07:56:11.674717+0000","last_peered":"2024-02-10T07:56:11.674717+0000","last_clean":"2024-02-10T07:56:11.674717+0000","last_became_active":"2024-02-10T07:54:35.652681+0000","last_became_peered":"2024-02-10T07:54:35.652681+0000","last_unstale":"2024-02-10T07:56:11.674717+0000","last_undegraded":"2024-02-10T07:56:11.674717+0000","last_fullsized":"2024-02-10T07:56:11.674717+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'42","last_scrub_stamp":"2024-02-10T07:54:55.373497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:54:55.373497+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:20.015616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":970,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436861+0000","last_change":"2024-02-10T07:55:54.183746+0000","last_active":"2024-02-10T07:56:11.436861+0000","last_peered":"2024-02-10T07:56:11.436861+0000","last_clean":"2024-02-10T07:56:11.436861+0000","last_became_active":"2024-02-10T07:49:06.239190+0000","last_became_peered":"2024-02-10T07:49:06.239190+0000","last_unstale":"2024-02-10T07:56:11.436861+0000","last_undegraded":"2024-02-10T07:56:11.436861+0000","last_fullsized":"2024-02-10T07:56:11.436861+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:54.183627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:54.183627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:54.841818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":623,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676667+0000","last_change":"2024-02-10T07:56:08.315139+0000","last_active":"2024-02-10T07:56:11.676667+0000","last_peered":"2024-02-10T07:56:11.676667+0000","last_clean":"2024-02-10T07:56:11.676667+0000","last_became_active":"2024-02-10T07:55:31.313595+0000","last_became_peered":"2024-02-10T07:55:31.313595+0000","last_unstale":"2024-02-10T07:56:11.676667+0000","last_undegraded":"2024-02-10T07:56:11.676667+0000","last_fullsized":"2024-02-10T07:56:11.676667+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:08.314994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:49.704024+0000","last_clean_scrub_stamp":"2024-02-10T07:56:08.314994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:29.920163+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"395'12","reported_seq":409,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676490+0000","last_change":"2024-02-10T07:54:51.114540+0000","last_active":"2024-02-10T07:56:11.676490+0000","last_peered":"2024-02-10T07:56:11.676490+0000","last_clean":"2024-02-10T07:56:11.676490+0000","last_became_active":"2024-02-10T07:48:35.071916+0000","last_became_peered":"2024-02-10T07:48:35.071916+0000","last_unstale":"2024-02-10T07:56:11.676490+0000","last_undegraded":"2024-02-10T07:56:11.676490+0000","last_fullsized":"2024-02-10T07:56:11.676490+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'12","last_scrub_stamp":"2024-02-10T07:54:51.114463+0000","last_deep_scrub":"395'12","last_deep_scrub_stamp":"2024-02-10T07:52:24.850269+0000","last_clean_scrub_stamp":"2024-02-10T07:54:51.114463+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:21.024599+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":290,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436106+0000","last_change":"2024-02-10T07:55:02.973698+0000","last_active":"2024-02-10T07:56:11.436106+0000","last_peered":"2024-02-10T07:56:11.436106+0000","last_clean":"2024-02-10T07:56:11.436106+0000","last_became_active":"2024-02-10T07:49:06.251038+0000","last_became_peered":"2024-02-10T07:49:06.251038+0000","last_unstale":"2024-02-10T07:56:11.436106+0000","last_undegraded":"2024-02-10T07:56:11.436106+0000","last_fullsized":"2024-02-10T07:56:11.436106+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:02.973607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:02.973607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"430'39","reported_seq":1036,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642634+0000","last_change":"2024-02-10T07:55:26.743799+0000","last_active":"2024-02-10T07:56:19.642634+0000","last_peered":"2024-02-10T07:56:19.642634+0000","last_clean":"2024-02-10T07:56:19.642634+0000","last_became_active":"2024-02-10T07:47:47.967427+0000","last_became_peered":"2024-02-10T07:47:47.967427+0000","last_unstale":"2024-02-10T07:56:19.642634+0000","last_undegraded":"2024-02-10T07:56:19.642634+0000","last_fullsized":"2024-02-10T07:56:19.642634+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'39","last_scrub_stamp":"2024-02-10T07:55:26.743712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:26.743712+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:52.234899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":399,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436886+0000","last_change":"2024-02-10T07:55:37.329516+0000","last_active":"2024-02-10T07:56:11.436886+0000","last_peered":"2024-02-10T07:56:11.436886+0000","last_clean":"2024-02-10T07:56:11.436886+0000","last_became_active":"2024-02-10T07:55:37.329226+0000","last_became_peered":"2024-02-10T07:55:37.329226+0000","last_unstale":"2024-02-10T07:56:11.436886+0000","last_undegraded":"2024-02-10T07:56:11.436886+0000","last_fullsized":"2024-02-10T07:56:11.436886+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:39.281620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:39.231605+0000","last_clean_scrub_stamp":"2024-02-10T07:54:39.281620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":597,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.677090+0000","last_change":"2024-02-10T07:55:31.312458+0000","last_active":"2024-02-10T07:56:11.677090+0000","last_peered":"2024-02-10T07:56:11.677090+0000","last_clean":"2024-02-10T07:56:11.677090+0000","last_became_active":"2024-02-10T07:55:31.311729+0000","last_became_peered":"2024-02-10T07:55:31.311729+0000","last_unstale":"2024-02-10T07:56:11.677090+0000","last_undegraded":"2024-02-10T07:56:11.677090+0000","last_fullsized":"2024-02-10T07:56:11.677090+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:14.813005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:14.813005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:31.799191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":975,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437017+0000","last_change":"2024-02-10T07:55:33.012220+0000","last_active":"2024-02-10T07:56:11.437017+0000","last_peered":"2024-02-10T07:56:11.437017+0000","last_clean":"2024-02-10T07:56:11.437017+0000","last_became_active":"2024-02-10T07:50:43.283104+0000","last_became_peered":"2024-02-10T07:50:43.283104+0000","last_unstale":"2024-02-10T07:56:11.437017+0000","last_undegraded":"2024-02-10T07:56:11.437017+0000","last_fullsized":"2024-02-10T07:56:11.437017+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:33.012118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:55:33.012118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:37.908808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"430'32","reported_seq":1052,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.438072+0000","last_change":"2024-02-10T07:54:54.904449+0000","last_active":"2024-02-10T07:56:11.438072+0000","last_peered":"2024-02-10T07:56:11.438072+0000","last_clean":"2024-02-10T07:56:11.438072+0000","last_became_active":"2024-02-10T07:48:35.062573+0000","last_became_peered":"2024-02-10T07:48:35.062573+0000","last_unstale":"2024-02-10T07:56:11.438072+0000","last_undegraded":"2024-02-10T07:56:11.438072+0000","last_fullsized":"2024-02-10T07:56:11.438072+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'32","last_scrub_stamp":"2024-02-10T07:54:54.904336+0000","last_deep_scrub":"430'32","last_deep_scrub_stamp":"2024-02-10T07:54:54.904336+0000","last_clean_scrub_stamp":"2024-02-10T07:54:54.904336+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":223,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:18.147696+0000","last_change":"2024-02-10T07:56:18.147722+0000","last_active":"2024-02-10T07:56:18.147696+0000","last_peered":"2024-02-10T07:56:18.147696+0000","last_clean":"2024-02-10T07:56:18.147696+0000","last_became_active":"2024-02-10T07:50:43.290295+0000","last_became_peered":"2024-02-10T07:50:43.290295+0000","last_unstale":"2024-02-10T07:56:18.147696+0000","last_undegraded":"2024-02-10T07:56:18.147696+0000","last_fullsized":"2024-02-10T07:56:18.147696+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:18.147617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:56:18.147617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:18.989042+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":962,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.641984+0000","last_change":"2024-02-10T07:56:05.822657+0000","last_active":"2024-02-10T07:56:19.641984+0000","last_peered":"2024-02-10T07:56:19.641984+0000","last_clean":"2024-02-10T07:56:19.641984+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:56:19.641984+0000","last_undegraded":"2024-02-10T07:56:19.641984+0000","last_fullsized":"2024-02-10T07:56:19.641984+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:05.822569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:56:05.822569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:19.346262+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":993,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675757+0000","last_change":"2024-02-10T07:55:16.346642+0000","last_active":"2024-02-10T07:56:11.675757+0000","last_peered":"2024-02-10T07:56:11.675757+0000","last_clean":"2024-02-10T07:56:11.675757+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:56:11.675757+0000","last_undegraded":"2024-02-10T07:56:11.675757+0000","last_fullsized":"2024-02-10T07:56:11.675757+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:16.346542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:03.803361+0000","last_clean_scrub_stamp":"2024-02-10T07:55:16.346542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:25.873581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"430'53","reported_seq":1108,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437983+0000","last_change":"2024-02-10T07:55:52.213939+0000","last_active":"2024-02-10T07:56:11.437983+0000","last_peered":"2024-02-10T07:56:11.437983+0000","last_clean":"2024-02-10T07:56:11.437983+0000","last_became_active":"2024-02-10T07:50:43.281879+0000","last_became_peered":"2024-02-10T07:50:43.281879+0000","last_unstale":"2024-02-10T07:56:11.437983+0000","last_undegraded":"2024-02-10T07:56:11.437983+0000","last_fullsized":"2024-02-10T07:56:11.437983+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'53","last_scrub_stamp":"2024-02-10T07:55:52.213839+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_clean_scrub_stamp":"2024-02-10T07:55:52.213839+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:20.262420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":954,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.438318+0000","last_change":"2024-02-10T07:55:03.992982+0000","last_active":"2024-02-10T07:56:11.438318+0000","last_peered":"2024-02-10T07:56:11.438318+0000","last_clean":"2024-02-10T07:56:11.438318+0000","last_became_active":"2024-02-10T07:50:43.290334+0000","last_became_peered":"2024-02-10T07:50:43.290334+0000","last_unstale":"2024-02-10T07:56:11.438318+0000","last_undegraded":"2024-02-10T07:56:11.438318+0000","last_fullsized":"2024-02-10T07:56:11.438318+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:03.992885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:03.992885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"405'12","reported_seq":287,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676349+0000","last_change":"2024-02-10T07:54:50.289202+0000","last_active":"2024-02-10T07:56:11.676349+0000","last_peered":"2024-02-10T07:56:11.676349+0000","last_clean":"2024-02-10T07:56:11.676349+0000","last_became_active":"2024-02-10T07:50:43.280142+0000","last_became_peered":"2024-02-10T07:50:43.280142+0000","last_unstale":"2024-02-10T07:56:11.676349+0000","last_undegraded":"2024-02-10T07:56:11.676349+0000","last_fullsized":"2024-02-10T07:56:11.676349+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'12","last_scrub_stamp":"2024-02-10T07:54:50.289092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:54:50.289092+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":203,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436539+0000","last_change":"2024-02-10T07:55:37.331676+0000","last_active":"2024-02-10T07:56:11.436539+0000","last_peered":"2024-02-10T07:56:11.436539+0000","last_clean":"2024-02-10T07:56:11.436539+0000","last_became_active":"2024-02-10T07:55:37.330932+0000","last_became_peered":"2024-02-10T07:55:37.330932+0000","last_unstale":"2024-02-10T07:56:11.436539+0000","last_undegraded":"2024-02-10T07:56:11.436539+0000","last_fullsized":"2024-02-10T07:56:11.436539+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:14.391787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_clean_scrub_stamp":"2024-02-10T07:55:14.391787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:31.045571+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"430'30","reported_seq":1042,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437471+0000","last_change":"2024-02-10T07:55:48.127773+0000","last_active":"2024-02-10T07:56:11.437471+0000","last_peered":"2024-02-10T07:56:11.437471+0000","last_clean":"2024-02-10T07:56:11.437471+0000","last_became_active":"2024-02-10T07:49:06.238858+0000","last_became_peered":"2024-02-10T07:49:06.238858+0000","last_unstale":"2024-02-10T07:56:11.437471+0000","last_undegraded":"2024-02-10T07:56:11.437471+0000","last_fullsized":"2024-02-10T07:56:11.437471+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'30","last_scrub_stamp":"2024-02-10T07:55:48.127615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:48.127615+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:16.583177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":1455,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1533,"num_read_kb":1290,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":6602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1065,"ondisk_log_size":1065,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038059,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":840900,"kb_used_data":21180,"kb_used_omap":1,"kb_used_meta":819710,"kb_avail":92904252,"statfs":{"total":95995035648,"available":95133954048,"internally_reserved":0,"allocated":21688320,"data_stored":21269048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":839383498},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":1.3129999999999999,"15min":1.3600000000000001},"min":{"1min":0.34300000000000003,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.90500000000000003,"5min":18.202999999999999,"15min":39.494},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.641,"15min":1.446},"min":{"1min":0.372,"5min":0.255,"15min":0.23000000000000001},"max":{"1min":18.158000000000001,"5min":19.745000000000001,"15min":38.524000000000001},"last":0.372}]},{"osd":1,"last update":"Sat Feb 10 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":2.375,"15min":1.798},"min":{"1min":0.35799999999999998,"5min":0.251,"15min":0.24399999999999999},"max":{"1min":0.81299999999999994,"5min":50.149000000000001,"15min":50.149000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":2.3839999999999999,"15min":1.77},"min":{"1min":0.37,"5min":0.26300000000000001,"15min":0.24399999999999999},"max":{"1min":0.81999999999999995,"5min":54.048000000000002,"15min":54.048000000000002},"last":0.41599999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.413,"15min":1.1180000000000001},"min":{"1min":0.35799999999999998,"5min":0.28199999999999997,"15min":0.255},"max":{"1min":16.815999999999999,"5min":16.815999999999999,"15min":16.815999999999999},"last":0.432},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.367,"15min":1.202},"min":{"1min":0.39900000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.745,"5min":8.3949999999999996,"15min":16.295000000000002},"last":0.44}]},{"osd":3,"last update":"Sat Feb 10 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":1.002,"15min":1.0840000000000001},"min":{"1min":0.433,"5min":0.32400000000000001,"15min":0.307},"max":{"1min":0.65200000000000002,"5min":18.053999999999998,"15min":18.053999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.999,"15min":0.97299999999999998},"min":{"1min":0.40300000000000002,"5min":0.31900000000000001,"15min":0.313},"max":{"1min":0.69999999999999996,"5min":6.2839999999999998,"15min":6.2839999999999998},"last":0.35399999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.68300000000000005,"15min":1.1659999999999999},"min":{"1min":0.27900000000000003,"5min":0.24399999999999999,"15min":0.22},"max":{"1min":0.77700000000000002,"5min":5.0679999999999996,"15min":26.001000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.59699999999999998,"15min":1.131},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.248},"max":{"1min":0.75900000000000001,"5min":3.1099999999999999,"15min":27.09},"last":0.308}]},{"osd":5,"last update":"Sat Feb 10 07:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.60099999999999998,"15min":1.002},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.245},"max":{"1min":0.65500000000000003,"5min":3.448,"15min":18.126999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64600000000000002,"15min":1.1180000000000001},"min":{"1min":0.51300000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.81999999999999995,"5min":2.4449999999999998,"15min":19.548999999999999},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.82599999999999996,"15min":1.1479999999999999},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":0.89200000000000002,"5min":7.4329999999999998,"15min":17.832000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.74199999999999999,"15min":1.167},"min":{"1min":0.36799999999999999,"5min":0.30399999999999999,"15min":0.29799999999999999},"max":{"1min":0.80400000000000005,"5min":3.4129999999999998,"15min":20.449999999999999},"last":0.40600000000000003}]}]},{"osd":6,"up_from":50,"seq":214748365104,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":646312,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93098840,"statfs":{"total":95995035648,"available":95333212160,"internally_reserved":0,"allocated":1540096,"data_stored":1148513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":660273606},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.4419999999999999,"15min":1.52},"min":{"1min":0.38900000000000001,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.75600000000000001,"5min":14.500999999999999,"15min":24.727},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.3939999999999999,"15min":1.2490000000000001},"min":{"1min":0.41799999999999998,"5min":0.25700000000000001,"15min":0.23100000000000001},"max":{"1min":0.73999999999999999,"5min":14.808,"15min":29.084},"last":0.63}]},{"osd":1,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":1.8899999999999999,"15min":1.7010000000000001},"min":{"1min":0.36699999999999999,"5min":0.26200000000000001,"15min":0.23799999999999999},"max":{"1min":0.84699999999999998,"5min":18.859000000000002,"15min":37.659999999999997},"last":0.439},{"interface":"front","average":{"1min":0.58299999999999996,"5min":1.474,"15min":1.7330000000000001},"min":{"1min":0.45600000000000002,"5min":0.28999999999999998,"15min":0.253},"max":{"1min":0.755,"5min":18.725999999999999,"15min":37.75},"last":0.61099999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.5089999999999999,"15min":1.3520000000000001},"min":{"1min":0.41599999999999998,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":0.70499999999999996,"5min":11.839,"15min":15.768000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":1.3520000000000001,"15min":1.151},"min":{"1min":0.434,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80700000000000005,"5min":12.013999999999999,"15min":15.728999999999999},"last":0.70899999999999996}]},{"osd":3,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":1.2250000000000001,"15min":1.444},"min":{"1min":0.47399999999999998,"5min":0.33100000000000002,"15min":0.31900000000000001},"max":{"1min":0.874,"5min":5.6500000000000004,"15min":13.279},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.622,"5min":1.0820000000000001,"15min":1.157},"min":{"1min":0.39700000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.89100000000000001,"5min":7.7949999999999999,"15min":7.7949999999999999},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.51800000000000002,"15min":0.83999999999999997},"min":{"1min":0.37,"5min":0.246,"15min":0.21099999999999999},"max":{"1min":0.82499999999999996,"5min":3.6099999999999999,"15min":7.6509999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54600000000000004,"15min":0.86099999999999999},"min":{"1min":0.41899999999999998,"5min":0.27700000000000002,"15min":0.26000000000000001},"max":{"1min":0.997,"5min":2.3220000000000001,"15min":7.0300000000000002},"last":0.498}]},{"osd":5,"last update":"Sat Feb 10 07:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56000000000000005,"15min":1.0509999999999999},"min":{"1min":0.42799999999999999,"5min":0.28799999999999998,"15min":0.24099999999999999},"max":{"1min":0.81100000000000005,"5min":4.5720000000000001,"15min":44.689},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.56799999999999995,"15min":1.169},"min":{"1min":0.38100000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80200000000000005,"5min":2.4460000000000002,"15min":44.465000000000003},"last":0.59499999999999997}]},{"osd":7,"last update":"Sat Feb 10 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.753,"15min":1.3069999999999999},"min":{"1min":0.38300000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.69199999999999995,"5min":4.335,"15min":42.631999999999998},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.77400000000000002,"15min":1.2230000000000001},"min":{"1min":0.374,"5min":0.32500000000000001,"15min":0.315},"max":{"1min":0.81399999999999995,"5min":4.9290000000000003,"15min":44.792999999999999},"last":0.65400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575178,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852616,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892536,"statfs":{"total":95995035648,"available":95121956864,"internally_reserved":0,"allocated":983040,"data_stored":558872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":872085958},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:55:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":2.008,"15min":1.371},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":0.56599999999999995,"5min":14.382999999999999,"15min":14.382999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":2.0430000000000001,"15min":1.3420000000000001},"min":{"1min":0.26600000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.59999999999999998,"5min":14.16,"15min":14.16},"last":0.51300000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":2.641,"15min":1.502},"min":{"1min":0.25900000000000001,"5min":0.246,"15min":0.20399999999999999},"max":{"1min":0.80500000000000005,"5min":20.129999999999999,"15min":20.129999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":2.302,"15min":1.448},"min":{"1min":0.30099999999999999,"5min":0.27000000000000002,"15min":0.224},"max":{"1min":0.65700000000000003,"5min":20.047000000000001,"15min":20.047000000000001},"last":0.57299999999999995}]},{"osd":3,"last update":"Sat Feb 10 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":1.8180000000000001,"15min":1.0449999999999999},"min":{"1min":0.27600000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.58099999999999996,"5min":14.452999999999999,"15min":22.367999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.41999999999999998,"5min":1.931,"15min":1.004},"min":{"1min":0.26700000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.65000000000000002,"5min":13.680999999999999,"15min":22.199999999999999},"last":0.46400000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.363,"15min":2.0590000000000002},"min":{"1min":0.48099999999999998,"5min":0.29499999999999998,"15min":0.28399999999999997},"max":{"1min":0.94999999999999996,"5min":9.2010000000000005,"15min":207.45699999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.62,"5min":1.28,"15min":1.2729999999999999},"min":{"1min":0.41899999999999998,"5min":0.29599999999999999,"15min":0.28899999999999998},"max":{"1min":0.83999999999999997,"5min":19.446999999999999,"15min":32.201000000000001},"last":0.55800000000000005}]},{"osd":5,"last update":"Sat Feb 10 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":1.4350000000000001,"15min":1.905},"min":{"1min":0.44,"5min":0.35899999999999999,"15min":0.308},"max":{"1min":1.0349999999999999,"5min":9.1419999999999995,"15min":98.703000000000003},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.377,"15min":1.861},"min":{"1min":0.432,"5min":0.32600000000000001,"15min":0.30199999999999999},"max":{"1min":3.1589999999999998,"5min":10.210000000000001,"15min":98.664000000000001},"last":0.627}]},{"osd":6,"last update":"Sat Feb 10 07:55:52 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.244,"15min":1.3660000000000001},"min":{"1min":0.442,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":1.0029999999999999,"5min":10.291,"15min":36.808},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":1.637,"15min":1.6140000000000001},"min":{"1min":0.47799999999999998,"5min":0.40100000000000002,"15min":0.33300000000000002},"max":{"1min":1.0549999999999999,"5min":9.1750000000000007,"15min":33.774999999999999},"last":0.76600000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":1.8640000000000001,"15min":1.5},"min":{"1min":0.42499999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98999999999999999,"5min":24.099,"15min":24.099},"last":0.753},{"interface":"front","average":{"1min":0.65800000000000003,"5min":1.3400000000000001,"15min":1.2569999999999999},"min":{"1min":0.48199999999999998,"5min":0.39300000000000002,"15min":0.34300000000000003},"max":{"1min":0.92900000000000005,"5min":10.356999999999999,"15min":10.356999999999999},"last":0.68600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738702,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799288,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92945864,"statfs":{"total":95995035648,"available":95176564736,"internally_reserved":0,"allocated":1568768,"data_stored":1149893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":816642501},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":2.8460000000000001,"15min":2.5089999999999999},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.182},"max":{"1min":0.68899999999999995,"5min":38.679000000000002,"15min":156.16200000000001},"last":0.246},{"interface":"front","average":{"1min":0.36899999999999999,"5min":2.2400000000000002,"15min":2.3650000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.19},"max":{"1min":0.66300000000000003,"5min":40.790999999999997,"15min":156.941},"last":0.48699999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":2.0600000000000001,"15min":1.5189999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.20000000000000001},"max":{"1min":0.53400000000000003,"5min":19.733000000000001,"15min":34.603000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":2.1030000000000002,"15min":1.427},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.751,"5min":10.116,"15min":34.718000000000004},"last":0.40899999999999997}]},{"osd":3,"last update":"Sat Feb 10 07:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":2.0710000000000002,"15min":1.1319999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.65100000000000002,"5min":29.792999999999999,"15min":29.792999999999999},"last":0.371},{"interface":"front","average":{"1min":0.40300000000000002,"5min":2.2930000000000001,"15min":1.202},"min":{"1min":0.29099999999999998,"5min":0.24099999999999999,"15min":0.214},"max":{"1min":0.57299999999999995,"5min":29.506,"15min":29.506},"last":0.39000000000000001}]},{"osd":4,"last update":"Sat Feb 10 07:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.5349999999999999,"15min":1.405},"min":{"1min":0.38700000000000001,"5min":0.28599999999999998,"15min":0.27600000000000002},"max":{"1min":0.873,"5min":16.991,"15min":37.286000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":1.3400000000000001,"15min":1.3129999999999999},"min":{"1min":0.38100000000000001,"5min":0.32300000000000001,"15min":0.27800000000000002},"max":{"1min":0.90900000000000003,"5min":8.5939999999999994,"15min":34.683},"last":0.50800000000000001}]},{"osd":5,"last update":"Sat Feb 10 07:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":1.6499999999999999,"15min":1.788},"min":{"1min":0.42699999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.95399999999999996,"5min":17.062999999999999,"15min":41.429000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":1.357,"15min":1.6659999999999999},"min":{"1min":0.49099999999999999,"5min":0.35999999999999999,"15min":0.308},"max":{"1min":0.95099999999999996,"5min":8.6460000000000008,"15min":41.456000000000003},"last":0.46600000000000003}]},{"osd":6,"last update":"Sat Feb 10 07:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":1.3939999999999999,"15min":1.514},"min":{"1min":0.44900000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.95199999999999996,"5min":8.5630000000000006,"15min":41.587000000000003},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":1.8660000000000001,"15min":1.7270000000000001},"min":{"1min":0.45700000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81499999999999995,"5min":17.027999999999999,"15min":41.256},"last":0.52100000000000002}]},{"osd":7,"last update":"Sat Feb 10 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":1.5680000000000001,"15min":1.4810000000000001},"min":{"1min":0.50900000000000001,"5min":0.39600000000000002,"15min":0.33300000000000002},"max":{"1min":0.88200000000000001,"5min":8.6809999999999992,"15min":34.710999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.67900000000000005,"5min":1.4690000000000001,"15min":1.347},"min":{"1min":0.48799999999999999,"5min":0.45400000000000001,"15min":0.379},"max":{"1min":0.95699999999999996,"5min":8.5999999999999996,"15min":34.997999999999998},"last":0.64900000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411652,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":694312,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":93050840,"statfs":{"total":95995035648,"available":95284060160,"internally_reserved":0,"allocated":950272,"data_stored":558955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":710015429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:55:27 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":2.2759999999999998,"15min":1.7549999999999999},"min":{"1min":0.23699999999999999,"5min":0.20699999999999999,"15min":0.191},"max":{"1min":0.51100000000000001,"5min":32.371000000000002,"15min":65.430999999999997},"last":0.223},{"interface":"front","average":{"1min":0.36399999999999999,"5min":2.4910000000000001,"15min":1.6539999999999999},"min":{"1min":0.23699999999999999,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":0.59199999999999997,"5min":38.173000000000002,"15min":38.173000000000002},"last":0.253}]},{"osd":1,"last update":"Sat Feb 10 07:55:27 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":2.0680000000000001,"15min":1.651},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54700000000000004,"5min":27.606000000000002,"15min":65.795000000000002},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":2.677,"15min":1.635},"min":{"1min":0.26100000000000001,"5min":0.23599999999999999,"15min":0.20499999999999999},"max":{"1min":0.56100000000000005,"5min":29.709,"15min":29.709},"last":0.39200000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":1.288,"15min":0.83399999999999996},"min":{"1min":0.314,"5min":0.217,"15min":0.217},"max":{"1min":0.56499999999999995,"5min":7.0819999999999999,"15min":7.0819999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":1.6759999999999999,"15min":0.90600000000000003},"min":{"1min":0.311,"5min":0.248,"15min":0.23999999999999999},"max":{"1min":0.56999999999999995,"5min":7.7530000000000001,"15min":7.7530000000000001},"last":0.308}]},{"osd":4,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":1.107,"15min":1.135},"min":{"1min":0.50900000000000001,"5min":0.33400000000000002,"15min":0.29599999999999999},"max":{"1min":0.84799999999999998,"5min":8.875,"15min":11.395},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.5640000000000001,"15min":1.262},"min":{"1min":0.45200000000000001,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":0.78900000000000003,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.45700000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":1.3160000000000001,"15min":1.325},"min":{"1min":0.39000000000000001,"5min":0.33400000000000002,"15min":0.32600000000000001},"max":{"1min":0.78800000000000003,"5min":10.807,"15min":25.533000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":1.1819999999999999,"15min":1.252},"min":{"1min":0.47799999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78200000000000003,"5min":8.9299999999999997,"15min":25.503},"last":0.57799999999999996}]},{"osd":6,"last update":"Sat Feb 10 07:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.167,"15min":1.381},"min":{"1min":0.439,"5min":0.36499999999999999,"15min":0.35199999999999998},"max":{"1min":4.4199999999999999,"5min":5.069,"15min":35.603999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":1.4530000000000001,"15min":1.3600000000000001},"min":{"1min":0.48799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80300000000000005,"5min":10.893000000000001,"15min":25.901},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Feb 10 07:55:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.609,"15min":1.4059999999999999},"min":{"1min":0.48899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":8.8239999999999998,"5min":10.859999999999999,"15min":10.859999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.171,"15min":1.4119999999999999},"min":{"1min":0.45900000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85299999999999998,"5min":5.0979999999999999,"15min":65.539000000000001},"last":0.55600000000000005}]}]},{"osd":3,"up_from":347,"seq":1490353651816,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":252160,"kb_used_data":952,"kb_used_omap":5,"kb_used_meta":251194,"kb_avail":93492992,"statfs":{"total":95995035648,"available":95736823808,"internally_reserved":0,"allocated":974848,"data_stored":558431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5625,"internal_metadata":257223175},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":1.8460000000000001,"15min":1.0029999999999999},"min":{"1min":0.20300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.63,"5min":28.247,"15min":28.247},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38,"5min":1.8169999999999999,"15min":1.0820000000000001},"min":{"1min":0.24299999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.55300000000000005,"5min":27.361999999999998,"15min":27.361999999999998},"last":0.47799999999999998}]},{"osd":1,"last update":"Sat Feb 10 07:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":2.6739999999999999,"15min":1.2230000000000001},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.20100000000000001},"max":{"1min":0.57699999999999996,"5min":30.718,"15min":30.718},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":2.319,"15min":1.1459999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.22},"max":{"1min":0.58799999999999997,"5min":32.927999999999997,"15min":32.927999999999997},"last":0.38200000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.7829999999999999,"15min":0.93400000000000005},"min":{"1min":0.30299999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55400000000000005,"5min":18.059000000000001,"15min":18.059000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":2.008,"15min":1.0509999999999999},"min":{"1min":0.24299999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56399999999999995,"5min":18.196000000000002,"15min":18.196000000000002},"last":0.42599999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":1.1990000000000001,"15min":2.0950000000000002},"min":{"1min":0.34300000000000003,"5min":0.29899999999999999,"15min":0.29599999999999999},"max":{"1min":1.038,"5min":7.782,"15min":28.707999999999998},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":1.163,"15min":2.2240000000000002},"min":{"1min":0.39700000000000002,"5min":0.33600000000000002,"15min":0.32400000000000001},"max":{"1min":1.024,"5min":7.8220000000000001,"15min":28.760999999999999},"last":0.59899999999999998}]},{"osd":5,"last update":"Sat Feb 10 07:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.2470000000000001,"15min":1.0860000000000001},"min":{"1min":0.51200000000000001,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":0.97399999999999998,"5min":8.2799999999999994,"15min":15.525},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.2110000000000001,"15min":0.97399999999999998},"min":{"1min":0.374,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94399999999999995,"5min":7.8280000000000003,"15min":10.298},"last":0.61699999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":1.8759999999999999,"15min":2.1819999999999999},"min":{"1min":0.42899999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.92700000000000005,"5min":55.750999999999998,"15min":55.750999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":2.0499999999999998,"15min":2.6070000000000002},"min":{"1min":0.40699999999999997,"5min":0.39800000000000002,"15min":0.39200000000000002},"max":{"1min":0.89200000000000002,"5min":55.807000000000002,"15min":55.807000000000002},"last":0.69099999999999995}]},{"osd":7,"last update":"Sat Feb 10 07:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":1.381,"15min":1.298},"min":{"1min":0.47199999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84199999999999997,"5min":7.8760000000000003,"15min":15.577999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.626,"5min":1.2869999999999999,"15min":1.29},"min":{"1min":0.439,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.95799999999999996,"5min":7.8650000000000002,"15min":7.8650000000000002},"last":0.49299999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953786,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758212,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92986940,"statfs":{"total":95995035648,"available":95218626560,"internally_reserved":0,"allocated":978944,"data_stored":558763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.141,"15min":1.2509999999999999},"min":{"1min":0.35799999999999998,"5min":0.26300000000000001,"15min":0.20499999999999999},"max":{"1min":0.875,"5min":20.004999999999999,"15min":45.610999999999997},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":1.4359999999999999,"15min":1.421},"min":{"1min":0.371,"5min":0.23499999999999999,"15min":0.22700000000000001},"max":{"1min":0.86299999999999999,"5min":20.050000000000001,"15min":45.524999999999999},"last":0.51400000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":1.9990000000000001,"15min":2.0649999999999999},"min":{"1min":0.28999999999999998,"5min":0.25800000000000001,"15min":0.23899999999999999},"max":{"1min":0.73999999999999999,"5min":35.673999999999999,"15min":66.343999999999994},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":2.0960000000000001,"15min":2.0329999999999999},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.83999999999999997,"5min":35.375,"15min":53.536000000000001},"last":0.47799999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.2430000000000001,"15min":1.151},"min":{"1min":0.35199999999999998,"5min":0.28000000000000003,"15min":0.26600000000000001},"max":{"1min":0.79100000000000004,"5min":25.32,"15min":25.32},"last":0.505},{"interface":"front","average":{"1min":0.57399999999999995,"5min":1.1599999999999999,"15min":1.0900000000000001},"min":{"1min":0.40500000000000003,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.82499999999999996,"5min":20.535,"15min":20.535},"last":0.497}]},{"osd":3,"last update":"Sat Feb 10 07:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.1140000000000001,"15min":3.0720000000000001},"min":{"1min":0.374,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.81000000000000005,"5min":20.318000000000001,"15min":66.379999999999995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.625,"5min":1.137,"15min":1.476},"min":{"1min":0.40899999999999997,"5min":0.314,"15min":0.309},"max":{"1min":0.92700000000000005,"5min":22.917000000000002,"15min":22.917000000000002},"last":0.53600000000000003}]},{"osd":5,"last update":"Sat Feb 10 07:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47099999999999997,"15min":0.94799999999999995},"min":{"1min":0.34799999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.69099999999999995,"5min":2.1389999999999998,"15min":10.502000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.48399999999999999,"15min":0.91300000000000003},"min":{"1min":0.35699999999999998,"5min":0.26000000000000001,"15min":0.247},"max":{"1min":0.72499999999999998,"5min":2.0289999999999999,"15min":9.2460000000000004},"last":0.31}]},{"osd":6,"last update":"Sat Feb 10 07:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51900000000000002,"15min":1.179},"min":{"1min":0.39200000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.73299999999999998,"5min":1.2090000000000001,"15min":67.433999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54700000000000004,"15min":1.244},"min":{"1min":0.32700000000000001,"5min":0.29199999999999998,"15min":0.26600000000000001},"max":{"1min":0.70599999999999996,"5min":1.7989999999999999,"15min":66.876000000000005},"last":0.42699999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.65900000000000003,"15min":0.97999999999999998},"min":{"1min":0.45500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75900000000000001,"5min":4.5730000000000004,"15min":7.4930000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.623,"15min":0.97499999999999998},"min":{"1min":0.371,"5min":0.314,"15min":0.29399999999999998},"max":{"1min":0.76200000000000001,"5min":3.8479999999999999,"15min":10.919},"last":0.52400000000000002}]}]},{"osd":5,"up_from":156,"seq":670014898415,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":697772,"kb_used_data":944,"kb_used_omap":4,"kb_used_meta":696571,"kb_avail":93047380,"statfs":{"total":95995035648,"available":95280517120,"internally_reserved":0,"allocated":966656,"data_stored":559577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":713289378},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":1.3320000000000001,"15min":1.637},"min":{"1min":0.38600000000000001,"5min":0.24299999999999999,"15min":0.20899999999999999},"max":{"1min":0.79100000000000004,"5min":24.292999999999999,"15min":52.938000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.458,"5min":1.4990000000000001,"15min":1.9119999999999999},"min":{"1min":0.41799999999999998,"5min":0.24199999999999999,"15min":0.22900000000000001},"max":{"1min":17.600999999999999,"5min":23.908000000000001,"15min":52.805},"last":0.70699999999999996}]},{"osd":1,"last update":"Sat Feb 10 07:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":2.1299999999999999,"15min":1.9399999999999999},"min":{"1min":0.39100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.79700000000000004,"5min":28.780999999999999,"15min":54.298000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.1259999999999999,"15min":1.891},"min":{"1min":0.34300000000000003,"5min":0.254,"15min":0.246},"max":{"1min":0.81399999999999995,"5min":28.356000000000002,"15min":52.969999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Feb 10 07:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":1.2909999999999999,"15min":1.194},"min":{"1min":0.433,"5min":0.29899999999999999,"15min":0.27900000000000003},"max":{"1min":0.872,"5min":10.039999999999999,"15min":10.039999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":1.228,"15min":1.22},"min":{"1min":0.39700000000000002,"5min":0.28799999999999998,"15min":0.28100000000000003},"max":{"1min":0.85099999999999998,"5min":9.6069999999999993,"15min":13.662000000000001},"last":0.59299999999999997}]},{"osd":3,"last update":"Sat Feb 10 07:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.998,"15min":1.712},"min":{"1min":0.48499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.66700000000000004,"5min":97.119,"15min":97.119},"last":0.621},{"interface":"front","average":{"1min":0.58199999999999996,"5min":2.23,"15min":1.661},"min":{"1min":0.375,"5min":0.32700000000000001,"15min":0.28699999999999998},"max":{"1min":0.76500000000000001,"5min":97.141999999999996,"15min":97.141999999999996},"last":0.61299999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.54600000000000004,"15min":1.0980000000000001},"min":{"1min":0.34599999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.73199999999999998,"5min":3.0760000000000001,"15min":8.952},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55300000000000005,"15min":1.0049999999999999},"min":{"1min":0.36399999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":3.5019999999999998,"15min":10.180999999999999},"last":0.42499999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.875,"15min":1.103},"min":{"1min":0.29299999999999998,"5min":0.26500000000000001,"15min":0.23999999999999999},"max":{"1min":0.77400000000000002,"5min":21.748000000000001,"15min":21.748000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.86199999999999999,"15min":1.0389999999999999},"min":{"1min":0.32300000000000001,"5min":0.28599999999999998,"15min":0.26300000000000001},"max":{"1min":0.72999999999999998,"5min":22.187999999999999,"15min":22.187999999999999},"last":0.48399999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.83399999999999996,"15min":1.181},"min":{"1min":0.38900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.94099999999999995,"5min":22.302,"15min":22.302},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.90200000000000002,"15min":1.105},"min":{"1min":0.40500000000000003,"5min":0.34699999999999998,"15min":0.26300000000000001},"max":{"1min":0.92000000000000004,"5min":21.779,"15min":21.779},"last":0.66000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:56:21.434 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T07:56:21.682 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:56:21.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:21 smithi008 ceph-mon[29816]: 3.5 scrub starts 2024-02-10T07:56:21.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:21 smithi008 ceph-mon[29816]: 3.5 scrub ok 2024-02-10T07:56:21.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:21 smithi008 ceph-mon[29816]: pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:56:21.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:21 smithi008 ceph-mon[29816]: 5.5 scrub starts 2024-02-10T07:56:21.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:21 smithi008 ceph-mon[29816]: 5.5 scrub ok 2024-02-10T07:56:21.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:21 smithi008 ceph-mon[29816]: 5.0 scrub starts 2024-02-10T07:56:21.804 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:21 smithi008 ceph-mon[29816]: 5.0 scrub ok 2024-02-10T07:56:21.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:21 smithi008 ceph-mon[40743]: 3.5 scrub starts 2024-02-10T07:56:21.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:21 smithi008 ceph-mon[40743]: 3.5 scrub ok 2024-02-10T07:56:21.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:21 smithi008 ceph-mon[40743]: pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:56:21.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:21 smithi008 ceph-mon[40743]: 5.5 scrub starts 2024-02-10T07:56:21.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:21 smithi008 ceph-mon[40743]: 5.5 scrub ok 2024-02-10T07:56:21.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:21 smithi008 ceph-mon[40743]: 5.0 scrub starts 2024-02-10T07:56:21.805 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:21 smithi008 ceph-mon[40743]: 5.0 scrub ok 2024-02-10T07:56:21.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:21 smithi170 ceph-mon[33394]: 3.5 scrub starts 2024-02-10T07:56:21.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:21 smithi170 ceph-mon[33394]: 3.5 scrub ok 2024-02-10T07:56:21.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:21 smithi170 ceph-mon[33394]: pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-10T07:56:21.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:21 smithi170 ceph-mon[33394]: 5.5 scrub starts 2024-02-10T07:56:21.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:21 smithi170 ceph-mon[33394]: 5.5 scrub ok 2024-02-10T07:56:21.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:21 smithi170 ceph-mon[33394]: 5.0 scrub starts 2024-02-10T07:56:21.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:21 smithi170 ceph-mon[33394]: 5.0 scrub ok 2024-02-10T07:56:22.824 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:22 smithi008 ceph-mon[29816]: from='client.16395 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:56:22.824 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:22 smithi008 ceph-mon[29816]: 6.c deep-scrub starts 2024-02-10T07:56:22.824 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:22 smithi008 ceph-mon[29816]: 6.c deep-scrub ok 2024-02-10T07:56:22.824 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:22 smithi008 ceph-mon[29816]: 5.1d scrub starts 2024-02-10T07:56:22.824 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:22 smithi008 ceph-mon[29816]: 5.1d scrub ok 2024-02-10T07:56:22.825 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:22 smithi008 ceph-mon[29816]: 5.d scrub starts 2024-02-10T07:56:22.825 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:22 smithi008 ceph-mon[29816]: 5.d scrub ok 2024-02-10T07:56:22.825 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:22 smithi008 ceph-mon[40743]: from='client.16395 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:56:22.825 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:22 smithi008 ceph-mon[40743]: 6.c deep-scrub starts 2024-02-10T07:56:22.825 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:22 smithi008 ceph-mon[40743]: 6.c deep-scrub ok 2024-02-10T07:56:22.825 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:22 smithi008 ceph-mon[40743]: 5.1d scrub starts 2024-02-10T07:56:22.825 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:22 smithi008 ceph-mon[40743]: 5.1d scrub ok 2024-02-10T07:56:22.825 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:22 smithi008 ceph-mon[40743]: 5.d scrub starts 2024-02-10T07:56:22.825 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:22 smithi008 ceph-mon[40743]: 5.d scrub ok 2024-02-10T07:56:22.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:22 smithi170 ceph-mon[33394]: from='client.16395 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:56:22.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:22 smithi170 ceph-mon[33394]: 6.c deep-scrub starts 2024-02-10T07:56:22.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:22 smithi170 ceph-mon[33394]: 6.c deep-scrub ok 2024-02-10T07:56:22.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:22 smithi170 ceph-mon[33394]: 5.1d scrub starts 2024-02-10T07:56:22.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:22 smithi170 ceph-mon[33394]: 5.1d scrub ok 2024-02-10T07:56:22.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:22 smithi170 ceph-mon[33394]: 5.d scrub starts 2024-02-10T07:56:22.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:22 smithi170 ceph-mon[33394]: 5.d scrub ok 2024-02-10T07:56:23.620 INFO:teuthology.orchestra.run.smithi008.stdout: 2024-02-10T07:56:23.623 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2024-02-10T07:56:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:23 smithi008 ceph-mon[29816]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:23 smithi008 ceph-mon[29816]: 5.17 scrub starts 2024-02-10T07:56:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:23 smithi008 ceph-mon[29816]: 5.17 scrub ok 2024-02-10T07:56:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:23 smithi008 ceph-mon[29816]: 5.e scrub starts 2024-02-10T07:56:23.832 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:23 smithi008 ceph-mon[29816]: 5.e scrub ok 2024-02-10T07:56:23.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:23 smithi008 ceph-mon[40743]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:23.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:23 smithi008 ceph-mon[40743]: 5.17 scrub starts 2024-02-10T07:56:23.832 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:23 smithi008 ceph-mon[40743]: 5.17 scrub ok 2024-02-10T07:56:23.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:23 smithi008 ceph-mon[40743]: 5.e scrub starts 2024-02-10T07:56:23.833 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:23 smithi008 ceph-mon[40743]: 5.e scrub ok 2024-02-10T07:56:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:23 smithi170 ceph-mon[33394]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:23 smithi170 ceph-mon[33394]: 5.17 scrub starts 2024-02-10T07:56:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:23 smithi170 ceph-mon[33394]: 5.17 scrub ok 2024-02-10T07:56:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:23 smithi170 ceph-mon[33394]: 5.e scrub starts 2024-02-10T07:56:23.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:23 smithi170 ceph-mon[33394]: 5.e scrub ok 2024-02-10T07:56:24.281 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":1139,"stamp":"2024-02-10T07:56:22.246597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3229,"num_read_kb":3075,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":1780561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1307,"ondisk_log_size":1307,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":397,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5541344,"kb_used_data":28764,"kb_used_omap":19,"kb_used_meta":5512044,"kb_avail":744419872,"statfs":{"total":767960285184,"available":762285948928,"internally_reserved":0,"allocated":29454336,"data_stored":26251429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19634,"internal_metadata":5644333902},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002726"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":950,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436958+0000","last_change":"2024-02-10T07:55:12.844519+0000","last_active":"2024-02-10T07:56:11.436958+0000","last_peered":"2024-02-10T07:56:11.436958+0000","last_clean":"2024-02-10T07:56:11.436958+0000","last_became_active":"2024-02-10T07:36:33.754373+0000","last_became_peered":"2024-02-10T07:36:33.754373+0000","last_unstale":"2024-02-10T07:56:11.436958+0000","last_undegraded":"2024-02-10T07:56:11.436958+0000","last_fullsized":"2024-02-10T07:56:11.436958+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:12.844419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:12.844419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:15.833463+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":971,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642227+0000","last_change":"2024-02-10T07:55:59.815245+0000","last_active":"2024-02-10T07:56:19.642227+0000","last_peered":"2024-02-10T07:56:19.642227+0000","last_clean":"2024-02-10T07:56:19.642227+0000","last_became_active":"2024-02-10T07:36:33.751604+0000","last_became_peered":"2024-02-10T07:36:33.751604+0000","last_unstale":"2024-02-10T07:56:19.642227+0000","last_undegraded":"2024-02-10T07:56:19.642227+0000","last_fullsized":"2024-02-10T07:56:19.642227+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:59.815161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:59.815161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:11.379146+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"496'2","reported_seq":1001,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.270870+0000","last_change":"2024-02-10T07:55:39.332198+0000","last_active":"2024-02-10T07:56:20.270870+0000","last_peered":"2024-02-10T07:56:20.270870+0000","last_clean":"2024-02-10T07:56:20.270870+0000","last_became_active":"2024-02-10T07:50:43.281905+0000","last_became_peered":"2024-02-10T07:50:43.281905+0000","last_unstale":"2024-02-10T07:56:20.270870+0000","last_undegraded":"2024-02-10T07:56:20.270870+0000","last_fullsized":"2024-02-10T07:56:20.270870+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'2","last_scrub_stamp":"2024-02-10T07:55:39.332082+0000","last_deep_scrub":"496'2","last_deep_scrub_stamp":"2024-02-10T07:55:39.332082+0000","last_clean_scrub_stamp":"2024-02-10T07:55:39.332082+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:42.221152+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"430'25","reported_seq":1028,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436414+0000","last_change":"2024-02-10T07:55:38.061236+0000","last_active":"2024-02-10T07:56:11.436414+0000","last_peered":"2024-02-10T07:56:11.436414+0000","last_clean":"2024-02-10T07:56:11.436414+0000","last_became_active":"2024-02-10T07:46:10.717475+0000","last_became_peered":"2024-02-10T07:46:10.717475+0000","last_unstale":"2024-02-10T07:56:11.436414+0000","last_undegraded":"2024-02-10T07:56:11.436414+0000","last_fullsized":"2024-02-10T07:56:11.436414+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'25","last_scrub_stamp":"2024-02-10T07:55:38.061140+0000","last_deep_scrub":"287'19","last_deep_scrub_stamp":"2024-02-10T07:44:13.402994+0000","last_clean_scrub_stamp":"2024-02-10T07:55:38.061140+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:51.039567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":291,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.460210+0000","last_change":"2024-02-10T07:54:58.107439+0000","last_active":"2024-02-10T07:56:19.460210+0000","last_peered":"2024-02-10T07:56:19.460210+0000","last_clean":"2024-02-10T07:56:19.460210+0000","last_became_active":"2024-02-10T07:49:06.236235+0000","last_became_peered":"2024-02-10T07:49:06.236235+0000","last_unstale":"2024-02-10T07:56:19.460210+0000","last_undegraded":"2024-02-10T07:56:19.460210+0000","last_fullsized":"2024-02-10T07:56:19.460210+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:58.107337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:15.079428+0000","last_clean_scrub_stamp":"2024-02-10T07:54:58.107337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":954,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436308+0000","last_change":"2024-02-10T07:55:16.854169+0000","last_active":"2024-02-10T07:56:11.436308+0000","last_peered":"2024-02-10T07:56:11.436308+0000","last_clean":"2024-02-10T07:56:11.436308+0000","last_became_active":"2024-02-10T07:31:55.300304+0000","last_became_peered":"2024-02-10T07:31:55.300304+0000","last_unstale":"2024-02-10T07:56:11.436308+0000","last_undegraded":"2024-02-10T07:56:11.436308+0000","last_fullsized":"2024-02-10T07:56:11.436308+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:16.854068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:16.854068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:22.589436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":289,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.460612+0000","last_change":"2024-02-10T07:55:58.207446+0000","last_active":"2024-02-10T07:56:19.460612+0000","last_peered":"2024-02-10T07:56:19.460612+0000","last_clean":"2024-02-10T07:56:19.460612+0000","last_became_active":"2024-02-10T07:55:37.333014+0000","last_became_peered":"2024-02-10T07:55:37.333014+0000","last_unstale":"2024-02-10T07:56:19.460612+0000","last_undegraded":"2024-02-10T07:56:19.460612+0000","last_fullsized":"2024-02-10T07:56:19.460612+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:58.207336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:24.458825+0000","last_clean_scrub_stamp":"2024-02-10T07:55:58.207336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:08.282823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"430'39","reported_seq":127,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.459773+0000","last_change":"2024-02-10T07:56:15.354517+0000","last_active":"2024-02-10T07:56:19.459773+0000","last_peered":"2024-02-10T07:56:19.459773+0000","last_clean":"2024-02-10T07:56:19.459773+0000","last_became_active":"2024-02-10T07:55:37.330931+0000","last_became_peered":"2024-02-10T07:55:37.330931+0000","last_unstale":"2024-02-10T07:56:19.459773+0000","last_undegraded":"2024-02-10T07:56:19.459773+0000","last_fullsized":"2024-02-10T07:56:19.459773+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'39","last_scrub_stamp":"2024-02-10T07:56:15.354439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:56:15.354439+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:16.256553+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":952,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.270317+0000","last_change":"2024-02-10T07:54:54.543925+0000","last_active":"2024-02-10T07:56:20.270317+0000","last_peered":"2024-02-10T07:56:20.270317+0000","last_clean":"2024-02-10T07:56:20.270317+0000","last_became_active":"2024-02-10T07:31:57.313421+0000","last_became_peered":"2024-02-10T07:31:57.313421+0000","last_unstale":"2024-02-10T07:56:20.270317+0000","last_undegraded":"2024-02-10T07:56:20.270317+0000","last_fullsized":"2024-02-10T07:56:20.270317+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:54.543830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:54:54.543830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":946,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675865+0000","last_change":"2024-02-10T07:55:39.378371+0000","last_active":"2024-02-10T07:56:11.675865+0000","last_peered":"2024-02-10T07:56:11.675865+0000","last_clean":"2024-02-10T07:56:11.675865+0000","last_became_active":"2024-02-10T07:36:33.760571+0000","last_became_peered":"2024-02-10T07:36:33.760571+0000","last_unstale":"2024-02-10T07:56:11.675865+0000","last_undegraded":"2024-02-10T07:56:11.675865+0000","last_fullsized":"2024-02-10T07:56:11.675865+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:39.378240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:39.378240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:56.084564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"119'2","reported_seq":609,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676520+0000","last_change":"2024-02-10T07:55:31.314516+0000","last_active":"2024-02-10T07:56:11.676520+0000","last_peered":"2024-02-10T07:56:11.676520+0000","last_clean":"2024-02-10T07:56:11.676520+0000","last_became_active":"2024-02-10T07:55:31.314182+0000","last_became_peered":"2024-02-10T07:55:31.314182+0000","last_unstale":"2024-02-10T07:56:11.676520+0000","last_undegraded":"2024-02-10T07:56:11.676520+0000","last_fullsized":"2024-02-10T07:56:11.676520+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'2","last_scrub_stamp":"2024-02-10T07:55:02.912660+0000","last_deep_scrub":"119'2","last_deep_scrub_stamp":"2024-02-10T07:49:13.218549+0000","last_clean_scrub_stamp":"2024-02-10T07:55:02.912660+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:19.640092+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"430'21","reported_seq":992,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437894+0000","last_change":"2024-02-10T07:55:55.145845+0000","last_active":"2024-02-10T07:56:11.437894+0000","last_peered":"2024-02-10T07:56:11.437894+0000","last_clean":"2024-02-10T07:56:11.437894+0000","last_became_active":"2024-02-10T07:47:47.965930+0000","last_became_peered":"2024-02-10T07:47:47.965930+0000","last_unstale":"2024-02-10T07:56:11.437894+0000","last_undegraded":"2024-02-10T07:56:11.437894+0000","last_fullsized":"2024-02-10T07:56:11.437894+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'21","last_scrub_stamp":"2024-02-10T07:55:55.145674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:55.145674+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:17.552998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":637,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675623+0000","last_change":"2024-02-10T07:55:31.309229+0000","last_active":"2024-02-10T07:56:11.675623+0000","last_peered":"2024-02-10T07:56:11.675623+0000","last_clean":"2024-02-10T07:56:11.675623+0000","last_became_active":"2024-02-10T07:55:31.308892+0000","last_became_peered":"2024-02-10T07:55:31.308892+0000","last_unstale":"2024-02-10T07:56:11.675623+0000","last_undegraded":"2024-02-10T07:56:11.675623+0000","last_fullsized":"2024-02-10T07:56:11.675623+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:16.806296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:16.806296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:41.616615+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":961,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436586+0000","last_change":"2024-02-10T07:56:09.521122+0000","last_active":"2024-02-10T07:56:11.436586+0000","last_peered":"2024-02-10T07:56:11.436586+0000","last_clean":"2024-02-10T07:56:11.436586+0000","last_became_active":"2024-02-10T07:31:55.302836+0000","last_became_peered":"2024-02-10T07:31:55.302836+0000","last_unstale":"2024-02-10T07:56:11.436586+0000","last_undegraded":"2024-02-10T07:56:11.436586+0000","last_fullsized":"2024-02-10T07:56:11.436586+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:09.520986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:56:09.520986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:22.476454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":640,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:14.222707+0000","last_change":"2024-02-10T07:56:14.222738+0000","last_active":"2024-02-10T07:56:14.222707+0000","last_peered":"2024-02-10T07:56:14.222707+0000","last_clean":"2024-02-10T07:56:14.222707+0000","last_became_active":"2024-02-10T07:55:31.309628+0000","last_became_peered":"2024-02-10T07:55:31.309628+0000","last_unstale":"2024-02-10T07:56:14.222707+0000","last_undegraded":"2024-02-10T07:56:14.222707+0000","last_fullsized":"2024-02-10T07:56:14.222707+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:14.222627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:40:31.473686+0000","last_clean_scrub_stamp":"2024-02-10T07:56:14.222627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:42.470351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"430'9","reported_seq":966,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437559+0000","last_change":"2024-02-10T07:54:40.819181+0000","last_active":"2024-02-10T07:56:11.437559+0000","last_peered":"2024-02-10T07:56:11.437559+0000","last_clean":"2024-02-10T07:56:11.437559+0000","last_became_active":"2024-02-10T07:47:47.966093+0000","last_became_peered":"2024-02-10T07:47:47.966093+0000","last_unstale":"2024-02-10T07:56:11.437559+0000","last_undegraded":"2024-02-10T07:56:11.437559+0000","last_fullsized":"2024-02-10T07:56:11.437559+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'9","last_scrub_stamp":"2024-02-10T07:54:40.819097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:54:40.819097+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":961,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:18.253463+0000","last_change":"2024-02-10T07:56:18.253486+0000","last_active":"2024-02-10T07:56:18.253463+0000","last_peered":"2024-02-10T07:56:18.253463+0000","last_clean":"2024-02-10T07:56:18.253463+0000","last_became_active":"2024-02-10T07:36:33.754634+0000","last_became_peered":"2024-02-10T07:36:33.754634+0000","last_unstale":"2024-02-10T07:56:18.253463+0000","last_undegraded":"2024-02-10T07:56:18.253463+0000","last_fullsized":"2024-02-10T07:56:18.253463+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:18.253405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:56:18.253405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:33.380146+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":961,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.641981+0000","last_change":"2024-02-10T07:54:59.789284+0000","last_active":"2024-02-10T07:56:19.641981+0000","last_peered":"2024-02-10T07:56:19.641981+0000","last_clean":"2024-02-10T07:56:19.641981+0000","last_became_active":"2024-02-10T07:31:55.298694+0000","last_became_peered":"2024-02-10T07:31:55.298694+0000","last_unstale":"2024-02-10T07:56:19.641981+0000","last_undegraded":"2024-02-10T07:56:19.641981+0000","last_fullsized":"2024-02-10T07:56:19.641981+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:59.789208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:54:59.789208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":971,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.270773+0000","last_change":"2024-02-10T07:56:15.342712+0000","last_active":"2024-02-10T07:56:20.270773+0000","last_peered":"2024-02-10T07:56:20.270773+0000","last_clean":"2024-02-10T07:56:20.270773+0000","last_became_active":"2024-02-10T07:31:51.286904+0000","last_became_peered":"2024-02-10T07:31:51.286904+0000","last_unstale":"2024-02-10T07:56:20.270773+0000","last_undegraded":"2024-02-10T07:56:20.270773+0000","last_fullsized":"2024-02-10T07:56:20.270773+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:15.342620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:56:15.342620+0000","last_clean_scrub_stamp":"2024-02-10T07:56:15.342620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:42.289567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"430'46","reported_seq":257,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:12.213289+0000","last_change":"2024-02-10T07:56:12.213323+0000","last_active":"2024-02-10T07:56:12.213289+0000","last_peered":"2024-02-10T07:56:12.213289+0000","last_clean":"2024-02-10T07:56:12.213289+0000","last_became_active":"2024-02-10T07:50:43.300153+0000","last_became_peered":"2024-02-10T07:50:43.300153+0000","last_unstale":"2024-02-10T07:56:12.213289+0000","last_undegraded":"2024-02-10T07:56:12.213289+0000","last_fullsized":"2024-02-10T07:56:12.213289+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'46","last_scrub_stamp":"2024-02-10T07:56:12.213228+0000","last_deep_scrub":"89'22","last_deep_scrub_stamp":"2024-02-10T07:33:14.170072+0000","last_clean_scrub_stamp":"2024-02-10T07:56:12.213228+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:23.851735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":401,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.460550+0000","last_change":"2024-02-10T07:55:56.318846+0000","last_active":"2024-02-10T07:56:19.460550+0000","last_peered":"2024-02-10T07:56:19.460550+0000","last_clean":"2024-02-10T07:56:19.460550+0000","last_became_active":"2024-02-10T07:55:37.329874+0000","last_became_peered":"2024-02-10T07:55:37.329874+0000","last_unstale":"2024-02-10T07:56:19.460550+0000","last_undegraded":"2024-02-10T07:56:19.460550+0000","last_fullsized":"2024-02-10T07:56:19.460550+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:56.318750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:56.318750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:19.181640+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":970,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437552+0000","last_change":"2024-02-10T07:54:44.867195+0000","last_active":"2024-02-10T07:56:11.437552+0000","last_peered":"2024-02-10T07:56:11.437552+0000","last_clean":"2024-02-10T07:56:11.437552+0000","last_became_active":"2024-02-10T07:50:43.285201+0000","last_became_peered":"2024-02-10T07:50:43.285201+0000","last_unstale":"2024-02-10T07:56:11.437552+0000","last_undegraded":"2024-02-10T07:56:11.437552+0000","last_fullsized":"2024-02-10T07:56:11.437552+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:44.867113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:41:28.853233+0000","last_clean_scrub_stamp":"2024-02-10T07:54:44.867113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":973,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437984+0000","last_change":"2024-02-10T07:55:11.841026+0000","last_active":"2024-02-10T07:56:11.437984+0000","last_peered":"2024-02-10T07:56:11.437984+0000","last_clean":"2024-02-10T07:56:11.437984+0000","last_became_active":"2024-02-10T07:50:43.281382+0000","last_became_peered":"2024-02-10T07:50:43.281382+0000","last_unstale":"2024-02-10T07:56:11.437984+0000","last_undegraded":"2024-02-10T07:56:11.437984+0000","last_fullsized":"2024-02-10T07:56:11.437984+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:11.840927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:11.840927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:14.450537+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"430'24","reported_seq":807,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.460395+0000","last_change":"2024-02-10T07:56:18.376998+0000","last_active":"2024-02-10T07:56:19.460395+0000","last_peered":"2024-02-10T07:56:19.460395+0000","last_clean":"2024-02-10T07:56:19.460395+0000","last_became_active":"2024-02-10T07:55:37.325786+0000","last_became_peered":"2024-02-10T07:55:37.325786+0000","last_unstale":"2024-02-10T07:56:19.460395+0000","last_undegraded":"2024-02-10T07:56:19.460395+0000","last_fullsized":"2024-02-10T07:56:19.460395+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'24","last_scrub_stamp":"2024-02-10T07:56:18.376910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:56:18.376910+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:42.931449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":993,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437387+0000","last_change":"2024-02-10T07:55:48.558565+0000","last_active":"2024-02-10T07:56:11.437387+0000","last_peered":"2024-02-10T07:56:11.437387+0000","last_clean":"2024-02-10T07:56:11.437387+0000","last_became_active":"2024-02-10T07:50:43.285254+0000","last_became_peered":"2024-02-10T07:50:43.285254+0000","last_unstale":"2024-02-10T07:56:11.437387+0000","last_undegraded":"2024-02-10T07:56:11.437387+0000","last_fullsized":"2024-02-10T07:56:11.437387+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-10T07:55:48.558470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:48.558470+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:15.581859+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"414'9","reported_seq":1299,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676554+0000","last_change":"2024-02-10T07:55:48.183624+0000","last_active":"2024-02-10T07:56:11.676554+0000","last_peered":"2024-02-10T07:56:11.676554+0000","last_clean":"2024-02-10T07:56:11.676554+0000","last_became_active":"2024-02-10T07:31:55.295569+0000","last_became_peered":"2024-02-10T07:31:55.295569+0000","last_unstale":"2024-02-10T07:56:11.676554+0000","last_undegraded":"2024-02-10T07:56:11.676554+0000","last_fullsized":"2024-02-10T07:56:11.676554+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'9","last_scrub_stamp":"2024-02-10T07:55:48.183527+0000","last_deep_scrub":"414'9","last_deep_scrub_stamp":"2024-02-10T07:51:13.306986+0000","last_clean_scrub_stamp":"2024-02-10T07:55:48.183527+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:11.823285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":980,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642179+0000","last_change":"2024-02-10T07:55:35.756308+0000","last_active":"2024-02-10T07:56:19.642179+0000","last_peered":"2024-02-10T07:56:19.642179+0000","last_clean":"2024-02-10T07:56:19.642179+0000","last_became_active":"2024-02-10T07:47:47.966127+0000","last_became_peered":"2024-02-10T07:47:47.966127+0000","last_unstale":"2024-02-10T07:56:19.642179+0000","last_undegraded":"2024-02-10T07:56:19.642179+0000","last_fullsized":"2024-02-10T07:56:19.642179+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-10T07:55:35.756214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:35.756214+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:58.483732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"430'35","reported_seq":1059,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.270850+0000","last_change":"2024-02-10T07:55:59.214996+0000","last_active":"2024-02-10T07:56:20.270850+0000","last_peered":"2024-02-10T07:56:20.270850+0000","last_clean":"2024-02-10T07:56:20.270850+0000","last_became_active":"2024-02-10T07:49:06.261487+0000","last_became_peered":"2024-02-10T07:49:06.261487+0000","last_unstale":"2024-02-10T07:56:20.270850+0000","last_undegraded":"2024-02-10T07:56:20.270850+0000","last_fullsized":"2024-02-10T07:56:20.270850+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'35","last_scrub_stamp":"2024-02-10T07:55:59.214906+0000","last_deep_scrub":"250'25","last_deep_scrub_stamp":"2024-02-10T07:42:17.820959+0000","last_clean_scrub_stamp":"2024-02-10T07:55:59.214906+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:21.584520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":966,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642846+0000","last_change":"2024-02-10T07:56:12.918823+0000","last_active":"2024-02-10T07:56:19.642846+0000","last_peered":"2024-02-10T07:56:19.642846+0000","last_clean":"2024-02-10T07:56:19.642846+0000","last_became_active":"2024-02-10T07:36:33.773085+0000","last_became_peered":"2024-02-10T07:36:33.773085+0000","last_unstale":"2024-02-10T07:56:19.642846+0000","last_undegraded":"2024-02-10T07:56:19.642846+0000","last_fullsized":"2024-02-10T07:56:19.642846+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:12.918725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:44:05.558659+0000","last_clean_scrub_stamp":"2024-02-10T07:56:12.918725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:34.344392+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":970,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.271057+0000","last_change":"2024-02-10T07:55:48.243563+0000","last_active":"2024-02-10T07:56:20.271057+0000","last_peered":"2024-02-10T07:56:20.271057+0000","last_clean":"2024-02-10T07:56:20.271057+0000","last_became_active":"2024-02-10T07:50:43.289011+0000","last_became_peered":"2024-02-10T07:50:43.289011+0000","last_unstale":"2024-02-10T07:56:20.271057+0000","last_undegraded":"2024-02-10T07:56:20.271057+0000","last_fullsized":"2024-02-10T07:56:20.271057+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:48.243485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:55:48.243485+0000","last_clean_scrub_stamp":"2024-02-10T07:55:48.243485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:56.752501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":211,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.459648+0000","last_change":"2024-02-10T07:55:37.330511+0000","last_active":"2024-02-10T07:56:19.459648+0000","last_peered":"2024-02-10T07:56:19.459648+0000","last_clean":"2024-02-10T07:56:19.459648+0000","last_became_active":"2024-02-10T07:55:37.330208+0000","last_became_peered":"2024-02-10T07:55:37.330208+0000","last_unstale":"2024-02-10T07:56:19.459648+0000","last_undegraded":"2024-02-10T07:56:19.459648+0000","last_fullsized":"2024-02-10T07:56:19.459648+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:01.490384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:55:01.490384+0000","last_clean_scrub_stamp":"2024-02-10T07:55:01.490384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:24.405129+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"499'32","reported_seq":1038,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676230+0000","last_change":"2024-02-10T07:55:52.183996+0000","last_active":"2024-02-10T07:56:11.676230+0000","last_peered":"2024-02-10T07:56:11.676230+0000","last_clean":"2024-02-10T07:56:11.676230+0000","last_became_active":"2024-02-10T07:36:33.760275+0000","last_became_peered":"2024-02-10T07:36:33.760275+0000","last_unstale":"2024-02-10T07:56:11.676230+0000","last_undegraded":"2024-02-10T07:56:11.676230+0000","last_fullsized":"2024-02-10T07:56:11.676230+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'32","last_scrub_stamp":"2024-02-10T07:55:52.183856+0000","last_deep_scrub":"499'32","last_deep_scrub_stamp":"2024-02-10T07:55:52.183856+0000","last_clean_scrub_stamp":"2024-02-10T07:55:52.183856+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:05.009917+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":980,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436474+0000","last_change":"2024-02-10T07:55:56.533465+0000","last_active":"2024-02-10T07:56:11.436474+0000","last_peered":"2024-02-10T07:56:11.436474+0000","last_clean":"2024-02-10T07:56:11.436474+0000","last_became_active":"2024-02-10T07:50:43.283688+0000","last_became_peered":"2024-02-10T07:50:43.283688+0000","last_unstale":"2024-02-10T07:56:11.436474+0000","last_undegraded":"2024-02-10T07:56:11.436474+0000","last_fullsized":"2024-02-10T07:56:11.436474+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:56.533379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:38.475358+0000","last_clean_scrub_stamp":"2024-02-10T07:55:56.533379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:59.462630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":984,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675599+0000","last_change":"2024-02-10T07:55:58.252182+0000","last_active":"2024-02-10T07:56:11.675599+0000","last_peered":"2024-02-10T07:56:11.675599+0000","last_clean":"2024-02-10T07:56:11.675599+0000","last_became_active":"2024-02-10T07:31:55.291253+0000","last_became_peered":"2024-02-10T07:31:55.291253+0000","last_unstale":"2024-02-10T07:56:11.675599+0000","last_undegraded":"2024-02-10T07:56:11.675599+0000","last_fullsized":"2024-02-10T07:56:11.675599+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:58.252063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:58.252063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:18.787601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":996,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.271004+0000","last_change":"2024-02-10T07:55:42.252188+0000","last_active":"2024-02-10T07:56:20.271004+0000","last_peered":"2024-02-10T07:56:20.271004+0000","last_clean":"2024-02-10T07:56:20.271004+0000","last_became_active":"2024-02-10T07:49:06.258763+0000","last_became_peered":"2024-02-10T07:49:06.258763+0000","last_unstale":"2024-02-10T07:56:20.271004+0000","last_undegraded":"2024-02-10T07:56:20.271004+0000","last_fullsized":"2024-02-10T07:56:20.271004+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:42.252112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:55:42.252112+0000","last_clean_scrub_stamp":"2024-02-10T07:55:42.252112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:06.348116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"430'14","reported_seq":312,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.459782+0000","last_change":"2024-02-10T07:56:14.318484+0000","last_active":"2024-02-10T07:56:19.459782+0000","last_peered":"2024-02-10T07:56:19.459782+0000","last_clean":"2024-02-10T07:56:19.459782+0000","last_became_active":"2024-02-10T07:49:06.250577+0000","last_became_peered":"2024-02-10T07:49:06.250577+0000","last_unstale":"2024-02-10T07:56:19.459782+0000","last_undegraded":"2024-02-10T07:56:19.459782+0000","last_fullsized":"2024-02-10T07:56:19.459782+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'14","last_scrub_stamp":"2024-02-10T07:56:14.318387+0000","last_deep_scrub":"95'6","last_deep_scrub_stamp":"2024-02-10T07:33:46.438117+0000","last_clean_scrub_stamp":"2024-02-10T07:56:14.318387+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:38.626302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":205,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.460035+0000","last_change":"2024-02-10T07:55:37.331956+0000","last_active":"2024-02-10T07:56:19.460035+0000","last_peered":"2024-02-10T07:56:19.460035+0000","last_clean":"2024-02-10T07:56:19.460035+0000","last_became_active":"2024-02-10T07:55:37.331369+0000","last_became_peered":"2024-02-10T07:55:37.331369+0000","last_unstale":"2024-02-10T07:56:19.460035+0000","last_undegraded":"2024-02-10T07:56:19.460035+0000","last_fullsized":"2024-02-10T07:56:19.460035+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:17.615043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:17.615043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:41.267467+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":962,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436702+0000","last_change":"2024-02-10T07:55:40.553297+0000","last_active":"2024-02-10T07:56:11.436702+0000","last_peered":"2024-02-10T07:56:11.436702+0000","last_clean":"2024-02-10T07:56:11.436702+0000","last_became_active":"2024-02-10T07:31:55.302034+0000","last_became_peered":"2024-02-10T07:31:55.302034+0000","last_unstale":"2024-02-10T07:56:11.436702+0000","last_undegraded":"2024-02-10T07:56:11.436702+0000","last_fullsized":"2024-02-10T07:56:11.436702+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:40.553198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:40.553198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:01.172275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":919,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676690+0000","last_change":"2024-02-10T07:55:57.070134+0000","last_active":"2024-02-10T07:56:11.676690+0000","last_peered":"2024-02-10T07:56:11.676690+0000","last_clean":"2024-02-10T07:56:11.676690+0000","last_became_active":"2024-02-10T07:36:33.752494+0000","last_became_peered":"2024-02-10T07:36:33.752494+0000","last_unstale":"2024-02-10T07:56:11.676690+0000","last_undegraded":"2024-02-10T07:56:11.676690+0000","last_fullsized":"2024-02-10T07:56:11.676690+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:57.069994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:32:53.469958+0000","last_clean_scrub_stamp":"2024-02-10T07:55:57.069994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:01.803000+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"430'27","reported_seq":802,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.459971+0000","last_change":"2024-02-10T07:55:37.328415+0000","last_active":"2024-02-10T07:56:19.459971+0000","last_peered":"2024-02-10T07:56:19.459971+0000","last_clean":"2024-02-10T07:56:19.459971+0000","last_became_active":"2024-02-10T07:55:37.327995+0000","last_became_peered":"2024-02-10T07:55:37.327995+0000","last_unstale":"2024-02-10T07:56:19.459971+0000","last_undegraded":"2024-02-10T07:56:19.459971+0000","last_fullsized":"2024-02-10T07:56:19.459971+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'27","last_scrub_stamp":"2024-02-10T07:55:14.660812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:14.660812+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:31.873409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":953,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.270124+0000","last_change":"2024-02-10T07:54:50.614286+0000","last_active":"2024-02-10T07:56:20.270124+0000","last_peered":"2024-02-10T07:56:20.270124+0000","last_clean":"2024-02-10T07:56:20.270124+0000","last_became_active":"2024-02-10T07:36:33.755697+0000","last_became_peered":"2024-02-10T07:36:33.755697+0000","last_unstale":"2024-02-10T07:56:20.270124+0000","last_undegraded":"2024-02-10T07:56:20.270124+0000","last_fullsized":"2024-02-10T07:56:20.270124+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:50.614178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:45:59.179342+0000","last_clean_scrub_stamp":"2024-02-10T07:54:50.614178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":205,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.459902+0000","last_change":"2024-02-10T07:55:37.331959+0000","last_active":"2024-02-10T07:56:19.459902+0000","last_peered":"2024-02-10T07:56:19.459902+0000","last_clean":"2024-02-10T07:56:19.459902+0000","last_became_active":"2024-02-10T07:55:37.331684+0000","last_became_peered":"2024-02-10T07:55:37.331684+0000","last_unstale":"2024-02-10T07:56:19.459902+0000","last_undegraded":"2024-02-10T07:56:19.459902+0000","last_fullsized":"2024-02-10T07:56:19.459902+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:02.441727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:02.441727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:30.783948+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":971,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.438129+0000","last_change":"2024-02-10T07:55:15.657294+0000","last_active":"2024-02-10T07:56:11.438129+0000","last_peered":"2024-02-10T07:56:11.438129+0000","last_clean":"2024-02-10T07:56:11.438129+0000","last_became_active":"2024-02-10T07:47:47.966472+0000","last_became_peered":"2024-02-10T07:47:47.966472+0000","last_unstale":"2024-02-10T07:56:11.438129+0000","last_undegraded":"2024-02-10T07:56:11.438129+0000","last_fullsized":"2024-02-10T07:56:11.438129+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:15.657153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:15.657153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:44.322928+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"430'25","reported_seq":814,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642222+0000","last_change":"2024-02-10T07:56:13.682987+0000","last_active":"2024-02-10T07:56:19.642222+0000","last_peered":"2024-02-10T07:56:19.642222+0000","last_clean":"2024-02-10T07:56:19.642222+0000","last_became_active":"2024-02-10T07:50:43.296447+0000","last_became_peered":"2024-02-10T07:50:43.296447+0000","last_unstale":"2024-02-10T07:56:19.642222+0000","last_undegraded":"2024-02-10T07:56:19.642222+0000","last_fullsized":"2024-02-10T07:56:19.642222+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'25","last_scrub_stamp":"2024-02-10T07:56:13.682903+0000","last_deep_scrub":"225'13","last_deep_scrub_stamp":"2024-02-10T07:39:32.934269+0000","last_clean_scrub_stamp":"2024-02-10T07:56:13.682903+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:18.120906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":393,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.459933+0000","last_change":"2024-02-10T07:55:37.331174+0000","last_active":"2024-02-10T07:56:19.459933+0000","last_peered":"2024-02-10T07:56:19.459933+0000","last_clean":"2024-02-10T07:56:19.459933+0000","last_became_active":"2024-02-10T07:55:37.330870+0000","last_became_peered":"2024-02-10T07:55:37.330870+0000","last_unstale":"2024-02-10T07:56:19.459933+0000","last_undegraded":"2024-02-10T07:56:19.459933+0000","last_fullsized":"2024-02-10T07:56:19.459933+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:15.492475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:15.492475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:29.933780+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":994,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.644097+0000","last_change":"2024-02-10T07:56:19.644118+0000","last_active":"2024-02-10T07:56:19.644097+0000","last_peered":"2024-02-10T07:56:19.644097+0000","last_clean":"2024-02-10T07:56:19.644097+0000","last_became_active":"2024-02-10T07:49:06.252380+0000","last_became_peered":"2024-02-10T07:49:06.252380+0000","last_unstale":"2024-02-10T07:56:19.644097+0000","last_undegraded":"2024-02-10T07:56:19.644097+0000","last_fullsized":"2024-02-10T07:56:19.644097+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:19.644032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:02.685831+0000","last_clean_scrub_stamp":"2024-02-10T07:56:19.644032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:35.020948+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":962,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437296+0000","last_change":"2024-02-10T07:55:17.821620+0000","last_active":"2024-02-10T07:56:11.437296+0000","last_peered":"2024-02-10T07:56:11.437296+0000","last_clean":"2024-02-10T07:56:11.437296+0000","last_became_active":"2024-02-10T07:31:51.285947+0000","last_became_peered":"2024-02-10T07:31:51.285947+0000","last_unstale":"2024-02-10T07:56:11.437296+0000","last_undegraded":"2024-02-10T07:56:11.437296+0000","last_fullsized":"2024-02-10T07:56:11.437296+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:17.821524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:17.821524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:26.982295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"430'32","reported_seq":1042,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437309+0000","last_change":"2024-02-10T07:55:59.165358+0000","last_active":"2024-02-10T07:56:11.437309+0000","last_peered":"2024-02-10T07:56:11.437309+0000","last_clean":"2024-02-10T07:56:11.437309+0000","last_became_active":"2024-02-10T07:46:10.721336+0000","last_became_peered":"2024-02-10T07:46:10.721336+0000","last_unstale":"2024-02-10T07:56:11.437309+0000","last_undegraded":"2024-02-10T07:56:11.437309+0000","last_fullsized":"2024-02-10T07:56:11.437309+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'32","last_scrub_stamp":"2024-02-10T07:55:59.165240+0000","last_deep_scrub":"120'16","last_deep_scrub_stamp":"2024-02-10T07:34:52.295064+0000","last_clean_scrub_stamp":"2024-02-10T07:55:59.165240+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:19.426052+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":967,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.270488+0000","last_change":"2024-02-10T07:55:37.461141+0000","last_active":"2024-02-10T07:56:20.270488+0000","last_peered":"2024-02-10T07:56:20.270488+0000","last_clean":"2024-02-10T07:56:20.270488+0000","last_became_active":"2024-02-10T07:49:06.261570+0000","last_became_peered":"2024-02-10T07:49:06.261570+0000","last_unstale":"2024-02-10T07:56:20.270488+0000","last_undegraded":"2024-02-10T07:56:20.270488+0000","last_fullsized":"2024-02-10T07:56:20.270488+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:37.461030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:58.487937+0000","last_clean_scrub_stamp":"2024-02-10T07:55:37.461030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:57.234558+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"493'12","reported_seq":797,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.677110+0000","last_change":"2024-02-10T07:55:55.083284+0000","last_active":"2024-02-10T07:56:11.677110+0000","last_peered":"2024-02-10T07:56:11.677110+0000","last_clean":"2024-02-10T07:56:11.677110+0000","last_became_active":"2024-02-10T07:55:31.311897+0000","last_became_peered":"2024-02-10T07:55:31.311897+0000","last_unstale":"2024-02-10T07:56:11.677110+0000","last_undegraded":"2024-02-10T07:56:11.677110+0000","last_fullsized":"2024-02-10T07:56:11.677110+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'12","last_scrub_stamp":"2024-02-10T07:55:55.083179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:55.083179+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:04.202185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":982,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436537+0000","last_change":"2024-02-10T07:55:03.764447+0000","last_active":"2024-02-10T07:56:11.436537+0000","last_peered":"2024-02-10T07:56:11.436537+0000","last_clean":"2024-02-10T07:56:11.436537+0000","last_became_active":"2024-02-10T07:50:43.282925+0000","last_became_peered":"2024-02-10T07:50:43.282925+0000","last_unstale":"2024-02-10T07:56:11.436537+0000","last_undegraded":"2024-02-10T07:56:11.436537+0000","last_fullsized":"2024-02-10T07:56:11.436537+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:03.764354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:06.348322+0000","last_clean_scrub_stamp":"2024-02-10T07:55:03.764354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:19.195051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"430'35","reported_seq":480,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676106+0000","last_change":"2024-02-10T07:56:00.101243+0000","last_active":"2024-02-10T07:56:11.676106+0000","last_peered":"2024-02-10T07:56:11.676106+0000","last_clean":"2024-02-10T07:56:11.676106+0000","last_became_active":"2024-02-10T07:55:31.309201+0000","last_became_peered":"2024-02-10T07:55:31.309201+0000","last_unstale":"2024-02-10T07:56:11.676106+0000","last_undegraded":"2024-02-10T07:56:11.676106+0000","last_fullsized":"2024-02-10T07:56:11.676106+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'35","last_scrub_stamp":"2024-02-10T07:56:00.101117+0000","last_deep_scrub":"430'35","last_deep_scrub_stamp":"2024-02-10T07:56:00.101117+0000","last_clean_scrub_stamp":"2024-02-10T07:56:00.101117+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:22.636831+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":963,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.438129+0000","last_change":"2024-02-10T07:55:09.058032+0000","last_active":"2024-02-10T07:56:11.438129+0000","last_peered":"2024-02-10T07:56:11.438129+0000","last_clean":"2024-02-10T07:56:11.438129+0000","last_became_active":"2024-02-10T07:50:43.284846+0000","last_became_peered":"2024-02-10T07:50:43.284846+0000","last_unstale":"2024-02-10T07:56:11.438129+0000","last_undegraded":"2024-02-10T07:56:11.438129+0000","last_fullsized":"2024-02-10T07:56:11.438129+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:09.057934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:09.057934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:24.605074+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"496'16","reported_seq":309,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.460404+0000","last_change":"2024-02-10T07:55:37.330498+0000","last_active":"2024-02-10T07:56:19.460404+0000","last_peered":"2024-02-10T07:56:19.460404+0000","last_clean":"2024-02-10T07:56:19.460404+0000","last_became_active":"2024-02-10T07:55:37.329477+0000","last_became_peered":"2024-02-10T07:55:37.329477+0000","last_unstale":"2024-02-10T07:56:19.460404+0000","last_undegraded":"2024-02-10T07:56:19.460404+0000","last_fullsized":"2024-02-10T07:56:19.460404+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'15","last_scrub_stamp":"2024-02-10T07:54:46.837590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:54:46.837590+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":987,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.270543+0000","last_change":"2024-02-10T07:56:11.336996+0000","last_active":"2024-02-10T07:56:20.270543+0000","last_peered":"2024-02-10T07:56:20.270543+0000","last_clean":"2024-02-10T07:56:20.270543+0000","last_became_active":"2024-02-10T07:49:06.257043+0000","last_became_peered":"2024-02-10T07:49:06.257043+0000","last_unstale":"2024-02-10T07:56:20.270543+0000","last_undegraded":"2024-02-10T07:56:20.270543+0000","last_fullsized":"2024-02-10T07:56:20.270543+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:11.336921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:56:11.336921+0000","last_clean_scrub_stamp":"2024-02-10T07:56:11.336921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:27.915986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"430'21","reported_seq":606,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676906+0000","last_change":"2024-02-10T07:55:33.321702+0000","last_active":"2024-02-10T07:56:11.676906+0000","last_peered":"2024-02-10T07:56:11.676906+0000","last_clean":"2024-02-10T07:56:11.676906+0000","last_became_active":"2024-02-10T07:55:31.312535+0000","last_became_peered":"2024-02-10T07:55:31.312535+0000","last_unstale":"2024-02-10T07:56:11.676906+0000","last_undegraded":"2024-02-10T07:56:11.676906+0000","last_fullsized":"2024-02-10T07:56:11.676906+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'21","last_scrub_stamp":"2024-02-10T07:55:33.321610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:33.321610+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:02.787894+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":26,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":223,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:17.041837+0000","last_change":"2024-02-10T07:56:17.041861+0000","last_active":"2024-02-10T07:56:17.041837+0000","last_peered":"2024-02-10T07:56:17.041837+0000","last_clean":"2024-02-10T07:56:17.041837+0000","last_became_active":"2024-02-10T07:50:43.281549+0000","last_became_peered":"2024-02-10T07:50:43.281549+0000","last_unstale":"2024-02-10T07:56:17.041837+0000","last_undegraded":"2024-02-10T07:56:17.041837+0000","last_fullsized":"2024-02-10T07:56:17.041837+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:17.041759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:56:17.041759+0000","last_clean_scrub_stamp":"2024-02-10T07:56:17.041759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:20.596782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":56,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.460832+0000","last_change":"2024-02-10T07:55:37.334058+0000","last_active":"2024-02-10T07:56:19.460832+0000","last_peered":"2024-02-10T07:56:19.460832+0000","last_clean":"2024-02-10T07:56:19.460832+0000","last_became_active":"2024-02-10T07:55:37.333766+0000","last_became_peered":"2024-02-10T07:55:37.333766+0000","last_unstale":"2024-02-10T07:56:19.460832+0000","last_undegraded":"2024-02-10T07:56:19.460832+0000","last_fullsized":"2024-02-10T07:56:19.460832+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:58.125605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:54:58.125605+0000","last_clean_scrub_stamp":"2024-02-10T07:54:58.125605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":970,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436672+0000","last_change":"2024-02-10T07:55:04.811613+0000","last_active":"2024-02-10T07:56:11.436672+0000","last_peered":"2024-02-10T07:56:11.436672+0000","last_clean":"2024-02-10T07:56:11.436672+0000","last_became_active":"2024-02-10T07:31:51.284017+0000","last_became_peered":"2024-02-10T07:31:51.284017+0000","last_unstale":"2024-02-10T07:56:11.436672+0000","last_undegraded":"2024-02-10T07:56:11.436672+0000","last_fullsized":"2024-02-10T07:56:11.436672+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:04.811514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:04.811514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:30.672774+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"430'25","reported_seq":1041,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.270881+0000","last_change":"2024-02-10T07:55:21.440664+0000","last_active":"2024-02-10T07:56:20.270881+0000","last_peered":"2024-02-10T07:56:20.270881+0000","last_clean":"2024-02-10T07:56:20.270881+0000","last_became_active":"2024-02-10T07:49:06.260962+0000","last_became_peered":"2024-02-10T07:49:06.260962+0000","last_unstale":"2024-02-10T07:56:20.270881+0000","last_undegraded":"2024-02-10T07:56:20.270881+0000","last_fullsized":"2024-02-10T07:56:20.270881+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'25","last_scrub_stamp":"2024-02-10T07:55:21.440575+0000","last_deep_scrub":"89'13","last_deep_scrub_stamp":"2024-02-10T07:32:56.584957+0000","last_clean_scrub_stamp":"2024-02-10T07:55:21.440575+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:48.008486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":958,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:12.603510+0000","last_change":"2024-02-10T07:56:12.603535+0000","last_active":"2024-02-10T07:56:12.603510+0000","last_peered":"2024-02-10T07:56:12.603510+0000","last_clean":"2024-02-10T07:56:12.603510+0000","last_became_active":"2024-02-10T07:31:57.304827+0000","last_became_peered":"2024-02-10T07:31:57.304827+0000","last_unstale":"2024-02-10T07:56:12.603510+0000","last_undegraded":"2024-02-10T07:56:12.603510+0000","last_fullsized":"2024-02-10T07:56:12.603510+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:12.603415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:56:12.603415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:33.929706+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":641,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676657+0000","last_change":"2024-02-10T07:56:11.265087+0000","last_active":"2024-02-10T07:56:11.676657+0000","last_peered":"2024-02-10T07:56:11.676657+0000","last_clean":"2024-02-10T07:56:11.676657+0000","last_became_active":"2024-02-10T07:55:31.311924+0000","last_became_peered":"2024-02-10T07:55:31.311924+0000","last_unstale":"2024-02-10T07:56:11.676657+0000","last_undegraded":"2024-02-10T07:56:11.676657+0000","last_fullsized":"2024-02-10T07:56:11.676657+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:11.264989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:56:11.264989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:35.015152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1009,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675236+0000","last_change":"2024-02-10T07:55:59.227276+0000","last_active":"2024-02-10T07:56:11.675236+0000","last_peered":"2024-02-10T07:56:11.675236+0000","last_clean":"2024-02-10T07:56:11.675236+0000","last_became_active":"2024-02-10T07:36:33.750625+0000","last_became_peered":"2024-02-10T07:36:33.750625+0000","last_unstale":"2024-02-10T07:56:11.675236+0000","last_undegraded":"2024-02-10T07:56:11.675236+0000","last_fullsized":"2024-02-10T07:56:11.675236+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:59.227189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:59.227189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:12.790306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"430'18","reported_seq":317,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.460711+0000","last_change":"2024-02-10T07:55:43.231671+0000","last_active":"2024-02-10T07:56:19.460711+0000","last_peered":"2024-02-10T07:56:19.460711+0000","last_clean":"2024-02-10T07:56:19.460711+0000","last_became_active":"2024-02-10T07:49:06.258281+0000","last_became_peered":"2024-02-10T07:49:06.258281+0000","last_unstale":"2024-02-10T07:56:19.460711+0000","last_undegraded":"2024-02-10T07:56:19.460711+0000","last_fullsized":"2024-02-10T07:56:19.460711+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'18","last_scrub_stamp":"2024-02-10T07:55:43.231207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:43.231207+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:07.365676+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":637,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:12.249068+0000","last_change":"2024-02-10T07:56:12.249110+0000","last_active":"2024-02-10T07:56:12.249068+0000","last_peered":"2024-02-10T07:56:12.249068+0000","last_clean":"2024-02-10T07:56:12.249068+0000","last_became_active":"2024-02-10T07:55:31.310282+0000","last_became_peered":"2024-02-10T07:55:31.310282+0000","last_unstale":"2024-02-10T07:56:12.249068+0000","last_undegraded":"2024-02-10T07:56:12.249068+0000","last_fullsized":"2024-02-10T07:56:12.249068+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:12.248988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:56:12.248988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:37.161675+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"245'4","reported_seq":992,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437869+0000","last_change":"2024-02-10T07:56:07.572440+0000","last_active":"2024-02-10T07:56:11.437869+0000","last_peered":"2024-02-10T07:56:11.437869+0000","last_clean":"2024-02-10T07:56:11.437869+0000","last_became_active":"2024-02-10T07:31:51.285945+0000","last_became_peered":"2024-02-10T07:31:51.285945+0000","last_unstale":"2024-02-10T07:56:11.437869+0000","last_undegraded":"2024-02-10T07:56:11.437869+0000","last_fullsized":"2024-02-10T07:56:11.437869+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'4","last_scrub_stamp":"2024-02-10T07:56:07.572335+0000","last_deep_scrub":"245'4","last_deep_scrub_stamp":"2024-02-10T07:56:07.572335+0000","last_clean_scrub_stamp":"2024-02-10T07:56:07.572335+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:27.261312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"430'46","reported_seq":610,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675368+0000","last_change":"2024-02-10T07:55:49.141082+0000","last_active":"2024-02-10T07:56:11.675368+0000","last_peered":"2024-02-10T07:56:11.675368+0000","last_clean":"2024-02-10T07:56:11.675368+0000","last_became_active":"2024-02-10T07:55:31.309045+0000","last_became_peered":"2024-02-10T07:55:31.309045+0000","last_unstale":"2024-02-10T07:56:11.675368+0000","last_undegraded":"2024-02-10T07:56:11.675368+0000","last_fullsized":"2024-02-10T07:56:11.675368+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'46","last_scrub_stamp":"2024-02-10T07:55:49.140951+0000","last_deep_scrub":"246'22","last_deep_scrub_stamp":"2024-02-10T07:41:26.771112+0000","last_clean_scrub_stamp":"2024-02-10T07:55:49.140951+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:11.343668+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":59,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":1273,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:15.550604+0000","last_change":"2024-02-10T07:56:15.550604+0000","last_active":"2024-02-10T07:56:15.550604+0000","last_peered":"2024-02-10T07:56:15.550604+0000","last_clean":"2024-02-10T07:56:15.550604+0000","last_became_active":"2024-02-10T07:31:55.301338+0000","last_became_peered":"2024-02-10T07:31:55.301338+0000","last_unstale":"2024-02-10T07:56:15.550604+0000","last_undegraded":"2024-02-10T07:56:15.550604+0000","last_fullsized":"2024-02-10T07:56:15.550604+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-10T07:56:15.550501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:56:15.550501+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:27.495869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":798,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.460287+0000","last_change":"2024-02-10T07:54:49.385794+0000","last_active":"2024-02-10T07:56:19.460287+0000","last_peered":"2024-02-10T07:56:19.460287+0000","last_clean":"2024-02-10T07:56:19.460287+0000","last_became_active":"2024-02-10T07:47:47.966183+0000","last_became_peered":"2024-02-10T07:47:47.966183+0000","last_unstale":"2024-02-10T07:56:19.460287+0000","last_undegraded":"2024-02-10T07:56:19.460287+0000","last_fullsized":"2024-02-10T07:56:19.460287+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:49.385673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:54:49.385673+0000","last_clean_scrub_stamp":"2024-02-10T07:54:49.385673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":970,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.438569+0000","last_change":"2024-02-10T07:55:55.559149+0000","last_active":"2024-02-10T07:56:11.438569+0000","last_peered":"2024-02-10T07:56:11.438569+0000","last_clean":"2024-02-10T07:56:11.438569+0000","last_became_active":"2024-02-10T07:31:51.286614+0000","last_became_peered":"2024-02-10T07:31:51.286614+0000","last_unstale":"2024-02-10T07:56:11.438569+0000","last_undegraded":"2024-02-10T07:56:11.438569+0000","last_fullsized":"2024-02-10T07:56:11.438569+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:55.559059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:31.170544+0000","last_clean_scrub_stamp":"2024-02-10T07:55:55.559059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:19.165727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"430'29","reported_seq":1078,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437414+0000","last_change":"2024-02-10T07:55:06.937722+0000","last_active":"2024-02-10T07:56:11.437414+0000","last_peered":"2024-02-10T07:56:11.437414+0000","last_clean":"2024-02-10T07:56:11.437414+0000","last_became_active":"2024-02-10T07:49:06.243569+0000","last_became_peered":"2024-02-10T07:49:06.243569+0000","last_unstale":"2024-02-10T07:56:11.437414+0000","last_undegraded":"2024-02-10T07:56:11.437414+0000","last_fullsized":"2024-02-10T07:56:11.437414+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'29","last_scrub_stamp":"2024-02-10T07:55:06.937614+0000","last_deep_scrub":"245'17","last_deep_scrub_stamp":"2024-02-10T07:41:21.884664+0000","last_clean_scrub_stamp":"2024-02-10T07:55:06.937614+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:12.612438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":223,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.674966+0000","last_change":"2024-02-10T07:56:10.186999+0000","last_active":"2024-02-10T07:56:11.674966+0000","last_peered":"2024-02-10T07:56:11.674966+0000","last_clean":"2024-02-10T07:56:11.674966+0000","last_became_active":"2024-02-10T07:50:43.284749+0000","last_became_peered":"2024-02-10T07:50:43.284749+0000","last_unstale":"2024-02-10T07:56:11.674966+0000","last_undegraded":"2024-02-10T07:56:11.674966+0000","last_fullsized":"2024-02-10T07:56:11.674966+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:10.186920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:56:10.186920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:27.968852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":984,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436261+0000","last_change":"2024-02-10T07:54:49.873554+0000","last_active":"2024-02-10T07:56:11.436261+0000","last_peered":"2024-02-10T07:56:11.436261+0000","last_clean":"2024-02-10T07:56:11.436261+0000","last_became_active":"2024-02-10T07:50:43.281262+0000","last_became_peered":"2024-02-10T07:50:43.281262+0000","last_unstale":"2024-02-10T07:56:11.436261+0000","last_undegraded":"2024-02-10T07:56:11.436261+0000","last_fullsized":"2024-02-10T07:56:11.436261+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:49.873476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:49:11.361139+0000","last_clean_scrub_stamp":"2024-02-10T07:54:49.873476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:19.596744+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":223,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675232+0000","last_change":"2024-02-10T07:55:51.304237+0000","last_active":"2024-02-10T07:56:11.675232+0000","last_peered":"2024-02-10T07:56:11.675232+0000","last_clean":"2024-02-10T07:56:11.675232+0000","last_became_active":"2024-02-10T07:50:43.280617+0000","last_became_peered":"2024-02-10T07:50:43.280617+0000","last_unstale":"2024-02-10T07:56:11.675232+0000","last_undegraded":"2024-02-10T07:56:11.675232+0000","last_fullsized":"2024-02-10T07:56:11.675232+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:51.304158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:55:51.304158+0000","last_clean_scrub_stamp":"2024-02-10T07:55:51.304158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:00.959208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":979,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.438341+0000","last_change":"2024-02-10T07:55:08.700382+0000","last_active":"2024-02-10T07:56:11.438341+0000","last_peered":"2024-02-10T07:56:11.438341+0000","last_clean":"2024-02-10T07:56:11.438341+0000","last_became_active":"2024-02-10T07:36:33.768710+0000","last_became_peered":"2024-02-10T07:36:33.768710+0000","last_unstale":"2024-02-10T07:56:11.438341+0000","last_undegraded":"2024-02-10T07:56:11.438341+0000","last_fullsized":"2024-02-10T07:56:11.438341+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:08.700256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:08.700256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:25.597166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"430'51","reported_seq":1068,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675551+0000","last_change":"2024-02-10T07:55:53.162057+0000","last_active":"2024-02-10T07:56:11.675551+0000","last_peered":"2024-02-10T07:56:11.675551+0000","last_clean":"2024-02-10T07:56:11.675551+0000","last_became_active":"2024-02-10T07:31:53.300867+0000","last_became_peered":"2024-02-10T07:31:53.300867+0000","last_unstale":"2024-02-10T07:56:11.675551+0000","last_undegraded":"2024-02-10T07:56:11.675551+0000","last_fullsized":"2024-02-10T07:56:11.675551+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'51","last_scrub_stamp":"2024-02-10T07:55:53.161924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:53.161924+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:57.866005+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":983,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.270411+0000","last_change":"2024-02-10T07:55:54.391918+0000","last_active":"2024-02-10T07:56:20.270411+0000","last_peered":"2024-02-10T07:56:20.270411+0000","last_clean":"2024-02-10T07:56:20.270411+0000","last_became_active":"2024-02-10T07:50:43.284289+0000","last_became_peered":"2024-02-10T07:50:43.284289+0000","last_unstale":"2024-02-10T07:56:20.270411+0000","last_undegraded":"2024-02-10T07:56:20.270411+0000","last_fullsized":"2024-02-10T07:56:20.270411+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:54.391813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:55:54.391813+0000","last_clean_scrub_stamp":"2024-02-10T07:55:54.391813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:55.249764+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":223,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675346+0000","last_change":"2024-02-10T07:55:38.144855+0000","last_active":"2024-02-10T07:56:11.675346+0000","last_peered":"2024-02-10T07:56:11.675346+0000","last_clean":"2024-02-10T07:56:11.675346+0000","last_became_active":"2024-02-10T07:50:43.294972+0000","last_became_peered":"2024-02-10T07:50:43.294972+0000","last_unstale":"2024-02-10T07:56:11.675346+0000","last_undegraded":"2024-02-10T07:56:11.675346+0000","last_fullsized":"2024-02-10T07:56:11.675346+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:38.144764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:38.144764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:53.577603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":642,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676614+0000","last_change":"2024-02-10T07:55:31.310852+0000","last_active":"2024-02-10T07:56:11.676614+0000","last_peered":"2024-02-10T07:56:11.676614+0000","last_clean":"2024-02-10T07:56:11.676614+0000","last_became_active":"2024-02-10T07:55:31.310032+0000","last_became_peered":"2024-02-10T07:55:31.310032+0000","last_unstale":"2024-02-10T07:56:11.676614+0000","last_undegraded":"2024-02-10T07:56:11.676614+0000","last_fullsized":"2024-02-10T07:56:11.676614+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:52.184663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:54:52.184663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"430'21","reported_seq":1002,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.270878+0000","last_change":"2024-02-10T07:56:18.295173+0000","last_active":"2024-02-10T07:56:20.270878+0000","last_peered":"2024-02-10T07:56:20.270878+0000","last_clean":"2024-02-10T07:56:20.270878+0000","last_became_active":"2024-02-10T07:31:53.292061+0000","last_became_peered":"2024-02-10T07:31:53.292061+0000","last_unstale":"2024-02-10T07:56:20.270878+0000","last_undegraded":"2024-02-10T07:56:20.270878+0000","last_fullsized":"2024-02-10T07:56:20.270878+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'21","last_scrub_stamp":"2024-02-10T07:56:18.295029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:56:18.295029+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:35.726032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":292,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.460735+0000","last_change":"2024-02-10T07:54:51.220395+0000","last_active":"2024-02-10T07:56:19.460735+0000","last_peered":"2024-02-10T07:56:19.460735+0000","last_clean":"2024-02-10T07:56:19.460735+0000","last_became_active":"2024-02-10T07:49:06.238367+0000","last_became_peered":"2024-02-10T07:49:06.238367+0000","last_unstale":"2024-02-10T07:56:19.460735+0000","last_undegraded":"2024-02-10T07:56:19.460735+0000","last_fullsized":"2024-02-10T07:56:19.460735+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:51.220315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:39.095411+0000","last_clean_scrub_stamp":"2024-02-10T07:54:51.220315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"70'2","reported_seq":624,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676085+0000","last_change":"2024-02-10T07:55:59.062555+0000","last_active":"2024-02-10T07:56:11.676085+0000","last_peered":"2024-02-10T07:56:11.676085+0000","last_clean":"2024-02-10T07:56:11.676085+0000","last_became_active":"2024-02-10T07:55:31.310657+0000","last_became_peered":"2024-02-10T07:55:31.310657+0000","last_unstale":"2024-02-10T07:56:11.676085+0000","last_undegraded":"2024-02-10T07:56:11.676085+0000","last_fullsized":"2024-02-10T07:56:11.676085+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-02-10T07:55:59.062446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:55:59.062446+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:20.920685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":955,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437175+0000","last_change":"2024-02-10T07:54:57.888563+0000","last_active":"2024-02-10T07:56:11.437175+0000","last_peered":"2024-02-10T07:56:11.437175+0000","last_clean":"2024-02-10T07:56:11.437175+0000","last_became_active":"2024-02-10T07:31:51.285652+0000","last_became_peered":"2024-02-10T07:31:51.285652+0000","last_unstale":"2024-02-10T07:56:11.437175+0000","last_undegraded":"2024-02-10T07:56:11.437175+0000","last_fullsized":"2024-02-10T07:56:11.437175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:57.888457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:54:57.888457+0000","last_clean_scrub_stamp":"2024-02-10T07:54:57.888457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:19.409712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"430'59","reported_seq":1455,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642625+0000","last_change":"2024-02-10T07:55:12.746702+0000","last_active":"2024-02-10T07:56:19.642625+0000","last_peered":"2024-02-10T07:56:19.642625+0000","last_clean":"2024-02-10T07:56:19.642625+0000","last_became_active":"2024-02-10T07:50:43.297262+0000","last_became_peered":"2024-02-10T07:50:43.297262+0000","last_unstale":"2024-02-10T07:56:19.642625+0000","last_undegraded":"2024-02-10T07:56:19.642625+0000","last_fullsized":"2024-02-10T07:56:19.642625+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'59","last_scrub_stamp":"2024-02-10T07:55:12.746601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:12.746601+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:40.616807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":973,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.272112+0000","last_change":"2024-02-10T07:56:20.272133+0000","last_active":"2024-02-10T07:56:20.272112+0000","last_peered":"2024-02-10T07:56:20.272112+0000","last_clean":"2024-02-10T07:56:20.272112+0000","last_became_active":"2024-02-10T07:31:55.296243+0000","last_became_peered":"2024-02-10T07:31:55.296243+0000","last_unstale":"2024-02-10T07:56:20.272112+0000","last_undegraded":"2024-02-10T07:56:20.272112+0000","last_fullsized":"2024-02-10T07:56:20.272112+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:20.272030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:42:22.786949+0000","last_clean_scrub_stamp":"2024-02-10T07:56:20.272030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:27.763718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":960,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.438453+0000","last_change":"2024-02-10T07:55:41.578626+0000","last_active":"2024-02-10T07:56:11.438453+0000","last_peered":"2024-02-10T07:56:11.438453+0000","last_clean":"2024-02-10T07:56:11.438453+0000","last_became_active":"2024-02-10T07:47:48.149256+0000","last_became_peered":"2024-02-10T07:47:48.149256+0000","last_unstale":"2024-02-10T07:56:11.438453+0000","last_undegraded":"2024-02-10T07:56:11.438453+0000","last_fullsized":"2024-02-10T07:56:11.438453+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:41.578516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:42.730875+0000","last_clean_scrub_stamp":"2024-02-10T07:55:41.578516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:01.514055+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":992,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642468+0000","last_change":"2024-02-10T07:56:06.806801+0000","last_active":"2024-02-10T07:56:19.642468+0000","last_peered":"2024-02-10T07:56:19.642468+0000","last_clean":"2024-02-10T07:56:19.642468+0000","last_became_active":"2024-02-10T07:50:43.288388+0000","last_became_peered":"2024-02-10T07:50:43.288388+0000","last_unstale":"2024-02-10T07:56:19.642468+0000","last_undegraded":"2024-02-10T07:56:19.642468+0000","last_fullsized":"2024-02-10T07:56:19.642468+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:06.806699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:56:06.806699+0000","last_clean_scrub_stamp":"2024-02-10T07:56:06.806699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:29.965184+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":223,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675039+0000","last_change":"2024-02-10T07:55:43.182777+0000","last_active":"2024-02-10T07:56:11.675039+0000","last_peered":"2024-02-10T07:56:11.675039+0000","last_clean":"2024-02-10T07:56:11.675039+0000","last_became_active":"2024-02-10T07:50:43.287043+0000","last_became_peered":"2024-02-10T07:50:43.287043+0000","last_unstale":"2024-02-10T07:56:11.675039+0000","last_undegraded":"2024-02-10T07:56:11.675039+0000","last_fullsized":"2024-02-10T07:56:11.675039+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:43.182633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:55:43.182633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:03.841170+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00049778500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"430'21","reported_seq":330,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437624+0000","last_change":"2024-02-10T07:55:59.454998+0000","last_active":"2024-02-10T07:56:11.437624+0000","last_peered":"2024-02-10T07:56:11.437624+0000","last_clean":"2024-02-10T07:56:11.437624+0000","last_became_active":"2024-02-10T07:48:35.057662+0000","last_became_peered":"2024-02-10T07:48:35.057662+0000","last_unstale":"2024-02-10T07:56:11.437624+0000","last_undegraded":"2024-02-10T07:56:11.437624+0000","last_fullsized":"2024-02-10T07:56:11.437624+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'21","last_scrub_stamp":"2024-02-10T07:55:59.454887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:59.454887+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:04.200114+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":40,"num_read_kb":29,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":215,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676483+0000","last_change":"2024-02-10T07:54:52.289867+0000","last_active":"2024-02-10T07:56:11.676483+0000","last_peered":"2024-02-10T07:56:11.676483+0000","last_clean":"2024-02-10T07:56:11.676483+0000","last_became_active":"2024-02-10T07:50:43.279125+0000","last_became_peered":"2024-02-10T07:50:43.279125+0000","last_unstale":"2024-02-10T07:56:11.676483+0000","last_undegraded":"2024-02-10T07:56:11.676483+0000","last_fullsized":"2024-02-10T07:56:11.676483+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:52.289795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:54:52.289795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:16.210220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"94'2","reported_seq":638,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.677014+0000","last_change":"2024-02-10T07:55:34.499797+0000","last_active":"2024-02-10T07:56:11.677014+0000","last_peered":"2024-02-10T07:56:11.677014+0000","last_clean":"2024-02-10T07:56:11.677014+0000","last_became_active":"2024-02-10T07:55:31.309335+0000","last_became_peered":"2024-02-10T07:55:31.309335+0000","last_unstale":"2024-02-10T07:56:11.677014+0000","last_undegraded":"2024-02-10T07:56:11.677014+0000","last_fullsized":"2024-02-10T07:56:11.677014+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-10T07:55:34.499623+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-10T07:42:17.190492+0000","last_clean_scrub_stamp":"2024-02-10T07:55:34.499623+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:48.456336+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1217,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436379+0000","last_change":"2024-02-10T07:55:49.600394+0000","last_active":"2024-02-10T07:56:11.436379+0000","last_peered":"2024-02-10T07:56:11.436379+0000","last_clean":"2024-02-10T07:56:11.436379+0000","last_became_active":"2024-02-10T07:31:40.614955+0000","last_became_peered":"2024-02-10T07:31:40.614955+0000","last_unstale":"2024-02-10T07:56:11.436379+0000","last_undegraded":"2024-02-10T07:56:11.436379+0000","last_fullsized":"2024-02-10T07:56:11.436379+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-10T07:55:49.600288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:29:33.358557+0000","last_clean_scrub_stamp":"2024-02-10T07:55:49.600288+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:03.438354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":366,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.459968+0000","last_change":"2024-02-10T07:55:39.252525+0000","last_active":"2024-02-10T07:56:19.459968+0000","last_peered":"2024-02-10T07:56:19.459968+0000","last_clean":"2024-02-10T07:56:19.459968+0000","last_became_active":"2024-02-10T07:55:37.329851+0000","last_became_peered":"2024-02-10T07:55:37.329851+0000","last_unstale":"2024-02-10T07:56:19.459968+0000","last_undegraded":"2024-02-10T07:56:19.459968+0000","last_fullsized":"2024-02-10T07:56:19.459968+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:39.252450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:39.252450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:54.582626+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"430'48","reported_seq":1349,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676829+0000","last_change":"2024-02-10T07:56:09.333014+0000","last_active":"2024-02-10T07:56:11.676829+0000","last_peered":"2024-02-10T07:56:11.676829+0000","last_clean":"2024-02-10T07:56:11.676829+0000","last_became_active":"2024-02-10T07:31:53.290542+0000","last_became_peered":"2024-02-10T07:31:53.290542+0000","last_unstale":"2024-02-10T07:56:11.676829+0000","last_undegraded":"2024-02-10T07:56:11.676829+0000","last_fullsized":"2024-02-10T07:56:11.676829+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'48","last_scrub_stamp":"2024-02-10T07:56:09.332879+0000","last_deep_scrub":"89'24","last_deep_scrub_stamp":"2024-02-10T07:33:11.841200+0000","last_clean_scrub_stamp":"2024-02-10T07:56:09.332879+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:12.229760+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":984,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437516+0000","last_change":"2024-02-10T07:54:50.914395+0000","last_active":"2024-02-10T07:56:11.437516+0000","last_peered":"2024-02-10T07:56:11.437516+0000","last_clean":"2024-02-10T07:56:11.437516+0000","last_became_active":"2024-02-10T07:50:43.284708+0000","last_became_peered":"2024-02-10T07:50:43.284708+0000","last_unstale":"2024-02-10T07:56:11.437516+0000","last_undegraded":"2024-02-10T07:56:11.437516+0000","last_fullsized":"2024-02-10T07:56:11.437516+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:50.914315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:54:50.914315+0000","last_clean_scrub_stamp":"2024-02-10T07:54:50.914315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:18.725665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":758,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642367+0000","last_change":"2024-02-10T07:55:17.639799+0000","last_active":"2024-02-10T07:56:19.642367+0000","last_peered":"2024-02-10T07:56:19.642367+0000","last_clean":"2024-02-10T07:56:19.642367+0000","last_became_active":"2024-02-10T07:49:06.270457+0000","last_became_peered":"2024-02-10T07:49:06.270457+0000","last_unstale":"2024-02-10T07:56:19.642367+0000","last_undegraded":"2024-02-10T07:56:19.642367+0000","last_fullsized":"2024-02-10T07:56:19.642367+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:17.639725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:34:54.269886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:17.639725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:21.835287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":980,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.270271+0000","last_change":"2024-02-10T07:55:19.449559+0000","last_active":"2024-02-10T07:56:20.270271+0000","last_peered":"2024-02-10T07:56:20.270271+0000","last_clean":"2024-02-10T07:56:20.270271+0000","last_became_active":"2024-02-10T07:49:06.266625+0000","last_became_peered":"2024-02-10T07:49:06.266625+0000","last_unstale":"2024-02-10T07:56:20.270271+0000","last_undegraded":"2024-02-10T07:56:20.270271+0000","last_fullsized":"2024-02-10T07:56:20.270271+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:19.449485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:47:41.181770+0000","last_clean_scrub_stamp":"2024-02-10T07:55:19.449485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:21.930276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2444,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:17.499887+0000","last_change":"2024-02-10T07:55:10.665617+0000","last_active":"2024-02-10T07:56:17.499887+0000","last_peered":"2024-02-10T07:56:17.499887+0000","last_clean":"2024-02-10T07:56:17.499887+0000","last_became_active":"2024-02-10T07:31:50.279801+0000","last_became_peered":"2024-02-10T07:31:50.279801+0000","last_unstale":"2024-02-10T07:56:17.499887+0000","last_undegraded":"2024-02-10T07:56:17.499887+0000","last_fullsized":"2024-02-10T07:56:17.499887+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-10T07:55:10.665441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:49.265852+0000","last_clean_scrub_stamp":"2024-02-10T07:55:10.665441+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:23.246477+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00052158499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1460,"num_read_kb":1455,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"430'42","reported_seq":1069,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.270720+0000","last_change":"2024-02-10T07:56:19.276387+0000","last_active":"2024-02-10T07:56:20.270720+0000","last_peered":"2024-02-10T07:56:20.270720+0000","last_clean":"2024-02-10T07:56:20.270720+0000","last_became_active":"2024-02-10T07:50:43.291771+0000","last_became_peered":"2024-02-10T07:50:43.291771+0000","last_unstale":"2024-02-10T07:56:20.270720+0000","last_undegraded":"2024-02-10T07:56:20.270720+0000","last_fullsized":"2024-02-10T07:56:20.270720+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'42","last_scrub_stamp":"2024-02-10T07:56:19.276241+0000","last_deep_scrub":"237'18","last_deep_scrub_stamp":"2024-02-10T07:40:31.005376+0000","last_clean_scrub_stamp":"2024-02-10T07:56:19.276241+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:44.692183+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":942,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.677224+0000","last_change":"2024-02-10T07:56:07.225072+0000","last_active":"2024-02-10T07:56:11.677224+0000","last_peered":"2024-02-10T07:56:11.677224+0000","last_clean":"2024-02-10T07:56:11.677224+0000","last_became_active":"2024-02-10T07:36:33.762643+0000","last_became_peered":"2024-02-10T07:36:33.762643+0000","last_unstale":"2024-02-10T07:56:11.677224+0000","last_undegraded":"2024-02-10T07:56:11.677224+0000","last_fullsized":"2024-02-10T07:56:11.677224+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:07.224956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:56:07.224956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:25.829280+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":979,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642307+0000","last_change":"2024-02-10T07:55:27.841804+0000","last_active":"2024-02-10T07:56:19.642307+0000","last_peered":"2024-02-10T07:56:19.642307+0000","last_clean":"2024-02-10T07:56:19.642307+0000","last_became_active":"2024-02-10T07:50:43.296488+0000","last_became_peered":"2024-02-10T07:50:43.296488+0000","last_unstale":"2024-02-10T07:56:19.642307+0000","last_undegraded":"2024-02-10T07:56:19.642307+0000","last_fullsized":"2024-02-10T07:56:19.642307+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:27.841703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:27.841703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:38.780615+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":992,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642511+0000","last_change":"2024-02-10T07:56:04.855805+0000","last_active":"2024-02-10T07:56:19.642511+0000","last_peered":"2024-02-10T07:56:19.642511+0000","last_clean":"2024-02-10T07:56:19.642511+0000","last_became_active":"2024-02-10T07:50:43.297861+0000","last_became_peered":"2024-02-10T07:50:43.297861+0000","last_unstale":"2024-02-10T07:56:19.642511+0000","last_undegraded":"2024-02-10T07:56:19.642511+0000","last_fullsized":"2024-02-10T07:56:19.642511+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:04.855726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:36.734185+0000","last_clean_scrub_stamp":"2024-02-10T07:56:04.855726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:33.028779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"430'47","reported_seq":1044,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.270715+0000","last_change":"2024-02-10T07:54:58.487627+0000","last_active":"2024-02-10T07:56:20.270715+0000","last_peered":"2024-02-10T07:56:20.270715+0000","last_clean":"2024-02-10T07:56:20.270715+0000","last_became_active":"2024-02-10T07:36:33.753229+0000","last_became_peered":"2024-02-10T07:36:33.753229+0000","last_unstale":"2024-02-10T07:56:20.270715+0000","last_undegraded":"2024-02-10T07:56:20.270715+0000","last_fullsized":"2024-02-10T07:56:20.270715+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'47","last_scrub_stamp":"2024-02-10T07:54:58.487485+0000","last_deep_scrub":"271'37","last_deep_scrub_stamp":"2024-02-10T07:43:10.047405+0000","last_clean_scrub_stamp":"2024-02-10T07:54:58.487485+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1000,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675652+0000","last_change":"2024-02-10T07:55:44.154328+0000","last_active":"2024-02-10T07:56:11.675652+0000","last_peered":"2024-02-10T07:56:11.675652+0000","last_clean":"2024-02-10T07:56:11.675652+0000","last_became_active":"2024-02-10T07:48:35.062118+0000","last_became_peered":"2024-02-10T07:48:35.062118+0000","last_unstale":"2024-02-10T07:56:11.675652+0000","last_undegraded":"2024-02-10T07:56:11.675652+0000","last_fullsized":"2024-02-10T07:56:11.675652+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:44.154228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:46:00.292320+0000","last_clean_scrub_stamp":"2024-02-10T07:55:44.154228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:53.412497+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":861,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:17.423496+0000","last_change":"2024-02-10T07:56:17.423521+0000","last_active":"2024-02-10T07:56:17.423496+0000","last_peered":"2024-02-10T07:56:17.423496+0000","last_clean":"2024-02-10T07:56:17.423496+0000","last_became_active":"2024-02-10T07:35:20.820400+0000","last_became_peered":"2024-02-10T07:35:20.820400+0000","last_unstale":"2024-02-10T07:56:17.423496+0000","last_undegraded":"2024-02-10T07:56:17.423496+0000","last_fullsized":"2024-02-10T07:56:17.423496+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:17.423386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:54:49.148869+0000","last_clean_scrub_stamp":"2024-02-10T07:56:17.423386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:45.221256+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":650,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675203+0000","last_change":"2024-02-10T07:55:31.495075+0000","last_active":"2024-02-10T07:56:11.675203+0000","last_peered":"2024-02-10T07:56:11.675203+0000","last_clean":"2024-02-10T07:56:11.675203+0000","last_became_active":"2024-02-10T07:55:31.494836+0000","last_became_peered":"2024-02-10T07:55:31.494836+0000","last_unstale":"2024-02-10T07:56:11.675203+0000","last_undegraded":"2024-02-10T07:56:11.675203+0000","last_fullsized":"2024-02-10T07:56:11.675203+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:54:53.334334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:54:53.334334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"430'35","reported_seq":798,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642725+0000","last_change":"2024-02-10T07:56:01.753567+0000","last_active":"2024-02-10T07:56:19.642725+0000","last_peered":"2024-02-10T07:56:19.642725+0000","last_clean":"2024-02-10T07:56:19.642725+0000","last_became_active":"2024-02-10T07:37:06.180581+0000","last_became_peered":"2024-02-10T07:37:06.180581+0000","last_unstale":"2024-02-10T07:56:19.642725+0000","last_undegraded":"2024-02-10T07:56:19.642725+0000","last_fullsized":"2024-02-10T07:56:19.642725+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'35","last_scrub_stamp":"2024-02-10T07:56:01.753456+0000","last_deep_scrub":"382'25","last_deep_scrub_stamp":"2024-02-10T07:49:19.083856+0000","last_clean_scrub_stamp":"2024-02-10T07:56:01.753456+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:02.056304+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"476'11","reported_seq":1270,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.271018+0000","last_change":"2024-02-10T07:54:49.551910+0000","last_active":"2024-02-10T07:56:20.271018+0000","last_peered":"2024-02-10T07:56:20.271018+0000","last_clean":"2024-02-10T07:56:20.271018+0000","last_became_active":"2024-02-10T07:36:33.754733+0000","last_became_peered":"2024-02-10T07:36:33.754733+0000","last_unstale":"2024-02-10T07:56:20.271018+0000","last_undegraded":"2024-02-10T07:56:20.271018+0000","last_fullsized":"2024-02-10T07:56:20.271018+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'11","last_scrub_stamp":"2024-02-10T07:54:49.551831+0000","last_deep_scrub":"476'11","last_deep_scrub_stamp":"2024-02-10T07:54:49.551831+0000","last_clean_scrub_stamp":"2024-02-10T07:54:49.551831+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":971,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436917+0000","last_change":"2024-02-10T07:55:50.743012+0000","last_active":"2024-02-10T07:56:11.436917+0000","last_peered":"2024-02-10T07:56:11.436917+0000","last_clean":"2024-02-10T07:56:11.436917+0000","last_became_active":"2024-02-10T07:50:43.283161+0000","last_became_peered":"2024-02-10T07:50:43.283161+0000","last_unstale":"2024-02-10T07:56:11.436917+0000","last_undegraded":"2024-02-10T07:56:11.436917+0000","last_fullsized":"2024-02-10T07:56:11.436917+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:50.742905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:48:23.271454+0000","last_clean_scrub_stamp":"2024-02-10T07:55:50.742905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:03.864495+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"296'2","reported_seq":975,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.270774+0000","last_change":"2024-02-10T07:54:57.480478+0000","last_active":"2024-02-10T07:56:20.270774+0000","last_peered":"2024-02-10T07:56:20.270774+0000","last_clean":"2024-02-10T07:56:20.270774+0000","last_became_active":"2024-02-10T07:31:51.285584+0000","last_became_peered":"2024-02-10T07:31:51.285584+0000","last_unstale":"2024-02-10T07:56:20.270774+0000","last_undegraded":"2024-02-10T07:56:20.270774+0000","last_fullsized":"2024-02-10T07:56:20.270774+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'2","last_scrub_stamp":"2024-02-10T07:54:57.480405+0000","last_deep_scrub":"296'2","last_deep_scrub_stamp":"2024-02-10T07:45:03.233375+0000","last_clean_scrub_stamp":"2024-02-10T07:54:57.480405+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:21.737320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"430'42","reported_seq":1031,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:13.141553+0000","last_change":"2024-02-10T07:56:13.141623+0000","last_active":"2024-02-10T07:56:13.141553+0000","last_peered":"2024-02-10T07:56:13.141553+0000","last_clean":"2024-02-10T07:56:13.141553+0000","last_became_active":"2024-02-10T07:36:33.747417+0000","last_became_peered":"2024-02-10T07:36:33.747417+0000","last_unstale":"2024-02-10T07:56:13.141553+0000","last_undegraded":"2024-02-10T07:56:13.141553+0000","last_fullsized":"2024-02-10T07:56:13.141553+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'42","last_scrub_stamp":"2024-02-10T07:56:13.141459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:56:13.141459+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:39.109637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"496'13","reported_seq":734,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:20.323215+0000","last_change":"2024-02-10T07:56:20.323252+0000","last_active":"2024-02-10T07:56:20.323215+0000","last_peered":"2024-02-10T07:56:20.323215+0000","last_clean":"2024-02-10T07:56:20.323215+0000","last_became_active":"2024-02-10T07:55:37.325698+0000","last_became_peered":"2024-02-10T07:55:37.325698+0000","last_unstale":"2024-02-10T07:56:20.323215+0000","last_undegraded":"2024-02-10T07:56:20.323215+0000","last_fullsized":"2024-02-10T07:56:20.323215+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'13","last_scrub_stamp":"2024-02-10T07:56:20.323135+0000","last_deep_scrub":"376'12","last_deep_scrub_stamp":"2024-02-10T07:54:43.773982+0000","last_clean_scrub_stamp":"2024-02-10T07:56:20.323135+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:47.175972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":968,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437067+0000","last_change":"2024-02-10T07:55:27.965230+0000","last_active":"2024-02-10T07:56:11.437067+0000","last_peered":"2024-02-10T07:56:11.437067+0000","last_clean":"2024-02-10T07:56:11.437067+0000","last_became_active":"2024-02-10T07:50:43.281465+0000","last_became_peered":"2024-02-10T07:50:43.281465+0000","last_unstale":"2024-02-10T07:56:11.437067+0000","last_undegraded":"2024-02-10T07:56:11.437067+0000","last_fullsized":"2024-02-10T07:56:11.437067+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:27.965137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:36:48.752552+0000","last_clean_scrub_stamp":"2024-02-10T07:55:27.965137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:31.429452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":527,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.460791+0000","last_change":"2024-02-10T07:55:41.250763+0000","last_active":"2024-02-10T07:56:19.460791+0000","last_peered":"2024-02-10T07:56:19.460791+0000","last_clean":"2024-02-10T07:56:19.460791+0000","last_became_active":"2024-02-10T07:55:37.330301+0000","last_became_peered":"2024-02-10T07:55:37.330301+0000","last_unstale":"2024-02-10T07:56:19.460791+0000","last_undegraded":"2024-02-10T07:56:19.460791+0000","last_fullsized":"2024-02-10T07:56:19.460791+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:41.250667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:38:50.299314+0000","last_clean_scrub_stamp":"2024-02-10T07:55:41.250667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:44.143151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"430'42","reported_seq":99,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.674717+0000","last_change":"2024-02-10T07:54:55.373700+0000","last_active":"2024-02-10T07:56:11.674717+0000","last_peered":"2024-02-10T07:56:11.674717+0000","last_clean":"2024-02-10T07:56:11.674717+0000","last_became_active":"2024-02-10T07:54:35.652681+0000","last_became_peered":"2024-02-10T07:54:35.652681+0000","last_unstale":"2024-02-10T07:56:11.674717+0000","last_undegraded":"2024-02-10T07:56:11.674717+0000","last_fullsized":"2024-02-10T07:56:11.674717+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'42","last_scrub_stamp":"2024-02-10T07:54:55.373497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:54:55.373497+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:20.015616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":970,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.436861+0000","last_change":"2024-02-10T07:55:54.183746+0000","last_active":"2024-02-10T07:56:11.436861+0000","last_peered":"2024-02-10T07:56:11.436861+0000","last_clean":"2024-02-10T07:56:11.436861+0000","last_became_active":"2024-02-10T07:49:06.239190+0000","last_became_peered":"2024-02-10T07:49:06.239190+0000","last_unstale":"2024-02-10T07:56:11.436861+0000","last_undegraded":"2024-02-10T07:56:11.436861+0000","last_fullsized":"2024-02-10T07:56:11.436861+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:54.183627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:54.183627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:54.841818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":623,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676667+0000","last_change":"2024-02-10T07:56:08.315139+0000","last_active":"2024-02-10T07:56:11.676667+0000","last_peered":"2024-02-10T07:56:11.676667+0000","last_clean":"2024-02-10T07:56:11.676667+0000","last_became_active":"2024-02-10T07:55:31.313595+0000","last_became_peered":"2024-02-10T07:55:31.313595+0000","last_unstale":"2024-02-10T07:56:11.676667+0000","last_undegraded":"2024-02-10T07:56:11.676667+0000","last_fullsized":"2024-02-10T07:56:11.676667+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:08.314994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:39:49.704024+0000","last_clean_scrub_stamp":"2024-02-10T07:56:08.314994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:29.920163+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"395'12","reported_seq":409,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676490+0000","last_change":"2024-02-10T07:54:51.114540+0000","last_active":"2024-02-10T07:56:11.676490+0000","last_peered":"2024-02-10T07:56:11.676490+0000","last_clean":"2024-02-10T07:56:11.676490+0000","last_became_active":"2024-02-10T07:48:35.071916+0000","last_became_peered":"2024-02-10T07:48:35.071916+0000","last_unstale":"2024-02-10T07:56:11.676490+0000","last_undegraded":"2024-02-10T07:56:11.676490+0000","last_fullsized":"2024-02-10T07:56:11.676490+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'12","last_scrub_stamp":"2024-02-10T07:54:51.114463+0000","last_deep_scrub":"395'12","last_deep_scrub_stamp":"2024-02-10T07:52:24.850269+0000","last_clean_scrub_stamp":"2024-02-10T07:54:51.114463+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:21.024599+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":292,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.459945+0000","last_change":"2024-02-10T07:55:02.973698+0000","last_active":"2024-02-10T07:56:19.459945+0000","last_peered":"2024-02-10T07:56:19.459945+0000","last_clean":"2024-02-10T07:56:19.459945+0000","last_became_active":"2024-02-10T07:49:06.251038+0000","last_became_peered":"2024-02-10T07:49:06.251038+0000","last_unstale":"2024-02-10T07:56:19.459945+0000","last_undegraded":"2024-02-10T07:56:19.459945+0000","last_fullsized":"2024-02-10T07:56:19.459945+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:02.973607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:50.269547+0000","last_clean_scrub_stamp":"2024-02-10T07:55:02.973607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"430'39","reported_seq":1036,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.642634+0000","last_change":"2024-02-10T07:55:26.743799+0000","last_active":"2024-02-10T07:56:19.642634+0000","last_peered":"2024-02-10T07:56:19.642634+0000","last_clean":"2024-02-10T07:56:19.642634+0000","last_became_active":"2024-02-10T07:47:47.967427+0000","last_became_peered":"2024-02-10T07:47:47.967427+0000","last_unstale":"2024-02-10T07:56:19.642634+0000","last_undegraded":"2024-02-10T07:56:19.642634+0000","last_fullsized":"2024-02-10T07:56:19.642634+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'39","last_scrub_stamp":"2024-02-10T07:55:26.743712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:26.743712+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:52.234899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":409,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:21.281292+0000","last_change":"2024-02-10T07:56:21.281314+0000","last_active":"2024-02-10T07:56:21.281292+0000","last_peered":"2024-02-10T07:56:21.281292+0000","last_clean":"2024-02-10T07:56:21.281292+0000","last_became_active":"2024-02-10T07:55:37.329226+0000","last_became_peered":"2024-02-10T07:55:37.329226+0000","last_unstale":"2024-02-10T07:56:21.281292+0000","last_undegraded":"2024-02-10T07:56:21.281292+0000","last_fullsized":"2024-02-10T07:56:21.281292+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:21.281227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:56:21.281227+0000","last_clean_scrub_stamp":"2024-02-10T07:56:21.281227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:25.364327+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":597,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.677090+0000","last_change":"2024-02-10T07:55:31.312458+0000","last_active":"2024-02-10T07:56:11.677090+0000","last_peered":"2024-02-10T07:56:11.677090+0000","last_clean":"2024-02-10T07:56:11.677090+0000","last_became_active":"2024-02-10T07:55:31.311729+0000","last_became_peered":"2024-02-10T07:55:31.311729+0000","last_unstale":"2024-02-10T07:56:11.677090+0000","last_undegraded":"2024-02-10T07:56:11.677090+0000","last_fullsized":"2024-02-10T07:56:11.677090+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:14.813005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:55:14.813005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:31.799191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":975,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437017+0000","last_change":"2024-02-10T07:55:33.012220+0000","last_active":"2024-02-10T07:56:11.437017+0000","last_peered":"2024-02-10T07:56:11.437017+0000","last_clean":"2024-02-10T07:56:11.437017+0000","last_became_active":"2024-02-10T07:50:43.283104+0000","last_became_peered":"2024-02-10T07:50:43.283104+0000","last_unstale":"2024-02-10T07:56:11.437017+0000","last_undegraded":"2024-02-10T07:56:11.437017+0000","last_fullsized":"2024-02-10T07:56:11.437017+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:33.012118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:37:36.765729+0000","last_clean_scrub_stamp":"2024-02-10T07:55:33.012118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:37.908808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"430'32","reported_seq":1052,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.438072+0000","last_change":"2024-02-10T07:54:54.904449+0000","last_active":"2024-02-10T07:56:11.438072+0000","last_peered":"2024-02-10T07:56:11.438072+0000","last_clean":"2024-02-10T07:56:11.438072+0000","last_became_active":"2024-02-10T07:48:35.062573+0000","last_became_peered":"2024-02-10T07:48:35.062573+0000","last_unstale":"2024-02-10T07:56:11.438072+0000","last_undegraded":"2024-02-10T07:56:11.438072+0000","last_fullsized":"2024-02-10T07:56:11.438072+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'32","last_scrub_stamp":"2024-02-10T07:54:54.904336+0000","last_deep_scrub":"430'32","last_deep_scrub_stamp":"2024-02-10T07:54:54.904336+0000","last_clean_scrub_stamp":"2024-02-10T07:54:54.904336+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":223,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:18.147696+0000","last_change":"2024-02-10T07:56:18.147722+0000","last_active":"2024-02-10T07:56:18.147696+0000","last_peered":"2024-02-10T07:56:18.147696+0000","last_clean":"2024-02-10T07:56:18.147696+0000","last_became_active":"2024-02-10T07:50:43.290295+0000","last_became_peered":"2024-02-10T07:50:43.290295+0000","last_unstale":"2024-02-10T07:56:18.147696+0000","last_undegraded":"2024-02-10T07:56:18.147696+0000","last_fullsized":"2024-02-10T07:56:18.147696+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:18.147617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:33:05.197642+0000","last_clean_scrub_stamp":"2024-02-10T07:56:18.147617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:18.989042+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":962,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.641984+0000","last_change":"2024-02-10T07:56:05.822657+0000","last_active":"2024-02-10T07:56:19.641984+0000","last_peered":"2024-02-10T07:56:19.641984+0000","last_clean":"2024-02-10T07:56:19.641984+0000","last_became_active":"2024-02-10T07:31:55.299294+0000","last_became_peered":"2024-02-10T07:31:55.299294+0000","last_unstale":"2024-02-10T07:56:19.641984+0000","last_undegraded":"2024-02-10T07:56:19.641984+0000","last_fullsized":"2024-02-10T07:56:19.641984+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:56:05.822569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:56:05.822569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:19.346262+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":993,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.675757+0000","last_change":"2024-02-10T07:55:16.346642+0000","last_active":"2024-02-10T07:56:11.675757+0000","last_peered":"2024-02-10T07:56:11.675757+0000","last_clean":"2024-02-10T07:56:11.675757+0000","last_became_active":"2024-02-10T07:31:51.286684+0000","last_became_peered":"2024-02-10T07:31:51.286684+0000","last_unstale":"2024-02-10T07:56:11.675757+0000","last_undegraded":"2024-02-10T07:56:11.675757+0000","last_fullsized":"2024-02-10T07:56:11.675757+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:16.346542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:50:03.803361+0000","last_clean_scrub_stamp":"2024-02-10T07:55:16.346542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:25.873581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"430'53","reported_seq":1108,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437983+0000","last_change":"2024-02-10T07:55:52.213939+0000","last_active":"2024-02-10T07:56:11.437983+0000","last_peered":"2024-02-10T07:56:11.437983+0000","last_clean":"2024-02-10T07:56:11.437983+0000","last_became_active":"2024-02-10T07:50:43.281879+0000","last_became_peered":"2024-02-10T07:50:43.281879+0000","last_unstale":"2024-02-10T07:56:11.437983+0000","last_undegraded":"2024-02-10T07:56:11.437983+0000","last_fullsized":"2024-02-10T07:56:11.437983+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'53","last_scrub_stamp":"2024-02-10T07:55:52.213839+0000","last_deep_scrub":"89'25","last_deep_scrub_stamp":"2024-02-10T07:41:25.891983+0000","last_clean_scrub_stamp":"2024-02-10T07:55:52.213839+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:20.262420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":954,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.438318+0000","last_change":"2024-02-10T07:55:03.992982+0000","last_active":"2024-02-10T07:56:11.438318+0000","last_peered":"2024-02-10T07:56:11.438318+0000","last_clean":"2024-02-10T07:56:11.438318+0000","last_became_active":"2024-02-10T07:50:43.290334+0000","last_became_peered":"2024-02-10T07:50:43.290334+0000","last_unstale":"2024-02-10T07:56:11.438318+0000","last_undegraded":"2024-02-10T07:56:11.438318+0000","last_fullsized":"2024-02-10T07:56:11.438318+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:03.992885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:56.282886+0000","last_clean_scrub_stamp":"2024-02-10T07:55:03.992885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"405'12","reported_seq":287,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.676349+0000","last_change":"2024-02-10T07:54:50.289202+0000","last_active":"2024-02-10T07:56:11.676349+0000","last_peered":"2024-02-10T07:56:11.676349+0000","last_clean":"2024-02-10T07:56:11.676349+0000","last_became_active":"2024-02-10T07:50:43.280142+0000","last_became_peered":"2024-02-10T07:50:43.280142+0000","last_unstale":"2024-02-10T07:56:11.676349+0000","last_undegraded":"2024-02-10T07:56:11.676349+0000","last_fullsized":"2024-02-10T07:56:11.676349+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'12","last_scrub_stamp":"2024-02-10T07:54:50.289092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:54.275917+0000","last_clean_scrub_stamp":"2024-02-10T07:54:50.289092+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":205,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-02-10T07:56:19.460142+0000","last_change":"2024-02-10T07:55:37.331676+0000","last_active":"2024-02-10T07:56:19.460142+0000","last_peered":"2024-02-10T07:56:19.460142+0000","last_clean":"2024-02-10T07:56:19.460142+0000","last_became_active":"2024-02-10T07:55:37.330932+0000","last_became_peered":"2024-02-10T07:55:37.330932+0000","last_unstale":"2024-02-10T07:56:19.460142+0000","last_undegraded":"2024-02-10T07:56:19.460142+0000","last_fullsized":"2024-02-10T07:56:19.460142+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T07:55:14.391787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:43:09.950640+0000","last_clean_scrub_stamp":"2024-02-10T07:55:14.391787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:56:31.045571+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"430'30","reported_seq":1042,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-02-10T07:56:11.437471+0000","last_change":"2024-02-10T07:55:48.127773+0000","last_active":"2024-02-10T07:56:11.437471+0000","last_peered":"2024-02-10T07:56:11.437471+0000","last_clean":"2024-02-10T07:56:11.437471+0000","last_became_active":"2024-02-10T07:49:06.238858+0000","last_became_peered":"2024-02-10T07:49:06.238858+0000","last_unstale":"2024-02-10T07:56:11.437471+0000","last_undegraded":"2024-02-10T07:56:11.437471+0000","last_fullsized":"2024-02-10T07:56:11.437471+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"430'30","last_scrub_stamp":"2024-02-10T07:55:48.127615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T07:31:52.272420+0000","last_clean_scrub_stamp":"2024-02-10T07:55:48.127615+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T07:57:16.583177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":1455,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1533,"num_read_kb":1290,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":6602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1065,"ondisk_log_size":1065,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038059,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":840900,"kb_used_data":21180,"kb_used_omap":1,"kb_used_meta":819710,"kb_avail":92904252,"statfs":{"total":95995035648,"available":95133954048,"internally_reserved":0,"allocated":21688320,"data_stored":21269048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":839383498},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":1.3129999999999999,"15min":1.3600000000000001},"min":{"1min":0.34300000000000003,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.90500000000000003,"5min":18.202999999999999,"15min":39.494},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.641,"15min":1.446},"min":{"1min":0.372,"5min":0.255,"15min":0.23000000000000001},"max":{"1min":18.158000000000001,"5min":19.745000000000001,"15min":38.524000000000001},"last":0.372}]},{"osd":1,"last update":"Sat Feb 10 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":2.375,"15min":1.798},"min":{"1min":0.35799999999999998,"5min":0.251,"15min":0.24399999999999999},"max":{"1min":0.81299999999999994,"5min":50.149000000000001,"15min":50.149000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":2.3839999999999999,"15min":1.77},"min":{"1min":0.37,"5min":0.26300000000000001,"15min":0.24399999999999999},"max":{"1min":0.81999999999999995,"5min":54.048000000000002,"15min":54.048000000000002},"last":0.41599999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.413,"15min":1.1180000000000001},"min":{"1min":0.35799999999999998,"5min":0.28199999999999997,"15min":0.255},"max":{"1min":16.815999999999999,"5min":16.815999999999999,"15min":16.815999999999999},"last":0.432},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.367,"15min":1.202},"min":{"1min":0.39900000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.745,"5min":8.3949999999999996,"15min":16.295000000000002},"last":0.44}]},{"osd":3,"last update":"Sat Feb 10 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":1.002,"15min":1.0840000000000001},"min":{"1min":0.433,"5min":0.32400000000000001,"15min":0.307},"max":{"1min":0.65200000000000002,"5min":18.053999999999998,"15min":18.053999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.999,"15min":0.97299999999999998},"min":{"1min":0.40300000000000002,"5min":0.31900000000000001,"15min":0.313},"max":{"1min":0.69999999999999996,"5min":6.2839999999999998,"15min":6.2839999999999998},"last":0.35399999999999998}]},{"osd":4,"last update":"Sat Feb 10 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.68300000000000005,"15min":1.1659999999999999},"min":{"1min":0.27900000000000003,"5min":0.24399999999999999,"15min":0.22},"max":{"1min":0.77700000000000002,"5min":5.0679999999999996,"15min":26.001000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.59699999999999998,"15min":1.131},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.248},"max":{"1min":0.75900000000000001,"5min":3.1099999999999999,"15min":27.09},"last":0.308}]},{"osd":5,"last update":"Sat Feb 10 07:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.60099999999999998,"15min":1.002},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.245},"max":{"1min":0.65500000000000003,"5min":3.448,"15min":18.126999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64600000000000002,"15min":1.1180000000000001},"min":{"1min":0.51300000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.81999999999999995,"5min":2.4449999999999998,"15min":19.548999999999999},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Feb 10 07:55:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.82599999999999996,"15min":1.1479999999999999},"min":{"1min":0.35199999999999998,"5min":0.30299999999999999,"15min":0.29399999999999998},"max":{"1min":0.89200000000000002,"5min":7.4329999999999998,"15min":17.832000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.74199999999999999,"15min":1.167},"min":{"1min":0.36799999999999999,"5min":0.30399999999999999,"15min":0.29799999999999999},"max":{"1min":0.80400000000000005,"5min":3.4129999999999998,"15min":20.449999999999999},"last":0.40600000000000003}]}]},{"osd":6,"up_from":50,"seq":214748365104,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":646312,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93098840,"statfs":{"total":95995035648,"available":95333212160,"internally_reserved":0,"allocated":1540096,"data_stored":1148513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":660273606},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":1.4419999999999999,"15min":1.52},"min":{"1min":0.38900000000000001,"5min":0.249,"15min":0.18099999999999999},"max":{"1min":0.75600000000000001,"5min":14.500999999999999,"15min":24.727},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.3939999999999999,"15min":1.2490000000000001},"min":{"1min":0.41799999999999998,"5min":0.25700000000000001,"15min":0.23100000000000001},"max":{"1min":0.73999999999999999,"5min":14.808,"15min":29.084},"last":0.63}]},{"osd":1,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":1.8899999999999999,"15min":1.7010000000000001},"min":{"1min":0.36699999999999999,"5min":0.26200000000000001,"15min":0.23799999999999999},"max":{"1min":0.84699999999999998,"5min":18.859000000000002,"15min":37.659999999999997},"last":0.439},{"interface":"front","average":{"1min":0.58299999999999996,"5min":1.474,"15min":1.7330000000000001},"min":{"1min":0.45600000000000002,"5min":0.28999999999999998,"15min":0.253},"max":{"1min":0.755,"5min":18.725999999999999,"15min":37.75},"last":0.61099999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.5089999999999999,"15min":1.3520000000000001},"min":{"1min":0.41599999999999998,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":0.70499999999999996,"5min":11.839,"15min":15.768000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":1.3520000000000001,"15min":1.151},"min":{"1min":0.434,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80700000000000005,"5min":12.013999999999999,"15min":15.728999999999999},"last":0.70899999999999996}]},{"osd":3,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":1.2250000000000001,"15min":1.444},"min":{"1min":0.47399999999999998,"5min":0.33100000000000002,"15min":0.31900000000000001},"max":{"1min":0.874,"5min":5.6500000000000004,"15min":13.279},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.622,"5min":1.0820000000000001,"15min":1.157},"min":{"1min":0.39700000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.89100000000000001,"5min":7.7949999999999999,"15min":7.7949999999999999},"last":0.54600000000000004}]},{"osd":4,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.51800000000000002,"15min":0.83999999999999997},"min":{"1min":0.37,"5min":0.246,"15min":0.21099999999999999},"max":{"1min":0.82499999999999996,"5min":3.6099999999999999,"15min":7.6509999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54600000000000004,"15min":0.86099999999999999},"min":{"1min":0.41899999999999998,"5min":0.27700000000000002,"15min":0.26000000000000001},"max":{"1min":0.997,"5min":2.3220000000000001,"15min":7.0300000000000002},"last":0.498}]},{"osd":5,"last update":"Sat Feb 10 07:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56000000000000005,"15min":1.0509999999999999},"min":{"1min":0.42799999999999999,"5min":0.28799999999999998,"15min":0.24099999999999999},"max":{"1min":0.81100000000000005,"5min":4.5720000000000001,"15min":44.689},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.56799999999999995,"15min":1.169},"min":{"1min":0.38100000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.80200000000000005,"5min":2.4460000000000002,"15min":44.465000000000003},"last":0.59499999999999997}]},{"osd":7,"last update":"Sat Feb 10 07:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.753,"15min":1.3069999999999999},"min":{"1min":0.38300000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.69199999999999995,"5min":4.335,"15min":42.631999999999998},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.77400000000000002,"15min":1.2230000000000001},"min":{"1min":0.374,"5min":0.32500000000000001,"15min":0.315},"max":{"1min":0.81399999999999995,"5min":4.9290000000000003,"15min":44.792999999999999},"last":0.65400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575178,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852616,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892536,"statfs":{"total":95995035648,"available":95121956864,"internally_reserved":0,"allocated":983040,"data_stored":558872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":872085958},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:55:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":2.008,"15min":1.371},"min":{"1min":0.23300000000000001,"5min":0.21299999999999999,"15min":0.17499999999999999},"max":{"1min":0.56599999999999995,"5min":14.382999999999999,"15min":14.382999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":2.0430000000000001,"15min":1.3420000000000001},"min":{"1min":0.26600000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.59999999999999998,"5min":14.16,"15min":14.16},"last":0.51300000000000001}]},{"osd":2,"last update":"Sat Feb 10 07:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":2.641,"15min":1.502},"min":{"1min":0.25900000000000001,"5min":0.246,"15min":0.20399999999999999},"max":{"1min":0.80500000000000005,"5min":20.129999999999999,"15min":20.129999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":2.302,"15min":1.448},"min":{"1min":0.30099999999999999,"5min":0.27000000000000002,"15min":0.224},"max":{"1min":0.65700000000000003,"5min":20.047000000000001,"15min":20.047000000000001},"last":0.57299999999999995}]},{"osd":3,"last update":"Sat Feb 10 07:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":1.8180000000000001,"15min":1.0449999999999999},"min":{"1min":0.27600000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.58099999999999996,"5min":14.452999999999999,"15min":22.367999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.41999999999999998,"5min":1.931,"15min":1.004},"min":{"1min":0.26700000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.65000000000000002,"5min":13.680999999999999,"15min":22.199999999999999},"last":0.46400000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.363,"15min":2.0590000000000002},"min":{"1min":0.48099999999999998,"5min":0.29499999999999998,"15min":0.28399999999999997},"max":{"1min":0.94999999999999996,"5min":9.2010000000000005,"15min":207.45699999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.62,"5min":1.28,"15min":1.2729999999999999},"min":{"1min":0.41899999999999998,"5min":0.29599999999999999,"15min":0.28899999999999998},"max":{"1min":0.83999999999999997,"5min":19.446999999999999,"15min":32.201000000000001},"last":0.55800000000000005}]},{"osd":5,"last update":"Sat Feb 10 07:56:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":1.4350000000000001,"15min":1.905},"min":{"1min":0.44,"5min":0.35899999999999999,"15min":0.308},"max":{"1min":1.0349999999999999,"5min":9.1419999999999995,"15min":98.703000000000003},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.377,"15min":1.861},"min":{"1min":0.432,"5min":0.32600000000000001,"15min":0.30199999999999999},"max":{"1min":3.1589999999999998,"5min":10.210000000000001,"15min":98.664000000000001},"last":0.627}]},{"osd":6,"last update":"Sat Feb 10 07:55:52 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.244,"15min":1.3660000000000001},"min":{"1min":0.442,"5min":0.33900000000000002,"15min":0.308},"max":{"1min":1.0029999999999999,"5min":10.291,"15min":36.808},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":1.637,"15min":1.6140000000000001},"min":{"1min":0.47799999999999998,"5min":0.40100000000000002,"15min":0.33300000000000002},"max":{"1min":1.0549999999999999,"5min":9.1750000000000007,"15min":33.774999999999999},"last":0.76600000000000001}]},{"osd":7,"last update":"Sat Feb 10 07:55:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":1.8640000000000001,"15min":1.5},"min":{"1min":0.42499999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.98999999999999999,"5min":24.099,"15min":24.099},"last":0.753},{"interface":"front","average":{"1min":0.65800000000000003,"5min":1.3400000000000001,"15min":1.2569999999999999},"min":{"1min":0.48199999999999998,"5min":0.39300000000000002,"15min":0.34300000000000003},"max":{"1min":0.92900000000000005,"5min":10.356999999999999,"15min":10.356999999999999},"last":0.68600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738703,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799044,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92946108,"statfs":{"total":95995035648,"available":95176814592,"internally_reserved":0,"allocated":1568768,"data_stored":1149893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":816642501},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 07:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":2.8460000000000001,"15min":2.5089999999999999},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.182},"max":{"1min":0.68899999999999995,"5min":38.679000000000002,"15min":156.16200000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":2.2400000000000002,"15min":2.3650000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.19},"max":{"1min":0.66300000000000003,"5min":40.790999999999997,"15min":156.941},"last":0.41899999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":2.0600000000000001,"15min":1.5189999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.20000000000000001},"max":{"1min":0.53400000000000003,"5min":19.733000000000001,"15min":34.603000000000002},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":2.1030000000000002,"15min":1.427},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.751,"5min":10.116,"15min":34.718000000000004},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Feb 10 07:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":2.0710000000000002,"15min":1.1319999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.65100000000000002,"5min":29.792999999999999,"15min":29.792999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":2.2930000000000001,"15min":1.202},"min":{"1min":0.29099999999999998,"5min":0.24099999999999999,"15min":0.214},"max":{"1min":0.57299999999999995,"5min":29.506,"15min":29.506},"last":0.434}]},{"osd":4,"last update":"Sat Feb 10 07:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.5349999999999999,"15min":1.405},"min":{"1min":0.38700000000000001,"5min":0.28599999999999998,"15min":0.27600000000000002},"max":{"1min":0.873,"5min":16.991,"15min":37.286000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":1.3400000000000001,"15min":1.3129999999999999},"min":{"1min":0.38100000000000001,"5min":0.32300000000000001,"15min":0.27800000000000002},"max":{"1min":0.90900000000000003,"5min":8.5939999999999994,"15min":34.683},"last":0.54600000000000004}]},{"osd":5,"last update":"Sat Feb 10 07:56:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":1.6499999999999999,"15min":1.788},"min":{"1min":0.42699999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.95399999999999996,"5min":17.062999999999999,"15min":41.429000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":1.357,"15min":1.6659999999999999},"min":{"1min":0.49099999999999999,"5min":0.35999999999999999,"15min":0.308},"max":{"1min":0.95099999999999996,"5min":8.6460000000000008,"15min":41.456000000000003},"last":0.59299999999999997}]},{"osd":6,"last update":"Sat Feb 10 07:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":1.3939999999999999,"15min":1.514},"min":{"1min":0.44900000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.95199999999999996,"5min":8.5630000000000006,"15min":41.587000000000003},"last":0.626},{"interface":"front","average":{"1min":0.61199999999999999,"5min":1.8660000000000001,"15min":1.7270000000000001},"min":{"1min":0.45700000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81499999999999995,"5min":17.027999999999999,"15min":41.256},"last":0.502}]},{"osd":7,"last update":"Sat Feb 10 07:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":1.5680000000000001,"15min":1.4810000000000001},"min":{"1min":0.50900000000000001,"5min":0.39600000000000002,"15min":0.33300000000000002},"max":{"1min":0.88200000000000001,"5min":8.6809999999999992,"15min":34.710999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.67900000000000005,"5min":1.4690000000000001,"15min":1.347},"min":{"1min":0.48799999999999999,"5min":0.45400000000000001,"15min":0.379},"max":{"1min":0.95699999999999996,"5min":8.5999999999999996,"15min":34.997999999999998},"last":0.56299999999999994}]}]},{"osd":2,"up_from":18,"seq":77309411652,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":694312,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":93050840,"statfs":{"total":95995035648,"available":95284060160,"internally_reserved":0,"allocated":950272,"data_stored":558955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":710015429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:55:27 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":2.2759999999999998,"15min":1.7549999999999999},"min":{"1min":0.23699999999999999,"5min":0.20699999999999999,"15min":0.191},"max":{"1min":0.51100000000000001,"5min":32.371000000000002,"15min":65.430999999999997},"last":0.223},{"interface":"front","average":{"1min":0.36399999999999999,"5min":2.4910000000000001,"15min":1.6539999999999999},"min":{"1min":0.23699999999999999,"5min":0.22600000000000001,"15min":0.20100000000000001},"max":{"1min":0.59199999999999997,"5min":38.173000000000002,"15min":38.173000000000002},"last":0.253}]},{"osd":1,"last update":"Sat Feb 10 07:55:27 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":2.0680000000000001,"15min":1.651},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54700000000000004,"5min":27.606000000000002,"15min":65.795000000000002},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":2.677,"15min":1.635},"min":{"1min":0.26100000000000001,"5min":0.23599999999999999,"15min":0.20499999999999999},"max":{"1min":0.56100000000000005,"5min":29.709,"15min":29.709},"last":0.39200000000000002}]},{"osd":3,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":1.288,"15min":0.83399999999999996},"min":{"1min":0.314,"5min":0.217,"15min":0.217},"max":{"1min":0.56499999999999995,"5min":7.0819999999999999,"15min":7.0819999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":1.6759999999999999,"15min":0.90600000000000003},"min":{"1min":0.311,"5min":0.248,"15min":0.23999999999999999},"max":{"1min":0.56999999999999995,"5min":7.7530000000000001,"15min":7.7530000000000001},"last":0.308}]},{"osd":4,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":1.107,"15min":1.135},"min":{"1min":0.50900000000000001,"5min":0.33400000000000002,"15min":0.29599999999999999},"max":{"1min":0.84799999999999998,"5min":8.875,"15min":11.395},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.5640000000000001,"15min":1.262},"min":{"1min":0.45200000000000001,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":0.78900000000000003,"5min":8.8510000000000009,"15min":8.8510000000000009},"last":0.45700000000000002}]},{"osd":5,"last update":"Sat Feb 10 07:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":1.3160000000000001,"15min":1.325},"min":{"1min":0.39000000000000001,"5min":0.33400000000000002,"15min":0.32600000000000001},"max":{"1min":0.78800000000000003,"5min":10.807,"15min":25.533000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":1.1819999999999999,"15min":1.252},"min":{"1min":0.47799999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78200000000000003,"5min":8.9299999999999997,"15min":25.503},"last":0.57799999999999996}]},{"osd":6,"last update":"Sat Feb 10 07:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.167,"15min":1.381},"min":{"1min":0.439,"5min":0.36499999999999999,"15min":0.35199999999999998},"max":{"1min":4.4199999999999999,"5min":5.069,"15min":35.603999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.63800000000000001,"5min":1.4530000000000001,"15min":1.3600000000000001},"min":{"1min":0.48799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80300000000000005,"5min":10.893000000000001,"15min":25.901},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Feb 10 07:55:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.609,"15min":1.4059999999999999},"min":{"1min":0.48899999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":8.8239999999999998,"5min":10.859999999999999,"15min":10.859999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.171,"15min":1.4119999999999999},"min":{"1min":0.45900000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85299999999999998,"5min":5.0979999999999999,"15min":65.539000000000001},"last":0.55600000000000005}]}]},{"osd":3,"up_from":347,"seq":1490353651817,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":252176,"kb_used_data":760,"kb_used_omap":5,"kb_used_meta":251194,"kb_avail":93492976,"statfs":{"total":95995035648,"available":95736807424,"internally_reserved":0,"allocated":778240,"data_stored":447808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5625,"internal_metadata":257223175},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":1.8460000000000001,"15min":1.0029999999999999},"min":{"1min":0.20300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.63,"5min":28.247,"15min":28.247},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.38,"5min":1.8169999999999999,"15min":1.0820000000000001},"min":{"1min":0.24299999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.55300000000000005,"5min":27.361999999999998,"15min":27.361999999999998},"last":0.379}]},{"osd":1,"last update":"Sat Feb 10 07:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":2.6739999999999999,"15min":1.2230000000000001},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.20100000000000001},"max":{"1min":0.57699999999999996,"5min":30.718,"15min":30.718},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":2.319,"15min":1.1459999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.22},"max":{"1min":0.58799999999999997,"5min":32.927999999999997,"15min":32.927999999999997},"last":0.29999999999999999}]},{"osd":2,"last update":"Sat Feb 10 07:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.7829999999999999,"15min":0.93400000000000005},"min":{"1min":0.30299999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.55400000000000005,"5min":18.059000000000001,"15min":18.059000000000001},"last":0.317},{"interface":"front","average":{"1min":0.42399999999999999,"5min":2.008,"15min":1.0509999999999999},"min":{"1min":0.24299999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56399999999999995,"5min":18.196000000000002,"15min":18.196000000000002},"last":0.33900000000000002}]},{"osd":4,"last update":"Sat Feb 10 07:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":1.1990000000000001,"15min":2.0950000000000002},"min":{"1min":0.34300000000000003,"5min":0.29899999999999999,"15min":0.29599999999999999},"max":{"1min":1.038,"5min":7.782,"15min":28.707999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":1.163,"15min":2.2240000000000002},"min":{"1min":0.39700000000000002,"5min":0.33600000000000002,"15min":0.32400000000000001},"max":{"1min":1.024,"5min":7.8220000000000001,"15min":28.760999999999999},"last":0.70499999999999996}]},{"osd":5,"last update":"Sat Feb 10 07:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.2470000000000001,"15min":1.0860000000000001},"min":{"1min":0.51200000000000001,"5min":0.32900000000000001,"15min":0.29699999999999999},"max":{"1min":0.97399999999999998,"5min":8.2799999999999994,"15min":15.525},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.2110000000000001,"15min":0.97399999999999998},"min":{"1min":0.374,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94399999999999995,"5min":7.8280000000000003,"15min":10.298},"last":0.69399999999999995}]},{"osd":6,"last update":"Sat Feb 10 07:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":1.8759999999999999,"15min":2.1819999999999999},"min":{"1min":0.42899999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.92700000000000005,"5min":55.750999999999998,"15min":55.750999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":2.0499999999999998,"15min":2.6070000000000002},"min":{"1min":0.40699999999999997,"5min":0.39800000000000002,"15min":0.39200000000000002},"max":{"1min":0.89200000000000002,"5min":55.807000000000002,"15min":55.807000000000002},"last":0.73599999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":1.381,"15min":1.298},"min":{"1min":0.47199999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84199999999999997,"5min":7.8760000000000003,"15min":15.577999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.626,"5min":1.2869999999999999,"15min":1.29},"min":{"1min":0.439,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.95799999999999996,"5min":7.8650000000000002,"15min":7.8650000000000002},"last":0.67600000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953786,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758212,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92986940,"statfs":{"total":95995035648,"available":95218626560,"internally_reserved":0,"allocated":978944,"data_stored":558763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.141,"15min":1.2509999999999999},"min":{"1min":0.35799999999999998,"5min":0.26300000000000001,"15min":0.20499999999999999},"max":{"1min":0.875,"5min":20.004999999999999,"15min":45.610999999999997},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":1.4359999999999999,"15min":1.421},"min":{"1min":0.371,"5min":0.23499999999999999,"15min":0.22700000000000001},"max":{"1min":0.86299999999999999,"5min":20.050000000000001,"15min":45.524999999999999},"last":0.51400000000000001}]},{"osd":1,"last update":"Sat Feb 10 07:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":1.9990000000000001,"15min":2.0649999999999999},"min":{"1min":0.28999999999999998,"5min":0.25800000000000001,"15min":0.23899999999999999},"max":{"1min":0.73999999999999999,"5min":35.673999999999999,"15min":66.343999999999994},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":2.0960000000000001,"15min":2.0329999999999999},"min":{"1min":0.27900000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.83999999999999997,"5min":35.375,"15min":53.536000000000001},"last":0.47799999999999998}]},{"osd":2,"last update":"Sat Feb 10 07:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.2430000000000001,"15min":1.151},"min":{"1min":0.35199999999999998,"5min":0.28000000000000003,"15min":0.26600000000000001},"max":{"1min":0.79100000000000004,"5min":25.32,"15min":25.32},"last":0.505},{"interface":"front","average":{"1min":0.57399999999999995,"5min":1.1599999999999999,"15min":1.0900000000000001},"min":{"1min":0.40500000000000003,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.82499999999999996,"5min":20.535,"15min":20.535},"last":0.497}]},{"osd":3,"last update":"Sat Feb 10 07:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.1140000000000001,"15min":3.0720000000000001},"min":{"1min":0.374,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.81000000000000005,"5min":20.318000000000001,"15min":66.379999999999995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.625,"5min":1.137,"15min":1.476},"min":{"1min":0.40899999999999997,"5min":0.314,"15min":0.309},"max":{"1min":0.92700000000000005,"5min":22.917000000000002,"15min":22.917000000000002},"last":0.53600000000000003}]},{"osd":5,"last update":"Sat Feb 10 07:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.47099999999999997,"15min":0.94799999999999995},"min":{"1min":0.34799999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.69099999999999995,"5min":2.1389999999999998,"15min":10.502000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.48399999999999999,"15min":0.91300000000000003},"min":{"1min":0.35699999999999998,"5min":0.26000000000000001,"15min":0.247},"max":{"1min":0.72499999999999998,"5min":2.0289999999999999,"15min":9.2460000000000004},"last":0.31}]},{"osd":6,"last update":"Sat Feb 10 07:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51900000000000002,"15min":1.179},"min":{"1min":0.39200000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.73299999999999998,"5min":1.2090000000000001,"15min":67.433999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54700000000000004,"15min":1.244},"min":{"1min":0.32700000000000001,"5min":0.29199999999999998,"15min":0.26600000000000001},"max":{"1min":0.70599999999999996,"5min":1.7989999999999999,"15min":66.876000000000005},"last":0.42699999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.65900000000000003,"15min":0.97999999999999998},"min":{"1min":0.45500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75900000000000001,"5min":4.5730000000000004,"15min":7.4930000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.623,"15min":0.97499999999999998},"min":{"1min":0.371,"5min":0.314,"15min":0.29399999999999998},"max":{"1min":0.76200000000000001,"5min":3.8479999999999999,"15min":10.919},"last":0.52400000000000002}]}]},{"osd":5,"up_from":156,"seq":670014898415,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":697772,"kb_used_data":944,"kb_used_omap":4,"kb_used_meta":696571,"kb_avail":93047380,"statfs":{"total":95995035648,"available":95280517120,"internally_reserved":0,"allocated":966656,"data_stored":559577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4446,"internal_metadata":713289378},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 07:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":1.3320000000000001,"15min":1.637},"min":{"1min":0.38600000000000001,"5min":0.24299999999999999,"15min":0.20899999999999999},"max":{"1min":0.79100000000000004,"5min":24.292999999999999,"15min":52.938000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.458,"5min":1.4990000000000001,"15min":1.9119999999999999},"min":{"1min":0.41799999999999998,"5min":0.24199999999999999,"15min":0.22900000000000001},"max":{"1min":17.600999999999999,"5min":23.908000000000001,"15min":52.805},"last":0.70699999999999996}]},{"osd":1,"last update":"Sat Feb 10 07:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":2.1299999999999999,"15min":1.9399999999999999},"min":{"1min":0.39100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.79700000000000004,"5min":28.780999999999999,"15min":54.298000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.1259999999999999,"15min":1.891},"min":{"1min":0.34300000000000003,"5min":0.254,"15min":0.246},"max":{"1min":0.81399999999999995,"5min":28.356000000000002,"15min":52.969999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Feb 10 07:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":1.2909999999999999,"15min":1.194},"min":{"1min":0.433,"5min":0.29899999999999999,"15min":0.27900000000000003},"max":{"1min":0.872,"5min":10.039999999999999,"15min":10.039999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":1.228,"15min":1.22},"min":{"1min":0.39700000000000002,"5min":0.28799999999999998,"15min":0.28100000000000003},"max":{"1min":0.85099999999999998,"5min":9.6069999999999993,"15min":13.662000000000001},"last":0.59299999999999997}]},{"osd":3,"last update":"Sat Feb 10 07:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.998,"15min":1.712},"min":{"1min":0.48499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.66700000000000004,"5min":97.119,"15min":97.119},"last":0.621},{"interface":"front","average":{"1min":0.58199999999999996,"5min":2.23,"15min":1.661},"min":{"1min":0.375,"5min":0.32700000000000001,"15min":0.28699999999999998},"max":{"1min":0.76500000000000001,"5min":97.141999999999996,"15min":97.141999999999996},"last":0.61299999999999999}]},{"osd":4,"last update":"Sat Feb 10 07:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.54600000000000004,"15min":1.0980000000000001},"min":{"1min":0.34599999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.73199999999999998,"5min":3.0760000000000001,"15min":8.952},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55300000000000005,"15min":1.0049999999999999},"min":{"1min":0.36399999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":3.5019999999999998,"15min":10.180999999999999},"last":0.42499999999999999}]},{"osd":6,"last update":"Sat Feb 10 07:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.875,"15min":1.103},"min":{"1min":0.29299999999999998,"5min":0.26500000000000001,"15min":0.23999999999999999},"max":{"1min":0.77400000000000002,"5min":21.748000000000001,"15min":21.748000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.86199999999999999,"15min":1.0389999999999999},"min":{"1min":0.32300000000000001,"5min":0.28599999999999998,"15min":0.26300000000000001},"max":{"1min":0.72999999999999998,"5min":22.187999999999999,"15min":22.187999999999999},"last":0.48399999999999999}]},{"osd":7,"last update":"Sat Feb 10 07:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.83399999999999996,"15min":1.181},"min":{"1min":0.38900000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.94099999999999995,"5min":22.302,"15min":22.302},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.90200000000000002,"15min":1.105},"min":{"1min":0.40500000000000003,"5min":0.34699999999999998,"15min":0.26300000000000001},"max":{"1min":0.92000000000000004,"5min":21.779,"15min":21.779},"last":0.66000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T07:56:24.287 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-10T07:56:24.287 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-02-10T07:56:24.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:56:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug there is no tcmu-runner data available 2024-02-10T07:56:24.540 INFO:teuthology.orchestra.run.smithi008.stderr:Inferring config /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T07:56:24.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:24 smithi008 ceph-mon[29816]: 6.18 scrub starts 2024-02-10T07:56:24.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:24 smithi008 ceph-mon[29816]: 6.18 scrub ok 2024-02-10T07:56:24.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:24 smithi008 ceph-mon[29816]: 5.4 scrub starts 2024-02-10T07:56:24.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:24 smithi008 ceph-mon[29816]: 5.4 scrub ok 2024-02-10T07:56:24.582 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:24 smithi008 ceph-mon[29816]: from='client.16401 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:56:24.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:24 smithi008 ceph-mon[40743]: 6.18 scrub starts 2024-02-10T07:56:24.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:24 smithi008 ceph-mon[40743]: 6.18 scrub ok 2024-02-10T07:56:24.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:24 smithi008 ceph-mon[40743]: 5.4 scrub starts 2024-02-10T07:56:24.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:24 smithi008 ceph-mon[40743]: 5.4 scrub ok 2024-02-10T07:56:24.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:24 smithi008 ceph-mon[40743]: from='client.16401 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:56:24.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:24 smithi170 ceph-mon[33394]: 6.18 scrub starts 2024-02-10T07:56:24.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:24 smithi170 ceph-mon[33394]: 6.18 scrub ok 2024-02-10T07:56:24.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:24 smithi170 ceph-mon[33394]: 5.4 scrub starts 2024-02-10T07:56:24.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:24 smithi170 ceph-mon[33394]: 5.4 scrub ok 2024-02-10T07:56:24.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:24 smithi170 ceph-mon[33394]: from='client.16401 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T07:56:25.764 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[29816]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:56:25.764 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[29816]: pgmap v1140: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:25.764 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[29816]: 6.10 scrub starts 2024-02-10T07:56:25.764 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[29816]: 6.10 scrub ok 2024-02-10T07:56:25.764 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[29816]: 3.13 scrub starts 2024-02-10T07:56:25.764 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[29816]: 3.13 scrub ok 2024-02-10T07:56:25.764 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[29816]: 5.6 deep-scrub starts 2024-02-10T07:56:25.764 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[29816]: 5.6 deep-scrub ok 2024-02-10T07:56:25.765 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[29816]: 4.3 deep-scrub starts 2024-02-10T07:56:25.765 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[29816]: 4.3 deep-scrub ok 2024-02-10T07:56:25.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[40743]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:56:25.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[40743]: pgmap v1140: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:25.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[40743]: 6.10 scrub starts 2024-02-10T07:56:25.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[40743]: 6.10 scrub ok 2024-02-10T07:56:25.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[40743]: 3.13 scrub starts 2024-02-10T07:56:25.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[40743]: 3.13 scrub ok 2024-02-10T07:56:25.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[40743]: 5.6 deep-scrub starts 2024-02-10T07:56:25.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[40743]: 5.6 deep-scrub ok 2024-02-10T07:56:25.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[40743]: 4.3 deep-scrub starts 2024-02-10T07:56:25.765 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:25 smithi008 ceph-mon[40743]: 4.3 deep-scrub ok 2024-02-10T07:56:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:25 smithi170 ceph-mon[33394]: from='client.24440 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T07:56:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:25 smithi170 ceph-mon[33394]: pgmap v1140: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:25 smithi170 ceph-mon[33394]: 6.10 scrub starts 2024-02-10T07:56:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:25 smithi170 ceph-mon[33394]: 6.10 scrub ok 2024-02-10T07:56:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:25 smithi170 ceph-mon[33394]: 3.13 scrub starts 2024-02-10T07:56:25.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:25 smithi170 ceph-mon[33394]: 3.13 scrub ok 2024-02-10T07:56:25.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:25 smithi170 ceph-mon[33394]: 5.6 deep-scrub starts 2024-02-10T07:56:25.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:25 smithi170 ceph-mon[33394]: 5.6 deep-scrub ok 2024-02-10T07:56:25.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:25 smithi170 ceph-mon[33394]: 4.3 deep-scrub starts 2024-02-10T07:56:25.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:25 smithi170 ceph-mon[33394]: 4.3 deep-scrub ok 2024-02-10T07:56:26.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[29816]: 6.b scrub starts 2024-02-10T07:56:26.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[29816]: 6.b scrub ok 2024-02-10T07:56:26.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[29816]: 6.8 scrub starts 2024-02-10T07:56:26.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[29816]: 6.8 scrub ok 2024-02-10T07:56:26.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[29816]: 4.0 scrub starts 2024-02-10T07:56:26.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[29816]: 4.0 scrub ok 2024-02-10T07:56:26.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[29816]: from='client.? 172.21.15.8:0/2426665513' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-10T07:56:26.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-10T07:56:26.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[29816]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-10T07:56:26.766 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:56:26.767 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:56:26.767 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:56:26.767 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[29816]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:56:26.767 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:26] ENGINE Bus STOPPING 2024-02-10T07:56:26.767 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:56:26.767 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:26] ENGINE Bus STOPPED 2024-02-10T07:56:26.767 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:26] ENGINE Bus STARTING 2024-02-10T07:56:26.767 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:26] ENGINE Serving on http://:::9283 2024-02-10T07:56:26.767 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:26] ENGINE Bus STARTED 2024-02-10T07:56:26.767 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[40743]: 6.b scrub starts 2024-02-10T07:56:26.768 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[40743]: 6.b scrub ok 2024-02-10T07:56:26.768 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[40743]: 6.8 scrub starts 2024-02-10T07:56:26.768 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[40743]: 6.8 scrub ok 2024-02-10T07:56:26.768 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[40743]: 4.0 scrub starts 2024-02-10T07:56:26.768 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[40743]: 4.0 scrub ok 2024-02-10T07:56:26.768 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[40743]: from='client.? 172.21.15.8:0/2426665513' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-10T07:56:26.768 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-10T07:56:26.768 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[40743]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-10T07:56:26.768 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:56:26.768 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:56:26.768 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:56:26.768 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:26 smithi008 ceph-mon[40743]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:56:26.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:26 smithi170 ceph-mon[33394]: 6.b scrub starts 2024-02-10T07:56:26.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:26 smithi170 ceph-mon[33394]: 6.b scrub ok 2024-02-10T07:56:26.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:26 smithi170 ceph-mon[33394]: 6.8 scrub starts 2024-02-10T07:56:26.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:26 smithi170 ceph-mon[33394]: 6.8 scrub ok 2024-02-10T07:56:26.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:26 smithi170 ceph-mon[33394]: 4.0 scrub starts 2024-02-10T07:56:26.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:26 smithi170 ceph-mon[33394]: 4.0 scrub ok 2024-02-10T07:56:26.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:26 smithi170 ceph-mon[33394]: from='client.? 172.21.15.8:0/2426665513' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-10T07:56:26.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:26 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-10T07:56:26.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:26 smithi170 ceph-mon[33394]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-10T07:56:26.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:26 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T07:56:26.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:26 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T07:56:26.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:26 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T07:56:26.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:26 smithi170 ceph-mon[33394]: from='mgr.24482 172.21.15.8:0/3571381297' entity='mgr.y' 2024-02-10T07:56:27.119 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-10T07:56:27.129 INFO:tasks.cephadm:Teardown begin 2024-02-10T07:56:27.130 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:56:27.177 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:56:27.222 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-10T07:56:27.222 DEBUG:teuthology.orchestra.run.smithi008:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T07:56:27.259 DEBUG:teuthology.orchestra.run.smithi170:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T07:56:27.284 INFO:tasks.cephadm:Stopping all daemons... 2024-02-10T07:56:27.285 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-10T07:56:27.285 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mon.a 2024-02-10T07:56:27.604 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-mon[40743]: pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:27.605 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-mon[40743]: 3.1d scrub starts 2024-02-10T07:56:27.605 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-mon[40743]: 3.1d scrub ok 2024-02-10T07:56:27.605 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-mon[40743]: 2.0 scrub starts 2024-02-10T07:56:27.605 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-mon[40743]: 2.0 scrub ok 2024-02-10T07:56:27.605 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-mon[40743]: 5.3 scrub starts 2024-02-10T07:56:27.605 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-mon[40743]: 5.3 scrub ok 2024-02-10T07:56:27.605 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:27 smithi008 systemd[1]: Stopping Ceph mon.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:56:27.605 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-mon[29816]: pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:27.605 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-mon[29816]: 3.1d scrub starts 2024-02-10T07:56:27.605 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-mon[29816]: 3.1d scrub ok 2024-02-10T07:56:27.605 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-mon[29816]: 2.0 scrub starts 2024-02-10T07:56:27.605 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-mon[29816]: 2.0 scrub ok 2024-02-10T07:56:27.606 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-mon[29816]: 5.3 scrub starts 2024-02-10T07:56:27.606 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-mon[29816]: 5.3 scrub ok 2024-02-10T07:56:27.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:27 smithi170 ceph-mon[33394]: pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T07:56:27.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:27 smithi170 ceph-mon[33394]: 3.1d scrub starts 2024-02-10T07:56:27.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:27 smithi170 ceph-mon[33394]: 3.1d scrub ok 2024-02-10T07:56:27.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:27 smithi170 ceph-mon[33394]: 2.0 scrub starts 2024-02-10T07:56:27.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:27 smithi170 ceph-mon[33394]: 2.0 scrub ok 2024-02-10T07:56:27.883 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:27 smithi170 ceph-mon[33394]: 5.3 scrub starts 2024-02-10T07:56:27.884 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:27 smithi170 ceph-mon[33394]: 5.3 scrub ok 2024-02-10T07:56:27.988 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a[29812]: 2024-02-10T07:56:27.604+0000 7ff40f429700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:56:27.988 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a[29812]: 2024-02-10T07:56:27.604+0000 7ff40f429700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-10T07:56:27.988 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:27 smithi008 podman[218720]: 2024-02-10 07:56:27.73012595 +0000 UTC m=+0.237325765 container died e36eae6aadf6ba90f7c33fe2c60e781372c02d434b497c4b5c70e8cc55434f70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, ceph=True, org.label-schema.build-date=20240131) 2024-02-10T07:56:27.989 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:27] ENGINE Bus STOPPING 2024-02-10T07:56:27.989 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:56:27.989 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:27] ENGINE Bus STOPPED 2024-02-10T07:56:27.989 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:27] ENGINE Bus STARTING 2024-02-10T07:56:27.989 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:27] ENGINE Serving on http://:::9283 2024-02-10T07:56:27.989 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:27 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:27] ENGINE Bus STARTED 2024-02-10T07:56:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:28 smithi008 podman[218720]: 2024-02-10 07:56:28.096052914 +0000 UTC m=+0.603252726 container cleanup e36eae6aadf6ba90f7c33fe2c60e781372c02d434b497c4b5c70e8cc55434f70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:56:28.332 INFO:journalctl@ceph.mon.a.smithi008.stdout:Feb 10 07:56:28 smithi008 bash[218720]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-a 2024-02-10T07:56:28.519 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mon.a.service' 2024-02-10T07:56:28.566 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:56:28.566 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-10T07:56:28.566 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-10T07:56:28.566 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mon.c 2024-02-10T07:56:28.790 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:28 smithi008 systemd[1]: Stopping Ceph mon.c for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:56:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-c[40738]: 2024-02-10T07:56:28.850+0000 7f453183d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:56:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-c[40738]: 2024-02-10T07:56:28.850+0000 7f453183d700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-02-10T07:56:29.082 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:28 smithi008 podman[218853]: 2024-02-10 07:56:28.984219967 +0000 UTC m=+0.237318667 container died c714c53367e5b61e6483c0842ba3f2f23d2232636b40b3ca53ae4bfce429c47e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-c, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-10T07:56:29.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: ::ffff:172.21.15.170 - - [10/Feb/2024:07:56:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T07:56:29.082 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:28] ENGINE Bus STOPPING 2024-02-10T07:56:29.083 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:28] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T07:56:29.083 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:28] ENGINE Bus STOPPED 2024-02-10T07:56:29.083 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:28] ENGINE Bus STARTING 2024-02-10T07:56:29.083 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:28] ENGINE Serving on http://:::9283 2024-02-10T07:56:29.083 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:28 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: [10/Feb/2024:07:56:28] ENGINE Bus STARTED 2024-02-10T07:56:29.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:29 smithi008 podman[218853]: 2024-02-10 07:56:29.359246725 +0000 UTC m=+0.612345434 container cleanup c714c53367e5b61e6483c0842ba3f2f23d2232636b40b3ca53ae4bfce429c47e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-c, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:56:29.582 INFO:journalctl@ceph.mon.c.smithi008.stdout:Feb 10 07:56:29 smithi008 bash[218853]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-c 2024-02-10T07:56:29.796 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mon.c.service' 2024-02-10T07:56:29.832 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:56:29.832 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-10T07:56:29.832 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-10T07:56:29.833 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mon.b 2024-02-10T07:56:30.133 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:29 smithi170 systemd[1]: Stopping Ceph mon.b for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:56:30.481 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-b[33378]: 2024-02-10T07:56:30.214+0000 7f24f930b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:56:30.481 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-b[33378]: 2024-02-10T07:56:30.214+0000 7f24f930b700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-02-10T07:56:30.741 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:30 smithi170 podman[160978]: 2024-02-10 07:56:30.481298386 +0000 UTC m=+0.413208725 container died 8c90b2d2d7b7ffb7b54b42b015a8ecd0d98d071bc1fcb01353935474e464d60d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-10T07:56:31.117 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:30 smithi170 podman[160978]: 2024-02-10 07:56:30.882889697 +0000 UTC m=+0.814800042 container cleanup 8c90b2d2d7b7ffb7b54b42b015a8ecd0d98d071bc1fcb01353935474e464d60d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-b, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-02-10T07:56:31.117 INFO:journalctl@ceph.mon.b.smithi170.stdout:Feb 10 07:56:30 smithi170 bash[160978]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mon-b 2024-02-10T07:56:31.264 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mon.b.service' 2024-02-10T07:56:31.315 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:56:31.315 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-10T07:56:31.315 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-10T07:56:31.315 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mgr.y 2024-02-10T07:56:31.614 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:31 smithi008 systemd[1]: Stopping Ceph mgr.y for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:56:31.906 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:31 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y[30044]: 2024-02-10T07:56:31.614+0000 7f408396c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:56:31.907 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:31 smithi008 podman[218986]: 2024-02-10 07:56:31.644860963 +0000 UTC m=+0.132560120 container died b30d99dabe008f67adb64f3b0b9b14a29a0e56a7327716ecde9b8afb7559403d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y, GIT_CLEAN=True, RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-10T07:56:32.332 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:32 smithi008 podman[218986]: 2024-02-10 07:56:32.098029097 +0000 UTC m=+0.585728253 container cleanup b30d99dabe008f67adb64f3b0b9b14a29a0e56a7327716ecde9b8afb7559403d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=) 2024-02-10T07:56:32.332 INFO:journalctl@ceph.mgr.y.smithi008.stdout:Feb 10 07:56:32 smithi008 bash[218986]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-y 2024-02-10T07:56:32.683 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mgr.y.service' 2024-02-10T07:56:32.721 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:56:32.721 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-10T07:56:32.721 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-10T07:56:32.721 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mgr.x 2024-02-10T07:56:33.060 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:56:32 smithi170 systemd[1]: Stopping Ceph mgr.x for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:56:33.366 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:56:33 smithi170 podman[161101]: 2024-02-10 07:56:33.05993916 +0000 UTC m=+0.146149306 container died 9015edea14abc905fd6808d7c9a43a9d7ad1cbc124f9b95ab4257ab30635cd06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a) 2024-02-10T07:56:33.633 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:56:33 smithi170 podman[161101]: 2024-02-10 07:56:33.554807118 +0000 UTC m=+0.641017263 container cleanup 9015edea14abc905fd6808d7c9a43a9d7ad1cbc124f9b95ab4257ab30635cd06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2) 2024-02-10T07:56:33.633 INFO:journalctl@ceph.mgr.x.smithi170.stdout:Feb 10 07:56:33 smithi170 bash[161101]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-mgr-x 2024-02-10T07:56:33.937 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@mgr.x.service' 2024-02-10T07:56:33.973 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:56:33.974 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-10T07:56:33.974 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-10T07:56:33.974 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.0 2024-02-10T07:56:34.278 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:56:34 smithi008 systemd[1]: Stopping Ceph osd.0 for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:56:34.582 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:56:34 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-0[44764]: 2024-02-10T07:56:34.276+0000 7fd19057d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:56:34.582 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:56:34 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-0[44764]: 2024-02-10T07:56:34.276+0000 7fd19057d700 -1 osd.0 503 *** Got signal Terminated *** 2024-02-10T07:56:34.582 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:56:34 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-0[44764]: 2024-02-10T07:56:34.276+0000 7fd19057d700 -1 osd.0 503 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:56:40.246 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:56:39 smithi008 podman[219109]: 2024-02-10 07:56:39.958160291 +0000 UTC m=+5.785186017 container died a123d6c92a846e4782fbc417c018253392874f373cf39855204a767f91371011 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-0, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=main-290212a) 2024-02-10T07:56:40.582 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:56:40 smithi008 podman[219109]: 2024-02-10 07:56:40.436775766 +0000 UTC m=+6.263801525 container cleanup a123d6c92a846e4782fbc417c018253392874f373cf39855204a767f91371011 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-0, io.buildah.version=1.33.2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T07:56:40.582 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:56:40 smithi008 bash[219109]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-0 2024-02-10T07:56:41.047 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:56:40 smithi008 podman[219126]: 2024-02-10 07:56:40.704944669 +0000 UTC m=+0.741363920 container remove a123d6c92a846e4782fbc417c018253392874f373cf39855204a767f91371011 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.schema-version=1.0) 2024-02-10T07:56:41.332 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:56:41 smithi008 podman[219219]: 2024-02-10 07:56:41.047477297 +0000 UTC m=+0.109745879 container create 2c99f85ce8414488569e1985bbb1d7f056fb2d2ba93a2cb14ed498992926150d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, RELEASE=main-290212a, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:56:41.332 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:56:41 smithi008 podman[219219]: 2024-02-10 07:56:40.955432779 +0000 UTC m=+0.017701364 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:56:41.332 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:56:41 smithi008 podman[219219]: 2024-02-10 07:56:41.291687217 +0000 UTC m=+0.353955800 container init 2c99f85ce8414488569e1985bbb1d7f056fb2d2ba93a2cb14ed498992926150d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-02-10T07:56:41.332 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:56:41 smithi008 podman[219219]: 2024-02-10 07:56:41.295065484 +0000 UTC m=+0.357334067 container start 2c99f85ce8414488569e1985bbb1d7f056fb2d2ba93a2cb14ed498992926150d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2) 2024-02-10T07:56:41.832 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:56:41 smithi008 podman[219219]: 2024-02-10 07:56:41.408674224 +0000 UTC m=+0.470942807 container attach 2c99f85ce8414488569e1985bbb1d7f056fb2d2ba93a2cb14ed498992926150d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2) 2024-02-10T07:56:42.262 INFO:journalctl@ceph.osd.0.smithi008.stdout:Feb 10 07:56:41 smithi008 podman[219219]: 2024-02-10 07:56:41.898282024 +0000 UTC m=+0.960550608 container died 2c99f85ce8414488569e1985bbb1d7f056fb2d2ba93a2cb14ed498992926150d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-10T07:56:42.553 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.0.service' 2024-02-10T07:56:42.594 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:56:42.595 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-10T07:56:42.595 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-10T07:56:42.595 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.1 2024-02-10T07:56:42.832 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:56:42 smithi008 systemd[1]: Stopping Ceph osd.1 for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:56:43.332 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:56:42 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-1[50308]: 2024-02-10T07:56:42.907+0000 7ff868745700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:56:43.332 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:56:42 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-1[50308]: 2024-02-10T07:56:42.907+0000 7ff868745700 -1 osd.1 503 *** Got signal Terminated *** 2024-02-10T07:56:43.332 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:56:42 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-1[50308]: 2024-02-10T07:56:42.907+0000 7ff868745700 -1 osd.1 503 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:56:48.832 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:56:48 smithi008 podman[219326]: 2024-02-10 07:56:48.535951363 +0000 UTC m=+5.719159708 container died 07cb566a11e93a15f3ecd5526b8070632107777bfe9fb938e20a336f9daf49ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-1, ceph=True, RELEASE=main-290212a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2) 2024-02-10T07:56:49.332 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:56:49 smithi008 podman[219326]: 2024-02-10 07:56:49.016577403 +0000 UTC m=+6.199785750 container cleanup 07cb566a11e93a15f3ecd5526b8070632107777bfe9fb938e20a336f9daf49ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131) 2024-02-10T07:56:49.332 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:56:49 smithi008 bash[219326]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-1 2024-02-10T07:56:49.383 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:56:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:56:49.112Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.8:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.8:8765: connect: connection refused" 2024-02-10T07:56:49.384 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:56:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:56:49.114Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.8:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.8:8765: connect: connection refused" 2024-02-10T07:56:49.384 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:56:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:56:49.115Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.8:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.8:8765: connect: connection refused" 2024-02-10T07:56:49.384 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:56:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:56:49.116Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.8:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.8:8765: connect: connection refused" 2024-02-10T07:56:49.657 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:56:49 smithi008 podman[219356]: 2024-02-10 07:56:49.335077031 +0000 UTC m=+0.794040645 container remove 07cb566a11e93a15f3ecd5526b8070632107777bfe9fb938e20a336f9daf49ff (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:56:50.082 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:56:49 smithi008 podman[219450]: 2024-02-10 07:56:49.561285095 +0000 UTC m=+0.018810582 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:56:50.082 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:56:49 smithi008 podman[219450]: 2024-02-10 07:56:49.661102501 +0000 UTC m=+0.118627979 container create d642a2724c3eaad30e71c3e2d0ddc570fb118722e3df2b1968a03f5ac4478375 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, maintainer=Guillaume Abrioux ) 2024-02-10T07:56:50.082 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:56:49 smithi008 podman[219450]: 2024-02-10 07:56:49.888538127 +0000 UTC m=+0.346063615 container init d642a2724c3eaad30e71c3e2d0ddc570fb118722e3df2b1968a03f5ac4478375 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240131, ceph=True, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-10T07:56:50.082 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:56:49 smithi008 podman[219450]: 2024-02-10 07:56:49.891703531 +0000 UTC m=+0.349229015 container start d642a2724c3eaad30e71c3e2d0ddc570fb118722e3df2b1968a03f5ac4478375 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-10T07:56:50.533 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:56:50 smithi008 podman[219450]: 2024-02-10 07:56:50.214205314 +0000 UTC m=+0.671730802 container attach d642a2724c3eaad30e71c3e2d0ddc570fb118722e3df2b1968a03f5ac4478375 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-10T07:56:50.533 INFO:journalctl@ceph.osd.1.smithi008.stdout:Feb 10 07:56:50 smithi008 podman[219450]: 2024-02-10 07:56:50.244593809 +0000 UTC m=+0.702119288 container died d642a2724c3eaad30e71c3e2d0ddc570fb118722e3df2b1968a03f5ac4478375 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-10T07:56:50.943 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.1.service' 2024-02-10T07:56:50.992 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:56:50.992 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-10T07:56:50.992 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-10T07:56:50.992 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.2 2024-02-10T07:56:51.288 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:56:51 smithi008 systemd[1]: Stopping Ceph osd.2 for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:56:51.582 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:56:51 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-2[56371]: 2024-02-10T07:56:51.287+0000 7fee49737700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:56:51.582 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:56:51 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-2[56371]: 2024-02-10T07:56:51.287+0000 7fee49737700 -1 osd.2 503 *** Got signal Terminated *** 2024-02-10T07:56:51.582 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:56:51 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-2[56371]: 2024-02-10T07:56:51.287+0000 7fee49737700 -1 osd.2 503 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:56:57.164 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:56:56 smithi008 podman[219556]: 2024-02-10 07:56:56.869735519 +0000 UTC m=+5.688477275 container died dad12ea2798e67df2142fb11573907fa9e9163340c74054c05959996d0146a3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.schema-version=1.0) 2024-02-10T07:56:57.582 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:56:57 smithi008 podman[219556]: 2024-02-10 07:56:57.329619481 +0000 UTC m=+6.148361292 container cleanup dad12ea2798e67df2142fb11573907fa9e9163340c74054c05959996d0146a3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-2, io.buildah.version=1.33.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:56:57.582 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:56:57 smithi008 bash[219556]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-2 2024-02-10T07:56:57.998 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:56:57 smithi008 podman[219574]: 2024-02-10 07:56:57.647458932 +0000 UTC m=+0.763525484 container remove dad12ea2798e67df2142fb11573907fa9e9163340c74054c05959996d0146a3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-10T07:56:58.318 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:56:57 smithi008 podman[219663]: 2024-02-10 07:56:57.902074912 +0000 UTC m=+0.018526436 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:56:58.318 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:56:58 smithi008 podman[219663]: 2024-02-10 07:56:58.015059756 +0000 UTC m=+0.131511278 container create 5d70217abcba7a28be09a37a7a92caf68da733efaf6736c121a47f1f349fafbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-10T07:56:58.318 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:56:58 smithi008 podman[219663]: 2024-02-10 07:56:58.234553435 +0000 UTC m=+0.351004955 container init 5d70217abcba7a28be09a37a7a92caf68da733efaf6736c121a47f1f349fafbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-290212a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, io.buildah.version=1.33.2) 2024-02-10T07:56:58.318 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:56:58 smithi008 podman[219663]: 2024-02-10 07:56:58.237648652 +0000 UTC m=+0.354100174 container start 5d70217abcba7a28be09a37a7a92caf68da733efaf6736c121a47f1f349fafbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-10T07:56:58.318 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:56:58 smithi008 podman[219663]: 2024-02-10 07:56:58.317852894 +0000 UTC m=+0.434304426 container attach 5d70217abcba7a28be09a37a7a92caf68da733efaf6736c121a47f1f349fafbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, RELEASE=main-290212a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-10T07:56:58.582 INFO:journalctl@ceph.osd.2.smithi008.stdout:Feb 10 07:56:58 smithi008 podman[219663]: 2024-02-10 07:56:58.52952672 +0000 UTC m=+0.645978231 container died 5d70217abcba7a28be09a37a7a92caf68da733efaf6736c121a47f1f349fafbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-10T07:56:59.108 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.2.service' 2024-02-10T07:56:59.155 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:56:59.155 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-10T07:56:59.155 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-10T07:56:59.155 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.3 2024-02-10T07:56:59.332 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:56:59 smithi008 systemd[1]: Stopping Ceph osd.3 for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:56:59.832 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:56:59 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3[169715]: 2024-02-10T07:56:59.457+0000 7fb49cd8e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:56:59.832 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:56:59 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3[169715]: 2024-02-10T07:56:59.457+0000 7fb49cd8e700 -1 osd.3 503 *** Got signal Terminated *** 2024-02-10T07:56:59.832 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:56:59 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3[169715]: 2024-02-10T07:56:59.457+0000 7fb49cd8e700 -1 osd.3 503 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:57:01.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:01 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:01.427+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:02.152 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:57:02.020+0000 7f98afd8e700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:33.378231+0000 front 2024-02-10T07:56:33.378285+0000 (oldest deadline 2024-02-10T07:57:01.578713+0000) 2024-02-10T07:57:02.579 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:02.398+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:02.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:02 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:02.578+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:03.383 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:57:03.034+0000 7f98afd8e700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:33.378231+0000 front 2024-02-10T07:56:33.378285+0000 (oldest deadline 2024-02-10T07:57:01.578713+0000) 2024-02-10T07:57:03.832 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:57:03 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3[169715]: 2024-02-10T07:57:03.452+0000 7fb49537a700 -1 osd.3 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.990320+0000 front 2024-02-10T07:56:38.990399+0000 (oldest deadline 2024-02-10T07:57:03.090417+0000) 2024-02-10T07:57:03.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:03.401+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:03.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:03 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:03.572+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:04.352 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:04 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:57:04.054+0000 7f98afd8e700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:33.378231+0000 front 2024-02-10T07:56:33.378285+0000 (oldest deadline 2024-02-10T07:57:01.578713+0000) 2024-02-10T07:57:04.352 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:04 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:04.289+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:04.353 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:04 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:04.351+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:04.633 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:04 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:04.548+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:05.168 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:57:04 smithi008 podman[219767]: 2024-02-10 07:57:04.896005869 +0000 UTC m=+5.554414563 container died e65d1d5399fbdba4f0c3e3332fe06d71c789e0a1011249073785e3bf4da79cbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-10T07:57:05.306 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:05 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:57:05.045+0000 7f98afd8e700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:33.378231+0000 front 2024-02-10T07:56:33.378285+0000 (oldest deadline 2024-02-10T07:57:01.578713+0000) 2024-02-10T07:57:05.560 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:05 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:05.364+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:05.560 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:05 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:05.305+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:05.582 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:57:05 smithi008 podman[219767]: 2024-02-10 07:57:05.391677281 +0000 UTC m=+6.050086011 container cleanup e65d1d5399fbdba4f0c3e3332fe06d71c789e0a1011249073785e3bf4da79cbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-10T07:57:05.582 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:57:05 smithi008 bash[219767]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3 2024-02-10T07:57:05.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:05 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:05.559+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:05.968 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:57:05 smithi008 podman[219784]: 2024-02-10 07:57:05.634855787 +0000 UTC m=+0.735329183 container remove e65d1d5399fbdba4f0c3e3332fe06d71c789e0a1011249073785e3bf4da79cbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-10T07:57:06.246 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:57:05 smithi008 podman[219876]: 2024-02-10 07:57:05.870818054 +0000 UTC m=+0.017429963 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:57:06.246 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:57:06 smithi008 podman[219876]: 2024-02-10 07:57:06.069022539 +0000 UTC m=+0.215634483 container create bb8f2fff843b307493668edc8038cc53fe924732100fd12d39d10eab5101ef1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a) 2024-02-10T07:57:06.246 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:57:06 smithi008 podman[219876]: 2024-02-10 07:57:06.246444522 +0000 UTC m=+0.393056425 container init bb8f2fff843b307493668edc8038cc53fe924732100fd12d39d10eab5101ef1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-10T07:57:06.341 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:06 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:57:06.059+0000 7f98afd8e700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:33.378231+0000 front 2024-02-10T07:56:33.378285+0000 (oldest deadline 2024-02-10T07:57:01.578713+0000) 2024-02-10T07:57:06.543 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:57:06 smithi008 podman[219876]: 2024-02-10 07:57:06.249520962 +0000 UTC m=+0.396132865 container start bb8f2fff843b307493668edc8038cc53fe924732100fd12d39d10eab5101ef1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:57:06.544 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:57:06 smithi008 podman[219876]: 2024-02-10 07:57:06.313106764 +0000 UTC m=+0.459718683 container attach bb8f2fff843b307493668edc8038cc53fe924732100fd12d39d10eab5101ef1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-10T07:57:06.633 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:06 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:06.347+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:06.633 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:06 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:06.341+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:06.634 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:06 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:06.574+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:06.832 INFO:journalctl@ceph.osd.3.smithi008.stdout:Feb 10 07:57:06 smithi008 podman[219876]: 2024-02-10 07:57:06.543996816 +0000 UTC m=+0.690608722 container died bb8f2fff843b307493668edc8038cc53fe924732100fd12d39d10eab5101ef1e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T07:57:07.138 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.3.service' 2024-02-10T07:57:07.181 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:57:07.182 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-10T07:57:07.182 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-10T07:57:07.182 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.4 2024-02-10T07:57:07.346 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:07 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:57:07.094+0000 7f98afd8e700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:33.378231+0000 front 2024-02-10T07:56:33.378285+0000 (oldest deadline 2024-02-10T07:57:01.578713+0000) 2024-02-10T07:57:07.346 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:07 smithi170 systemd[1]: Stopping Ceph osd.4 for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:57:07.633 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:07 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:07.344+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:07.633 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:07 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:07.351+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:07.634 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:07 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:07.558+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:07.634 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:07 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:57:07.481+0000 7f98b77a2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:57:07.634 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:07 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:57:07.481+0000 7f98b77a2700 -1 osd.4 503 *** Got signal Terminated *** 2024-02-10T07:57:07.634 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:07 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:57:07.481+0000 7f98b77a2700 -1 osd.4 503 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:57:08.383 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:08 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:57:08.130+0000 7f98afd8e700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:33.378231+0000 front 2024-02-10T07:56:33.378285+0000 (oldest deadline 2024-02-10T07:57:01.578713+0000) 2024-02-10T07:57:08.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:08 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:08.299+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:08.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:08 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:08.384+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:08.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:08 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:08.548+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:09.507 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:09 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:57:09.157+0000 7f98afd8e700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:33.378231+0000 front 2024-02-10T07:56:33.378285+0000 (oldest deadline 2024-02-10T07:57:01.578713+0000) 2024-02-10T07:57:09.507 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:09 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:09.310+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:09.507 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:09 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:09.310+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:43.862010+0000 front 2024-02-10T07:56:43.862099+0000 (oldest deadline 2024-02-10T07:57:08.562010+0000) 2024-02-10T07:57:09.508 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:09 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:09.406+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:09.508 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:09 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:09.406+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:09.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:09 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:09.506+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:10.383 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:10 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:57:10.120+0000 7f98afd8e700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:33.378231+0000 front 2024-02-10T07:56:33.378285+0000 (oldest deadline 2024-02-10T07:57:01.578713+0000) 2024-02-10T07:57:10.384 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:10 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:10.312+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:10.384 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:10 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:10.312+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:43.862010+0000 front 2024-02-10T07:56:43.862099+0000 (oldest deadline 2024-02-10T07:57:08.562010+0000) 2024-02-10T07:57:10.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:10 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:10.546+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:10.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:10 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:10.423+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:10.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:10 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:10.423+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:11.406 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:11 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:57:11.150+0000 7f98afd8e700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:33.378231+0000 front 2024-02-10T07:56:33.378285+0000 (oldest deadline 2024-02-10T07:57:01.578713+0000) 2024-02-10T07:57:11.406 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:11 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:11.287+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:11.407 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:11 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:11.287+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:43.862010+0000 front 2024-02-10T07:56:43.862099+0000 (oldest deadline 2024-02-10T07:57:08.562010+0000) 2024-02-10T07:57:11.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:11 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:11.405+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:11.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:11 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:11.405+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:11.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:11 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:11.513+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:11.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:11 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:11.513+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:12.545 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:12 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4[38313]: 2024-02-10T07:57:12.173+0000 7f98afd8e700 -1 osd.4 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:33.378231+0000 front 2024-02-10T07:56:33.378285+0000 (oldest deadline 2024-02-10T07:57:01.578713+0000) 2024-02-10T07:57:12.545 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:12 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:12.392+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:12.545 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:12 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:12.392+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:12.546 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:12 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:12.240+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:12.546 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:12 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:12.240+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:43.862010+0000 front 2024-02-10T07:56:43.862099+0000 (oldest deadline 2024-02-10T07:57:08.562010+0000) 2024-02-10T07:57:12.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:12 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:12.544+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:12.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:12 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:12.544+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:13.383 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:13 smithi170 podman[161241]: 2024-02-10 07:57:13.095433492 +0000 UTC m=+5.712999418 container died 466523c072a4741fe66556628931cc4caba877d9abc5b109228bb6862d552e2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131) 2024-02-10T07:57:13.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:13.245+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:13.384 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:13.246+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:43.862010+0000 front 2024-02-10T07:56:43.862099+0000 (oldest deadline 2024-02-10T07:57:08.562010+0000) 2024-02-10T07:57:13.816 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:13 smithi170 podman[161241]: 2024-02-10 07:57:13.648798181 +0000 UTC m=+6.266364115 container cleanup 466523c072a4741fe66556628931cc4caba877d9abc5b109228bb6862d552e2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-10T07:57:13.817 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:13 smithi170 bash[161241]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4 2024-02-10T07:57:13.817 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:13.431+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:13.817 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:13.431+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:13.817 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:13.584+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:13.817 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:13 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:13.584+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:14.133 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:13 smithi170 podman[161258]: 2024-02-10 07:57:13.816963366 +0000 UTC m=+0.717981625 container remove 466523c072a4741fe66556628931cc4caba877d9abc5b109228bb6862d552e2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:57:14.468 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:14 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:14.201+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:14.468 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:14 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:14.201+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:43.862010+0000 front 2024-02-10T07:56:43.862099+0000 (oldest deadline 2024-02-10T07:57:08.562010+0000) 2024-02-10T07:57:14.469 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:14 smithi170 podman[161350]: 2024-02-10 07:57:14.115741677 +0000 UTC m=+0.018310268 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:57:14.469 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:14 smithi170 podman[161350]: 2024-02-10 07:57:14.217903377 +0000 UTC m=+0.120471958 container create 139b3dd4dd35faa4ab34bbec1d18cd374aec6940821f1d30ba7b26c4e49070ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:57:14.469 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:14 smithi170 podman[161350]: 2024-02-10 07:57:14.445296116 +0000 UTC m=+0.347864707 container init 139b3dd4dd35faa4ab34bbec1d18cd374aec6940821f1d30ba7b26c4e49070ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:57:14.469 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:14 smithi170 podman[161350]: 2024-02-10 07:57:14.448445761 +0000 UTC m=+0.351014349 container start 139b3dd4dd35faa4ab34bbec1d18cd374aec6940821f1d30ba7b26c4e49070ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True) 2024-02-10T07:57:14.883 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:14 smithi170 podman[161350]: 2024-02-10 07:57:14.562531005 +0000 UTC m=+0.465099649 container attach 139b3dd4dd35faa4ab34bbec1d18cd374aec6940821f1d30ba7b26c4e49070ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:57:14.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:14 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:14.466+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:14.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:14 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:14.466+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:14.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:14 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:14.609+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:14.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:14 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:14.609+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:15.622 INFO:journalctl@ceph.osd.4.smithi170.stdout:Feb 10 07:57:15 smithi170 podman[161350]: 2024-02-10 07:57:15.367836946 +0000 UTC m=+1.270405537 container died 139b3dd4dd35faa4ab34bbec1d18cd374aec6940821f1d30ba7b26c4e49070ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-4-deactivate, ceph=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131) 2024-02-10T07:57:15.623 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:15 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:15.188+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:15.623 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:15 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:15.188+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:43.862010+0000 front 2024-02-10T07:56:43.862099+0000 (oldest deadline 2024-02-10T07:57:08.562010+0000) 2024-02-10T07:57:15.623 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:15 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:15.431+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:15.623 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:15 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:15.431+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:15.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:15 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:15.620+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:15.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:15 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:15.620+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:16.018 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.4.service' 2024-02-10T07:57:16.060 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:57:16.060 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-10T07:57:16.060 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-10T07:57:16.061 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.5 2024-02-10T07:57:16.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:16 smithi170 systemd[1]: Stopping Ceph osd.5 for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:57:16.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:16 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:16.180+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:16.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:16 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:16.180+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:43.862010+0000 front 2024-02-10T07:56:43.862099+0000 (oldest deadline 2024-02-10T07:57:08.562010+0000) 2024-02-10T07:57:16.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:16 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:16.416+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:16.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:16 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:16.416+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:16.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:16 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:16.612+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:16.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:16 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:16.612+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:16.884 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:16 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:16.409+0000 7f6fa4e2a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:57:16.884 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:16 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:16.409+0000 7f6fa4e2a700 -1 osd.5 503 *** Got signal Terminated *** 2024-02-10T07:57:16.884 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:16 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:16.409+0000 7f6fa4e2a700 -1 osd.5 503 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:57:17.460 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:17 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:17.175+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:17.460 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:17 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:17.175+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:43.862010+0000 front 2024-02-10T07:56:43.862099+0000 (oldest deadline 2024-02-10T07:57:08.562010+0000) 2024-02-10T07:57:17.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:17 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:17.459+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:17.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:17 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:17.459+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:17.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:17 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:17.614+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:17.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:17 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:17.614+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:18.507 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:18 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:18.142+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:18.507 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:18 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:18.142+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:43.862010+0000 front 2024-02-10T07:56:43.862099+0000 (oldest deadline 2024-02-10T07:57:08.562010+0000) 2024-02-10T07:57:18.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:18 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:18.622+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:18.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:18 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:18.622+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:18.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:18 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:18.506+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:18.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:18 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:18.506+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:18.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:18 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:18.506+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:53.736524+0000 front 2024-02-10T07:56:53.736656+0000 (oldest deadline 2024-02-10T07:57:17.836457+0000) 2024-02-10T07:57:19.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:19 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:19.096+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:19.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:19 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:19.096+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:43.862010+0000 front 2024-02-10T07:56:43.862099+0000 (oldest deadline 2024-02-10T07:57:08.562010+0000) 2024-02-10T07:57:19.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:19 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:19.096+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:53.163446+0000 front 2024-02-10T07:56:53.163497+0000 (oldest deadline 2024-02-10T07:57:19.063382+0000) 2024-02-10T07:57:19.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:19 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:19.640+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:19.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:19 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:19.640+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:19.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:19 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:19.527+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:19.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:19 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:19.527+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:19.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:19 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:19.527+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:53.736524+0000 front 2024-02-10T07:56:53.736656+0000 (oldest deadline 2024-02-10T07:57:17.836457+0000) 2024-02-10T07:57:20.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:20.079+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:20.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:20.079+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:43.862010+0000 front 2024-02-10T07:56:43.862099+0000 (oldest deadline 2024-02-10T07:57:08.562010+0000) 2024-02-10T07:57:20.384 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:20.079+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:53.163446+0000 front 2024-02-10T07:56:53.163497+0000 (oldest deadline 2024-02-10T07:57:19.063382+0000) 2024-02-10T07:57:20.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:20.641+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:20.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:20.641+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:20.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:20.641+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:20.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:20.546+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:20.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:20.546+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:20.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:20 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:20.546+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:53.736524+0000 front 2024-02-10T07:56:53.736656+0000 (oldest deadline 2024-02-10T07:57:17.836457+0000) 2024-02-10T07:57:21.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:21 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:21.055+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:38.561579+0000 front 2024-02-10T07:56:38.561624+0000 (oldest deadline 2024-02-10T07:57:03.861540+0000) 2024-02-10T07:57:21.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:21 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:21.055+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:43.862010+0000 front 2024-02-10T07:56:43.862099+0000 (oldest deadline 2024-02-10T07:57:08.562010+0000) 2024-02-10T07:57:21.383 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:21 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5[79865]: 2024-02-10T07:57:21.055+0000 7f6fa141e700 -1 osd.5 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:53.163446+0000 front 2024-02-10T07:56:53.163497+0000 (oldest deadline 2024-02-10T07:57:19.063382+0000) 2024-02-10T07:57:21.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:21 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:21.654+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:21.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:21 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:21.654+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:21.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:21 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:21.654+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:21.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:21 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:21.535+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:21.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:21 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:21.535+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:21.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:21 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:21.535+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:53.736524+0000 front 2024-02-10T07:56:53.736656+0000 (oldest deadline 2024-02-10T07:57:17.836457+0000) 2024-02-10T07:57:22.299 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:21 smithi170 podman[161468]: 2024-02-10 07:57:21.988093446 +0000 UTC m=+5.679474879 container died f398c377f85d658abc6c0040e1b95c12700080ea3a4f7a675b374d41d5b0e155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD) 2024-02-10T07:57:22.583 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:22 smithi170 podman[161468]: 2024-02-10 07:57:22.40208648 +0000 UTC m=+6.093467919 container cleanup f398c377f85d658abc6c0040e1b95c12700080ea3a4f7a675b374d41d5b0e155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux ) 2024-02-10T07:57:22.584 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:22 smithi170 bash[161468]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5 2024-02-10T07:57:22.883 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:22 smithi170 podman[161488]: 2024-02-10 07:57:22.636910236 +0000 UTC m=+0.644927988 container remove f398c377f85d658abc6c0040e1b95c12700080ea3a4f7a675b374d41d5b0e155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, ceph=True) 2024-02-10T07:57:22.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:22.582+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:22.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:22.582+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:22.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:22.582+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:53.736524+0000 front 2024-02-10T07:56:53.736656+0000 (oldest deadline 2024-02-10T07:57:17.836457+0000) 2024-02-10T07:57:22.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:22.621+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:22.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:22.621+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:22.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:22 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:22.621+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:23.299 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:22 smithi170 podman[161579]: 2024-02-10 07:57:22.873717373 +0000 UTC m=+0.018063929 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:57:23.299 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:23 smithi170 podman[161579]: 2024-02-10 07:57:23.004497291 +0000 UTC m=+0.148843839 container create 4723c461d7fff11277181a96811b0603e38451191ac370f915e1f57cb0406004 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-02-10T07:57:23.299 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:23 smithi170 podman[161579]: 2024-02-10 07:57:23.215145016 +0000 UTC m=+0.359491563 container init 4723c461d7fff11277181a96811b0603e38451191ac370f915e1f57cb0406004 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-290212a) 2024-02-10T07:57:23.299 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:23 smithi170 podman[161579]: 2024-02-10 07:57:23.218135361 +0000 UTC m=+0.362481916 container start 4723c461d7fff11277181a96811b0603e38451191ac370f915e1f57cb0406004 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-10T07:57:23.299 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:23 smithi170 podman[161579]: 2024-02-10 07:57:23.298357293 +0000 UTC m=+0.442703850 container attach 4723c461d7fff11277181a96811b0603e38451191ac370f915e1f57cb0406004 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS) 2024-02-10T07:57:23.587 INFO:journalctl@ceph.osd.5.smithi170.stdout:Feb 10 07:57:23 smithi170 podman[161579]: 2024-02-10 07:57:23.514645594 +0000 UTC m=+0.658992143 container died 4723c461d7fff11277181a96811b0603e38451191ac370f915e1f57cb0406004 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-10T07:57:23.877 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:23.598+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:23.877 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:23.598+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:23.877 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:23.598+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:23.878 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:23.586+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:23.878 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:23.586+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:23.878 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:23 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:23.586+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:53.736524+0000 front 2024-02-10T07:56:53.736656+0000 (oldest deadline 2024-02-10T07:57:17.836457+0000) 2024-02-10T07:57:24.172 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.5.service' 2024-02-10T07:57:24.213 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:57:24.213 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-10T07:57:24.214 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-10T07:57:24.214 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.6 2024-02-10T07:57:24.547 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:24 smithi170 systemd[1]: Stopping Ceph osd.6 for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:57:24.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:24.604+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:24.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:24.604+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:24.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:24.604+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:24.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:24.545+0000 7fc48b6aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:57:24.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:24.545+0000 7fc48b6aa700 -1 osd.6 503 *** Got signal Terminated *** 2024-02-10T07:57:24.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:24.545+0000 7fc48b6aa700 -1 osd.6 503 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:57:24.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:24.598+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:24.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:24.598+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:24.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:24 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:24.598+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:53.736524+0000 front 2024-02-10T07:56:53.736656+0000 (oldest deadline 2024-02-10T07:57:17.836457+0000) 2024-02-10T07:57:25.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:25.563+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:25.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:25.563+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:25.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:25.563+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:53.736524+0000 front 2024-02-10T07:56:53.736656+0000 (oldest deadline 2024-02-10T07:57:17.836457+0000) 2024-02-10T07:57:25.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:25.642+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:25.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:25.642+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:25.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:25.642+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:25.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:25 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:25.642+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:57:02.606821+0000 front 2024-02-10T07:57:02.606867+0000 (oldest deadline 2024-02-10T07:57:25.506900+0000) 2024-02-10T07:57:26.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:26 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:26.564+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:26.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:26 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:26.564+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:26.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:26 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:26.564+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:53.736524+0000 front 2024-02-10T07:56:53.736656+0000 (oldest deadline 2024-02-10T07:57:17.836457+0000) 2024-02-10T07:57:26.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:26 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:26.690+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:26.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:26 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:26.690+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:26.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:26 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:26.690+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:26.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:26 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:26.690+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:57:02.606821+0000 front 2024-02-10T07:57:02.606867+0000 (oldest deadline 2024-02-10T07:57:25.506900+0000) 2024-02-10T07:57:27.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:27 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:27.588+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:27.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:27 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:27.588+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:27.883 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:27 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:27.588+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:53.736524+0000 front 2024-02-10T07:56:53.736656+0000 (oldest deadline 2024-02-10T07:57:17.836457+0000) 2024-02-10T07:57:27.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:27 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:27.677+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:27.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:27 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:27.677+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:27.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:27 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:27.677+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:27.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:27 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:27.677+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:57:02.606821+0000 front 2024-02-10T07:57:02.606867+0000 (oldest deadline 2024-02-10T07:57:25.506900+0000) 2024-02-10T07:57:28.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:28 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:28.659+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:28.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:28 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:28.659+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:28.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:28 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:28.659+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:28.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:28 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:28.659+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:57:02.606821+0000 front 2024-02-10T07:57:02.606867+0000 (oldest deadline 2024-02-10T07:57:25.506900+0000) 2024-02-10T07:57:28.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:28 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:28.600+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:37.633414+0000 front 2024-02-10T07:56:37.633386+0000 (oldest deadline 2024-02-10T07:57:01.133175+0000) 2024-02-10T07:57:28.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:28 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:28.600+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:44.034097+0000 front 2024-02-10T07:56:44.034071+0000 (oldest deadline 2024-02-10T07:57:08.634661+0000) 2024-02-10T07:57:28.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:28 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:28.600+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:53.736524+0000 front 2024-02-10T07:56:53.736656+0000 (oldest deadline 2024-02-10T07:57:17.836457+0000) 2024-02-10T07:57:28.884 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:28 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6[50241]: 2024-02-10T07:57:28.600+0000 7fc487c9e700 -1 osd.6 503 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:57:01.937317+0000 front 2024-02-10T07:57:01.937412+0000 (oldest deadline 2024-02-10T07:57:27.837229+0000) 2024-02-10T07:57:30.021 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:29.699+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:30.021 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:29.699+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:30.021 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:29.699+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:30.021 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:29 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:29.699+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:57:02.606821+0000 front 2024-02-10T07:57:02.606867+0000 (oldest deadline 2024-02-10T07:57:25.506900+0000) 2024-02-10T07:57:30.339 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:30 smithi170 podman[161684]: 2024-02-10 07:57:30.020469149 +0000 UTC m=+5.579096227 container died 4ea2d62f63b91b16cec8a0eba14ec7d9069c05d92169083807736bd98ed6bcd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-10T07:57:30.633 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:30 smithi170 podman[161684]: 2024-02-10 07:57:30.463029996 +0000 UTC m=+6.021657074 container cleanup 4ea2d62f63b91b16cec8a0eba14ec7d9069c05d92169083807736bd98ed6bcd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True, RELEASE=main-290212a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=) 2024-02-10T07:57:30.633 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:30 smithi170 bash[161684]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6 2024-02-10T07:57:31.065 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:30 smithi170 podman[161704]: 2024-02-10 07:57:30.672717227 +0000 UTC m=+0.640177020 container remove 4ea2d62f63b91b16cec8a0eba14ec7d9069c05d92169083807736bd98ed6bcd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD) 2024-02-10T07:57:31.065 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:30.676+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:31.065 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:30.676+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:31.065 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:30.676+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:31.065 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:30 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:30.676+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:57:02.606821+0000 front 2024-02-10T07:57:02.606867+0000 (oldest deadline 2024-02-10T07:57:25.506900+0000) 2024-02-10T07:57:31.383 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:31 smithi170 podman[161797]: 2024-02-10 07:57:31.06524277 +0000 UTC m=+0.113737308 container create 57e8606eb4d19754eb7913c1ce8d8599e7d09cede9d90ed47748630641577eca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-10T07:57:31.384 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:31 smithi170 podman[161797]: 2024-02-10 07:57:30.970109993 +0000 UTC m=+0.018604490 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:57:31.384 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:31 smithi170 podman[161797]: 2024-02-10 07:57:31.309775852 +0000 UTC m=+0.358270344 container init 57e8606eb4d19754eb7913c1ce8d8599e7d09cede9d90ed47748630641577eca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131) 2024-02-10T07:57:31.384 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:31 smithi170 podman[161797]: 2024-02-10 07:57:31.313612492 +0000 UTC m=+0.362106990 container start 57e8606eb4d19754eb7913c1ce8d8599e7d09cede9d90ed47748630641577eca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-10T07:57:31.716 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:31 smithi170 podman[161797]: 2024-02-10 07:57:31.384410426 +0000 UTC m=+0.432904930 container attach 57e8606eb4d19754eb7913c1ce8d8599e7d09cede9d90ed47748630641577eca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a) 2024-02-10T07:57:31.717 INFO:journalctl@ceph.osd.6.smithi170.stdout:Feb 10 07:57:31 smithi170 podman[161797]: 2024-02-10 07:57:31.607698096 +0000 UTC m=+0.656192604 container died 57e8606eb4d19754eb7913c1ce8d8599e7d09cede9d90ed47748630641577eca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True) 2024-02-10T07:57:32.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:31 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:31.716+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:32.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:31 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:31.716+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:32.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:31 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:31.716+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:32.134 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:31 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:31.716+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:57:02.606821+0000 front 2024-02-10T07:57:02.606867+0000 (oldest deadline 2024-02-10T07:57:25.506900+0000) 2024-02-10T07:57:32.316 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.6.service' 2024-02-10T07:57:32.360 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:57:32.361 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-10T07:57:32.361 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-10T07:57:32.361 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.7 2024-02-10T07:57:32.583 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:32 smithi170 systemd[1]: Stopping Ceph osd.7 for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:57:32.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:32 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:32.673+0000 7f082f5cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:57:32.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:32 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:32.673+0000 7f082f5cc700 -1 osd.7 503 *** Got signal Terminated *** 2024-02-10T07:57:32.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:32 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:32.673+0000 7f082f5cc700 -1 osd.7 503 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T07:57:32.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:32 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:32.711+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:32.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:32 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:32.711+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:32.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:32 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:32.711+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:32.884 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:32 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:32.711+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:57:02.606821+0000 front 2024-02-10T07:57:02.606867+0000 (oldest deadline 2024-02-10T07:57:25.506900+0000) 2024-02-10T07:57:33.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:33.734+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:33.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:33.734+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:33.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:33.734+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:33.883 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:33 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:33.734+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:57:02.606821+0000 front 2024-02-10T07:57:02.606867+0000 (oldest deadline 2024-02-10T07:57:25.506900+0000) 2024-02-10T07:57:35.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:34 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:34.782+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:35.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:34 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:34.782+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:35.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:34 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:34.782+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:35.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:34 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:34.782+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:57:02.606821+0000 front 2024-02-10T07:57:02.606867+0000 (oldest deadline 2024-02-10T07:57:25.506900+0000) 2024-02-10T07:57:36.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:35 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:35.792+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:36.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:35 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:35.792+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:36.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:35 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:35.792+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:36.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:35 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:35.792+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:57:02.606821+0000 front 2024-02-10T07:57:02.606867+0000 (oldest deadline 2024-02-10T07:57:25.506900+0000) 2024-02-10T07:57:37.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:36 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:36.833+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6806 osd.0 since back 2024-02-10T07:56:36.503113+0000 front 2024-02-10T07:56:36.503171+0000 (oldest deadline 2024-02-10T07:57:02.402918+0000) 2024-02-10T07:57:37.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:36 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:36.833+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6814 osd.1 since back 2024-02-10T07:56:45.303913+0000 front 2024-02-10T07:56:45.303901+0000 (oldest deadline 2024-02-10T07:57:10.603818+0000) 2024-02-10T07:57:37.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:36 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:36.833+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6822 osd.2 since back 2024-02-10T07:56:55.705759+0000 front 2024-02-10T07:56:55.705646+0000 (oldest deadline 2024-02-10T07:57:19.805604+0000) 2024-02-10T07:57:37.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:36 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:36.833+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.8:6830 osd.3 since back 2024-02-10T07:57:02.606821+0000 front 2024-02-10T07:57:02.606867+0000 (oldest deadline 2024-02-10T07:57:25.506900+0000) 2024-02-10T07:57:37.133 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:36 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7[56307]: 2024-02-10T07:57:36.833+0000 7f082bbc0700 -1 osd.7 503 heartbeat_check: no reply from 172.21.15.170:6804 osd.4 since back 2024-02-10T07:57:10.608755+0000 front 2024-02-10T07:57:10.608718+0000 (oldest deadline 2024-02-10T07:57:35.908639+0000) 2024-02-10T07:57:38.633 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:38 smithi170 podman[161904]: 2024-02-10 07:57:38.354168271 +0000 UTC m=+5.768659833 container died 623262f9b38bf2dbabf273a0d6f3e0e744380e404d2faa7eee3190d206773c6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2) 2024-02-10T07:57:39.025 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:38 smithi170 podman[161904]: 2024-02-10 07:57:38.732223681 +0000 UTC m=+6.146715232 container cleanup 623262f9b38bf2dbabf273a0d6f3e0e744380e404d2faa7eee3190d206773c6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T07:57:39.025 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:38 smithi170 bash[161904]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7 2024-02-10T07:57:39.343 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:39 smithi170 podman[161931]: 2024-02-10 07:57:39.025172162 +0000 UTC m=+0.670965408 container remove 623262f9b38bf2dbabf273a0d6f3e0e744380e404d2faa7eee3190d206773c6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-290212a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:57:39.633 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:39 smithi170 podman[162022]: 2024-02-10 07:57:39.246237734 +0000 UTC m=+0.018748310 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T07:57:39.633 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:39 smithi170 podman[162022]: 2024-02-10 07:57:39.350957803 +0000 UTC m=+0.123468370 container create cd28be6650d3c4ac83b56174c02c5c838638ee5437af06f58d04741a4e642ed8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131) 2024-02-10T07:57:39.633 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:39 smithi170 podman[162022]: 2024-02-10 07:57:39.561706041 +0000 UTC m=+0.334216608 container init cd28be6650d3c4ac83b56174c02c5c838638ee5437af06f58d04741a4e642ed8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T07:57:39.633 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:39 smithi170 podman[162022]: 2024-02-10 07:57:39.56486001 +0000 UTC m=+0.337370584 container start cd28be6650d3c4ac83b56174c02c5c838638ee5437af06f58d04741a4e642ed8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-10T07:57:40.131 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:39 smithi170 podman[162022]: 2024-02-10 07:57:39.636563446 +0000 UTC m=+0.409074019 container attach cd28be6650d3c4ac83b56174c02c5c838638ee5437af06f58d04741a4e642ed8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-10T07:57:40.131 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:39 smithi170 podman[162022]: 2024-02-10 07:57:39.857952603 +0000 UTC m=+0.630463169 container died cd28be6650d3c4ac83b56174c02c5c838638ee5437af06f58d04741a4e642ed8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:57:40.383 INFO:journalctl@ceph.osd.7.smithi170.stdout:Feb 10 07:57:40 smithi170 podman[162049]: 2024-02-10 07:57:40.373201443 +0000 UTC m=+0.508138921 container remove cd28be6650d3c4ac83b56174c02c5c838638ee5437af06f58d04741a4e642ed8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-10T07:57:40.404 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@osd.7.service' 2024-02-10T07:57:40.446 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:57:40.446 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-10T07:57:40.446 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-02-10T07:57:40.446 DEBUG:teuthology.orchestra.run.smithi008:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@rgw.foo.a 2024-02-10T07:57:40.758 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:57:40 smithi008 systemd[1]: Stopping Ceph rgw.foo.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:57:41.025 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:57:40 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-rgw-foo-a[67625]: 2024-02-10T07:57:40.756+0000 7feb1469f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T07:57:41.025 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:57:40 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-rgw-foo-a[67625]: 2024-02-10T07:57:40.756+0000 7feb22f10ac0 -1 shutting down 2024-02-10T07:57:49.383 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:49.112Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.8:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.8:8765: connect: connection refused" 2024-02-10T07:57:49.384 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:49.114Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.8:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.8:8765: connect: connection refused" 2024-02-10T07:57:49.384 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:49.114Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.8:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.8:8765: connect: connection refused" 2024-02-10T07:57:49.384 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:49 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:49.117Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.8:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.8:8765: connect: connection refused" 2024-02-10T07:57:51.049 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:57:50 smithi008 bash[219982]: time="2024-02-10T07:57:50Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-02-10T07:57:51.049 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:57:50 smithi008 podman[219982]: 2024-02-10 07:57:50.779814645 +0000 UTC m=+10.134807700 container died d8d2986f826582c7030e8f62dc19a7dd3b2266c70bed9bef59f60b3dee216164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-rgw-foo-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-10T07:57:51.332 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:57:51 smithi008 podman[219982]: 2024-02-10 07:57:51.240226513 +0000 UTC m=+10.595219565 container cleanup d8d2986f826582c7030e8f62dc19a7dd3b2266c70bed9bef59f60b3dee216164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-10T07:57:51.332 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:57:51 smithi008 bash[219982]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-rgw-foo-a 2024-02-10T07:57:51.804 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:57:51 smithi008 podman[220009]: 2024-02-10 07:57:51.666548187 +0000 UTC m=+0.872722424 container remove d8d2986f826582c7030e8f62dc19a7dd3b2266c70bed9bef59f60b3dee216164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-rgw-foo-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True) 2024-02-10T07:57:51.804 INFO:journalctl@ceph.rgw.foo.a.smithi008.stdout:Feb 10 07:57:51 smithi008 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-02-10T07:57:51.811 DEBUG:teuthology.orchestra.run.smithi008:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@rgw.foo.a.service' 2024-02-10T07:57:51.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:57:51.849 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-02-10T07:57:51.849 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-10T07:57:51.849 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@prometheus.a 2024-02-10T07:57:52.133 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:51 smithi170 systemd[1]: Stopping Ceph prometheus.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:57:52.458 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:52.166Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T07:57:52.458 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:52.166Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-10T07:57:52.458 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:52.166Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-10T07:57:52.458 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:52.166Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T07:57:52.458 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:52.166Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-10T07:57:52.459 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:52.166Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-10T07:57:52.459 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:52.166Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T07:57:52.459 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:52.166Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-10T07:57:52.459 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:52.175Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-10T07:57:52.459 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:52.175Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T07:57:52.459 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:52.175Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-10T07:57:52.459 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a[70479]: ts=2024-02-10T07:57:52.175Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-10T07:57:52.459 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 podman[162128]: 2024-02-10 07:57:52.187046059 +0000 UTC m=+0.140215352 container died 78459537e7c8487a980f395a10ee3dec052a7412d81f3b409456d11d156888e0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:57:52.842 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 podman[162128]: 2024-02-10 07:57:52.557663398 +0000 UTC m=+0.510832691 container cleanup 78459537e7c8487a980f395a10ee3dec052a7412d81f3b409456d11d156888e0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T07:57:52.842 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:Feb 10 07:57:52 smithi170 bash[162128]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T07:57:52.960 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@prometheus.a.service' 2024-02-10T07:57:52.997 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T07:57:52.998 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-10T07:57:52.998 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T07:57:53.194 INFO:teuthology.orchestra.run.smithi008.stdout:Deleting cluster with fsid: 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:57:55.788 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:57:55 smithi008 systemd[1]: Stopping Ceph alertmanager.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:57:56.082 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:57:55 smithi008 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a[84970]: ts=2024-02-10T07:57:55.787Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-10T07:57:56.082 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:57:55 smithi008 podman[220485]: 2024-02-10 07:57:55.838602393 +0000 UTC m=+0.162896020 container died e3bc4fcf4b68cd4f0371ec05657febd21721e4ee3cef7dd31b8be95ed0fbe53c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:57:56.455 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:57:56 smithi008 podman[220485]: 2024-02-10 07:57:56.246144712 +0000 UTC m=+0.570438347 container cleanup e3bc4fcf4b68cd4f0371ec05657febd21721e4ee3cef7dd31b8be95ed0fbe53c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:57:56.456 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:57:56 smithi008 bash[220485]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T07:57:56.832 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:57:56 smithi008 podman[220500]: 2024-02-10 07:57:56.45578044 +0000 UTC m=+0.611034876 container remove e3bc4fcf4b68cd4f0371ec05657febd21721e4ee3cef7dd31b8be95ed0fbe53c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T07:57:56.832 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:57:56 smithi008 podman[220500]: 2024-02-10 07:57:56.556969475 +0000 UTC m=+0.712223911 volume remove 6f52805fe07df4a3c3f4ceec93c4773fd3e35d9ac95517bd57890e8ee651164f 2024-02-10T07:57:56.832 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:57:56 smithi008 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-10T07:57:56.832 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:57:56 smithi008 systemd[1]: Stopped Ceph alertmanager.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:57:56.833 INFO:journalctl@ceph.alertmanager.a.smithi008.stdout:Feb 10 07:57:56 smithi008 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@alertmanager.a.service: Consumed 1.744s CPU time. 2024-02-10T07:57:57.332 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:57:56 smithi008 systemd[1]: Stopping Ceph node-exporter.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:57:57.332 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:57:57 smithi008 podman[220612]: 2024-02-10 07:57:57.111655548 +0000 UTC m=+0.095369630 container died 521ef0a9ea22a179bd36f96098b20f8be9d25b9c76f254e1d6d80baaf78765be (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:57:57.832 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:57:57 smithi008 podman[220612]: 2024-02-10 07:57:57.559451361 +0000 UTC m=+0.543165441 container cleanup 521ef0a9ea22a179bd36f96098b20f8be9d25b9c76f254e1d6d80baaf78765be (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:57:57.832 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:57:57 smithi008 bash[220612]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a 2024-02-10T07:57:58.188 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:57:57 smithi008 podman[220625]: 2024-02-10 07:57:57.870875348 +0000 UTC m=+0.762001014 container remove 521ef0a9ea22a179bd36f96098b20f8be9d25b9c76f254e1d6d80baaf78765be (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T07:57:58.188 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:57:57 smithi008 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T07:57:58.188 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:57:58 smithi008 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T07:57:58.188 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:57:58 smithi008 systemd[1]: Stopped Ceph node-exporter.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:57:58.188 INFO:journalctl@ceph.node-exporter.a.smithi008.stdout:Feb 10 07:57:58 smithi008 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@node-exporter.a.service: Consumed 4.846s CPU time. 2024-02-10T07:57:58.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T07:57:58.980 INFO:teuthology.orchestra.run.smithi170.stdout:Deleting cluster with fsid: 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T07:58:01.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:58:00 smithi170 systemd[1]: Stopping Ceph iscsi.iscsi.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:58:01.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:58:01 smithi170 ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a[61105]: debug Shutdown received 2024-02-10T07:58:11.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:58:11 smithi170 bash[162540]: time="2024-02-10T07:58:11Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-02-10T07:58:11.436 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:58:11 smithi170 podman[162540]: 2024-02-10 07:58:11.170038004 +0000 UTC m=+10.119486444 container died cf83e29530758e183eaffe0dbe745fb0aea6ce6bd0631843e0622e89d8e8407d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.2) 2024-02-10T07:58:11.811 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:58:11 smithi170 podman[162540]: 2024-02-10 07:58:11.526871302 +0000 UTC m=+10.476319751 container cleanup cf83e29530758e183eaffe0dbe745fb0aea6ce6bd0631843e0622e89d8e8407d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T07:58:11.811 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:58:11 smithi170 bash[162540]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a 2024-02-10T07:58:12.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:58:11 smithi170 podman[162567]: 2024-02-10 07:58:11.811259527 +0000 UTC m=+0.641611078 container remove cf83e29530758e183eaffe0dbe745fb0aea6ce6bd0631843e0622e89d8e8407d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, RELEASE=main-290212a) 2024-02-10T07:58:12.134 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:58:11 smithi170 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-02-10T07:58:12.134 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:58:11 smithi170 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-02-10T07:58:12.134 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:58:11 smithi170 systemd[1]: Stopped Ceph iscsi.iscsi.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:58:12.134 INFO:journalctl@ceph.iscsi.iscsi.a.smithi170.stdout:Feb 10 07:58:11 smithi170 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@iscsi.iscsi.a.service: Consumed 3.544s CPU time. 2024-02-10T07:58:13.704 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:58:13 smithi170 systemd[1]: Stopping Ceph grafana.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:58:13.977 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:58:13 smithi170 podman[162834]: 2024-02-10 07:58:13.704193859 +0000 UTC m=+0.153583325 container died a7f8d6fdc1d75e1189f6eba7fe3b975ad7ddab8443acde5a0941f36d293847c2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, name=ubi8, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, distribution-scope=public, description=Ceph Grafana Container, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897) 2024-02-10T07:58:14.361 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:58:14 smithi170 podman[162834]: 2024-02-10 07:58:14.068527037 +0000 UTC m=+0.517916504 container cleanup a7f8d6fdc1d75e1189f6eba7fe3b975ad7ddab8443acde5a0941f36d293847c2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a, distribution-scope=public, io.buildah.version=1.32.0, release=239.1651231664, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, name=ubi8, architecture=x86_64, maintainer=Paul Cuzner , io.openshift.expose-services=, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, vcs-type=git) 2024-02-10T07:58:14.361 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:58:14 smithi170 bash[162834]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a 2024-02-10T07:58:14.633 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:58:14 smithi170 podman[162849]: 2024-02-10 07:58:14.361301803 +0000 UTC m=+0.656231618 container remove a7f8d6fdc1d75e1189f6eba7fe3b975ad7ddab8443acde5a0941f36d293847c2 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , io.openshift.expose-services=, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.buildah.version=1.32.0, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc.) 2024-02-10T07:58:14.633 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:58:14 smithi170 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T07:58:14.633 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:58:14 smithi170 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T07:58:14.633 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:58:14 smithi170 systemd[1]: Stopped Ceph grafana.a for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:58:14.633 INFO:journalctl@ceph.grafana.a.smithi170.stdout:Feb 10 07:58:14 smithi170 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@grafana.a.service: Consumed 13.101s CPU time. 2024-02-10T07:58:14.924 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:58:14 smithi170 systemd[1]: Stopping Ceph node-exporter.b for 8c8244e6-c7e5-11ee-95b8-87774f69a715... 2024-02-10T07:58:15.189 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:58:14 smithi170 podman[162959]: 2024-02-10 07:58:14.924877224 +0000 UTC m=+0.119429822 container died dc68e79a865eb180681ca6b7e9e5654042761d7294ed5f320d6de9f7c834d1c9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:58:15.548 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:58:15 smithi170 podman[162959]: 2024-02-10 07:58:15.272343744 +0000 UTC m=+0.466896343 container cleanup dc68e79a865eb180681ca6b7e9e5654042761d7294ed5f320d6de9f7c834d1c9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:58:15.549 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:58:15 smithi170 bash[162959]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b 2024-02-10T07:58:15.847 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:58:15 smithi170 podman[162972]: 2024-02-10 07:58:15.548364577 +0000 UTC m=+0.624505667 container remove dc68e79a865eb180681ca6b7e9e5654042761d7294ed5f320d6de9f7c834d1c9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T07:58:15.847 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:58:15 smithi170 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-10T07:58:15.847 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:58:15 smithi170 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T07:58:15.847 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:58:15 smithi170 systemd[1]: Stopped Ceph node-exporter.b for 8c8244e6-c7e5-11ee-95b8-87774f69a715. 2024-02-10T07:58:15.848 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:Feb 10 07:58:15 smithi170 systemd[1]: ceph-8c8244e6-c7e5-11ee-95b8-87774f69a715@node-exporter.b.service: Consumed 4.427s CPU time. 2024-02-10T07:58:16.390 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:58:16.425 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T07:58:16.454 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-10T07:58:16.457 DEBUG:teuthology.misc:Transferring archived files from smithi008:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555212/remote/smithi008/crash 2024-02-10T07:58:16.458 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/crash -- . 2024-02-10T07:58:16.516 INFO:teuthology.orchestra.run.smithi008.stderr:tar: /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T07:58:16.516 INFO:teuthology.orchestra.run.smithi008.stderr:tar: Error is not recoverable: exiting now 2024-02-10T07:58:16.517 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555212/remote/smithi170/crash 2024-02-10T07:58:16.518 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/crash -- . 2024-02-10T07:58:16.566 INFO:teuthology.orchestra.run.smithi170.stderr:tar: /var/lib/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T07:58:16.566 INFO:teuthology.orchestra.run.smithi170.stderr:tar: Error is not recoverable: exiting now 2024-02-10T07:58:16.568 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-10T07:58:16.568 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-10T07:58:16.654 INFO:teuthology.orchestra.run.smithi008.stdout:2024-02-10T07:40:00.000146+0000 mon.a (mon.0) 1241 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-10T07:58:16.655 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-10T07:58:16.655 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[SEC\]' /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-10T07:58:16.686 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[ERR\]' /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-10T07:58:16.757 DEBUG:teuthology.orchestra.run.smithi008:> sudo egrep '\[WRN\]' /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-10T07:58:16.826 INFO:teuthology.orchestra.run.smithi008.stdout:2024-02-10T07:40:00.000146+0000 mon.a (mon.0) 1241 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-10T07:58:16.826 INFO:tasks.cephadm:Compressing logs... 2024-02-10T07:58:16.827 DEBUG:teuthology.orchestra.run.smithi008:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T07:58:16.868 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T07:58:16.918 INFO:teuthology.orchestra.run.smithi008.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-10T07:58:16.918 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T07:58:16.922 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mon.a.log 2024-02-10T07:58:16.924 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T07:58:16.924 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log 2024-02-10T07:58:16.924 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mon.a.log: /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.1.log 2024-02-10T07:58:16.924 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.3.log 2024-02-10T07:58:16.924 INFO:teuthology.orchestra.run.smithi008.stderr: 57.9% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-02-10T07:58:16.924 INFO:teuthology.orchestra.run.smithi008.stderr:gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mgr.y.log 2024-02-10T07:58:16.926 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.1.log: /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.0.log 2024-02-10T07:58:16.926 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T07:58:16.926 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.log 2024-02-10T07:58:16.926 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.2.log 2024-02-10T07:58:16.926 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T07:58:16.926 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/objectstore_tool.2.log 2024-02-10T07:58:16.926 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mon.c.log 2024-02-10T07:58:16.930 INFO:teuthology.orchestra.run.smithi170.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T07:58:16.930 INFO:teuthology.orchestra.run.smithi170.stderr:: No such file or directory 2024-02-10T07:58:16.937 INFO:teuthology.orchestra.run.smithi008.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/objectstore_tool.2.log: /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mon.c.log: 91.6% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T07:58:16.949 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.6.log 2024-02-10T07:58:16.949 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T07:58:16.949 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.4.log 2024-02-10T07:58:16.949 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.7.log 2024-02-10T07:58:16.949 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.5.log 2024-02-10T07:58:16.950 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mgr.x.log 2024-02-10T07:58:16.950 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/tcmu-runner.log 2024-02-10T07:58:16.950 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T07:58:16.950 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.log 2024-02-10T07:58:16.950 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T07:58:16.950 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mon.b.log 2024-02-10T07:58:16.950 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/objectstore_tool.2.log 2024-02-10T07:58:17.030 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mon.b.log: /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/objectstore_tool.2.log: 67.8% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/tcmu-runner.log.gz 2024-02-10T07:58:17.035 INFO:teuthology.orchestra.run.smithi008.stderr: 88.9% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T07:58:17.110 INFO:teuthology.orchestra.run.smithi008.stderr: 94.0% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T07:58:17.119 INFO:teuthology.orchestra.run.smithi008.stderr: 91.2% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T07:58:17.129 INFO:teuthology.orchestra.run.smithi170.stderr: 91.1% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T07:58:17.135 INFO:teuthology.orchestra.run.smithi008.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T07:58:17.161 INFO:teuthology.orchestra.run.smithi170.stderr: 94.8% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/objectstore_tool.2.log.gz 2024-02-10T07:58:17.185 INFO:teuthology.orchestra.run.smithi170.stderr: 90.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T07:58:17.187 INFO:teuthology.orchestra.run.smithi170.stderr: 81.5% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T07:58:17.196 INFO:teuthology.orchestra.run.smithi170.stderr: 85.2% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T07:58:17.209 INFO:teuthology.orchestra.run.smithi170.stderr: 92.4% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mgr.x.log.gz 2024-02-10T07:58:17.261 INFO:teuthology.orchestra.run.smithi170.stderr: 89.9% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T07:58:17.620 INFO:teuthology.orchestra.run.smithi008.stderr: 89.6% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mgr.y.log.gz 2024-02-10T07:58:17.949 INFO:teuthology.orchestra.run.smithi008.stderr: 94.3% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/objectstore_tool.2.log.gz 2024-02-10T07:58:24.230 INFO:teuthology.orchestra.run.smithi170.stderr: 91.2% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mon.b.log.gz 2024-02-10T07:58:28.862 INFO:teuthology.orchestra.run.smithi008.stderr: 91.2% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mon.c.log.gz 2024-02-10T07:58:42.826 INFO:teuthology.orchestra.run.smithi008.stderr: 90.4% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-mon.a.log.gz 2024-02-10T08:06:34.250 INFO:teuthology.orchestra.run.smithi008.stderr: 93.7% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.2.log.gz 2024-02-10T08:06:50.315 INFO:teuthology.orchestra.run.smithi008.stderr: 92.8% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.1.log.gz 2024-02-10T08:07:25.170 INFO:teuthology.orchestra.run.smithi170.stderr: 92.8% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.7.log.gz 2024-02-10T08:09:30.647 INFO:teuthology.orchestra.run.smithi008.stderr: 93.4% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.3.log.gz 2024-02-10T08:09:59.625 INFO:teuthology.orchestra.run.smithi170.stderr: 93.8% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.4.log.gz 2024-02-10T08:10:57.513 INFO:teuthology.orchestra.run.smithi008.stderr: 92.7% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.0.log.gz 2024-02-10T08:10:57.515 INFO:teuthology.orchestra.run.smithi008.stderr: 2024-02-10T08:10:57.515 INFO:teuthology.orchestra.run.smithi008.stderr:real 12m40.634s 2024-02-10T08:10:57.515 INFO:teuthology.orchestra.run.smithi008.stderr:user 6m41.139s 2024-02-10T08:10:57.515 INFO:teuthology.orchestra.run.smithi008.stderr:sys 0m17.881s 2024-02-10T08:11:21.963 INFO:teuthology.orchestra.run.smithi170.stderr: 93.6% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.5.log.gz 2024-02-10T08:11:40.639 INFO:teuthology.orchestra.run.smithi170.stderr: 93.5% -- replaced with /var/log/ceph/8c8244e6-c7e5-11ee-95b8-87774f69a715/ceph-osd.6.log.gz 2024-02-10T08:11:40.641 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-02-10T08:11:40.641 INFO:teuthology.orchestra.run.smithi170.stderr:real 13m23.758s 2024-02-10T08:11:40.641 INFO:teuthology.orchestra.run.smithi170.stderr:user 5m21.521s 2024-02-10T08:11:40.641 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m15.994s 2024-02-10T08:11:40.641 INFO:tasks.cephadm:Archiving logs... 2024-02-10T08:11:40.642 DEBUG:teuthology.misc:Transferring archived files from smithi008:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555212/remote/smithi008/log 2024-02-10T08:11:40.643 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T08:12:42.730 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555212/remote/smithi170/log 2024-02-10T08:12:42.732 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T08:13:33.521 INFO:tasks.cephadm:Removing cluster... 2024-02-10T08:13:33.522 DEBUG:teuthology.orchestra.run.smithi008:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 --force 2024-02-10T08:13:33.749 INFO:teuthology.orchestra.run.smithi008.stdout:Deleting cluster with fsid: 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T08:13:34.438 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8c8244e6-c7e5-11ee-95b8-87774f69a715 --force 2024-02-10T08:13:34.923 INFO:teuthology.orchestra.run.smithi170.stdout:Deleting cluster with fsid: 8c8244e6-c7e5-11ee-95b8-87774f69a715 2024-02-10T08:13:35.607 INFO:tasks.cephadm:Removing cephadm ... 2024-02-10T08:13:35.607 DEBUG:teuthology.orchestra.run.smithi008:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T08:13:35.632 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T08:13:35.647 INFO:tasks.cephadm:Teardown complete 2024-02-10T08:13:35.648 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-10T08:13:35.658 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-10T08:13:35.658 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-10T08:13:35.675 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-10T08:13:35.788 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T08:13:35.798 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T08:13:35.798 DEBUG:teuthology.orchestra.run.smithi008:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T08:13:35.801 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T08:13:35.815 INFO:teuthology.orchestra.run.smithi008.stderr:bash: line 1: ntpq: command not found 2024-02-10T08:13:35.826 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-02-10T08:13:35.850 INFO:teuthology.orchestra.run.smithi008.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T08:13:35.850 INFO:teuthology.orchestra.run.smithi008.stdout:=============================================================================== 2024-02-10T08:13:35.850 INFO:teuthology.orchestra.run.smithi008.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 +511us[ +507us] +/- 82ms 2024-02-10T08:13:35.850 INFO:teuthology.orchestra.run.smithi008.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -155us[ -159us] +/- 62ms 2024-02-10T08:13:35.850 INFO:teuthology.orchestra.run.smithi008.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -316us[ -320us] +/- 86ms 2024-02-10T08:13:35.850 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:13:35.853 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T08:13:35.853 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-02-10T08:13:35.853 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 +513us[ +513us] +/- 82ms 2024-02-10T08:13:35.853 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -137us[ -138us] +/- 62ms 2024-02-10T08:13:35.853 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -318us[ -318us] +/- 86ms 2024-02-10T08:13:35.853 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:13:35.854 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T08:13:35.863 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T08:13:35.863 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T08:13:35.872 DEBUG:teuthology.orchestra.run.smithi008:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T08:13:35.991 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T08:13:36.082 DEBUG:teuthology.orchestra.run.smithi008:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T08:13:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:13:36.114 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T08:13:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:13:36.142 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T08:13:36.153 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T08:13:36.162 INFO:teuthology.task.internal:Duration was 3326.780370 seconds 2024-02-10T08:13:36.162 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T08:13:36.171 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T08:13:36.171 DEBUG:teuthology.orchestra.run.smithi008:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T08:13:36.174 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T08:13:36.240 INFO:teuthology.orchestra.run.smithi008.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T08:13:36.250 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T08:13:36.638 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-10T08:13:36.639 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi008.front.sepia.ceph.com 2024-02-10T08:13:36.639 DEBUG:teuthology.orchestra.run.smithi008:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T08:13:36.662 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2024-02-10T08:13:36.662 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T08:13:36.694 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-10T08:13:36.694 DEBUG:teuthology.orchestra.run.smithi008:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T08:13:36.705 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T08:13:36.764 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-10T08:13:36.764 DEBUG:teuthology.orchestra.run.smithi008:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T08:13:36.775 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T08:13:38.113 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T08:13:38.123 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T08:13:38.123 DEBUG:teuthology.orchestra.run.smithi008:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T08:13:38.150 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T08:13:38.180 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T08:13:38.190 DEBUG:teuthology.orchestra.run.smithi008:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T08:13:38.193 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T08:13:38.227 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern = core 2024-02-10T08:13:38.276 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2024-02-10T08:13:38.303 DEBUG:teuthology.orchestra.run.smithi008:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T08:13:38.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:13:38.316 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T08:13:38.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:13:38.332 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T08:13:38.342 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T08:13:38.343 DEBUG:teuthology.misc:Transferring archived files from smithi008:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555212/remote/smithi008 2024-02-10T08:13:38.344 DEBUG:teuthology.orchestra.run.smithi008:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T08:13:38.439 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555212/remote/smithi170 2024-02-10T08:13:38.439 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T08:13:38.491 INFO:teuthology.task.internal:Removing archive directory... 2024-02-10T08:13:38.491 DEBUG:teuthology.orchestra.run.smithi008:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T08:13:38.494 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T08:13:38.538 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T08:13:38.549 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T08:13:38.550 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T08:13:38.559 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T08:13:38.559 DEBUG:teuthology.orchestra.run.smithi008:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T08:13:38.562 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T08:13:38.573 INFO:teuthology.orchestra.run.smithi008.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 08:13 /home/ubuntu/cephtest 2024-02-10T08:13:38.594 INFO:teuthology.orchestra.run.smithi170.stdout: 266030 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 08:13 /home/ubuntu/cephtest 2024-02-10T08:13:38.595 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T08:13:38.604 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T08:13:38.624 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T08:13:38.652 DEBUG:teuthology.nuke:shortname: smithi008 2024-02-10T08:13:38.652 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T08:13:38.677 DEBUG:teuthology.nuke:shortname: smithi170 2024-02-10T08:13:38.677 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T08:13:38.680 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi008.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555212', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 07:08:43.556003', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDT5Z0nOVs84klw14kkTk/VwcgXuWxydE/hPpmGCsA3HeEftC0FePofIpD9URIhSthaYNQ1YNqtlHhVMxfvVml4='} 2024-02-10T08:13:38.703 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555212', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 07:08:43.557917', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEQupySyFXI1xahb593YgRdd8Bp/ewEbjvYKu5rp2civZem77CWEcEaJeIr9jnQ94tTz2l6w7sK4peMzZNkIZco='} 2024-02-10T08:13:38.733 INFO:teuthology.orchestra.console.smithi008:Power off 2024-02-10T08:13:38.733 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T08:13:38.757 DEBUG:teuthology.orchestra.console.smithi008:power off output: Chassis Power Control: Down/Off 2024-02-10T08:13:38.757 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T08:13:38.779 INFO:teuthology.orchestra.console.smithi170:Power off 2024-02-10T08:13:38.780 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T08:13:38.789 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is on 2024-02-10T08:13:38.802 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2024-02-10T08:13:38.802 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T08:13:38.829 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2024-02-10T08:13:42.791 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T08:13:42.831 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T08:13:42.903 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is on 2024-02-10T08:13:42.943 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2024-02-10T08:13:46.904 DEBUG:teuthology.orchestra.console.smithi008:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T08:13:46.944 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T08:13:47.015 DEBUG:teuthology.orchestra.console.smithi008:check power output: Chassis Power is off 2024-02-10T08:13:47.055 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2024-02-10T08:13:47.116 INFO:teuthology.orchestra.console.smithi008:Power off completed 2024-02-10T08:13:47.156 INFO:teuthology.orchestra.console.smithi170:Power off completed 2024-02-10T08:13:47.260 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} duration: 3326.7803699970245 failure_reason: '"2024-02-10T07:40:00.000146+0000 mon.a (mon.0) 1241 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2024-02-10T08:13:47.261 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T08:13:47.347 INFO:teuthology.run:FAIL